diff --git a/.claude/commands/ship.md b/.claude/commands/ship.md index 30fcbce..935a26a 100644 --- a/.claude/commands/ship.md +++ b/.claude/commands/ship.md @@ -29,7 +29,7 @@ Review the changes on this branch (`git diff origin/main...HEAD` and - Positive tests: happy path, valid inputs, expected state transitions - Negative tests: invalid inputs, error conditions, boundary cases - Security tests: if network, URL parsing, filtering, or HTML conversion changed, - add or extend tests tied to `specs/threat-model.md` + add or extend tests tied to `knowledge/security/threat-model.md` 4. Run all tests: `cargo test --workspace` 5. If any test fails, fix the code or test until green @@ -37,8 +37,8 @@ Review the changes on this branch (`git diff origin/main...HEAD` and Review the change and update affected artifacts. Skip items that are not touched. -1. Specs in `specs/` -2. Threat model in `specs/threat-model.md` for new attack surfaces or mitigations +1. Relevant concepts in `knowledge/` +2. Threat model in `knowledge/security/threat-model.md` for new attack surfaces or mitigations 3. Release process docs/spec if shipping or release behavior changed 4. `AGENTS.md` if workflow, commands, or repo guidance changed 5. Public docs in `docs/` if user-facing behavior changed @@ -68,7 +68,7 @@ Analyze all changed code for security vulnerabilities. 6. Unsafe code usage If security issues are found, fix them, add regression tests, and update -`specs/threat-model.md` if a new threat must be tracked. +`knowledge/security/threat-model.md` if a new threat must be tracked. ### Phase 4: Smoke Testing diff --git a/.claude/skills/process-issues/SKILL.md b/.claude/skills/process-issues/SKILL.md index 42aa9f0..d513a18 100644 --- a/.claude/skills/process-issues/SKILL.md +++ b/.claude/skills/process-issues/SKILL.md @@ -42,8 +42,8 @@ For each qualifying issue (ordered by issue number), achieve ALL of these before - Minimal, focused changes - Positive and negative tests pass -- Security tests added if change touches URL parsing, fetchers, HTML conversion, network, or user input (per `specs/threat-model.md`) -- Threat model updated if new attack surface (per `specs/threat-model.md`) +- Security tests added if change touches URL parsing, fetchers, HTML conversion, network, or user input (per `knowledge/security/threat-model.md`) +- Threat model updated if new attack surface (per `knowledge/security/threat-model.md`) ### 4. Ship via `/ship` diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 20bb81e..6ce5088 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -19,4 +19,4 @@ when possible). For changes with no observable behavior (pure refactor, docs), s - [ ] Unit tests are passed - [ ] Smoke tests are passed - [ ] Documentation is updated -- [ ] Specs are up to date and not in conflict +- [ ] Knowledge is up to date and not in conflict diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ab396e4..682f277 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -33,6 +33,14 @@ jobs: run: cargo fmt --all -- --check - name: Clippy run: cargo clippy --workspace --all-targets -- -D warnings + - name: Install okf-lint + run: cargo install okf-lint --version 0.1.1 --locked + - name: Check knowledge bundle + run: | + python3 scripts/check_okf.py knowledge + okf-lint knowledge --max-line-length 10000 + - name: Test repository scripts + run: python3 -m unittest discover -s scripts/tests -p 'test_*.py' test: name: Test diff --git a/AGENTS.md b/AGENTS.md index 3c15e64..1bb071f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,19 +35,19 @@ Key capabilities: - URL filtering via allow/block lists - MCP server for AI tool integration -### Specs +### Knowledge -`specs/` folder contains feature specifications outlining requirements for specific features and components. New code should comply with these specifications or propose changes to them. +`knowledge/` is the canonical OKF v0.2 bundle and persistent project memory. Read relevant knowledge before changing behavior. Update it in the same change when decisions, behavior, constraints, threats, tests, or operations change. See `knowledge/knowledge-contract.md` for maintenance rules and run `python3 scripts/check_okf.py knowledge` after edits. -Available specs: -- `specs/initial.md` - WebFetch tool specification (types, behavior, conversions, error handling) -- `specs/fetchers.md` - Pluggable fetcher system for URL-specific handling -- `specs/release-process.md` - Agent-driven release and publish workflow -- `specs/maintenance.md` - Periodic maintenance checklist (deps, docs, spec-code alignment) -- `specs/threat-model.md` - Security threat model (SSRF, network, input validation, DoS) -- `specs/bot-auth.md` - Web Bot Authentication (draft-meunier-web-bot-auth-architecture) - -Specification format: Abstract and Requirements sections. +Available knowledge: +- `knowledge/knowledge-contract.md` - Knowledge maintenance and OKF conformance rules +- `knowledge/foundations/tool-contract.md` - Library, CLI, MCP, and Python behavior +- `knowledge/foundations/fetchers.md` - Pluggable fetcher and content processor system +- `knowledge/integrations/agent-discovery.md` - Bounded agent resource discovery +- `knowledge/security/threat-model.md` - Security threats and mitigations +- `knowledge/security/bot-auth.md` - Web Bot Authentication design +- `knowledge/operations/maintenance.md` - Periodic maintenance checklist +- `knowledge/operations/release-process.md` - Agent-driven release and publish workflow ### Shipping @@ -104,7 +104,7 @@ crates/ ├── fetchkit/ # Core library - types, fetch logic, HTML conversion ├── fetchkit-cli/ # CLI binary and MCP server └── fetchkit-python/ # Python bindings (PyO3) -specs/ # Feature specifications +knowledge/ # Canonical OKF v0.2 engineering knowledge ``` ### Naming @@ -126,7 +126,7 @@ specs/ # Feature specifications ### Releasing -See `specs/release-process.md` for the release contract. +See `knowledge/operations/release-process.md` for the release contract. Quick summary: 1. Human asks agent: "Create release v0.2.0" @@ -212,7 +212,7 @@ Before creating a pull request, ensure: 8. **PR comments resolved**: No unaddressed review comments in PR -9. **Specs**: If changes affect system behavior, update specs in `specs/` +9. **Knowledge**: If changes affect durable engineering knowledge, update `knowledge/` and run its OKF checks 10. **Docs**: If changes affect usage or configuration, update public docs in `docs/` @@ -290,7 +290,7 @@ when possible). For changes with no observable behavior (pure refactor, docs), s - [ ] Unit tests are passed - [ ] Smoke tests are passed - [ ] Documentation is updated -- [ ] Specs are up to date and not in conflict +- [ ] Knowledge is up to date and not in conflict ``` ### Testing the system @@ -312,4 +312,4 @@ cargo run -p webfetch-cli -- --url https://example.com --as-markdown cargo run -p webfetch-cli -- mcp ``` -Tests use `wiremock` for HTTP mocking (no real external network calls). See `specs/initial.md` for test requirements. +Tests use `wiremock` for HTTP mocking (no real external network calls). See `knowledge/foundations/tool-contract.md` for test requirements. diff --git a/CHANGELOG.md b/CHANGELOG.md index aadcd73..07c584f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Replace the specification directory with an indexed OKF v0.2 knowledge + bundle, maintenance contract, update log, and CI conformance checks. - Add a pluggable post-download `ContentProcessor` registry and built-in `PdfProcessor` that extracts Markdown from text-based PDFs with local `pdf-inspector` processing and explicit OCR guidance for unsupported pages. diff --git a/README.md b/README.md index 877942d..30f42f6 100644 --- a/README.md +++ b/README.md @@ -316,7 +316,7 @@ DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canon Redirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as `allowed.example.com.evil.test`. Proxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with `ToolBuilder::respect_proxy_env(true)` only when it is part of an intentional egress design. -See [`specs/threat-model.md`](specs/threat-model.md) for the full threat model. +See the [`knowledge/security/threat-model.md`](knowledge/security/threat-model.md) concept for the full threat model. See [`docs/hardening.md`](docs/hardening.md) for deployment guidance. ## Configuration diff --git a/crates/fetchkit/tests/ssrf_security.rs b/crates/fetchkit/tests/ssrf_security.rs index e2788c9..58f0c63 100644 --- a/crates/fetchkit/tests/ssrf_security.rs +++ b/crates/fetchkit/tests/ssrf_security.rs @@ -2,7 +2,7 @@ //! //! Tests that validate the resolve-then-check DNS policy prevents //! server-side request forgery attacks. These tests verify the threat -//! mitigations documented in specs/threat-model.md. +//! mitigations documented in knowledge/security/threat-model.md. //! //! Safe-by-default: Tool::default() and fetch() block private IPs. //! Tests that need loopback (wiremock) must explicitly opt out. diff --git a/docs/security.md b/docs/security.md index 6732f86..e638dc5 100644 --- a/docs/security.md +++ b/docs/security.md @@ -31,7 +31,7 @@ let tool = ToolBuilder::new() .build(); ``` -See [`specs/threat-model.md`](../specs/threat-model.md) for the full threat inventory. +See [`knowledge/security/threat-model.md`](../knowledge/security/threat-model.md) for the full threat inventory. ## Web Bot Authentication @@ -59,4 +59,4 @@ CLI usage: fetchkit fetch https://example.com --bot-auth-key --bot-auth-agent bot.example.com ``` -See [`specs/bot-auth.md`](../specs/bot-auth.md) for the full specification. +See [`knowledge/security/bot-auth.md`](../knowledge/security/bot-auth.md) for the full protocol design. diff --git a/specs/fetchers.md b/knowledge/foundations/fetchers.md similarity index 97% rename from specs/fetchers.md rename to knowledge/foundations/fetchers.md index f4fce99..89816a7 100644 --- a/specs/fetchers.md +++ b/knowledge/foundations/fetchers.md @@ -1,4 +1,14 @@ -# Fetcher System Specification +--- +type: Subsystem Design +title: Fetcher System +description: URL-specific fetchers, content processors, transport policy, extension points, and tests. +tags: + - fetchkit + - fetching + - architecture +--- + +# Fetcher System ## Abstract @@ -354,7 +364,7 @@ Both built-in fetchers integrate resolve-then-check DNS validation: - Enabled by default via `DnsPolicy::default()` (blocks private IPs) - Ignore ambient proxy env by default so shared runtimes do not silently route traffic through operator-provided proxies unless explicitly enabled -- See `specs/threat-model.md` for threat IDs: TM-SSRF-001 through TM-SSRF-010 +- See the [Threat Model](../security/threat-model.md) for threat IDs: TM-SSRF-001 through TM-SSRF-010. ## Module Structure @@ -448,3 +458,8 @@ Tests real URLs: 3. Add `mod {name};` and `pub use {name}::*;` to `mod.rs` 4. Register in `FetcherRegistry::with_defaults()` (before DefaultFetcher) 5. Add test cases to `examples/fetch_urls.rs` + +## See also + +- [Fetchkit Tool Contract](tool-contract.md) — shared request, response, and policy behavior +- [Threat Model](../security/threat-model.md) — network and SSRF requirements applied to every fetcher diff --git a/knowledge/foundations/index.md b/knowledge/foundations/index.md new file mode 100644 index 0000000..065d315 --- /dev/null +++ b/knowledge/foundations/index.md @@ -0,0 +1,4 @@ +# Foundations + +* [Fetchkit Tool Contract](tool-contract.md) - Public library, CLI, MCP, and Python behavior for fetching and converting web content. +* [Fetcher System](fetchers.md) - URL-specific fetchers, content processors, transport policy, extension points, and tests. diff --git a/specs/initial.md b/knowledge/foundations/tool-contract.md similarity index 97% rename from specs/initial.md rename to knowledge/foundations/tool-contract.md index 6136f40..c198277 100644 --- a/specs/initial.md +++ b/knowledge/foundations/tool-contract.md @@ -1,10 +1,20 @@ +--- +type: Interface Contract +title: Fetchkit Tool Contract +description: Public library, CLI, MCP, and Python behavior for fetching and converting web content. +tags: + - fetchkit + - api + - fetching +--- + # Decisions: -# - Spec mirrors current Fetchkit tool behavior (no new features) unless noted below. +# - Contract mirrors current Fetchkit tool behavior (no new features) unless noted below. # - Rust is the source of truth: library + CLI + MCP server + Python bindings. # - HTML conversion is built-in (no external HTML conversion deps). # - `FetchRequest` and `FetchResponse` are defined in this crate (no external dependency). -# Fetchkit Specification +# Fetchkit Tool Contract ## Abstract @@ -251,7 +261,7 @@ By default, Fetchkit blocks connections to private/reserved IP ranges: - Handles IPv6-mapped IPv4 addresses via canonicalization. - Pins validated IP via `reqwest::ClientBuilder::resolve()` to prevent DNS rebinding. - Blocked by default; opt out via `ToolBuilder::block_private_ips(false)`. -- See `specs/threat-model.md` for full threat analysis. +- See the [Threat Model](../security/threat-model.md) for full threat analysis. ### HTTP Behavior @@ -453,3 +463,8 @@ SSRF security: - Default-blocks-loopback verification. - Explicit opt-out verification. - Script stripping in converted content. + +## See also + +- [Fetcher System](fetchers.md) — URL-specific retrieval and content processing architecture +- [Threat Model](../security/threat-model.md) — security boundaries and mitigation requirements diff --git a/knowledge/index.md b/knowledge/index.md new file mode 100644 index 0000000..1b4881f --- /dev/null +++ b/knowledge/index.md @@ -0,0 +1,15 @@ +--- +okf_version: "0.2" +--- + +# Fetchkit Knowledge + +* [Knowledge Maintenance Contract](knowledge-contract.md) - Rules for maintaining Fetchkit's OKF bundle. +* [Update Log](log.md) - Chronological history of changes to this bundle. + +# Domains + +* [foundations/](foundations/) - Public tool contracts and core fetching architecture. +* [integrations/](integrations/) - Agent-facing discovery and integration contracts. +* [security/](security/) - Threat analysis and authentication protocol design. +* [operations/](operations/) - Maintenance and release playbooks. diff --git a/specs/agent-discovery.md b/knowledge/integrations/agent-discovery.md similarity index 78% rename from specs/agent-discovery.md rename to knowledge/integrations/agent-discovery.md index e927d1e..95eb1b5 100644 --- a/specs/agent-discovery.md +++ b/knowledge/integrations/agent-discovery.md @@ -1,3 +1,13 @@ +--- +type: Interface Contract +title: Agent Resource Discovery +description: Bounded discovery and reporting of same-origin resources intended for AI agents. +tags: + - fetchkit + - agents + - discovery +--- + # Agent Resource Discovery ## Abstract @@ -27,3 +37,8 @@ navigation links to Markdown output. requested and validated that exact resource. 11. Discovery MUST NOT invoke APIs, authorization flows, payment protocols, or agent capabilities. + +## See also + +- [Fetcher System](../foundations/fetchers.md) — transport and URL policy used by discovery probes +- [Threat Model](../security/threat-model.md) — discovery amplification and network-policy threats diff --git a/knowledge/integrations/index.md b/knowledge/integrations/index.md new file mode 100644 index 0000000..b4bcad1 --- /dev/null +++ b/knowledge/integrations/index.md @@ -0,0 +1,3 @@ +# Integrations + +* [Agent Resource Discovery](agent-discovery.md) - Bounded discovery and reporting of same-origin resources intended for AI agents. diff --git a/knowledge/knowledge-contract.md b/knowledge/knowledge-contract.md new file mode 100644 index 0000000..09c07e3 --- /dev/null +++ b/knowledge/knowledge-contract.md @@ -0,0 +1,63 @@ +--- +type: Playbook +title: Knowledge Maintenance Contract +description: Rules for maintaining the Fetchkit knowledge bundle and its OKF conformance. +tags: + - fetchkit + - knowledge + - okf + - process +--- + +# Knowledge Maintenance Contract + +`knowledge/` is Fetchkit's canonical [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle and persistent project memory. + +## Maintenance rules + +- Treat this knowledge as part of the implementation, not as historical documentation. +- Before changing behavior, read the relevant concepts and follow their decisions or update them in the same change. +- When code changes a documented behavior, design decision, invariant, limitation, threat, test strategy, or operational process, update the affected knowledge in the same pull request. +- Record important decisions that are not recoverable from code. Prefer links to source and tests over duplicating volatile implementation details. +- Keep stable identifiers such as `TM-*` and `R-*`; never renumber them. +- Add durable engineering knowledge here. User-facing guides remain in `docs/`. + +## OKF conformance rules + +The bundle targets OKF v0.2, declared as `okf_version: "0.2"` in the bundle-root [index](index.md). + +- Every Markdown file except reserved `index.md` and `log.md` files is a concept and starts with YAML frontmatter containing a non-empty `type`. +- Concepts also carry `title`, a single-line `description`, and useful `tags`. +- Directory indexes contain link lists for concepts and immediate subdirectories only. +- The update log uses `## YYYY-MM-DD` headings, newest first. +- Links between concepts are relative and resolve inside the bundle. +- Every concept links to another concept so agents can traverse the bundle as a graph. +- Reference bundle documents with relative Markdown links, not repository-path text that can silently rot. + +OKF provenance, trust, lifecycle, and attestation metadata remain optional. If generated concepts are added, they must identify their `resource` and `generated.by` actor so readers can distinguish generated facts from hand-maintained knowledge. + +## Layout + +| Directory | Holds | +|---|---| +| [foundations/](foundations/) | Tool behavior and fetcher architecture | +| [integrations/](integrations/) | Agent-facing integration contracts | +| [security/](security/) | Threat model and authentication design | +| [operations/](operations/) | Maintenance and release playbooks | + +## Enforcement + +Run both checks after changing the bundle: + +```console +$ python3 scripts/check_okf.py knowledge +knowledge: OKF v0.2 conformant (8 concepts, 5 index files, 1 log file) +$ okf-lint knowledge --max-line-length 10000 +``` + +The upstream linter enforces OKF v0.2. The local checker adds bundle conventions the format intentionally leaves soft: complete indexes, resolvable graph links, required descriptions, and generated-resource metadata. CI pins `okf-lint` to a reviewed version. + +## See also + +- [Periodic Maintenance](operations/maintenance.md) — broader repository drift and health checks +- [Fetchkit Tool Contract](foundations/tool-contract.md) — primary behavior contract maintained in this bundle diff --git a/knowledge/log.md b/knowledge/log.md new file mode 100644 index 0000000..a78317a --- /dev/null +++ b/knowledge/log.md @@ -0,0 +1,6 @@ +# Fetchkit Knowledge Update Log + +## 2026-08-08 + +* **Migration**: Replaced the unindexed specification directory with an OKF v0.2 bundle organized by foundations, integrations, security, and operations. +* **Process**: Added a maintenance contract and automated conformance checks so durable engineering knowledge changes with the implementation. diff --git a/knowledge/operations/index.md b/knowledge/operations/index.md new file mode 100644 index 0000000..e88d513 --- /dev/null +++ b/knowledge/operations/index.md @@ -0,0 +1,4 @@ +# Operations + +* [Periodic Maintenance](maintenance.md) - Recurring dependency, documentation, security, compatibility, and release-alignment checks. +* [Release Process](release-process.md) - Agent-driven version preparation, validation, GitHub release creation, and crates.io publishing. diff --git a/specs/maintenance.md b/knowledge/operations/maintenance.md similarity index 77% rename from specs/maintenance.md rename to knowledge/operations/maintenance.md index 3e33167..a41c5ae 100644 --- a/specs/maintenance.md +++ b/knowledge/operations/maintenance.md @@ -1,8 +1,18 @@ -# Periodic Maintenance Specification +--- +type: Playbook +title: Periodic Maintenance +description: Recurring dependency, documentation, security, compatibility, and release-alignment checks. +tags: + - fetchkit + - maintenance + - operations +--- + +# Periodic Maintenance ## Abstract -Define recurring maintenance tasks to keep the fetchkit repository healthy, up-to-date, and well-documented. This spec is intended to be executed periodically (e.g., monthly or before each release) by a human or coding agent. +Define recurring maintenance tasks to keep the fetchkit repository healthy, up-to-date, and well-documented. This playbook is intended to be executed periodically (e.g., monthly or before each release) by a human or coding agent. ## Requirements @@ -41,18 +51,18 @@ Ensure all public items have good documentation suitable for docs.rs rendering. 4. **No doc warnings** - `RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps` must pass 5. **README sync** - Root README.md code snippets should be consistent with actual API -### 3. Spec-Code Alignment +### 3. Knowledge-Code Alignment -Ensure specifications in `specs/` accurately describe the current code, and code conforms to specs. +Ensure the canonical knowledge accurately describes the current code, and code conforms to documented contracts. -1. **Type definitions** - Verify struct/enum fields in code match spec definitions (field names, types, optionality) -2. **Error variants** - Verify `FetchError` variants in code match spec -3. **Behavior** - Verify timeouts, binary detection, HTML conversion rules match spec descriptions -4. **Fetcher system** - Verify fetcher trait, registry, and built-in fetchers match `specs/fetchers.md` -5. **CLI flags** - Verify CLI argument names and behavior match spec -6. **MCP protocol** - Verify MCP method names and schemas match spec -7. **Update stale specs** - If code intentionally diverges from spec, update the spec to match -8. **Update stale code** - If spec describes required behavior not in code, flag for implementation +1. **Type definitions** - Verify struct/enum fields in code match contract definitions (field names, types, optionality) +2. **Error variants** - Verify `FetchError` variants in code match the tool contract +3. **Behavior** - Verify timeouts, binary detection, HTML conversion rules match documented behavior +4. **Fetcher system** - Verify fetcher trait, registry, and built-in fetchers match the [Fetcher System](../foundations/fetchers.md) +5. **CLI flags** - Verify CLI argument names and behavior match the tool contract +6. **MCP protocol** - Verify MCP method names and schemas match the tool contract +7. **Update stale knowledge** - If code intentionally diverges from documented behavior, update the knowledge to match +8. **Update stale code** - If knowledge describes required behavior not in code, flag for implementation ### 4. Example Verification @@ -84,11 +94,11 @@ Verify CI pipeline and development tooling are current. 1. **Unreleased section** - `CHANGELOG.md` has an `[Unreleased]` section for pending changes 2. **Version consistency** - Workspace version in root `Cargo.toml` matches latest changelog entry 3. **Inter-crate versions** - Internal dependency versions (e.g., `fetchkit-cli` depending on `fetchkit`) are consistent -4. **Release format** - Released changelog sections follow `specs/release-process.md` +4. **Release format** - Released changelog sections follow the [Release Process](release-process.md) ### 8. Release Automation Alignment -1. **Release spec sync** - `.claude/commands/ship.md` still matches `specs/release-process.md` +1. **Release process sync** - [`.claude/commands/ship.md`](../../.claude/commands/ship.md) still matches the [Release Process](release-process.md) 2. **Workflow triggers** - `.github/workflows/release.yml` and `.github/workflows/publish.yml` still reflect the documented release handoff 3. **Publish scope** - crates.io publishing still targets `fetchkit` and `fetchkit-cli` only 4. **Manual retry path** - `workflow_dispatch` remains available for release/publish recovery @@ -96,3 +106,8 @@ Verify CI pipeline and development tooling are current. ## Execution Run this checklist by working through sections 1-8 in order. Fix issues as encountered. Commit fixes in logical groups following conventional commits. After completion, all CI checks should pass. + +## See also + +- [Release Process](release-process.md) — release preparation, verification, and publication contract +- [Knowledge Maintenance Contract](../knowledge-contract.md) — rules for keeping this bundle synchronized diff --git a/specs/release-process.md b/knowledge/operations/release-process.md similarity index 90% rename from specs/release-process.md rename to knowledge/operations/release-process.md index e149d2c..6ec5e3d 100644 --- a/specs/release-process.md +++ b/knowledge/operations/release-process.md @@ -1,9 +1,19 @@ +--- +type: Playbook +title: Release Process +description: Agent-driven version preparation, validation, GitHub release creation, and crates.io publishing. +tags: + - fetchkit + - release + - operations +--- + # Decisions: -# - Spec mirrors Bashkit's agent-driven release flow, adapted to Fetchkit's crates. +# - Process mirrors Bashkit's agent-driven release flow, adapted to Fetchkit's crates. # - GitHub Release creation is the handoff point to publishing; publish retries use `workflow_dispatch`. # - `fetchkit-python` is explicitly out of the crates.io publish flow until PyPI packaging exists. -# Release Process Specification +# Release Process ## Abstract @@ -137,3 +147,8 @@ cargo build --workspace --exclude fetchkit-python --release ### Alignment - `.claude/commands/ship.md` must remain compatible with this release workflow + +## See also + +- [Periodic Maintenance](maintenance.md) — pre-release health and alignment checks +- [Knowledge Maintenance Contract](../knowledge-contract.md) — synchronized knowledge update requirements diff --git a/specs/bot-auth.md b/knowledge/security/bot-auth.md similarity index 88% rename from specs/bot-auth.md rename to knowledge/security/bot-auth.md index afa8811..5933b55 100644 --- a/specs/bot-auth.md +++ b/knowledge/security/bot-auth.md @@ -1,3 +1,13 @@ +--- +type: Protocol Design +title: Web Bot Authentication +description: Optional Ed25519 HTTP message signing for cryptographically verifiable bot identity. +tags: + - fetchkit + - authentication + - security +--- + # Web Bot Authentication ## Abstract @@ -84,3 +94,8 @@ key directory if they want origins to discover keys via `Signature-Agent`. and `Signature-Agent` headers are present on outgoing requests. - All tests run under `#[cfg(feature = "bot-auth")]` or with the feature enabled in dev-dependencies. + +## See also + +- [Threat Model](threat-model.md) — signing-key, replay, identity, and failure-mode analysis +- [Fetchkit Tool Contract](../foundations/tool-contract.md) — configuration and outbound request behavior diff --git a/knowledge/security/index.md b/knowledge/security/index.md new file mode 100644 index 0000000..0935bf1 --- /dev/null +++ b/knowledge/security/index.md @@ -0,0 +1,4 @@ +# Security + +* [Fetchkit Threat Model](threat-model.md) - Assets, trust boundaries, threats, mitigations, and stable security identifiers for Fetchkit. +* [Web Bot Authentication](bot-auth.md) - Optional Ed25519 HTTP message signing for cryptographically verifiable bot identity. diff --git a/specs/threat-model.md b/knowledge/security/threat-model.md similarity index 98% rename from specs/threat-model.md rename to knowledge/security/threat-model.md index dc07c90..10e220d 100644 --- a/specs/threat-model.md +++ b/knowledge/security/threat-model.md @@ -1,3 +1,13 @@ +--- +type: Threat Model +title: Fetchkit Threat Model +description: Assets, trust boundaries, threats, mitigations, and stable security identifiers for Fetchkit. +tags: + - fetchkit + - security + - threat-model +--- + # Threat Model ## Abstract @@ -464,8 +474,8 @@ None — all previously open threats have been mitigated. ## References -- `specs/initial.md` — Fetchkit tool specification -- `specs/fetchers.md` — Pluggable fetcher system +- [Fetchkit Tool Contract](../foundations/tool-contract.md) — public tool behavior and security controls +- [Fetcher System](../foundations/fetchers.md) — pluggable fetchers and transport architecture - [OWASP SSRF Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html) - [CWE-918: Server-Side Request Forgery](https://cwe.mitre.org/data/definitions/918.html) @@ -496,3 +506,8 @@ redirects, or receive credentials intended for another origin. **Verification**: Discovery uses the shared request transport and tests use a private-address policy override explicitly. + +## See also + +- [Web Bot Authentication](bot-auth.md) — request-signing protocol design +- [Agent Resource Discovery](../integrations/agent-discovery.md) — bounded discovery behavior diff --git a/scripts/check_okf.py b/scripts/check_okf.py new file mode 100644 index 0000000..e1aa60a --- /dev/null +++ b/scripts/check_okf.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +"""Validate Fetchkit's Open Knowledge Format (OKF) v0.2 bundle. + +The upstream specification intentionally keeps most structure optional. This +checker enforces the bundle-local contract in knowledge/knowledge-contract.md: +metadata, complete indexes, dated logs, resolvable graph links, and sound +metadata for generated concepts. It has no third-party Python dependencies. +""" + +from __future__ import annotations + +import argparse +import pathlib +import re +import sys + +RESERVED = ("index.md", "log.md") +DATE_HEADING = re.compile(r"^## \d{4}-\d{2}-\d{2}\s*$") +LINK = re.compile(r"\[[^\]]*\]\(([^)]+)\)") +EXTERNAL = re.compile(r"\A(?:[a-z][a-z0-9+.-]*:|//|#)") +DATE = re.compile(r"\A\d{4}-\d{2}-\d{2}\Z") +ACTOR = re.compile(r"\A(?:human:\S+|process:\S+|[^/\s]+/[^/\s]+)\Z") +STATUSES = ("draft", "stable", "deprecated") +BUNDLE_PATH = re.compile(r"knowledge/[A-Za-z0-9_./-]+\.(?:md|json)") +CODE = re.compile(r"^```.*?^```|``.*?``|`[^`\n]*`", re.DOTALL | re.MULTILINE) + + +def strip_code(text: str) -> str: + """Blank code spans and fences so their examples are not parsed as links.""" + return CODE.sub(lambda match: "\n" * match.group(0).count("\n"), text) + + +def split_frontmatter(text: str) -> tuple[str | None, str]: + """Return frontmatter and body, or None and the full text when absent.""" + if not text.startswith("---\n"): + return None, text + end = text.find("\n---\n", 3) + if end == -1: + raise ValueError("unterminated frontmatter block") + return text[4:end], text[end + 5 :] + + +def parse_frontmatter(frontmatter: str) -> dict[str, object]: + """Parse the scalar, scalar-list, and one-level mapping subset in use.""" + data: dict[str, object] = {} + key: str | None = None + for lineno, raw in enumerate(frontmatter.splitlines(), start=2): + line = raw.rstrip() + if not line or line.lstrip().startswith("#"): + continue + if line.startswith(" "): + if key is None: + raise ValueError(f"line {lineno}: indented entry without a key") + item = line.strip() + if item.startswith("- "): + existing = data.get(key) + values = existing if isinstance(existing, list) else [] + data[key] = values + [item[2:].strip()] + elif ":" in item: + nested = data.get(key) + if not isinstance(nested, dict): + nested = {} + data[key] = nested + subkey, _, value = item.partition(":") + nested[subkey.strip()] = value.strip() + else: + raise ValueError(f"line {lineno}: unparseable entry {item!r}") + continue + if ":" not in line: + raise ValueError(f"line {lineno}: unparseable key line {line!r}") + key, _, value = line.partition(":") + key = key.strip() + data[key] = value.strip() + return data + + +def index_targets(path: pathlib.Path) -> set[str]: + if not path.exists(): + return set() + _, body = split_frontmatter(path.read_text()) + return { + target.split("#", 1)[0].rstrip("/") + for target in LINK.findall(strip_code(body)) + } + + +def check_links(path: pathlib.Path, rel: str, errors: list[str]) -> None: + _, body = split_frontmatter(path.read_text()) + for target in LINK.findall(strip_code(body)): + if EXTERNAL.match(target): + continue + resolved = target.split("#", 1)[0] + if resolved and not (path.parent / resolved).exists(): + errors.append(f"{rel}: link target does not exist: {target}") + + +def check_trust( + path: pathlib.Path, + rel: str, + metadata: dict[str, object], + errors: list[str], +) -> None: + generated = metadata.get("generated") + resource = metadata.get("resource") + if metadata.get("type") == "Generated Inventory": + if not isinstance(generated, dict) or not generated.get("by"): + errors.append( + f"{rel}: a 'Generated Inventory' must declare 'generated.by'" + ) + if not resource: + errors.append(f"{rel}: a 'Generated Inventory' must declare 'resource'") + if isinstance(generated, dict): + actor = generated.get("by") + if actor and not ACTOR.match(str(actor)): + errors.append(f"{rel}: 'generated.by' is not an OKF actor: {actor!r}") + if isinstance(resource, str) and resource and not EXTERNAL.match(resource): + if not (path.parent / resource).exists(): + errors.append(f"{rel}: 'resource' does not exist: {resource}") + status = metadata.get("status") + if status and status not in STATUSES: + errors.append(f"{rel}: 'status' must be one of {STATUSES}, got {status!r}") + stale_after = metadata.get("stale_after") + if stale_after and not DATE.match(str(stale_after)): + errors.append( + f"{rel}: 'stale_after' must be YYYY-MM-DD, got {stale_after!r}" + ) + + +def check_cross_links( + path: pathlib.Path, rel: str, body: str, errors: list[str] +) -> None: + for target in LINK.findall(strip_code(body)): + if EXTERNAL.match(target): + continue + resolved = (path.parent / target.split("#", 1)[0]).resolve() + if ( + resolved.suffix == ".md" + and resolved.name not in RESERVED + and resolved != path.resolve() + and resolved.exists() + ): + return + errors.append(f"{rel}: links to no other concept") + + +def check_concept(path: pathlib.Path, rel: str, errors: list[str]) -> None: + text = path.read_text() + frontmatter, body = split_frontmatter(text) + if frontmatter is None: + errors.append(f"{rel}: missing YAML frontmatter block") + return + metadata = parse_frontmatter(frontmatter) + if not metadata.get("type"): + errors.append(f"{rel}: frontmatter must contain a non-empty 'type'") + for field in ("title", "description"): + if not metadata.get(field): + errors.append(f"{rel}: frontmatter must contain a non-empty '{field}'") + if "summary" in metadata: + errors.append(f"{rel}: 'summary' is not an OKF field; use 'description'") + check_trust(path, rel, metadata, errors) + check_cross_links(path, rel, body, errors) + + +def check_index(path: pathlib.Path, rel: str, is_root: bool, errors: list[str]) -> None: + frontmatter, body = split_frontmatter(path.read_text()) + if frontmatter is not None: + keys = set(parse_frontmatter(frontmatter)) + allowed = {"okf_version"} if is_root else set() + extra = sorted(keys - allowed) + if extra: + errors.append(f"{rel}: index.md may not carry frontmatter keys {extra}") + if not body.strip(): + errors.append(f"{rel}: index.md body is empty") + + +def check_log(path: pathlib.Path, rel: str, errors: list[str]) -> None: + frontmatter, body = split_frontmatter(path.read_text()) + if frontmatter is not None: + errors.append(f"{rel}: log.md may not carry frontmatter") + headings = [line for line in body.splitlines() if line.startswith("## ")] + if not headings: + errors.append(f"{rel}: log.md needs at least one '## YYYY-MM-DD' heading") + for heading in headings: + if not DATE_HEADING.match(heading): + errors.append(f"{rel}: log heading {heading!r} is not '## YYYY-MM-DD'") + + +def check_bundle_paths(rel: str, text: str, errors: list[str]) -> None: + for match in sorted(set(BUNDLE_PATH.findall(text))): + errors.append( + f"{rel}: reference bundle documents as relative markdown links, " + f"not as repository paths: {match}" + ) + + +def check_bundle(root: pathlib.Path) -> tuple[list[str], dict[str, int]]: + errors: list[str] = [] + counts = {"concepts": 0, "indexes": 0, "logs": 0} + if not (root / "index.md").exists(): + errors.append("index.md: bundle root index is missing") + + directories = sorted(path for path in root.rglob("*") if path.is_dir()) + [root] + for directory in directories: + listed = index_targets(directory / "index.md") + for child in sorted(directory.iterdir()): + rel = child.relative_to(root).as_posix() + if child.is_dir(): + if not (child / "index.md").exists(): + errors.append(f"{rel}/: subdirectory has no index.md") + if child.name not in listed: + errors.append(f"{rel}/: not listed in {directory.name}/index.md") + continue + if child.suffix != ".md" or child.name in RESERVED: + continue + if child.name not in listed: + index_rel = (directory / "index.md").relative_to(root).as_posix() + errors.append(f"{rel}: not listed in {index_rel}") + + for path in sorted(root.rglob("*.md")): + rel = path.relative_to(root).as_posix() + try: + if path.name == "index.md": + counts["indexes"] += 1 + check_index(path, rel, path.parent == root, errors) + elif path.name == "log.md": + counts["logs"] += 1 + check_log(path, rel, errors) + else: + counts["concepts"] += 1 + check_concept(path, rel, errors) + check_links(path, rel, errors) + check_bundle_paths(rel, path.read_text(), errors) + except ValueError as error: + errors.append(f"{rel}: {error}") + + return errors, counts + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("bundle", nargs="?", default="knowledge", type=pathlib.Path) + args = parser.parse_args(argv) + if not args.bundle.is_dir(): + print(f"error: {args.bundle} is not a directory", file=sys.stderr) + return 2 + + errors, counts = check_bundle(args.bundle) + if errors: + print( + f"{args.bundle}: {len(errors)} OKF conformance error(s)", + file=sys.stderr, + ) + for error in errors: + print(f" {error}", file=sys.stderr) + return 1 + + print( + f"{args.bundle}: OKF v0.2 conformant ({counts['concepts']} concepts, " + f"{counts['indexes']} index files, {counts['logs']} log file" + f"{'s' if counts['logs'] != 1 else ''})" + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/tests/test_check_okf.py b/scripts/tests/test_check_okf.py new file mode 100644 index 0000000..c7694b7 --- /dev/null +++ b/scripts/tests/test_check_okf.py @@ -0,0 +1,131 @@ +"""Regression tests for scripts/check_okf.py.""" + +import pathlib +import subprocess +import sys +import tempfile +import unittest + +REPO = pathlib.Path(__file__).resolve().parents[2] +SCRIPT = REPO / "scripts" / "check_okf.py" + +CONCEPT = """\ +--- +type: Subsystem Design +title: Widget +description: One sentence about the widget. +--- + +# Widget + +See [Gadget](gadget.md). +""" + +GADGET = """\ +--- +type: Subsystem Design +title: Gadget +description: One sentence about the gadget. +--- + +# Gadget + +See [Widget](widget.md). +""" + +ROOT_INDEX = """\ +--- +okf_version: "0.2" +--- + +# Bundle + +* [Widget](widget.md) - One sentence about the widget. +* [Gadget](gadget.md) - One sentence about the gadget. +""" + +LOG = """\ +# Bundle Update Log + +## 2026-08-08 + +* **Creation**: Added [Widget](widget.md). +""" + + +def run(bundle: pathlib.Path) -> subprocess.CompletedProcess[str]: + return subprocess.run( + [sys.executable, str(SCRIPT), str(bundle)], + capture_output=True, + text=True, + check=False, + ) + + +class CheckOkfTest(unittest.TestCase): + def setUp(self) -> None: + self._temp = tempfile.TemporaryDirectory() + self.addCleanup(self._temp.cleanup) + self.bundle = pathlib.Path(self._temp.name) / "bundle" + self.bundle.mkdir() + (self.bundle / "index.md").write_text(ROOT_INDEX) + (self.bundle / "log.md").write_text(LOG) + (self.bundle / "widget.md").write_text(CONCEPT) + (self.bundle / "gadget.md").write_text(GADGET) + + def test_conformant_bundle_is_accepted(self) -> None: + result = run(self.bundle) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIn("OKF v0.2 conformant", result.stdout) + + def test_missing_frontmatter_is_rejected(self) -> None: + (self.bundle / "widget.md").write_text("# Widget\n") + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("missing YAML frontmatter", result.stderr) + + def test_missing_type_is_rejected(self) -> None: + (self.bundle / "widget.md").write_text( + CONCEPT.replace("type: Subsystem Design\n", "") + ) + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("non-empty 'type'", result.stderr) + + def test_unlisted_concept_is_rejected(self) -> None: + (self.bundle / "orphan.md").write_text(CONCEPT) + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("orphan.md: not listed", result.stderr) + + def test_dangling_link_is_rejected(self) -> None: + (self.bundle / "widget.md").write_text( + CONCEPT.replace("gadget.md", "missing.md") + ) + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("link target does not exist", result.stderr) + + def test_disconnected_concept_is_rejected(self) -> None: + (self.bundle / "widget.md").write_text(CONCEPT.replace("See [Gadget](gadget.md).", "")) + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("links to no other concept", result.stderr) + + def test_repository_path_reference_is_rejected(self) -> None: + (self.bundle / "widget.md").write_text( + CONCEPT + "\nRead `knowledge/gadget.md`.\n" + ) + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("not as repository paths", result.stderr) + + def test_bad_log_heading_is_rejected(self) -> None: + (self.bundle / "log.md").write_text("# Log\n\n## August 2026\n") + result = run(self.bundle) + self.assertEqual(result.returncode, 1) + self.assertIn("is not '## YYYY-MM-DD'", result.stderr) + + +if __name__ == "__main__": + unittest.main()