chore(knowledge): migrate specs to OKF

.claude/commands/ship.md

@@ -29,16 +29,16 @@ Review the changes on this branch (`git diff origin/main...HEAD` and
    - Positive tests: happy path, valid inputs, expected state transitions
    - Negative tests: invalid inputs, error conditions, boundary cases
    - Security tests: if network, URL parsing, filtering, or HTML conversion changed,
-     add or extend tests tied to `specs/threat-model.md`
+     add or extend tests tied to `knowledge/security/threat-model.md`
 4. Run all tests: `cargo test --workspace`
 5. If any test fails, fix the code or test until green
 
 ### Phase 3: Artifact Updates
 
 Review the change and update affected artifacts. Skip items that are not touched.
 
-1. Specs in `specs/`
-2. Threat model in `specs/threat-model.md` for new attack surfaces or mitigations
+1. Relevant concepts in `knowledge/`
+2. Threat model in `knowledge/security/threat-model.md` for new attack surfaces or mitigations
 3. Release process docs/spec if shipping or release behavior changed
 4. `AGENTS.md` if workflow, commands, or repo guidance changed
 5. Public docs in `docs/` if user-facing behavior changed
@@ -68,7 +68,7 @@ Analyze all changed code for security vulnerabilities.
 6. Unsafe code usage
 
 If security issues are found, fix them, add regression tests, and update
-`specs/threat-model.md` if a new threat must be tracked.
+`knowledge/security/threat-model.md` if a new threat must be tracked.
 
 ### Phase 4: Smoke Testing
 

.claude/skills/process-issues/SKILL.md

@@ -42,8 +42,8 @@ For each qualifying issue (ordered by issue number), achieve ALL of these before
 
 - Minimal, focused changes
 - Positive and negative tests pass
-- Security tests added if change touches URL parsing, fetchers, HTML conversion, network, or user input (per `specs/threat-model.md`)
-- Threat model updated if new attack surface (per `specs/threat-model.md`)
+- Security tests added if change touches URL parsing, fetchers, HTML conversion, network, or user input (per `knowledge/security/threat-model.md`)
+- Threat model updated if new attack surface (per `knowledge/security/threat-model.md`)
 
 ### 4. Ship via `/ship`
 

.github/pull_request_template.md

@@ -19,4 +19,4 @@ when possible). For changes with no observable behavior (pure refactor, docs), s
 - [ ] Unit tests are passed
 - [ ] Smoke tests are passed
 - [ ] Documentation is updated
-- [ ] Specs are up to date and not in conflict
+- [ ] Knowledge is up to date and not in conflict

.github/workflows/ci.yml

@@ -33,6 +33,14 @@ jobs:
         run: cargo fmt --all -- --check
       - name: Clippy
         run: cargo clippy --workspace --all-targets -- -D warnings
+      - name: Install okf-lint
+        run: cargo install okf-lint --version 0.1.1 --locked
+      - name: Check knowledge bundle
+        run: |
+          python3 scripts/check_okf.py knowledge
+          okf-lint knowledge --max-line-length 10000
+      - name: Test repository scripts
+        run: python3 -m unittest discover -s scripts/tests -p 'test_*.py'
 
   test:
     name: Test

AGENTS.md

@@ -35,19 +35,19 @@ Key capabilities:
 - URL filtering via allow/block lists
 - MCP server for AI tool integration
 
-### Specs
+### Knowledge
 
-`specs/` folder contains feature specifications outlining requirements for specific features and components. New code should comply with these specifications or propose changes to them.
+`knowledge/` is the canonical OKF v0.2 bundle and persistent project memory. Read relevant knowledge before changing behavior. Update it in the same change when decisions, behavior, constraints, threats, tests, or operations change. See `knowledge/knowledge-contract.md` for maintenance rules and run `python3 scripts/check_okf.py knowledge` after edits.
 
-Available specs:
-- `specs/initial.md` - WebFetch tool specification (types, behavior, conversions, error handling)
-- `specs/fetchers.md` - Pluggable fetcher system for URL-specific handling
-- `specs/release-process.md` - Agent-driven release and publish workflow
-- `specs/maintenance.md` - Periodic maintenance checklist (deps, docs, spec-code alignment)
-- `specs/threat-model.md` - Security threat model (SSRF, network, input validation, DoS)
-- `specs/bot-auth.md` - Web Bot Authentication (draft-meunier-web-bot-auth-architecture)
-
-Specification format: Abstract and Requirements sections.
+Available knowledge:
+- `knowledge/knowledge-contract.md` - Knowledge maintenance and OKF conformance rules
+- `knowledge/foundations/tool-contract.md` - Library, CLI, MCP, and Python behavior
+- `knowledge/foundations/fetchers.md` - Pluggable fetcher and content processor system
+- `knowledge/integrations/agent-discovery.md` - Bounded agent resource discovery
+- `knowledge/security/threat-model.md` - Security threats and mitigations
+- `knowledge/security/bot-auth.md` - Web Bot Authentication design
+- `knowledge/operations/maintenance.md` - Periodic maintenance checklist
+- `knowledge/operations/release-process.md` - Agent-driven release and publish workflow
 
 ### Shipping
 
@@ -104,7 +104,7 @@ crates/
 ├── fetchkit/           # Core library - types, fetch logic, HTML conversion
 ├── fetchkit-cli/       # CLI binary and MCP server
 └── fetchkit-python/    # Python bindings (PyO3)
-specs/                  # Feature specifications
+knowledge/              # Canonical OKF v0.2 engineering knowledge
 ```
 
 ### Naming
@@ -126,7 +126,7 @@ specs/                  # Feature specifications
 
 ### Releasing
 
-See `specs/release-process.md` for the release contract.
+See `knowledge/operations/release-process.md` for the release contract.
 
 Quick summary:
 1. Human asks agent: "Create release v0.2.0"
@@ -212,7 +212,7 @@ Before creating a pull request, ensure:
 
 8. **PR comments resolved**: No unaddressed review comments in PR
 
-9. **Specs**: If changes affect system behavior, update specs in `specs/`
+9. **Knowledge**: If changes affect durable engineering knowledge, update `knowledge/` and run its OKF checks
 
 10. **Docs**: If changes affect usage or configuration, update public docs in `docs/`
 
@@ -290,7 +290,7 @@ when possible). For changes with no observable behavior (pure refactor, docs), s
 - [ ] Unit tests are passed
 - [ ] Smoke tests are passed
 - [ ] Documentation is updated
-- [ ] Specs are up to date and not in conflict
+- [ ] Knowledge is up to date and not in conflict
 ```
 
 ### Testing the system
@@ -312,4 +312,4 @@ cargo run -p webfetch-cli -- --url https://example.com --as-markdown
 cargo run -p webfetch-cli -- mcp
 ```
 
-Tests use `wiremock` for HTTP mocking (no real external network calls). See `specs/initial.md` for test requirements.
+Tests use `wiremock` for HTTP mocking (no real external network calls). See `knowledge/foundations/tool-contract.md` for test requirements.

CHANGELOG.md

@@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Replace the specification directory with an indexed OKF v0.2 knowledge
+  bundle, maintenance contract, update log, and CI conformance checks.
 - Add a pluggable post-download `ContentProcessor` registry and built-in
   `PdfProcessor` that extracts Markdown from text-based PDFs with local
   `pdf-inspector` processing and explicit OCR guidance for unsupported pages.

README.md

@@ -316,7 +316,7 @@ DNS pinning prevents DNS rebinding attacks. IPv6-mapped IPv4 addresses are canon
 Redirects are followed manually in the default fetcher so each hop is revalidated against scheme and DNS policy. Allow/block prefixes are matched against parsed URLs rather than raw strings, which prevents lookalike host overmatches such as `allowed.example.com.evil.test`.
 Proxy environment variables are ignored by default. Use the hardened profile for cluster-facing deployments and opt in with `ToolBuilder::respect_proxy_env(true)` only when it is part of an intentional egress design.
 
-See [`specs/threat-model.md`](specs/threat-model.md) for the full threat model.
+See the [`knowledge/security/threat-model.md`](knowledge/security/threat-model.md) concept for the full threat model.
 See [`docs/hardening.md`](docs/hardening.md) for deployment guidance.
 
 ## Configuration

crates/fetchkit/tests/ssrf_security.rs

@@ -2,7 +2,7 @@
 //!
 //! Tests that validate the resolve-then-check DNS policy prevents
 //! server-side request forgery attacks. These tests verify the threat
-//! mitigations documented in specs/threat-model.md.
+//! mitigations documented in knowledge/security/threat-model.md.
 //!
 //! Safe-by-default: Tool::default() and fetch() block private IPs.
 //! Tests that need loopback (wiremock) must explicitly opt out.

docs/security.md

@@ -31,7 +31,7 @@ let tool = ToolBuilder::new()
     .build();
 ```
 
-See [`specs/threat-model.md`](../specs/threat-model.md) for the full threat inventory.
+See [`knowledge/security/threat-model.md`](../knowledge/security/threat-model.md) for the full threat inventory.
 
 ## Web Bot Authentication
 
@@ -59,4 +59,4 @@ CLI usage:
 fetchkit fetch https://example.com --bot-auth-key <base64url-seed> --bot-auth-agent bot.example.com
 ```
 
-See [`specs/bot-auth.md`](../specs/bot-auth.md) for the full specification.
+See [`knowledge/security/bot-auth.md`](../knowledge/security/bot-auth.md) for the full protocol design.

specs/fetchers.md → knowledge/foundations/fetchers.md

@@ -1,4 +1,14 @@
-# Fetcher System Specification
+---
+type: Subsystem Design
+title: Fetcher System
+description: URL-specific fetchers, content processors, transport policy, extension points, and tests.
+tags:
+  - fetchkit
+  - fetching
+  - architecture
+---
+
+# Fetcher System
 
 ## Abstract
 
@@ -354,7 +364,7 @@ Both built-in fetchers integrate resolve-then-check DNS validation:
 - Enabled by default via `DnsPolicy::default()` (blocks private IPs)
 - Ignore ambient proxy env by default so shared runtimes do not silently route
   traffic through operator-provided proxies unless explicitly enabled
-- See `specs/threat-model.md` for threat IDs: TM-SSRF-001 through TM-SSRF-010
+- See the [Threat Model](../security/threat-model.md) for threat IDs: TM-SSRF-001 through TM-SSRF-010.
 
 ## Module Structure
 
@@ -448,3 +458,8 @@ Tests real URLs:
 3. Add `mod {name};` and `pub use {name}::*;` to `mod.rs`
 4. Register in `FetcherRegistry::with_defaults()` (before DefaultFetcher)
 5. Add test cases to `examples/fetch_urls.rs`
+
+## See also
+
+- [Fetchkit Tool Contract](tool-contract.md) — shared request, response, and policy behavior
+- [Threat Model](../security/threat-model.md) — network and SSRF requirements applied to every fetcher

knowledge/foundations/index.md

@@ -0,0 +1,4 @@
+# Foundations
+
+* [Fetchkit Tool Contract](tool-contract.md) - Public library, CLI, MCP, and Python behavior for fetching and converting web content.
+* [Fetcher System](fetchers.md) - URL-specific fetchers, content processors, transport policy, extension points, and tests.

specs/initial.md → knowledge/foundations/tool-contract.md

@@ -1,10 +1,20 @@
+---
+type: Interface Contract
+title: Fetchkit Tool Contract
+description: Public library, CLI, MCP, and Python behavior for fetching and converting web content.
+tags:
+  - fetchkit
+  - api
+  - fetching
+---
+
 # Decisions:
-# - Spec mirrors current Fetchkit tool behavior (no new features) unless noted below.
+# - Contract mirrors current Fetchkit tool behavior (no new features) unless noted below.
 # - Rust is the source of truth: library + CLI + MCP server + Python bindings.
 # - HTML conversion is built-in (no external HTML conversion deps).
 # - `FetchRequest` and `FetchResponse` are defined in this crate (no external dependency).
 
-# Fetchkit Specification
+# Fetchkit Tool Contract
 
 ## Abstract
 
@@ -251,7 +261,7 @@ By default, Fetchkit blocks connections to private/reserved IP ranges:
 - Handles IPv6-mapped IPv4 addresses via canonicalization.
 - Pins validated IP via `reqwest::ClientBuilder::resolve()` to prevent DNS rebinding.
 - Blocked by default; opt out via `ToolBuilder::block_private_ips(false)`.
-- See `specs/threat-model.md` for full threat analysis.
+- See the [Threat Model](../security/threat-model.md) for full threat analysis.
 
 ### HTTP Behavior
 
@@ -453,3 +463,8 @@ SSRF security:
 - Default-blocks-loopback verification.
 - Explicit opt-out verification.
 - Script stripping in converted content.
+
+## See also
+
+- [Fetcher System](fetchers.md) — URL-specific retrieval and content processing architecture
+- [Threat Model](../security/threat-model.md) — security boundaries and mitigation requirements

knowledge/index.md

@@ -0,0 +1,15 @@
+---
+okf_version: "0.2"
+---
+
+# Fetchkit Knowledge
+
+* [Knowledge Maintenance Contract](knowledge-contract.md) - Rules for maintaining Fetchkit's OKF bundle.
+* [Update Log](log.md) - Chronological history of changes to this bundle.
+
+# Domains
+
+* [foundations/](foundations/) - Public tool contracts and core fetching architecture.
+* [integrations/](integrations/) - Agent-facing discovery and integration contracts.
+* [security/](security/) - Threat analysis and authentication protocol design.
+* [operations/](operations/) - Maintenance and release playbooks.

specs/agent-discovery.md → knowledge/integrations/agent-discovery.md

@@ -1,3 +1,13 @@
+---
+type: Interface Contract
+title: Agent Resource Discovery
+description: Bounded discovery and reporting of same-origin resources intended for AI agents.
+tags:
+  - fetchkit
+  - agents
+  - discovery
+---
+
 # Agent Resource Discovery
 
 ## Abstract
@@ -27,3 +37,8 @@ navigation links to Markdown output.
     requested and validated that exact resource.
 11. Discovery MUST NOT invoke APIs, authorization flows, payment protocols, or
     agent capabilities.
+
+## See also
+
+- [Fetcher System](../foundations/fetchers.md) — transport and URL policy used by discovery probes
+- [Threat Model](../security/threat-model.md) — discovery amplification and network-policy threats

knowledge/integrations/index.md

@@ -0,0 +1,3 @@
+# Integrations
+
+* [Agent Resource Discovery](agent-discovery.md) - Bounded discovery and reporting of same-origin resources intended for AI agents.

knowledge/knowledge-contract.md

@@ -0,0 +1,63 @@
+---
+type: Playbook
+title: Knowledge Maintenance Contract
+description: Rules for maintaining the Fetchkit knowledge bundle and its OKF conformance.
+tags:
+  - fetchkit
+  - knowledge
+  - okf
+  - process
+---
+
+# Knowledge Maintenance Contract
+
+`knowledge/` is Fetchkit's canonical [Open Knowledge Format (OKF) v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) bundle and persistent project memory.
+
+## Maintenance rules
+
+- Treat this knowledge as part of the implementation, not as historical documentation.
+- Before changing behavior, read the relevant concepts and follow their decisions or update them in the same change.
+- When code changes a documented behavior, design decision, invariant, limitation, threat, test strategy, or operational process, update the affected knowledge in the same pull request.
+- Record important decisions that are not recoverable from code. Prefer links to source and tests over duplicating volatile implementation details.
+- Keep stable identifiers such as `TM-*` and `R-*`; never renumber them.
+- Add durable engineering knowledge here. User-facing guides remain in `docs/`.
+
+## OKF conformance rules
+
+The bundle targets OKF v0.2, declared as `okf_version: "0.2"` in the bundle-root [index](index.md).
+
+- Every Markdown file except reserved `index.md` and `log.md` files is a concept and starts with YAML frontmatter containing a non-empty `type`.
+- Concepts also carry `title`, a single-line `description`, and useful `tags`.
+- Directory indexes contain link lists for concepts and immediate subdirectories only.
+- The update log uses `## YYYY-MM-DD` headings, newest first.
+- Links between concepts are relative and resolve inside the bundle.
+- Every concept links to another concept so agents can traverse the bundle as a graph.
+- Reference bundle documents with relative Markdown links, not repository-path text that can silently rot.
+
+OKF provenance, trust, lifecycle, and attestation metadata remain optional. If generated concepts are added, they must identify their `resource` and `generated.by` actor so readers can distinguish generated facts from hand-maintained knowledge.
+
+## Layout
+
+| Directory | Holds |
+|---|---|
+| [foundations/](foundations/) | Tool behavior and fetcher architecture |
+| [integrations/](integrations/) | Agent-facing integration contracts |
+| [security/](security/) | Threat model and authentication design |
+| [operations/](operations/) | Maintenance and release playbooks |
+
+## Enforcement
+
+Run both checks after changing the bundle:
+
+```console
+$ python3 scripts/check_okf.py knowledge
+knowledge: OKF v0.2 conformant (8 concepts, 5 index files, 1 log file)
+$ okf-lint knowledge --max-line-length 10000
+```
+
+The upstream linter enforces OKF v0.2. The local checker adds bundle conventions the format intentionally leaves soft: complete indexes, resolvable graph links, required descriptions, and generated-resource metadata. CI pins `okf-lint` to a reviewed version.
+
+## See also
+
+- [Periodic Maintenance](operations/maintenance.md) — broader repository drift and health checks
+- [Fetchkit Tool Contract](foundations/tool-contract.md) — primary behavior contract maintained in this bundle

knowledge/log.md

@@ -0,0 +1,6 @@
+# Fetchkit Knowledge Update Log
+
+## 2026-08-08
+
+* **Migration**: Replaced the unindexed specification directory with an OKF v0.2 bundle organized by foundations, integrations, security, and operations.
+* **Process**: Added a maintenance contract and automated conformance checks so durable engineering knowledge changes with the implementation.

knowledge/operations/index.md

@@ -0,0 +1,4 @@
+# Operations
+
+* [Periodic Maintenance](maintenance.md) - Recurring dependency, documentation, security, compatibility, and release-alignment checks.
+* [Release Process](release-process.md) - Agent-driven version preparation, validation, GitHub release creation, and crates.io publishing.