engineering-quality

Make coding agents prove their patches.

RECON → CONTRACT → CHANGE → VERIFY → REVIEW → EVIDENCE

Less vibes. More invariants.

What it is

engineering-quality is a portable Agent Skill for implementation, review, debugging, refactoring, compatibility, security, and verification work.

It gives coding agents a compact operating protocol:

read the repo
    ↓
model the contract
    ↓
patch the smallest coherent surface
    ↓
try to break it
    ↓
inspect the diff
    ↓
show the evidence

The point is simple:

plausible patch ≠ correct patch
green test       ≠ complete evidence
clean diff        ≠ safe rollout
works             ≠ verified

Install

Codex

$skill-installer Install engineering-quality from https://github.com/GeoGeekLab/engineering-quality

Manual install:

git clone https://github.com/GeoGeekLab/engineering-quality.git \
  ~/.agents/skills/engineering-quality

Claude Code

Standalone Skill:

git clone https://github.com/GeoGeekLab/engineering-quality.git \
  ~/.claude/skills/engineering-quality

Plugin development:

claude --plugin-dir /path/to/engineering-quality

ChatGPT

1. Open the latest release.
2. Download engineering-quality-<version>.zip.
3. Open Plugins → Skills → Create → Upload from your computer.
4. Upload the ZIP.

Cross-agent installer

npx -y skills@1.7.0 add GeoGeekLab/engineering-quality

Host setup notes: docs/compatibility.md

Protocol

1. RECON

Read before editing.

• repository instructions
• architecture and ownership
• nearby tests
• public behavior
• data flow
• existing patterns
• verification commands
• compatibility and security surfaces

2. CONTRACT

Write down the invariant.

must change
must stay stable
must reject
must remain compatible
must be verified

3. CHANGE

Keep the patch tight.

• no drive-by cleanup
• no abstraction for cosmetic deduplication
• no dependency without a reason
• no behavior change hidden inside a refactor
• no hand-editing generated output when a source generator exists

4. VERIFY

Start narrow, then widen.

regression
→ focused tests
→ static checks
→ type / compile
→ broader suite
→ build / package
→ risk-specific probes

5. REVIEW

Read the diff like an attacker and a maintainer.

Check:

• correctness
• edge cases
• failure behavior
• compatibility
• authorization
• path and trust boundaries
• concurrency
• cleanup
• migrations
• generated code
• scope creep

6. EVIDENCE

Report what actually happened.

Verified     = executed and observed
Reasoned     = supported by inspection
Not verified = still open

Where it bites

The Skill is intentionally strict around failure modes that frequently survive a normal patch review:

path traversal        → symlink escape → TOCTOU
object lookup         → authorization → cross-tenant access
schema rename         → mixed versions → rollback
parallelism           → ordering → cancellation / cleanup
public API change     → error type / code / message compatibility
generated code        → source of truth → generator drift
test failure          → retry luck → real flake cause
verification          → command exists → command actually ran
focused task          → unrelated red test → scope expansion

Behavioral evals

The repository ships 17 executable miniature-repository scenarios.

Current coverage includes:

• minimal bug fixes
• public contract evolution
• wrong abstractions
• verification evidence
• path traversal and symlink escape
• TOCTOU file replacement
• tenant authorization
• speculative performance work
• bounded concurrency and cancellation
• behavior-preserving refactors
• dependency pressure
• flaky tests
• mixed-version and rollback-safe migrations
• generated-code drift
• swallowed errors
• public error compatibility
• scope expansion under unrelated failures

Validate the suite:

python scripts/run_evals.py --validate-only

Run one case:

python scripts/run_evals.py \
  --case security-toctou \
  --agent-command 'my-agent --prompt {task}' \
  --adapter-label my-agent \
  --allow-workspace-execution

Run the full suite:

python scripts/run_evals.py \
  --agent-command 'my-agent --prompt {task}' \
  --adapter-label my-agent \
  --allow-workspace-execution \
  --output eval-results/my-agent.json

Skill vs no-Skill

Native Codex and Claude Code adapters support a clean A/B switch:

--skill-mode disabled
--skill-mode enabled

Repeat runs with fresh workspaces:

python scripts/run_evals.py \
  --agent-command '{python} {repo}/scripts/host_eval_adapter.py codex --model MODEL --skill-mode disabled' \
  --adapter-label codex-MODEL-no-skill \
  --repeat 5 \
  --allow-workspace-execution \
  --output eval-results/no-skill.json

python scripts/run_evals.py \
  --agent-command '{python} {repo}/scripts/host_eval_adapter.py codex --model MODEL --skill-mode enabled' \
  --adapter-label codex-MODEL-skill \
  --repeat 5 \
  --allow-workspace-execution \
  --output eval-results/skill.json

Compare:

python scripts/compare_eval_results.py \
  eval-results/no-skill.json \
  eval-results/skill.json \
  --output eval-results/comparison.md

The comparison reports:

case pass rate
check-type pass rate
changed-file scope
agent wall-clock time

Qualitative rubric items stay visible for review instead of being flattened into one score.

More: evals/README.md

Hidden checks

The eval harness keeps second-order probes outside the fixture shown to the agent.

Examples:

security-boundary
  visible:  ../ traversal
  hidden:   symlink escape

security-toctou
  visible:  normal nested read
  hidden:   replace validated file before open

concurrent-worker
  visible:  bounded concurrency + order
  hidden:   propagate failure + stop pending work

database-migration-rollout
  visible:  additive schema migration
  hidden:   backfill + old writer + rollback-safe write

generated-code-change
  visible:  generator + generated file
  hidden:   rerun generator and require zero diff

Two harness checks exist specifically for these cases:

• final_not_claim_any — rejects positive verification claims while allowing negated wording such as not fully verified.
• command_no_changes — executes a command and fails if it changes the workspace.

Inline hidden Python checks are compiled during --validate-only.

Repository checks

Discover likely project checks:

python scripts/project_checks.py /path/to/repository

Run discovered checks after reviewing them:

python scripts/project_checks.py /path/to/repository \
  --run --trust-repository

Run this repository's full quality contract:

make check

That covers:

repository integrity
├── Skill + host metadata
├── local links
├── governance contract
├── immutable GitHub Action pins
├── VERSION + CHANGELOG consistency
├── behavioral eval schema
├── behavioral eval fixtures
├── Python compilation
├── unit tests
├── package verification
└── release invariants

CI runs on Python 3.10, 3.12, and 3.14.

Release integrity

Tagged releases publish:

engineering-quality-<version>.zip
engineering-quality-<version>.zip.sha256

The ZIP includes MANIFEST.sha256.

Verify:

sha256sum -c engineering-quality-<version>.zip.sha256

gh attestation verify engineering-quality-<version>.zip \
  --repo GeoGeekLab/engineering-quality

Package CI does a full round trip:

build
→ SHA-256
→ upload
→ delete local copy
→ download
→ SHA-256

More: docs/release.md

Review protocol

Review findings are ranked by impact.

Severity	Meaning
Blocker	Incorrect behavior, data loss, security exposure, broken contract, or reliably failing verification.
Major	Material failure under realistic conditions or significant operational / maintenance risk.
Minor	Bounded clarity, resilience, test-quality, or consistency issue.
Note	Optional improvement, question, or follow-up.

A useful finding answers four questions:

what can fail?
under what condition?
why does this diff allow it?
what evidence closes the finding?

Quality model

Not this:

quality =
    more abstraction
  + more comments
  + more tests
  + more patterns

Closer to this:

quality =
    correctness
  + clarity
  + compatibility
  + security
  + verifiability
  - accidental complexity
  - unnecessary churn

No magic number. No style points.

Repository layout

SKILL.md                    core protocol
agents/openai.yaml          OpenAI Skill metadata
.claude-plugin/plugin.json  Claude Code plugin metadata

workflows/
├── feature.md
├── bug-fix.md
├── refactor.md
├── review.md
├── debug.md
└── performance.md

references/
├── principles.md
├── testing.md
├── verification.md
├── security.md
├── api-compatibility.md
├── performance-concurrency.md
└── change-discipline.md

scripts/
├── project_checks.py
├── run_evals.py
├── host_eval_adapter.py
├── compare_eval_results.py
├── package_skill.py
└── validate_skill.py

evals/
├── cases.json
├── schema.json
└── result-schema.json

docs/                       compatibility / release / distribution
assets/mascot/              Evi
.github/                    CI / release / repo policy

Project status

• GitHub Actions are pinned to immutable commit SHAs.
• main and v*.*.* are protected by repository rulesets.
• Releases ship SHA-256 checksums and GitHub artifact attestations.
• Behavioral evals use fresh temporary Git repositories.
• The staged Skill payload is hashed before and after each run.
• Eval credentials are forwarded only when explicitly selected.
• Distribution ZIPs contain runtime Skill files, not repository-maintenance tooling.

Contributing

Bring a failure mode, an invariant, or a measurable maintenance win.

Start with CONTRIBUTING.md.

Security issues: SECURITY.md
Governance: GOVERNANCE.md
Code of conduct: CODE_OF_CONDUCT.md

License

MIT — see LICENSE.

---

Read the repo. Respect the contract. Keep the patch tight. Prove the result.

works ≠ verified