feat(skill-analysis): add --analyze-skill mode for token efficiency scanning - #17
Merged
Conversation
…canning Adds static token analysis for AI agent skill packages, enabling Tank's security pipeline to also report token usage findings. New: - src/skill-analyzer.ts: engine that discovers skill files and runs rules - src/skill-rules/prompt-size.ts: flags oversized SKILL.md/atom files - src/skill-rules/claude-md-size.ts: flags large CLAUDE.md injections - src/skill-rules/tool-overhead.ts: counts tool definitions in manifest - src/skill-rules/large-files.ts: flags files >500 lines - src/skill-rules/redundant-instructions.ts: detects duplicated instructions - tests/skill-analyzer.test.ts: 26 tests covering all rules - 9 files changed, 26 tests added (208 total passing) - Bump version to 2.3.0 Co-Authored-By: Claude <noreply@anthropic.com>
… to v2.3 - Add Skill Analysis section with output format, rules table, efficiency score - Add integration example for calling from Python (Tank pipeline) - Add --analyze-skill to Options table - Update version description from v2.2 to v2.3 Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…y detection New section-analysis rule that parses markdown headings and provides: - Per-section token count breakdown - Cross-section redundancy detection (Jaccard similarity >40%) - Shortening tips: repeated lines, excessive code blocks, long lists, long paragraphs - Sections array in findings for structured consumption - src/skill-rules/section-analysis.ts: new rule with markdown section parser - src/types.ts: added SkillSection interface with tokens, redundantWith, shorteningTip - src/skill-analyzer.ts: wired section-analysis rule - tests/skill-analyzer.test.ts: 10 new tests (36 total for skill analyzer) Co-Authored-By: Claude <noreply@anthropic.com>
…breakdown Major improvements to section-analysis rule based on testing against real Tank skills (@tank/bdd-e2e-testing, @tank/idd-bdd-pack): - Cross-file redundancy: detects when SKILL.md sections duplicate content from references/*.md files, with specific file+section attribution - Always-report: files over 1000 tokens get section breakdown even if no issues found (info severity) - Lowered Jaccard threshold from 40% to 25% for better recall on real skills - Table-heavy section detection (>15 table rows) - Top-5 section token breakdown in description for quick scan - Fixed within-file vs cross-file distinction in remediation messages Co-Authored-By: Claude <noreply@anthropic.com>
New content-aware tips beyond structural suggestions: - Concept repetition: detects key phrases restated 3+ times across a section — suggests stating the rule once and referencing it - Restatement detection: flags "remember that", "note that", "in other words", "this is important because" — AI follows instructions without persuasion, so justifications are token waste - Bad/good example pairs: when a section shows both wrong and right patterns extensively, suggests stating the rule as a negative constraint instead of doubling content with examples - Specific file path overenumeration: flags sections referencing 6+ file paths — suggests glob patterns or naming conventions - Verbose justification: flags "this is important because" / "the reason for this is" — remove persuasion, keep the rule - Large directory trees: flags ASCII trees over 10 lines — suggest keeping only top-level and linking to reference Tested against @tank/bdd-e2e-testing and @tank/idd-bdd-pack skills. Co-Authored-By: Claude <noreply@anthropic.com>
Enriched the analysis report with human-facing metrics:
- Cost per use: real money estimates (~$0.18 Sonnet / ~$0.91 Opus for
avg skill) based on 80/20 input/output split pricing
- Grade: single letter A/B/C/D instead of raw 0-100 score
- One-liner: plain English summary at the top
("Slightly above average size. Works fine, could be leaner.")
- Comparison: token count vs average (~20k tokens)
- Size bar: visual indicator with avg marker
- "What this means": 2-sentence blurb about cost/speed tradeoff
- Human-readable terminal output by default; JSON only with --json
Co-Authored-By: Claude <noreply@anthropic.com>
Replaced 120-char/100-char truncation with word-wrapping at 76 chars. All finding descriptions and remediation lines are now fully visible in terminal output, wrapped to fit the display. Co-Authored-By: Claude <noreply@anthropic.com>
- Terminal output example with grade, size bar, cost per use - JSON output example with all new fields - Report fields table (one_liner, grade, comparison, cost_per_use, etc.) - Section analysis details: cross-file redundancy, content-level tips table - Cost estimation table with real money examples - Bump terminal example version to v2.3.0 Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tokenomics --analyze-skill <dir> --jsonCLI mode for static token analysis of AI agent skill packagesWhat's new
src/skill-analyzer.tssrc/skill-rules/prompt-size.tssrc/skill-rules/claude-md-size.tssrc/skill-rules/tool-overhead.tssrc/skill-rules/large-files.tssrc/skill-rules/redundant-instructions.tstests/skill-analyzer.test.tsModified
src/analyze.ts--analyze-skill <dir>CLI flagsrc/types.tsSkillFinding,SkillAnalysisResult,SkillRuletypespackage.jsonTest plan
tokenomics --analyze-skill <dir>produces valid JSON🤖 Generated with Claude Code