Skip to content

Create llm_enhance repo: extract LLM enhancement from phpMan with Source/Transform/Sink abstraction #222

Description

@chedong

Context

Following 2026-07-14 architectural simplification (docs/01-PRODUCT.md §2.12 + docs/05-PLAN.md §"Migration Plan"), LLM enhancement moves out of phpMan into a standalone repo. Must serve multiple content sources, not just phpMan:

  • phpMan: SQLite cache → MD files → enhanced MD
  • myblog: MySQL (Movable Type) → MD files → enhanced MD
  • future: filesystem, HTTP API, etc.

Goal

llm_enhance repo with source-agnostic abstraction. Same LLM transforms + task manager serves all projects.

Key design (full spec docs/05-PLAN.md §1.2-1.5)

Three ABCs:

  • Source: yields Entry objects. Impls: MysqlExportSource (myblog), PhpmanCacheSource (phpMan), FilesystemSource, HttpApiSource.
  • Transform: Entry → enhanced Entry. Versioned (e.g. emoji@v2). Impls: EmojiTransform, OkfTransform, TranslationTransform.
  • Sink: persists results. Impls: FilesystemSink, SqliteSink (phpMan compat), HttpApiSink.

Plus:

  • TaskManager: SQLite-backed queue, resume, rate limiting, cost tracking
  • LLMClient: multi-provider (OpenAI-compat, Anthropic), retry, truncation detection
  • Prompt loader: versioned prompts in src/llm_enhance/prompts/

Migration source map (Phase B)

From phpMan.phpllm_enhance:

From To
enhanceManPage() transform/emoji.py:EmojiTransform.apply()
callLLM() llm/client.py:LLMClient.call()
cleanEmojiHtml() transform/emoji.py:post_process()
cli/batch-enhance.php cli.py:run_command()
LLM config (5 keys) per-project YAML configs

Deleted (only existed for LLM path):

  • formatMarkdownToHTML(), formatInlineMarkdown()
  • renderTocSidebar()

Steps (Phase A+B+C+D, ~3-4 weeks)

  1. W1: Create repo, pyproject.toml, src layout, ABCs + tests
  2. W2: Port cli/batch-enhance.php → Python CLI (queue/resume/rate-limit)
  3. W2-3: Implement MysqlExportSource for myblog MT 4.x/5.x schema
  4. W3: Implement PhpmanCacheSource reading ~/.phpman/cache/db.sqlite
  5. Test: llm_enhance run --config=myblog.yaml --limit=10 --dry-run and same for phpMan

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions