Skip to content
BranchesTags

Repository files navigation

MergeLens

AI-powered GitHub PR risk analysis and code review intelligence platform.

ReviewForge is a production-grade engineering tool that analyzes GitHub pull requests, calculates merge risk, recommends impacted tests, and posts a structured AI-generated report back to the PR.

It is designed to answer one practical reviewer question:

Is this PR risky, what can break, and what should reviewers or testers focus on before merging?

ReviewForge is not intended to be a generic AI code comment bot. The goal is to combine GitHub App integration, diff parsing, repository context, rule-based risk scoring, async workers, PostgreSQL persistence, and LLM-generated reports into a realistic full-stack system.

Why ReviewForge exists

Most AI PR tools only summarize a diff or leave generic comments. Real engineering teams need more useful signals:

  • which modules are affected
  • whether risky files changed
  • whether tests were updated
  • what should QA validate
  • whether a PR touches auth, payment, APIs, database migrations, or config
  • how risky the merge looks before reviewers spend time on it

ReviewForge turns raw pull request changes into a structured risk report that is useful to developers, reviewers, QA engineers, and engineering managers.

Core features

GitHub App integration

ReviewForge runs as a GitHub App. After installation on a repository, GitHub sends pull request webhook events to the ReviewForge API.

Supported pull request events:

  • pull_request.opened
  • pull_request.synchronize
  • pull_request.reopened
  • pull_request.ready_for_review

The backend verifies GitHub webhook signatures before accepting events.

Queue-based PR analysis

Webhook handlers do not perform heavy analysis directly. They validate the webhook, persist the event, and enqueue an analysis job.

This keeps webhook responses fast and makes the system easier to retry, scale, and monitor.

Diff parsing

ReviewForge parses PR diffs to extract useful signals:

  • changed files
  • additions and deletions
  • changed patches
  • file path categories
  • test files changed or missing
  • config changes
  • package/dependency changes
  • migration changes
  • auth/payment/API-related changes

Repository context indexing

ReviewForge analyzes repository structure so risk scoring is not based only on isolated diffs.

The initial MVP uses heuristic import parsing to build relationships such as:

  • file imports
  • route to controller relationships
  • controller to service relationships
  • service to repository relationships
  • nearby test file mapping

The worker now augments path and patch heuristics with Tree-sitter AST context for changed JavaScript and TypeScript files.

Risk scoring engine

Each PR receives a risk level:

  • Low
  • Medium
  • High
  • Critical

Risk is calculated using rules such as:

  • number of changed files
  • total line changes
  • sensitive paths changed
  • database migration changes
  • dependency changes
  • config/environment changes
  • public API changes
  • missing test updates
  • auth/payment/security-related changes
  • core service changes

ReviewForge separates generic risk tags from evidence-backed findings. Generic tags identify review scope; confirmed findings require changed-line evidence, such as leaked environment data, credential-bearing URLs, interpolated SQL, API response contract changes, or unsafe type escapes.

Impacted test recommendations

ReviewForge recommends tests that should be run or updated based on changed files, nearby tests, naming conventions, related directories, and repository context.

Example output:

Recommended tests:
- auth.middleware.test.ts
- checkout-flow.e2e.ts
- payment-validation.integration.test.ts

Multi-pass AI-generated PR report

The LLM receives a carefully prepared context payload instead of the whole repository. For real AI generation, ReviewForge runs focused Gemini Pro reviewer passes before the final synthesis:

  • security review
  • correctness review
  • API contract review
  • final structured report synthesis

The report includes:

  • summary of changes
  • risk level
  • risk reasons
  • affected modules
  • suggested tests
  • reviewer checklist
  • possible breaking changes
  • suggested improvements
  • final recommendation

GitHub PR comment automation

After analysis, ReviewForge posts the generated report as a GitHub PR comment.

Each analysis run creates a fresh ReviewForge comment so reruns and new commits are visible in the PR timeline.

When static analysis finds concrete evidence-backed defects on changed lines, ReviewForge also creates a GitHub pull request review with inline comments on those exact lines. Inline comments are not generated from generic risk signals such as missing tests or API-route changes.

Dashboard

The Next.js dashboard gives a visual view of repositories, pull requests, analysis status, risk levels, generated reports, and trends.

Planned dashboard pages:

  • Login
  • Repositories
  • Pull Requests
  • PR Detail
  • Risk Report
  • Settings
  • Demo Mode

Public demo mode

Recruiters and interviewers may not want to install a GitHub App. ReviewForge includes a demo mode with sample repositories and sample PRs so the full analysis flow can be shown without external setup.

Demo mode includes:

  • sample repository
  • sample pull requests
  • Run Analysis button
  • generated risk report
  • dashboard preview
  • mock GitHub PR comment output

Architecture overview

flowchart TD
    A[GitHub Pull Request Event] --> B[Webhook Receiver]
    B --> C[Verify GitHub Signature]
    C --> D[Persist Webhook Event]
    D --> E[Enqueue Analysis Job]
    E --> F[Redis + BullMQ Queue]
    F --> G[PR Analysis Worker]
    G --> H[Fetch PR Files, Commits, Diff]
    H --> I[Diff Parser]
    I --> J[Repository Context Indexer]
    J --> K[Risk Scoring Engine]
    K --> L[Impacted Test Recommender]
    L --> M[Specialist Gemini Pro Review Passes]
    M --> N[Structured Report Synthesis]
    N --> O[PostgreSQL]
    O --> P[Next.js Dashboard]
    N --> Q[GitHub PR Comment]
Loading

See docs/ARCHITECTURE.md for the detailed component design.

Suggested tech stack

Frontend

  • Next.js
  • TypeScript
  • Tailwind CSS
  • shadcn/ui
  • TanStack Query
  • Recharts

Backend

  • Node.js
  • TypeScript
  • Express, Fastify, or NestJS

Database

  • PostgreSQL
  • Prisma or Drizzle ORM

Queue

  • Redis
  • BullMQ

GitHub integration

  • GitHub App
  • GitHub Webhooks
  • GitHub REST API
  • GitHub GraphQL API later if needed

AI

  • Gemini API, OpenAI API, or Claude API
  • Provider abstraction layer to avoid lock-in

Infrastructure

  • Docker
  • Docker Compose
  • Optional AWS deployment later

Testing

  • Vitest or Jest
  • Playwright for end-to-end testing
  • API integration tests

Monorepo structure

ReviewForge uses npm workspaces defined in the root package.json. The committed lockfile should be package-lock.json.

reviewforge/
  package.json
  package-lock.json
  apps/
    web/
      src/
        app/
        components/
        features/
        lib/
    api/
      src/
        routes/
        controllers/
        services/
        workers/
        queues/
        github/
        diff/
        ai/
        risk/
        db/
  packages/
    shared/
      src/
        types/
        constants/
        utils/
        config/
  docs/
    ARCHITECTURE.md
    SYSTEM_DESIGN.md
    DECISIONS.md
    API.md
  docker-compose.yml
  README.md

Local setup

Prerequisites

Install:

  • Node.js 20+
  • npm
  • Docker
  • Docker Compose
  • GitHub account
  • GitHub App credentials
  • LLM API key

1. Clone the repository

git clone https://github.com/<your-username>/reviewforge.git
cd reviewforge

2. Install dependencies

npm install

3. Start local infrastructure

docker compose up -d

Expected services:

  • PostgreSQL
  • Redis

4. Configure environment variables

Local .env files are created for Docker Compose, the API app, and the web app. Keep real secrets out of source control and fill in the blank GitHub and LLM credentials only for your local environment.

5. Run database migrations

npm run db:migrate

6. Start development processes

npm run dev

In a separate terminal, start the analysis worker:

npm run worker

Expected local services:

  • Web dashboard: http://localhost:3000
  • API server: http://localhost:4000
  • Worker process: BullMQ consumer for the pr-analysis queue

Environment variables

API

NODE_ENV=development
PORT=4000
DATABASE_URL=postgresql://reviewforge:reviewforge@localhost:5432/reviewforge
REDIS_URL=redis://localhost:6379

GITHUB_APP_ID=
GITHUB_APP_PRIVATE_KEY=
GITHUB_WEBHOOK_SECRET=
GITHUB_CLIENT_ID=
GITHUB_CLIENT_SECRET=
SESSION_SECRET=

LLM_PROVIDER=vertex
GEMINI_API_KEY=
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
GOOGLE_CLOUD_PROJECT=project-demo-500016
GOOGLE_CLOUD_LOCATION=global
GOOGLE_GENAI_MODEL=gemini-2.5-pro

APP_BASE_URL=http://localhost:3000
API_BASE_URL=http://localhost:4000

Web

NEXT_PUBLIC_API_BASE_URL=http://localhost:4000
NEXT_PUBLIC_APP_NAME=ReviewForge

Do not commit real secrets. Local .env files should contain development defaults and blank placeholders for credentials until you configure them yourself.

GitHub App setup

Create a GitHub App from GitHub Developer Settings. For local development, expose the API server with a tunnel such as ngrok or Cloudflare Tunnel and use that public URL as the webhook target.

Required app settings

Use these values for local development:

GitHub App name: ReviewForge Local
Homepage URL: http://localhost:3000
Webhook URL: https://<your-tunnel-url>/api/github/webhooks
Callback URL: http://localhost:4000/api/auth/github/callback
Webhook secret: any strong random string, also set as GITHUB_WEBHOOK_SECRET

The webhook URL must use an ngrok or Cloudflare Tunnel HTTPS URL because GitHub sends webhook HTTP requests from GitHub servers. The OAuth callback is a browser redirect; for simplest local login, use http://localhost:4000/api/auth/github/callback so the API session cookie is stored on the same origin used by NEXT_PUBLIC_API_BASE_URL=http://localhost:4000.

If you instead use an ngrok callback URL, set the web app's NEXT_PUBLIC_API_BASE_URL to that same ngrok origin so browser session cookies are sent to the same API host.

Generate a private key for the app, then set:

GITHUB_APP_ID=<numeric app id>
GITHUB_APP_PRIVATE_KEY="<private key with \n escaped newlines>"
GITHUB_WEBHOOK_SECRET=<same secret configured in GitHub>
GITHUB_CLIENT_ID=<GitHub App client id>
GITHUB_CLIENT_SECRET=<GitHub App client secret>
SESSION_SECRET=<64-byte random hex string>

Install the GitHub App on the repositories you want ReviewForge to analyze.

If GitHub redirects to /api/auth/github/callback?installation_id=...&setup_action=install after installation, that only means the app installation flow completed. It is not a trusted OAuth login because it was not started from ReviewForge's signed state flow. ReviewForge redirects that case back to /login?installed=1; click Continue with GitHub there to create your local session.

Required permissions

Repository permissions:

Permission Access Why
Pull requests Read & write Read PR data and post report comments
Contents Read-only Fetch changed files and repository context
Metadata Read-only Required by GitHub Apps
Issues Read & write PR comments use the Issues comments API
Checks Optional read/write Future check-run integration

Webhook events

Subscribe to:

  • Pull request
  • Issue comments

Supported pull request actions are opened, synchronize, reopened, and ready_for_review.

Supported PR comment commands are:

  • /reviewforge review
  • /reviewforge recheck
  • /reviewforge rerun
  • @reviewforge[bot] please review

Comment commands force a reanalysis for the current tracked PR head SHA and post a new ReviewForge report comment instead of editing the previous report.

Local webhook testing

Use a tunneling tool such as ngrok or Cloudflare Tunnel.

Example webhook URL:

https://<tunnel-url>/api/github/webhooks

Set the same webhook secret in GitHub and GITHUB_WEBHOOK_SECRET.

Local login testing

Use the GitHub App OAuth callback configured above and start the API/web apps:

npm run dev

Then open:

http://localhost:3000/login

Click Continue with GitHub. A successful login creates a hashed session record in PostgreSQL and sets an HTTP-only reviewforge_session cookie on the API origin.

For the MVP, personal GitHub App installations are linked to a user when the installation account login matches the GitHub OAuth login. Organization installation authorization requires a later membership/ownership check before showing org repositories.

Vertex AI / Google Gen AI setup

ReviewForge uses the current @google/genai SDK with Vertex AI enabled for real multi-pass structured report generation. Local development uses Google Application Default Credentials from the gcloud CLI.

  1. Install and authenticate the gcloud CLI.
  2. Select the GCP project.
  3. Create local Application Default Credentials.
  4. Enable the Vertex AI API.
gcloud auth application-default login
gcloud config set project project-demo-500016
gcloud auth application-default set-quota-project project-demo-500016
gcloud services enable aiplatform.googleapis.com

Configure the API environment:

LLM_PROVIDER=vertex
GOOGLE_CLOUD_PROJECT=project-demo-500016
GOOGLE_CLOUD_LOCATION=global
GOOGLE_GENAI_MODEL=gemini-2.5-pro

Do not commit service account JSON keys. For local development, prefer gcloud auth application-default login.

Dashboard API

The Next.js dashboard reads from the API server. Key MVP endpoints:

  • GET /api/dashboard - dashboard overview, repositories, recent PRs, and latest analyses
  • GET /api/dashboard/analyses/:analysisId - detailed report data for one analysis run
  • GET /api/demo/dashboard - explicit recruiter-friendly demo dashboard data
  • GET /api/demo/dashboard/analyses/:analysisId - demo analysis detail

The main dashboard at / shows only real database-backed data. If no GitHub repositories are connected yet, it shows setup instructions. Demo data is only shown on /demo.

Authentication status

ReviewForge does not fake account creation or login. The current /login page uses GitHub OAuth and stores HTTP-only session cookies. Real dashboard APIs require authentication and scope data to installations linked to the authenticated user. Demo routes remain public.

Review Quality Evaluation

ReviewForge includes a curated deterministic eval dataset for review quality. It checks that known unsafe PR diffs produce evidence-backed findings, while safe PR diffs avoid false-positive findings.

Run the eval with:

npm run eval:quality --workspace @reviewforge/api

The current eval reports:

  • case pass rate
  • evidence-finding precision
  • evidence-finding recall
  • false-positive finding count
  • missing finding count

The dataset lives in apps/api/src/evals/review-quality.dataset.ts.

Example PR report

## ReviewForge Report

**Risk:** High

### Summary
This PR updates checkout validation and modifies payment service behavior.

### Main Risks
1. Payment validation logic changed without integration test updates.
2. Database migration added, which may affect order persistence.
3. API response shape changed for checkout failures.
4. More than 800 lines changed across multiple modules.

### Affected Modules
- Checkout
- Payment
- Order service
- Database schema

### Suggested Tests
- checkout-flow.e2e.ts
- payment-validation.integration.test.ts
- order-service.test.ts

### Reviewer Checklist
- Verify failed payment behavior.
- Confirm API response compatibility.
- Check rollback behavior for failed orders.
- Validate migration behavior locally.

### Final Recommendation
Merge only after payment integration tests and checkout E2E tests pass.

Screenshots

Add screenshots after the first dashboard implementation.

Recommended screenshots:

  1. Repository list with risk summary
  2. Pull request list with analysis status
  3. PR detail page with risk report
  4. Demo mode analysis result
  5. GitHub PR comment generated by ReviewForge
![Dashboard screenshot](./docs/assets/dashboard.png)
![PR report screenshot](./docs/assets/pr-report.png)

Demo video

Add a short demo video after MVP completion.

Recommended flow:

  1. Open dashboard
  2. Show demo repository
  3. Run sample PR analysis
  4. Show risk score and suggested tests
  5. Show generated GitHub-style PR comment
  6. Explain async worker flow briefly
Demo video: <add link here>

Live demo

Live demo: <add link here>

The public demo should not require GitHub App installation.

MVP scope

The first version should include:

  • GitHub webhook receiver
  • PR data fetching
  • Redis/BullMQ analysis queue
  • PR analysis worker
  • diff parser
  • rule-based risk scoring
  • impacted test recommendation
  • LLM-generated PR report
  • GitHub PR comment posting
  • evidence-backed inline GitHub review comments
  • PostgreSQL storage
  • basic dashboard
  • public demo mode

Future improvements

  • deeper AST-based code analysis beyond changed JavaScript/TypeScript files
  • Tree-sitter dependency graph expansion
  • dependency graph visualization
  • semantic code search
  • historical PR risk trends
  • organization/team dashboard
  • Slack alerts
  • custom rules per repository
  • reviewer assignment suggestions
  • test coverage integration
  • CI/CD failure prediction
  • security vulnerability detection
  • self-hosted deployment mode

Engineering principles

ReviewForge should look like an engineered system, not a one-shot AI wrapper.

Project standards:

  • meaningful commits
  • clean architecture
  • documented decisions
  • environment variables for secrets
  • webhook signature verification
  • background jobs for heavy analysis
  • PostgreSQL persistence
  • Redis/BullMQ queueing
  • GitHub API rate-limit handling
  • structured LLM output
  • review quality eval dataset
  • graceful failure handling
  • test coverage for core logic
  • polished recruiter-friendly dashboard

Documentation

  • docs/ARCHITECTURE.md explains components, data flow, database design, risk scoring, AI report generation, and GitHub comments.
  • docs/SYSTEM_DESIGN.md explains scalability, queues, failure handling, retries, idempotency, LLM limits, and security.
  • docs/DECISIONS.md records important engineering decisions and trade-offs.

Interview talking points

After building ReviewForge, you should be able to confidently explain:

  • how GitHub Apps and webhooks work
  • how webhook signature verification protects the backend
  • why webhook handlers should be fast
  • why async workers are used
  • how PR diffs are fetched and parsed
  • how risk scoring is calculated
  • how impacted tests are recommended
  • how repository context is indexed
  • how LLM context is prepared
  • how the system avoids sending the whole repo to an LLM
  • how GitHub API rate limits are handled
  • how duplicate PR comments are avoided
  • how jobs are retried safely
  • how the database schema supports analysis history
  • how the system can scale to many repositories
  • how tokens and secrets are secured

License

MIT License. Update this section if a different license is chosen.

About

MergeLens is an AI-powered GitHub App for pull request risk analysis. It reviews diffs asynchronously, scores merge risk, recommends impacted tests, and generates structured evidence-backed reports. Results are posted back to GitHub, stored in PostgreSQL, and visible in a dashboard.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages