AI Code Review with DeepSeek — Headless PR Review Automation

AI code review with DeepSeek: headless PR review automation that verifies
PR descriptions claim-by-claim against real code, checks docs against reality,
and flags requirement impact — with human-in-the-loop only when it matters.

Why

PR descriptions lie. Docs go stale. Manual code review is slow and
inconsistent. This tool runs a DeepSeek Harness agent that:

• Verifies PR descriptions claim-by-claim — each sentence of the
  description is checked against the actual code, with file:line evidence
• Detects stale and fabricated docs — up to 60% of repo docs are wrong;
  the agent compares them against real code (MATCH / STALE / WRONG / FABRICATED)
• Flags requirement impact — which business requirements a change touches,
  and whether it breaks something (CHANGED / BROKEN / RISK)
• Runs headless — one command, or an auto-review poller that watches every
  new PR

Demo

Dashboard: per-repo KPIs, verdict distribution, and every open PR with review
status. Live demo data is included — see Web dashboard.

Features

Install

Requirements: Python 3.10+ (recommended 3.11), gh CLI already authenticated.

python -m venv .venv && . .venv/bin/activate
pip install -e '.[dev]'   # zsh needs quotes; SDK comes from PyPI (deepseek-harness-sdk)
gh auth login          # required
export DEEPSEEK_API_KEY=sk-...   # see .env.example

Usage

After pip install -e '.[dev]' you get two commands:

harness-pr-review doctor                # check readiness: Python, gh, API key, SDK
harness-pr-review owner/repo 123        # review one PR (interactive)
harness-pr-review owner/repo 123 --skip-human   # batch, no questions
harness-pr-review owner/repo 123 --no-post      # don't post a comment
autoreview --once                       # auto review: single pass
autoreview --daemon                     # auto review: every interval_minutes

(Or run from source: PYTHONPATH=src python -m src.run owner/repo 123)

Results land in sessions/<owner>/<repo>/pr-<n>/report.md (change the directory with DSH_SESSION_ROOT).

Pipeline

1. Snapshot — fetch PR metadata, diff files, commits, review threads (GitHub REST + GraphQL)
2. Claims — LLM splits the description into verifiable claims
3. Verify — DeepSeek Harness agent deep-dives in a disposable worktree:
   verifies each claim, docs reality-check (MATCH/STALE/WRONG/FABRICATED),
   requirement impact, review thread status
4. Human gate — asks for confirmation (≤20 words/question) when docs are wrong or claims are uncertain
5. Synthesize — English report.md + one English comment on the PR (idempotent)

Running tests

python -m pytest -v

Web dashboard

Read-only dashboard for review metrics (PRs reviewed, bugs, doc errors, verdicts
per repo). Reads sessions/ directly — no database.

pip install -e '.[web]'
DSH_SESSION_ROOT=sessions python -m web.server
# open http://127.0.0.1:6789

Pages: repo list → repo detail (KPIs + verdict donut + PR table) → PR detail
(tabs: Claims / Docs / Impact / Threads / Confirm). The PR table lists ALL open
PRs from GitHub with review status (Not reviewed / Reviewing / Reviewed N
rounds); Bugs counts FAIL + PARTIAL claims and BROKEN + RISK impacts; Doc
errors counts WRONG + FABRICATED + STALE docs. Each open PR row has a
Review now / Re-review button that runs the review synchronously using
the repo's auto-review config (skip-human + post-comment flags from
autoreview.yml).

Demo data is checked into sessions/demo/app/ — start the server and open
http://127.0.0.1:6789/repos/demo/app/pr/7 for a sample review (PR #8 shows a
MISLEADING verdict + FABRICATED doc), useful for screenshots and documentation.

Auto review

Poll GitHub for new PRs (and head-SHA changes) and review them automatically in
batch mode. Each repo is configured auto (poller reviews its PRs) or manual
(poller skips it; review via CLI). Edit autoreview.yml directly, via CLI, or
from the web dashboard (repo list page → toggle Auto/Manual).

# autoreview.yml
org: sample-org            # default org for repo discovery
default_mode: manual        # repos not listed → manual
interval_minutes: 2
post_comment: true
skip_human: true
drafts: false
repos:
  sample-app: auto
  sample-api: manual

python -m src.autoreview --add-repo sample-app --mode auto   # enable auto
python -m src.autoreview --rm-repo sample-app                # remove
python -m src.autoreview --repos                             # list status
python -m src.autoreview --once          # single pass (cron/launchd)
python -m src.autoreview --daemon        # loop every interval_minutes

launchd example (auto-start on login, every 2 minutes):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key><string>com.nexpeak.pr-review</string>
  <key>ProgramArguments</key>
  <array>
    <string>/Users/gianglh/work/harness/scripts/autoreview-once.sh</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>StartInterval</key><integer>120</integer>
  <key>StandardOutPath</key>
  <string>/Users/gianglh/work/harness/autoreview.log</string>
  <key>StandardErrorPath</key>
  <string>/Users/gianglh/work/harness/autoreview.log</string>
</dict>
</plist>

scripts/autoreview-once.sh sources .env (API key stays out of the plist).
Install:

cp com.nexpeak.pr-review.plist ~/Library/LaunchAgents/
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.nexpeak.pr-review.plist

Re-review rules: head SHA in the PR changed vs the last snapshot → all phases
re-run with --force; the PR comment is updated in place (never duplicated).

Configuration

License

MIT © 2026 Nexpeak