vulnify

A CVE ingestion and enrichment pipeline that builds a single normalised SQLite
database stitched from CVEProject, NVD, CISA KEV, EPSS, and OSV — plus a
Streamlit explorer for the result.

Ships as a fully populated SQLite DB on each release so you don't have to wait
hours for the initial ingest.

What's in it

• Full CVE corpus from the upstream CVEProject cvelistV5 zip
• NVD bulk merge for CVSS metrics, CPE configurations, vuln status, and
  references
• CISA Known Exploited Vulnerabilities with date_added, due_date,
  ransomware flag, and required action
• EPSS exploit probability + percentile, batched
• OSV package mapping per CVE
• Streamlit explorer with ~70 pre-built views over the data
• Resume-safe pipeline state — interruptions pick up where they left off

Current snapshot (May 2026 release):

Documentation

Deeper references live in docs/:

• Data sources — every upstream feed, endpoint, and licence.
• Pipeline — ingestion → enrichment, watermarks/resume, the cascade+heal model, tri-state discipline.
• Schema — the 20-table layout and query gotchas.
• MCP server — the 7 tools, per-agent setup, troubleshooting.
• Explorer — the Streamlit dashboard, tab by tab.

Quick start (use the pre-built DB)

If you just want to query the data without running the pipeline:

# 1. Clone and pick the extras you need
git clone https://github.com/mez-0/vulnify.git
cd vulnify
uv sync --extra mcp          # for agents / MCP server
# or:  uv sync --extra explore   # for the Streamlit dashboard
# or:  uv sync --all-extras      # for everything

# 2. Grab the latest release DB
gh release download -R mez-0/vulnify --pattern vulnify.db.zst
zstd -d vulnify.db.zst -o vulnify.db

# 3. Point .env at the DB
cat > .env <<EOF
VULNIFY_SQLITE_PATH=$(pwd)/vulnify.db
VULNIFY_SCHEMA_SQL_PATH=vulnify/db/schema.sql
EOF

# 4a. Use it from an agent (see "MCP server" below for full setup)
uv run vulnify-mcp </dev/null     # smoke test — should start then exit on EOF

# 4b. Or browse it in the Streamlit explorer
uv run streamlit run explore/app.py

# 4c. Or just query the SQLite directly
sqlite3 vulnify.db "SELECT cve_id, summary FROM cve WHERE cve_id = 'CVE-2024-3094';"

Building the DB from scratch

uv sync
cp .env.example .env       # edit paths as needed
uv run vulnify-gather

The first run takes several hours without an NVD API key (the public NVD
rate limit is ~5 requests / 30 seconds). With a key it's roughly 10x faster.

Get a free key at https://nvd.nist.gov/developers/request-an-api-key and set
VULNIFY_NVD_API_KEY in .env.

Skipping phases

vulnify-gather exposes flags so you can re-run individual enrichment stages
without re-ingesting:

vulnify-gather --skip-ingestion        # don't re-download the CVEProject zip
vulnify-gather --skip-nvd              # skip NVD bulk merge
vulnify-gather --skip-kev              # skip CISA KEV
vulnify-gather --skip-epss             # skip EPSS batch
vulnify-gather --skip-osv              # skip OSV per-CVE lookups
vulnify-gather --skip-nuclei          # skip Nuclei exploit-template ingest
vulnify-gather --skip-exploitdb       # skip Exploit-DB artefact ingest
vulnify-gather --skip-metasploit      # skip Metasploit module-metadata ingest

The pipeline writes phase watermarks to the pipeline_run table, so re-runs
are idempotent — NVD only fetches changes since the last watermark, KEV
re-syncs in full, EPSS only updates stale scores.

The exploit phases (Nuclei / Exploit-DB / Metasploit) fetch and cache their
corpora under ~/.vulnify/cache and watermark on corpus version (commit SHA /
content hash). They re-parse whenever cvelistV5 ingestion ran this gather — an
ingest cascade-deletes a re-touched CVE's artefacts, so they're healed on the
next run — and otherwise auto-skip an unmoved corpus. An explicit --skip-*
always wins. The tri-state exploit.public_poc / exploit.metasploit summary
bools are materialised from the artefacts: null until the contributing
sources have run, then true/false.

Configuration

All config lives in .env at the project root (loaded by vulnify.settings).

Relative paths are resolved against the project root.

Architecture

                  ┌───────────────────┐
                  │  gather.py (CLI)  │
                  └─────────┬─────────┘
                            │
                            ▼
              ┌─────────────────────────────┐
              │   providers/cveproject.py   │  bulk download + parse
              └──────────────┬──────────────┘
                             │
                             ▼
              ┌─────────────────────────────┐
              │   db/sqlite_store.py        │  upsert into normalised tables
              └──────────────┬──────────────┘
                             │
                             ▼
         ┌───────────────────┴────────────────────┐
         │   providers/enrichment.py              │
         │   ├── cisa_kev   (CISA KEV catalog)    │
         │   ├── nvd        (CVSS, CPE, status)   │
         │   ├── epss       (probability scores)  │
         │   ├── osv        (package mappings)    │
         │   └── exploit_ingest (Nuclei/EDB/MSF)  │
         └────────────────────┬───────────────────┘
                              │
                              ▼
         ┌────────────────────┴────────────────────┐
         ▼                                         ▼
┌───────────────────────────────┐   ┌───────────────────────────────┐
│   explore/app.py (Streamlit)  │   │   vulnify/mcp.py (FastMCP)    │
└───────────────────────────────┘   └───────────────────────────────┘

Source layout

vulnify/
├── vulnify/
│   ├── gather.py             # `vulnify-gather` CLI (ingestion + enrichment)
│   ├── mcp.py                # `vulnify-mcp` FastMCP server (stdio)
│   ├── settings.py           # .env loader + path resolvers
│   ├── constants.py          # Severity, SourceTrust, ExploitMaturity enums
│   ├── cvss_severity.py      # CVSS-base -> severity bucketing
│   ├── http.py               # shared httpx client + file/zip download helpers
│   ├── models/               # dataclass models (CVE, Vendor, Product, CVSS, …)
│   ├── providers/
│   │   ├── cveproject.py     # CVEProject bulk parser
│   │   ├── nvd.py            # NVD 2.0 API client + per-CVE merge
│   │   ├── cisa_kev.py       # KEV catalog ingest
│   │   ├── epss.py           # FIRST EPSS batch enrichment
│   │   ├── osv.py            # OSV per-CVE package lookup
│   │   ├── exploit_ingest.py # Nuclei/Exploit-DB/Metasploit artefact ingest
│   │   ├── vendor_advisories.py
│   │   ├── enrichment.py     # phase orchestrator
│   │   └── enrichment_resume.py
│   └── db/
│       ├── schema.sql        # 20-table normalised schema + FTS5 index
│       ├── sqlite_store.py   # connection + upsert API
│       ├── readonly.py       # shared read-only sqlite3 helper
│       ├── cve_upsert.py     # CVE/vendor/product writes
│       ├── migrate.py        # idempotent migrations + FTS5 backfill
│       ├── pipeline_state.py # phase watermarks
│       └── sql_conversion.py # model <-> row helpers
├── explore/
│   ├── app.py                # Streamlit dashboard
│   ├── data.py               # overview queries
│   └── data_views.py         # per-chart queries
└── tests/                    # pytest suite (no network, no real DB)

Schema overview

20 tables, all normalised, foreign keys enabled. Top-level CVE plus separate
tables for the high-cardinality bits (CPE matches, references, CVSS metrics,
exploit signals + artefacts, KEV listing, intel) so you can join cleanly.

cve                — primary CVE record
vendor             — deduped vendors (case/whitespace-normalised)
product            — products owned by a vendor
cve_vendor         — many-to-many with `role` (assigner, affected, source, …)
affected_product   — CVE <-> product affected/not-affected with notes
version_range      — per affected_product version constraints
cwe / cve_cwe      — weakness taxonomy
cvss               — every CVSS vector from any source, deduped on (source, version, vector)
cpe_match          — NVD configuration leaf matches
exploit            — exploit availability + maturity (tri-state summary bools)
exploit_artefact   — per-CVE exploit evidence (Nuclei/Exploit-DB/Metasploit)
kev                — CISA KEV listing fields per CVE
intel              — generic threat-intel rows (kind/value/source)
intel_string_list  — backing list values for intel rows
reference          — URL references with trust + source
reference_tag      — many-to-many for ref tag vocabulary
tag / cve_tag      — CVE-level tags
pipeline_run       — per-phase watermarks for resume

Full DDL is in vulnify/db/schema.sql; an annotated
walkthrough with query gotchas is in docs/SCHEMA.md.

MCP server (vulnify-mcp)

vulnify.mcp exposes the SQLite database to AI agents over the
Model Context Protocol. It's a FastMCP
server speaking stdio, so agents like Claude Code, Claude Desktop, and
Cursor spawn it as a subprocess on demand.

Full tool reference, per-agent setup, and troubleshooting:
docs/MCP.md.

Step 1 — set up the project

git clone https://github.com/mez-0/vulnify.git
cd vulnify
uv sync --extra mcp

Step 2 — get a SQLite database

Two options:

Option A — pre-built (recommended). Download the release asset and
decompress:

gh release download -R mez-0/vulnify --pattern vulnify.db.zst
zstd -d vulnify.db.zst -o vulnify.db

Option B — build from source. Takes hours; see Building the DB from
scratch.

Step 3 — point the server at the DB

Create .env at the project root with an absolute path:

echo "VULNIFY_SQLITE_PATH=$(pwd)/vulnify.db" > .env
echo "VULNIFY_SCHEMA_SQL_PATH=vulnify/db/schema.sql" >> .env

Step 4 — smoke test

Confirm the server starts and shuts down cleanly. It speaks stdio, so just
sending Ctrl-D (EOF) exits it:

uv run vulnify-mcp </dev/null

If you used a release DB built before FTS5 shipped, you'll see a one-off
~8 second backfill on stderr:

INFO   Backfilling FTS5 index over 347248 CVE rows — one-off, takes ~30s
INFO   FTS5 backfill complete in 8.5s

That's it — releases from v2026.05.21.1 onwards ship with the FTS index
pre-built, so this step is silent.

Step 5 — wire it into your agent

Claude Code

Easiest is the CLI:

claude mcp add vulnify \
  --scope user \
  -- uv run --directory "$(pwd)" vulnify-mcp

Or edit ~/.claude.json directly:

{
  "mcpServers": {
    "vulnify": {
      "command": "uv",
      "args": ["run", "--directory", "/absolute/path/to/vulnify", "vulnify-mcp"]
    }
  }
}

Verify it's connected:

claude mcp list

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json
(macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "vulnify": {
      "command": "uv",
      "args": ["run", "--directory", "/absolute/path/to/vulnify", "vulnify-mcp"]
    }
  }
}

Restart Claude Desktop after editing.

Cursor

Edit ~/.cursor/mcp.json (or the project-local .cursor/mcp.json):

{
  "mcpServers": {
    "vulnify": {
      "command": "uv",
      "args": ["run", "--directory", "/absolute/path/to/vulnify", "vulnify-mcp"]
    }
  }
}

Tools the agent can call

All tools return JSON-serialisable dicts. Limits cap at 100 rows.

Example agent prompts

• "What's CVE-2024-3094?"
• "Show me KEV entries added in the last 60 days about Cisco."
• "Find CVEs about container escape with CVSS ≥ 8."
• "Which products have the most CVEs published in 2025?"

Troubleshooting

• Tools don't show up in an already-running agent — MCP servers are
  attached at session start, not when claude mcp add runs. Exit your
  current Claude Code / Cursor session and start a fresh one — don't
  /resume an old one, that re-reads the old tool list.
• claude mcp list shows ✗ Failed to connect — first thing to
  check is the exact Args: field with claude mcp get vulnify; a
  copy-paste truncation that turns vulnify-mcp into vulnify-mc is
  a common one. Smoke test the spawned command verbatim in a shell:
  uv run --directory /path/to/vulnify vulnify-mcp </dev/null.
• uv: command not found in the agent's process — agents inherit a
  minimal PATH. Either use an absolute path to uv in the config
  (command: "/home/you/.cargo/bin/uv") or symlink uv into
  /usr/local/bin.
• VULNIFY_SQLITE_PATH is unset at startup — your .env isn't being
  picked up. The server resolves .env relative to the vulnify package
  root; pass an absolute --directory in the agent config so it lands in
  the right place.
• Tool calls return nothing — confirm the DB has data with
  uv run python -c "import sqlite3; print(sqlite3.connect('vulnify.db').execute('SELECT COUNT(*) FROM cve').fetchone())".
• Logs disappearing — vulnify-mcp logs to stderr (so it doesn't
  corrupt the MCP protocol on stdout). Most agent UIs surface stderr in
  their MCP logs view; in Claude Code that's /mcp then pick the server.

Explorer

explore/app.py is a Streamlit dashboard with roughly 70 pre-built views over
the data: severity distributions, KEV listing lag, EPSS density, CWE
co-occurrence heatmaps, vendor / product pressure, CVSS vector breakdowns,
publication recency, exploit-in-the-wild monthly trends, and more.

uv sync --extra explore
uv run streamlit run explore/app.py

Views are organised into 14 tabs (Overview, Vendors, Trends, Vendor links,
Severity & CWE, Enrichment, Exploit landscape, KEV deep dive, CVSS vector,
Products & CPEs, References, Threat intel, Lifecycle, Pipeline health) and split
across two query modules: explore/data.py (overview / landing) and
explore/data_views.py (the per-chart queries). Full tab-by-tab breakdown and
how to add a view: docs/EXPLORE.md.

Tests

uv run pytest

Tests are hermetic — no network, no real DB. They cover:

• SQLite store upsert + dedup behaviour
• Schema migration + backfill
• NVD bulk merge
• Pipeline state watermarks
• Enrichment resume logic
• Explorer query shapes
• MCP tool surface (structured + FTS5 search)

Releases

Each release ships:

• vulnify.db.zst — the full SQLite DB, zstd-compressed. Includes the
  FTS5 index pre-built from v2026.05.21.1 onwards so vulnify-mcp is
  ready to serve immediately on first start. Decompress with
  zstd -d vulnify.db.zst.
• schema.sql — the DDL, in case you only want the structure.

Tag scheme:

• vYYYY.MM.DD — major snapshot, data captured on that date.
• vYYYY.MM.DD.N — code patch against the same data snapshot
  (e.g. v2026.05.21.1 is the 2026-05-21 data with the vulnify-mcp
  server added).

The code version in pyproject.toml follows SemVer independently.

License

Released under the MIT License.