gtm-research

Status: Reference implementation — extracted from a private production GTM system; tenant data, provider adapters, and company-specific policy stay private. · Layer: Intelligence · Portfolio map ›

A cost-optimized web-research engine for GTM enrichment — free-first, cached, source-verified.

Give it a company or person (or a list) and the fields you want, and it returns
source-verified rows — each fact carrying the URL that proves it — by spending the
cheapest provider rung that works and caching results across runs. It's the research
layer: the part that turns "who are these accounts, really?" into cited, structured data
you can drop into a CSV or CRM.

It runs standalone, and it's a drop-in cached upgrade for
gtm-pipeline's web_research capability —
same .claude/workflows runtime, same DATABASE_URL, same BYOK/local-env conventions.

/research Acme Corp, Globex, Initech — find HQ city, employee count, CEO, and their CRM

The idea: spend money only when free fails

Web research costs money (search credits are a capped resource). So every step tries the
cheapest rung first and only escalates on failure — and a shared cache means the second
run of any URL or query is free.

Dead URLs (404/410/401) hard-stop and are negative-cached — never retried. Long pages are
compressed to quoted facts by a cheap model before they reach your research agent. The full
rung-by-rung tables are in docs/the-waterfall.md.

Rung order, budgets, and gates live in config/research-waterfall.yaml
— add or reorder a rung by editing config, not code. Every rung auto-skips when its API
key is absent, so a partial key set (even zero keys — DuckDuckGo + native fetch need none)
still gives a working, cheaper waterfall.

What it actually costs

Worked example: enrich 1,000 companies, ~3 web operations each (one search + ~two page
reads) = 3,000 operations. The waterfall absorbs the bulk on free rungs and pays only on
the tail that falls through.

Against an all-premium baseline (a paid search/enrich API at ~$0.005/call): 3,000 × $0.005
= $15.00 / 1,000 — and that's before re-runs. The waterfall is ~6–20× cheaper cold, and
because the cache is shared across runs, the second pass over an overlapping set of domains
is mostly cache hits — ≈ $0. The more you research, the wider the gap.

These resolution rates are an illustrative model, not a benchmark — your mix depends on how
many target sites are static (free) vs. JS-walled (a credit). The point is structural: free
rungs carry the body, paid is the long tail, and research.v_run_cost_split shows you the
real split for your corpus after any run.

What you get that generic enrichment doesn't

• Source discipline. Every field is value + verified=true + source_url, or blank +
  "NOT FOUND — searched X", or a labeled UNVERIFIED guess. Never invented. Emails are
  confirmed against a real directory, never guessed-and-verified.
• A shared cache. Web facts are client-agnostic, so one cache serves every run — the CEO
  you fetched last week is free this week.
• Cost telemetry (optional). When a Postgres DSN is set, every run/entity/rung is recorded,
  and research.v_run_cost_split tells you whether search credits or model turns were the real
  spend. A verified-rate watchdog pauses a run if a provider silently goes dark.

Known accounts: public facts, meet your world

Generic enrichment tells you what's public about a company. The optional internal
cross-reference tells you what's yours — whether the account is already a customer, a prior
prospect, or net-new — by matching each company against your own known_companies table.

That single internal_status column is what connects web research to GTM execution:

• don't prospect a company you already serve,
• route a prior-prospect-replied account to the rep who owns it,
• suppress do-not-contact domains before they ever reach a sequencer.

It's a deterministic database join, not an LLM call: you populate known_companies from your
CRM / sending tool, and bin/known-xref.py reads it (the company-level analog of
gtm-pipeline's contact suppression list). With no table configured, every company resolves to
net-new and the research still runs. Setup and the status convention:
docs/known-companies.md.

Generic tools can tell you what's true on the web. They can't tell you the account is already
in your pipeline. That column is the line between a research tool and a GTM tool.

Quick start

git clone https://github.com/kkrlstrm/gtm-research && cd gtm-research
pip install -r requirements.txt          # requests + PyYAML (+ optional extras)
cp .env.example .env                      # fill in only the keys you have (zero is fine)
set -a && source .env && set +a

# A free search and a free page read — no keys, no database needed:
python3 bin/research-search.py query "Acme Corp headquarters" --json
python3 bin/page-digest.py "https://www.acme.com/about" --entity "Acme Corp" --want "HQ city, CEO"

For a full multi-entity run, invoke the entity-research workflow from Claude Code
(see .claude/commands/research.md) or wire it into gtm-pipeline
(docs/integrating-with-gtm-pipeline.md).

Telemetry + cache (optional, local or hosted Postgres)

Unset = it just runs (no cache, no telemetry). To turn it on, point at any Postgres —
local or hosted:

createdb gtm_research                     # local Postgres
export RESEARCH_DATABASE_URL=postgresql://localhost:5432/gtm_research
psql "$RESEARCH_DATABASE_URL" -f storage/postgres/schema.sql
# optional "do we already know them?" table:
psql "$RESEARCH_DATABASE_URL" -f storage/postgres/known-companies-optional.sql

It falls back to DATABASE_URL, so dropped into a gtm-pipeline checkout it shares that
project's database automatically. See docs/local-postgres.md.

Layout

research_engine/   the engine: waterfall policy, web fetch/search, cache+telemetry, free search rungs
providers/         optional escalation rungs (exa, jina, parallel) — each auto-skips without its key
bin/               the CLIs: research-search, page-digest, research-run, known-xref
config/            research-waterfall.yaml — the ordered cost policy
storage/postgres/  schema.sql (cache + telemetry) + known-companies-optional.sql (internal xref)
.claude/           the entity-research workflow + the /research command
docs/              quickstart · architecture · the-waterfall · known-companies · local-postgres · integrating-with-gtm-pipeline

How it relates to gtm-pipeline

gtm-pipeline builds the list (discover → source → qualify → enrich → activate).
gtm-research is the research engine underneath the enrichment: a cached, free-first,
source-verified company_search / company_enrich / people_search provider. Use it on its
own for ad-hoc account research, or drop it in as a cached web_research upgrade.

License

Apache 2.0. BYOK, local-env-only: secrets are read from your environment and sent
only to each provider's own API — this engine never fetches a key over the network. See
SECURITY.md.

Where this fits

Part of a portfolio of governed, AI-native GTM systems — reference implementations and reusable patterns extracted from a private production stack. In that system this is the source-verified web-research engine that feeds targeting.

Full portfolio map → github.com/kkrlstrm

Works with:

• gtm-pipeline — drops in as its cached research capability