croft

The whole data pipeline, in TypeScript.

croft is one tool that pulls data from APIs and files, transforms it, keeps it fresh on a schedule, and serves it to your app. Pipelines are TypeScript, with SQL where you want it, and everything lives in one DuckDB file on your machine. No Python, no servers, no glue between tools.

Website · Quickstart · Docs · npm · For AI agents: llms.txt

bun add -g @zabaca/croft
croft init my-data && cd my-data
croft query "from example_sales limit 5"

croft needs Bun 1.3.14 or newer, on macOS or Linux (WSL works).

Why croft

Every project has data worth asking about: signups, payments, usage, support, the spreadsheet someone keeps by hand. A BI layer, a warehouse with all of it in one place and a pipeline that keeps it fresh, is not only for big companies with a data team. A vibe-coded app, a side project, a company of two just getting started: all of them benefit from one.

That used to mean a stack of tools to host and wire together, so most projects went without. croft makes it one folder and one file, small enough to start on day one. With everything that matters in one warehouse, you and your coding agent can both query it, build reports and find insights.

One tool, end to end

A croft project is a folder. Each file in assets/ becomes one table, and croft run brings them up to date in order.

Ingest. Bring data in from any API or from files. Incremental, typed, and safe to rerun.

// assets/stripe_charges.ts
export default ingest({
  key: "id",
  schedule: "every hour",
  async *rows({ http }) { … },
});

Transform. One SQL query per table, or a TypeScript function per row, with checks that run before anything is saved.

-- assets/revenue_by_month.sql
-- check: revenue >= 0
SELECT month, sum(amount) AS revenue
FROM stripe_charges
GROUP BY month

Schedule. A schedule in plain words. Everything downstream updates after it.

Serve. Read the tables from any Bun or Node app, typed by table name.

import { query } from "@zabaca/croft/read";

const rows = await query<"revenue_by_month">("from revenue_by_month order by month");

Typed across SQL and TypeScript

After every run, croft writes a TypeScript type for each table from its real columns. Rename a column in SQL, and croft validate --types flags the TypeScript that still reads the old name, before anything runs:

error UNKNOWN_INPUT_COLUMN  assets/region_labels.ts:9:65
      revenue_by_region has no column "revenue"
      fix: revenue_by_region has these columns: region, orders, total, _loaded_at

What it replaces

croft does these jobs; it does not read those tools' projects. It is built for one machine and one DuckDB file. If your team already runs a warehouse with dbt, keep it.

Built for coding agents

Claude Code, or any coding agent, can run a croft project from a cold start:

• Every command takes --json, and every problem has a stable code, a hint with the next command, and often a fix the agent can apply as is.

• croft validate checks every change without running it; croft preview builds it in a sandbox and diffs it against the live table.

• Anything that could lose data stops and waits for a person: every destructive step ends in croft confirm, so one permission rule gates them all.

• croft init writes a skill, a croft agent built on it, and a CLAUDE.md block, pinned to the project's croft version. Claude hands the agent the project's data work (a new source, a report, a question) as a subagent, which runs the loop on its own and returns anything that needs a person's yes; or start a session as it with claude --agent croft. Without a project yet, the repository's Claude Code plugin teaches the start, as a skill and as the same agent:

  /plugin marketplace add Zabaca/croft
  /plugin install croft@croft
  

Your data stays safe

Some data can't be fetched again, so croft never throws it away on its own. Every write is all-or-nothing, and a failed check keeps the old data. Anything replaced or deleted goes to a trash first, and croft restore brings it back. A crash, even kill -9, never leaves half a write. croft backup copies the warehouse, its history and the trash to another folder, even while croft is using them.

This repository

Develop

From the repository root:

bun install
bun --cwd packages/croft test             # the CLI's suite
bun --cwd packages/croft run typecheck
bun --cwd packages/site run dev           # the website, with hot reload
bun --cwd packages/site test              # the skill and docs agree with the CLI

License

MIT