Proven compression, not projections

All three results use exact o200k_base counts in balanced mode. The
JSON-data results are lossless. The schema benchmark preserves required
fields and descriptions while trimming redundant examples. Repetitive,
structured data benefits most.

What it does

• CLI — tokenfold compress / inspect files or stdin, and compare
  payloads with tokenfold diff; tokenfold wrap -- <command> compresses a
  command's output in place.
• Library — the same Rust engine and receipt shape from pip install tokenfold
  or npm install tokenfold, so behavior never drifts between languages.
• Proxy — tokenfold-proxy sits in front of your provider, compresses
  requests, and streams responses through untouched.
• MCP server — tokenfold mcp serve exposes compress/inspect/retrieve/stats
  to any MCP-compatible agent or editor.
• Receipts, not guesses — every call returns exact token counts, the
  transforms it applied, and any warnings, so you can audit what changed.
• No ML tax — one static Rust binary compresses a JSON payload; no Python
  runtime, no model download, no GPU.

Quick start

Install the interface that fits your stack:

pip install tokenfold       # Python 3.9+
npm install tokenfold       # Node.js 22+
cargo add tokenfold-core    # Rust library
cargo install tokenfold-cli # Rust CLI

Or download the CLI for Linux, macOS, or Windows from
GitHub Releases,
then verify it with the adjacent .sha256 file.

Preview the savings without changing the input, then write the compressed
payload when you are ready:

tokenfold inspect payload.json --format json
tokenfold compress payload.json --format json --output payload.compact.json

Compress an OpenAI-style request before sending it to your provider:

import json
from pathlib import Path

from tokenfold import CompressionMode, compress_openai_payload

result = compress_openai_payload(
    Path("request.json").read_text(),
    mode=CompressionMode.BALANCED,
)
compressed_request = json.loads(result.payload)

print(f"saved {result.report.saved_tokens} tokens ({result.saved_pct():.1f}%)")
# Pass compressed_request to your existing OpenAI client.

The TypeScript package calls the same local Rust engine and returns bytes plus
the canonical compression receipt:

import { compress } from "tokenfold";

const input = new TextEncoder().encode(JSON.stringify({
  results: [
    { id: 101, region: "us-east-1", plan: "pro" },
    { id: 102, region: "us-east-1", plan: "pro" },
    { id: 103, region: "us-east-1", plan: "pro" },
  ],
}, null, 2));

const { payload, report } = await compress(input, {
  format: "json",
  mode: "balanced",
});

console.log(`saved ${report.saved_tokens} tokens`);
console.log(new TextDecoder().decode(payload));

Want to try the CLI from source? Inspect the bundled request without changing it:

git clone https://github.com/snchimata/tokenfold.git
cd tokenfold
cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/openai_payload.json --format openai

Across 100 requests with the same payload shape, those savings add up to:

json_minify          34,600 → 22,900   saved 11,700
schema_compaction    22,900 → 21,300   saved  1,600
TOTAL                34,600 → 21,300   saved 13,300 (38.4% reduction, estimated)

Why tokenfold

Models do not need the same object key hundreds of times. Providers still
count every token. Tokenfold removes that structural waste before the model
call, so you get:

• Lower input cost — send fewer billable tokens without changing providers.
• More useful context — reclaim room for instructions, evidence, and
  conversation history.
• Less data movement — shrink payloads crossing queues, proxies, logs,
  and evaluation runs.
• Fewer blind spots — inspect counts, transforms, and warnings.
• No new data processor — run locally, in-process, or behind your own
  loopback proxy.

messages · schemas · JSON · logs · diffs
                    │
                    ▼
                tokenfold ──────▶ any LLM provider
                    │
                    └───────────▶ compressed payload + receipt

When to use tokenfold · when to look elsewhere

Good fit if you...

• want an exact, auditable receipt for every call instead of an estimate
• need lossless guarantees on structured data — JSON, schemas, tool
  arguments — before it reaches a billing meter
• want to ship a single static binary with no ML runtime, model download,
  or GPU in the dependency graph
• are shrinking logs, diffs, and repetitive tool output before they hit an
  LLM, a log store, or a queue

Look elsewhere if you...

• need query-aware summarization of unstructured prose — that calls for a
  model, and tokenfold stays deterministic on purpose
• want a hosted API to call — tokenfold runs entirely on your machine or
  your infrastructure; there is nothing to sign up for

What it improves

Pick your integration

One Rust engine powers every surface, so policies and receipts stay
consistent as your stack changes.

Compress generic JSON

Use format="JSON" for API responses, record dumps, and other data that is
not an LLM request:

import json
import tokenfold

result = tokenfold.compress(
    json.dumps({
        "results": [
            {"id": 101, "region": "us-east-1", "plan": "pro"},
            {"id": 102, "region": "us-east-1", "plan": "pro"},
            {"id": 103, "region": "us-east-1", "plan": "pro"},
        ]
    }),
    format="JSON",
    mode="BALANCED",
)

print(f"saved {result.report.saved_tokens} tokens")

Run the proxy

cargo build --release --locked -p tokenfold-proxy
target/release/tokenfold-proxy \
  --upstream https://api.openai.com \
  --target-tokens 12000

The proxy listens on 127.0.0.1:8787 by default, streams SSE responses, and
returns the compression receipt in X-TokenFold-* headers.

Safety you can inspect

Tokenfold recounts after every stage and stops when the target is met or the
allowed transform set is exhausted.

• Never larger: a transform stays only when it reduces the token count.
• Reversible JSON: every structural rewrite must pass an exact round trip.
• Clear provenance: exact tokenizer results and estimates are labeled separately.
• Actionable receipts: every result lists savings, transforms, warnings,
  and final status.
• Honest limits: unreachable targets return an explicit status instead of
  silently deleting more content.

Lossy log and diff transforms remain policy-gated. Optional originals can be
stored by SHA-256 hash; detected secret-shaped content is excluded.

Reproduce the numbers

Run the regression benchmark:

cargo bench -p tokenfold-core

Or inspect the small bundled JSON sample:

cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/api_response.json --format json

The sample reports 382 → 206 estimated tokens, a 46.1% reduction. Ragged
or compact inputs may save little; Tokenfold reports that result honestly.

Contributing

Issues and pull requests are welcome. Run the core checks before opening a PR:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
python eval/run_fidelity.py --gate --profile smoke-first-consumer
cd packages/tokenfold && npm ci && npm test

License

Apache-2.0

Reclaim your context window

Start with one representative payload. Install tokenfold, inspect the
receipt, and see how many tokens your application can stop sending today.

pip install tokenfold

If tokenfold earns a place in your stack, a ⭐ on
GitHub helps the next team
find it.