Sniff Test

Does the draft pass the sniff test?

Sniff Test reads Markdown and plain text and checks it against your house
rules. It prints one line per flag with the file, the line, the rule that
tripped and how sure it is.

There are two kinds of rule. Countable rules are regular expressions.
Long dashes, a paragraph with three colons, a banned word, every sentence the
same length. They run on your machine, cost nothing and send nothing anywhere.
Judgment rules need something read rather than counted. A closing sentence
that only restates the paragraph, a claim hedged three times, a line that talks
the reader out of the offer. For those it sends one paragraph at a time to a
small hosted judgment model and gets back one probability per rule. It only
does that after you have said yes.

Two things it is not. There is no general model in the loop; the judgment
model returns one probability per question and nothing else. And it will not
rewrite your text. A flag is a sentence for you to fix.

npm install --global snifftest
snifftest check --dry-run draft.md

That runs the countable rules and nothing leaves your machine. To run the
judgment rules too, put a TypeSafe key in TYPESAFE_API_KEY and drop
--dry-run. The first time, it prints exactly what it is about to send and
where, and waits for your answer.

It exits 0 when nothing tripped, 1 when at least one rule did, 2 when the tool
could not do its job, and 3 when the judgment rules needed a yes and did not
get one.

The nose is the mascot. It is one hand-drawn SVG with six expressions, in
assets/nose/, and snifftest serve opens a local page where you type and the
nose reacts.

Reading a flag

draft.md:14 dash_present 1.00 A long dash. Give the sentence a full stop instead.
draft.md:31 restating_closer 0.84 The last sentence says the paragraph again. End one sentence earlier.

The number is how sure the checker is, between 0 and 1, and it says nothing
about how bad the problem is. A countable rule always scores 1.00. A judgment
rule counts as a flag at or above the threshold, which is 0.7 unless the
ruleset or --threshold says otherwise. A reading between 0.4 and 0.6 is
reported as no judgment and never becomes a flag or a pass. That band exists
because the judgment model answers about 0.5 on text it cannot read, and a
flat middle number is the one failure that looks like a clean draft.

After a judgment run, check prints a usage line on stderr with the request
count, input tokens and estimated cost at the disclosed rate. The price has
no recorded published source, and the line says so. --quiet hides that line
without hiding flags or diagnostics. A dry run prints no usage line.

--format json prints the same flags for a machine, with every reading
including the ones under the threshold. snifftest rules prints the ruleset in
force and which file each rule came from, which is the quickest answer to "why
did that flag".

The rules

The default ruleset ships in rules/default.yaml. Fifteen rules, and the file
itself is the documentation. Ten apply to most prose and five are conventions
of copy meant to sell.

The five rules tagged marketing sit out an ordinary run. Ask for them on a
landing page with --only marketing, which runs those five and nothing else.
To run everything, write a project file that extends the default and sets
off_by_default: [].

Fenced code is dropped before either kind of rule sees it. Headings, tables,
front matter, link definitions and HTML comments are never sent to the
judgment rules, and the countable rules only read the ones a rule names.

Writing your own

A .snifftest.yaml in your project root is picked up on every run. It can
extend the default ruleset and change the rules it disagrees with, matched by
id, so a house that wants five colons instead of three writes one rule and
inherits the other fourteen.

version: 1
extends: default
threshold: 0.7

rules:
  # A countable rule of your own is a pattern and a message.
  - id: no_utilize
    kind: regex
    pattern: "\\butili[sz]e"
    flags: i
    message: "Use. The word is use."

  # Override a shipped rule by reusing its id.
  - id: colon_heavy
    kind: regex
    builtin: colon_count
    min: 5
    message: "Five colons in one paragraph."

  # A judgment rule is a description, its near misses, and two criteria.
  - id: passive_apology
    kind: judgment
    what: |
      A sentence that apologises for the document itself: for its length,
      its lateness, or the writer's lack of expertise.
    not_for: |
      An apology to a person for a thing that happened.
      A stated limit of scope.
    examples:
      - "Sorry this is so long."
      - "I am no expert, but here goes."
    criteria:
      true: "At least one sentence apologises for the document or the writer."
      false: "No sentence apologises for the document or the writer."
    message: "The draft is apologising for itself. Cut the line."

A countable rule names either a builtin or a pattern, never both. The
built-ins are dash_present, colon_count, sentence_rhythm, slop_vocab
and banned_words, and each takes the tuning keys the default file shows. A
rule may carry tags, and a countable rule may name the kinds of block it
applies to with chunks. A judgment rule's not_for is what keeps it honest.
A rule with a what and no not_for will flag things you did not mean.

A pattern with a nested quantifier is refused when the ruleset loads, and
every pattern runs against a capped slice of text. Neither is a proof that a
pattern cannot run for a long time. See SECURITY.md.

To propose a rule for the shipped set, open a rule proposal issue.
CONTRIBUTING.md says what one needs.

Consent, and what leaves the machine

Nothing is sent before the answer to one question is yes. The countable rules
run first, on your machine, whatever happens next. Then, if the ruleset has
judgment rules and you did not pass --dry-run, the checker looks for a key in
TYPESAFE_API_KEY. Without one it prints the countable verdict, says the
judgment rules could not run, and exits 2. With one, it prints this and waits:

Sniff Test is about to use the judgment rules, which run on a model.

  What leaves this machine: one paragraph of your text at a time, from 3 files,
    together with the wording of these rules: not_x_but_y, tricolon, stacked_hedging, rhetorical_opener, restating_closer.
  Where it goes: https://api.typesafe.ai/v1/systemone (TypeSafe), over HTTPS, with your TYPESAFE_API_KEY.
  What comes back: one probability per rule per paragraph.
  What is never sent: file names, file paths, anything outside the text you pointed at.
  There is no telemetry, no analytics, and nothing is stored by this tool.

  Countable rules never leave the machine. Run with --dry-run to use only those.

Send paragraphs to TypeSafe? [y/N]

A yes is remembered under ~/.config/snifftest/consent.json, or under
$XDG_CONFIG_HOME when that is set. It is never written into the directory
being checked, because a consent file committed to a repository would answer
for everyone who clones it. --yes answers and remembers without the prompt.
SNIFFTEST_SEND=1 answers for one run in CI and writes nothing down. With no
terminal to ask in and no answer in the environment, the checker exits 3 and
sends nothing.

Each request carries one paragraph and every judgment rule's wording. It goes
over HTTPS with your key as a bearer token, and the reply is a probability per
rule. The wire format and the guards around it are in src/jev.ts, which is
the only file in the tool that opens a network connection for check.

Answers are cached on disk so that a run interrupted by an outage does not pay
for every paragraph again. The cache lives under ~/.cache/snifftest, or
$XDG_CACHE_HOME, or SNIFFTEST_CACHE_DIR. Each entry is named by a hash of
the paragraph, the exact wording of the questions asked about it and the model
they were sent to, and it holds rule ids, probabilities, the model name and a
date. The paragraph itself is never
written to disk. Entries expire after a fortnight. --no-cache skips the cache
for a run, and SNIFFTEST_CACHE_DIR=off turns it off for good.

SECURITY.md has the full account, including the limits.

Before a commit

One command, from the root of your repository:

curl -fsSL https://raw.githubusercontent.com/DanRWilloughby/snifftest/v0.1.0/hooks/pre-commit \
  -o .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit

Read it before you trust it. The hook checks the Markdown and text files you
staged, as you staged them, with the countable rules only. It sends nothing
unless SNIFFTEST_SEND=1 and TYPESAFE_API_KEY are both in your environment.
It blocks a commit on a flag. When the checker itself is missing, broken or
could not be fetched, it says so and steps aside, because a failed download is
not evidence about your prose. SNIFFTEST_STRICT=1 reverses that.

Skip it once with SNIFFTEST_SKIP=1 git commit or git commit --no-verify.

The same script works under husky, and there are two entries for the
pre-commit framework, snifftest for the free rules and snifftest-send under
the manual stage for the rest. docs/husky.md has all three.

In CI

name: Prose
on: pull_request

permissions: {}

jobs:
  snifftest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.0.0
      - uses: DanRWilloughby/[email protected]
        with:
          paths: docs

The Action runs the countable rules and nothing else unless you set send and
pass a key in as api-key. A pull request from a fork always gets the
countable rules only, whatever the workflow says. The Action refuses to run on
any event whose name ends in _target, refuses a floating version, and
fetches the exact version named from the npm registry into a scratch directory
outside the checkout, so the repository being checked cannot hand it a checker
by committing one. It needs no token of its own. With comment: true and a github-token it posts
the checker's output on the pull request, and no line of the draft is copied
into that comment.

Pin the Action by commit SHA if you want the stronger guarantee. The inputs and
the exit codes are tabled in docs/husky.md.

In Claude Code

/plugin marketplace add DanRWilloughby/snifftest
/plugin install snifftest@snifftest

Then /snifftest draft.md, or hand Claude a draft and ask whether it passes.
The skill runs the countable rules by default. The judgment pass is the user's
call and the skill never answers the sending question on their behalf: it adds
no flag that skips the prompt and never touches the key. docs/claude-code.md
has the two other ways to install it.

Measuring it

snifftest eval plants one known fault per rule into copies of your own clean
paragraphs, runs the corpus three ways, and reports what each way caught and
what it flagged on the clean originals. Arm A is no tool. Arm B is the
countable rules. Arm C is the countable rules plus the judgment rules, and it
is the only arm that costs anything. Each run writes a Markdown report and a
scores.json, and it writes the clean and seeded paragraphs beside them with
a .gitignore, because they carry your prose in full.

snifftest bench asks a panel of general models the same questions over the
same corpus, through OpenRouter and one direct Anthropic call, and puts cost,
speed and accuracy side by side. An OpenAI row can be called directly too, on
OpenAI's own API with OPENAI_API_KEY, so a model can be measured without an
OpenRouter account and its latency is the provider's rather than a proxy's.
bench/panel-direct.yaml is a panel of nothing but direct rows. It asks for
consent per destination, and a yes given for TypeSafe is never a yes for anyone
else.

The run committed under bench/results/2026-09-17/ is the eval over the
packaged corpus with eight seeds per rule, and beside it the bench over the
same 166 paragraphs with bench/panel-direct.yaml: four general models, each
on its own provider's API, on the same day.

Every row is one paragraph per request. The Sniff Test row is the eval's arm
C, flags counted at 0.7, cost and latency from provider-reported usage. The
model rows are snifftest bench on 2026-09-17, one repeat, cache off, and
they count each model's own yes, which is the decision the model made; their
cost is the returned token usage at the list prices recorded under
bench/prices/ with no prompt caching, and their latency is the wall clock at
this end. Haiku, Sonnet and Opus were given 2,000 completion tokens, because
at the default 900 Sonnet 5 ran out of room on 20 of its 166 replies on an
earlier pass of the same run and a cut reply counts as unanswered; gpt-5.6-sol
was given 4,000 at low reasoning effort. bench-tables.md in the results
folder carries p95, the per-rule detail, every failure and the budget each row
was sent. Jev ran in the same rotation as the models, where it read 59 of 80,
2 of 54 and 198 ms, with one rule missing from its reply on 13 paragraphs;
across the runs of this corpus on 2026-09-17 the judgment model read between
59 and 64 of 80, and that swing is the size to read every row against.

What that supports. On this corpus the judgment model catches about as many
faults as the cheapest general model and fewer than the mid and top tiers,
raises far fewer false alarms than the cheapest one, and does it at about a
thirtieth of that model's cost and a tenth of its latency, with no general
model in the loop. The top models catch more, at twenty to thirty-five times
the latency and a hundred to two hundred and forty times the cost.

Three things to read beside the table. Eight seeds per rule is a small sample,
so the per-rule figures are direction rather than measurement. The eval
measures every rule in the file, including the five marketing rules an
ordinary check sits out. And two of the ten judgment rules were reworded after
a first run on these seeds and measured again on the same seeds;
docs/eval-notes.md carries both numbers and the reasoning.

snifftest eval --twins examples/adversarial measures what a sentence written
to the checker does to the readings around it. examples/CORPUS.md describes
the corpus and where every paragraph came from.

Configuration

snifftest --help lists the options for eval and bench as well.

The package has no runtime dependencies and needs Node 20 or newer. Installing
from the npm registry needs nothing else. Installing from a git URL needs Bun
on the machine, because the executable is built on install.

Contributing

CONTRIBUTING.md has the ground rules, the test setup and what a rule proposal
needs. Security problems go to a private advisory, never to a public issue.
SECURITY.md says how.

Licence

MIT. See LICENSE.