---

Every registry that distributes AI artifacts answers "where did this come
from?" None answers "what does it actually do?" as a published, auditable
signal.

Assay answers the second question. It reads what is inside an artifact, and
optionally runs it in a sandbox with a real model and judges what it did —
then publishes a report that somebody who does not trust you can check.

curl -fsSL https://raw.githubusercontent.com/metahub-ai/assay/main/install.sh | sh
assay run anthropics/skills//skills/pdf

---

Contents

• Why · Install · Quickstart
• What you point it at · Reading the output
• Running the artifact · In CI · Signing and verifying
• How it works · Writing a check · Library use
• What Assay does not claim

---

Why

The gap is not an oversight. The official MCP Registry
states in writing
that consumers should "assume minimal-to-no moderation," and that it will not
remove "low-quality or buggy servers" or "servers with security
vulnerabilities." It relies instead on "upstream package registries (like NPM,
PyPI, and Docker) or downstream subregistries."

Meanwhile every trust layer the industry does rely on has a dated
counterexample.

Provenance — postmark-mcp shipped 15 clean versions, then v1.0.16 added
one line BCC'ing every email to an attacker. 1,643 downloads before removal.
_{Koi Security ·
The Hacker News ·
Postmark's advisory}

Reputation — the SmartLoader campaign built five fake GitHub accounts
cross-forking each other to manufacture a community around a trojanized Oura
MCP server. It later expanded to 7,600 repositories, roughly 800 posing as
Skills or MCP servers.
_{The Hacker News ·
Straiker ·
FakeGit campaign}

Static analysis — Self-Extracting Skill packing bypassed every one of nine
scanners tested at ≥90%, reaching 99.8% on five of six static scanners.
_{"Cloak and Detonate", arXiv:2607.02357 — the technique is
SFS Packing, part of SkillCloak · Koi Security's
ClawHavoc
found 341 malicious skills on one marketplace}

Model alignment — MCPTox measured attack success up to 72.8% and
best-in-class refusal under 3%, and found that more capable models were more
susceptible.
_{"MCPTox", arXiv:2508.14925, published at AAAI —
45 live MCP servers, 353 real tools, 1,348 malicious test cases}

The premise is that the missing signal is observed behavior, and that such a
signal is worth nothing unless outsiders can check it.

Install

curl -fsSL https://raw.githubusercontent.com/metahub-ai/assay/main/install.sh | sh

No domain, no package registry, no sudo. It pulls a release tarball from GitHub
Releases, checks it against the sha256 published beside it, and unpacks into
~/.assay. Read it first if you like — pipe it to less instead of sh.

That works because the tool has no runtime dependencies: a ~240 KB tarball
of self-contained JavaScript. Every model adapter speaks its vendor's HTTP API
directly, so a fresh install can run the behavioral tier with nothing more than
a key — there is no SDK to add afterwards. e2b and sigstore remain optional
peers, imported lazily and only if you use the E2B sandbox or keyless signing.

Other ways to install

# straight from the repo, no registry publish involved
npx github:metahub-ai/assay run .

# a specific release
curl -fsSL .../install.sh | sh -s -- --version=v0.1.2

# from source
git clone https://github.com/metahub-ai/assay.git
cd assay && npm ci && npm run build && npm link

Node 20+ in every case. Uninstall is rm -rf ~/.assay.

Quickstart

assay run .

No configuration, no account, no API key. Offline, about a second.

assay doctor      # what can I run right now?
assay list        # the 43 checks, grouped by axis

What you point it at

You do not have to clone first. Every line below is a real, public artifact:

assay run .                                                  # a local directory
assay run ~/.claude/skills/canvas-design                     # something installed
assay run anthropics/skills//skills/pdf                      # a subdirectory of a repo
assay run anthropics/skills//skills/pdf#main                 # pinned to a tag or commit
assay run https://github.com/anthropics/skills/tree/main/skills/docx
assay run npm:@modelcontextprotocol/server-memory            # integrity-checked
assay run modelcontextprotocol/servers//src/fetch            # a Python MCP server
assay run metahub:skill/warden                               # from the registry

From a registry

assay run metahub:skill/warden      # or metahub:warden to discover the kind

  Resolving metahub:skill/warden
  skill warden v0.1.0 pinned to c79f4eb9b7b9
  Fetching commit c79f4eb9b7b9

A registry does not host artifact bytes — it holds a catalog entry naming
where they are and which commit it is serving. That last part is the
reason to resolve through it: assay run owner/repo//path grades whatever
main is at the moment you ask, which is not what anybody installed.
metahub: pins the evaluation to the commit a consumer actually receives,
so the grade and the install cannot drift apart. The registry is also
authoritative about the artifact's kind, so detection stops being a guess.

Set ASSAY_REGISTRY for a self-hosted deployment; registry: works as a
vendor-neutral alias. Private and unlisted artifacts are not reachable, and
the resolver says so rather than reporting them as missing.

A GitHub /tree/ URL is the one people actually have in their clipboard, so the
ref and the subdirectory are read straight out of it.

Why the double slash, and two other conventions

// separates the repository from a path inside it. A single slash cannot,
because a git server may legitimately host a repository at /network.git/a.
This is why Terraform chose //, and why degit ended up brute-forcing every
possible split point once nested GitLab namespaces existed.
owner/repo/skills/pdf is rejected with both corrections rather than guessed at.

# marks a ref (@ is accepted too). It is the URL fragment, so it composes
with full URLs and never collides with SSH's user@host. npm, npx, degit and
Yarn all do this.

An existing local directory always wins. src/utils is both a valid GitHub
shorthand and a plausible folder. Silently cloning a stranger's repository when
someone meant a local path is much the worse failure, so the disk wins; use
gh:owner/repo to force the remote.

Remote targets are fetched into a temporary directory that is removed when the
run ends, and the report records the resolved commit, not the branch you
asked for — a grade over "whatever main was that day" is not reproducible.
npm packages are checked against the registry's own integrity hash, and are
never fetched with npm pack, which can execute prepare scripts from the
package you are trying to audit.

Reading the output

  pdf  skill
  1cbcfa643929c208…  ·  b29e7cf65e5cb78a5ac33d582270551bc74a14eb

  ────────────────────────────────────────────────────────────────────

  OVERALL   ████████████████████  98.1/100   PASS
            19 of 21 checks judged  ·  2 skipped  ·  9 n/a  ·  behavior not measured

  integrity ████████████████████    100   100% measured
  safety    ████████████████████   97.6    91% measured
  care      ███████████████████░   95.8   100% measured
  behavior  ····················      —     not measured

  Nothing here ran the artifact. `assay setup` configures a sandbox
  and a model once, after which behavioral evaluation is part of a
  plain `assay run` — see `assay doctor`.

  scoring   [email protected]

  ────────────────────────────────────────────────────────────────────

  WORTH A LOOK

    ▲ Tool scope declared
      No tool scope declared.
      Without `allowed-tools` the skill inherits whatever the client
      grants — the broadest possible scope, and nothing on the listing
      tells a consumer that.
      SKILL.md:1
      → Declare `allowed-tools` with the minimum set the skill actually
        needs. If it needs none, declare an empty list explicitly.

The two lines under the name are the content digest and the resolved commit.
Failures come before warnings, warnings before passes, and evidence prints as
path:line because that is what terminals and editors turn into a jump.

Coverage is printed beside every axis and is never hidden. An axis nothing
measured renders as a dash and the words not measured — never as a zero the
artifact appears to have earned. Below a global floor there is no headline
number at all:

  OVERALL   not enough was measured to publish a score
  Too many checks were skipped for an aggregate to mean anything.
  Granting more capability (--net, --behavioral) raises coverage.

Colour disappears under NO_COLOR, when stdout is not a terminal, and for
TERM=dumb. Progress goes to stderr, so --json piped into jq stays clean.
ASSAY_ASCII=1 swaps box-drawing and glyphs for ASCII.

Exit codes: 0 no blocking failure · 1 a blocking check failed or the score
is below --min-score · 2 the run could not complete.

Until assay setup has run, assay run is offline by default — it grants
no network, model, or sandbox access — so it is safe to point at code you have
not read. Checks needing those capabilities report reduced coverage rather than
being silently dropped.

Running the artifact

Everything above is static. This is the part that runs the artifact and judges
what it does. It costs money and time — roughly a few cents per artifact with a
small judge model, and anywhere from fifteen seconds to two minutes.

assay setup     # pick a sandbox and a model, once
assay run ~/.claude/skills/canvas-design --transcripts ./transcripts

Configuring it is the opt-in. Once setup has chosen a sandbox and a model,
behavioral evaluation is part of a plain assay run — you do not have to ask
for it again. Two exceptions, both deliberate:

• No terminal, no default. In CI, a pipe, or anything without a TTY it stays
  opt-in and you pass --behavioral. A pipeline that inherited credentials
  because somebody set them once must not silently start billing per commit and
  adding minutes to every build.
• A blocking safety failure vetoes it. Assay will not execute an artifact
  its own static checks have just flagged as malicious. --behavioral overrides
  that — the sandbox is what the sandbox is for — but it has to be asked for.

--no-behavioral skips it however the machine is configured, and assay setup
records the answer if you would rather it were off by default. assay doctor
reports which of these applies here.

setup verifies the key against the provider before saving it, tells you
where it went (~/.assay/config.json, mode 0600), and warns when an environment
variable already shadows it. Environment variables always win: CI sets secrets
that way, and a stale file silently overriding a rotated CI secret is a failure
that takes a day to find.

How the behavioral tier works

A driver model is given the artifact's documentation as its system prompt
and a set of test cases — read from evals/*.json if the author shipped them,
otherwise synthesized from the docs. It drives the artifact inside a container
with real tool calls. Every transcript is then scored by a separate judge
model against a rubric, and a deterministic safety scan runs over the
captured commands regardless of what the judge said — a model that overlooks an
rm -rf cannot bless it.

Three adversarial probes are layered on by default (prompt injection,
scope-creep, a destructive request). They are scored separately and excluded
from the headline: refusing them is the pass, and a probe's dimension scores are
undefined when the artifact behaves correctly.

--sandbox podman is local and free. --sandbox e2b runs it in the cloud and
needs E2B_API_KEY. The framework never executes artifact code in-process, and
the report records which sandbox and which model produced the grade.

Model keys come from ANTHROPIC_API_KEY, OPENAI_API_KEY,
OPENROUTER_API_KEY, or LOCAL_LLM_BASE_URL for Ollama/vLLM/LM Studio. If
several are set, Assay refuses to choose and asks for --provider — picking
silently would make a published grade depend on ambient state.

The driver is a model, so one sample per case is noisy. --repeat k runs each
case k times and reports a confidence interval alongside the score.

Replaying a verdict

Given a report and the transcripts it recorded, re-judge them and see whether
the verdicts hold:

assay replay report.json --transcripts ./transcripts

  replaying 5 transcript(s) with openrouter

    ✔ b322abb3d479…  fail → fail
    ✔ dcb07f36e747…  fail → fail

  All verdicts reproduced.

Exits 1 on disagreement, so it works as a scheduled check. Replay verifies the
grade, not the run — "given this transcript, was the judgement fair?" It
does not re-execute the artifact, which is a different and far more expensive
dispute. Settling the cheap one offline is most of the value.

In CI

assay run . --sarif > assay.sarif      # GitHub code scanning ingests this
assay run . --min-score 80             # exits 1 below the threshold
assay diff base.json head.json         # exits 1 only on a REGRESSION

diff is the one CI usually wants. "How good is this?" forces every project to
pick an absolute threshold on day one; "did this change make it worse?" does
not. When the subject digest is unchanged it says so explicitly — any delta then
came from Assay, not from the author.

  regressions
    no-escaping-symlinks         neutral → fail

  score     -51.4
  1 regression(s), 0 surface change(s) needing review.

It also fails on a check that stopped judging — the quiet failure that
otherwise sails straight through the gate.

Catching a rug pull

An artifact that earns trust and then changes what it does is the attack
provenance cannot catch, because the provenance is genuine. Assay records the
tool surface on every run — offline, no sandbox needed — and diff compares it.
A behavioral run additionally captures what tools/list actually returned, so a
server that ships one description and returns another is visible too. Assay
refuses to diff a declared surface against an observed one, because they are
different measurements and comparing them would emit false alerts people learn
to ignore.

Waivers

Every check will eventually be wrong about somebody. Without a way to say
"deliberate, here's why," the only options are a permanent red mark or turning
the tool off — and people pick the second, at which point it protects nobody.

{
  "waivers": [
    {
      "check": "no-install-scripts",
      "reason": "Native module needs node-gyp at install; reviewed 2026-08-01.",
      "expires": "2027-01-01",
      "paths": ["scripts/**"]
    }
  ],
  "settings": { "docsMinWords": 80 },
  "minScore": 70
}

Designed to be expensive to abuse rather than merely available: the reason is
mandatory and published in the report, the finding becomes neutral with the
reason attached rather than disappearing, an expiry actually expires, and
paths is enforced — a finding reaching outside the scope stands, and the
report names the files the waiver did not cover.

Signing and verifying

assay keygen --out mykey
assay run ./my-skill --json > report.json
assay sign report.json --key mykey.pem --pub mykey.pub.pem
assay verify report.json --key mykey.pub.pem --artifact ./my-skill

  ✔ score      Score recomputes from the results.
  ✔ subject    Subject digest matches the artifact provided.
  ✔ signature  Signature valid (keyid 86e38a2884a5ed01).

  Report verified.

Three independent questions, three independent answers. The middle one is the
interesting one: it holds even against someone holding the signing key.
Re-signing an altered report produces a valid signature over an inconsistent
document, and recomputing the score from the findings catches it anyway.

Without --key, a valid signature only proves somebody signed, so that case
is reported as an open question rather than a pass. --require-signature makes
an unsigned report a failure.

Keyless signing via Sigstore

assay sign report.json --keyless

An ephemeral key is generated, an OIDC identity proves who is running, Fulcio
issues a ~10-minute certificate, the signature goes into Rekor's public
transparency log, and the private key is discarded. There is nothing left to
steal, and issuance is publicly auditable rather than asserted — which matters
for a party publishing verdicts about other people's work.

Verification requires an expected identity:

assay verify report.json --bundle report.sigstore.json \
  --issuer https://token.actions.githubusercontent.com \
  --identity "https://github.com/me/repo/.github/workflows/release.yml@refs/heads/main"

Needs an OIDC identity — ambient in CI with id-token: write, a browser flow
locally. The local ed25519 path above still works entirely offline.

Known-vulnerable dependencies

assay run . --net        # allows api.osv.dev only. No model, no cost.

OSV rather than NVD, deliberately: NVD describes affected software with CPE
strings, a taxonomy invented for enterprise inventory with no canonical mapping
to an npm or PyPI name. --net grants only the advisory lookup, through an
allowlisted client that refuses plaintext, re-checks the allowlist on every
redirect hop, and records every request into the report.

How it works

Determinism is enforced, not asserted

Every check declares a tier, and the tier is mechanically true because
capabilities are granted by declaration. A deterministic check that requests a
capability fails to load — not a convention, a throw.

Tier	Promise
deterministic	Pure function of the artifact bytes and the check version. Re-running in 2030 is byte-identical.
replayable	Expensive to produce, cheap to verify. The transcript is recorded, so the grade can be re-derived.
sampled	Irreducibly stochastic. Must report n, mean, spread, and pass^k.

Seeding is not the reproducibility story. Nondeterminism at temperature 0
comes from a lack of batch invariance in GPU kernels: many kernels produce
different numerics depending on the batch they land in, so a request's output
depends on what else was in the forward pass at that moment. Sampling one
prompt 1,000 times at temperature 0 produced 80 distinct completions.
It is fixable — with batch-invariant kernels, in your own serving stack — but
not by anyone calling a hosted API, which is what an evaluator is doing.
Recording and replaying is.

Judgements carry their evidence

A finding that says "found a hardcoded secret" without saying where is
unfalsifiable and unfixable. Every result carries Evidence — a file path and
line, an external URL with a response digest, a transcript digest, or a
structured metric. Secret findings deliberately report the path and never the
matched text, because a report that quotes the credential has republished it.

Four axes, not one number

Axis	Question
integrity	Is it well-formed and does it load?
safety	Could installing this hurt me?
care	Is it maintained by someone paying attention?
behavior	Does it actually do what it claims?

Four rules, each executable as a test:

1. Coverage is reported, never assumed. An axis that could not be measured
   says so, in the human output and as null in the JSON.
2. error never counts against the subject. Our sandbox timing out is our
   problem, not a mark on someone's work.
3. Safety does not average. A blocking safety failure floors the axis —
   credential exfiltration is not offset by a tidy README.
4. The formula is versioned and published, and the whole computation runs
   off the report, so anyone can recompute the number or show that they can't.

The field has been moving away from composite scores. OpenSSF Scorecard's own
v6 roadmap reframes the project around producing "trusted, structured security
evidence," with per-probe results that consumers can act on instead of an
aggregate out of ten. And a peer-reviewed study of 679 projects found reported
vulnerabilities increased as the aggregate Scorecard score increased.

_{Scorecard v6 roadmap ·
Beyond Scores with OpenSSF Scorecard ·
"Do Software Security Practices Yield Fewer Vulnerabilities?", arXiv:2210.14884}

That is why the headline sits behind a coverage floor, why the four axes are
reported alongside it, and why --json is the surface a consumer is expected to
build on.

Writing a check

A check is a standalone, versioned module. That is the whole point — it turns
"trust our score" into "read the check that produced it."

import { defineCheck } from "@metahub-ai/assay";

export default defineCheck({
  id: "acme/no-eval",
  version: "1.0.0",
  title: "No dynamic eval",
  category: "safety",
  axis: "safety",
  determinism: "deterministic",
  weight: 3,
  async run(ctx) {
    const src = (await ctx.source.readFile("index.js")) ?? "";
    const line = src.split("\n").findIndex((l) => l.includes("eval("));
    if (line < 0) return { status: "pass", summary: "No eval() calls." };
    return {
      status: "fail",
      summary: "Uses eval().",
      evidence: [{ type: "file", path: "index.js", line: line + 1 }],
      remediation: "Replace eval() with an explicit dispatch table.",
    };
  },
});

Namespace third-party ids as vendor/check-name. See
CONTRIBUTING.md for the bar a check has to clear.

Every check in the default suite is documented in
docs/CHECKS.md, generated from the registry so it cannot drift
from the code. assay explain <check-id> prints the same thing in your
terminal, including the waiver snippet.

Library use

import {
  runAssay,
  CheckRegistry,
  CORE_CHECKS,
  DirectorySource,
  digestTree,
} from "@metahub-ai/assay";

const source = new DirectorySource("./my-skill");
const report = await runAssay({
  subject: {
    kind: "skill",
    name: "my-skill",
    source: { type: "directory", path: "./my-skill" },
    digest: { sha256: await digestTree(source) },
  },
  source,
  registry: CheckRegistry.from(CORE_CHECKS),
  suite: { id: "assay-starter", version: "0.1.0" },
  environment: { runner: "assay/0.1.0" },
});

Sandbox adapters ship for podman (local, default) and E2B (cloud). Model
adapters ship for Anthropic, OpenAI, OpenRouter, and any
OpenAI-compatible local endpoint; each speaks its vendor's HTTP API directly and
needs no SDK installed. The core framework depends on no vendor.

What Assay does not claim

Stated up front, because overclaiming is the fastest way to lose the credibility
this project exists to build:

• Dynamic analysis only observes paths it triggers. Code on an untaken
  branch is unobserved.

• Sandbox-evasive and time-delayed payloads defeat it. This is the standing
  limit of the technique, not a bug we can close.

• Models detect evaluation. Anthropic reported (February 2026) that Claude
  Opus 4.6 independently inferred it was being benchmarked, identified which
  benchmark, and located the answer key — and that suppressing its
  eval-awareness increased misaligned behavior, meaning models behave better
  when they believe they are being watched. That is an unresolved validity
  problem for every behavioral number, including these.
  _{Eval awareness in Claude Opus 4.6 ·
  Claude Opus 4.6 system card}

• The judge is consistent, not validated. Replayed transcripts reproduce
  their verdicts. Nothing yet measures those verdicts against human labels.

• The lethal trifecta is not a property of any artifact. Private data +
  untrusted content + external communication is a property of the user's whole
  tool graph. Every individual artifact can pass and the combination still be
  unsafe.

docs/COVERAGE.md is the full accounting, including the parts
that are weaker than you would like.

Documentation

Document	What's in it
docs/COVERAGE.md	What Assay measures, what it doesn't, and what it structurally cannot
docs/CHECKS.md	Every check in the default suite, generated from the registry
CONTRIBUTING.md	How to write a check, the review bar, the versioning rules
SECURITY.md	Reporting a vulnerability, and Assay's own threat model

Development

npm install
npm test          # 1007 tests
npm run typecheck
npm run lint
npm run build

Every core check is testable from a plain object literal via MemorySource, and
all four behavioral harnesses run offline against in-memory doubles. If a
contribution can only be exercised against a live cloud sandbox and a paid
model, it is coupled to a transport and the design has failed.

License

Apache-2.0 — chosen over MIT for the explicit patent grant, which an
open standard benefits from.