Trustabl — Azure DevOps Extension

An Azure Pipelines task that runs trustabl,
the agent reliability scanner for AI agent repos (Claude Agent SDK, OpenAI
Agents SDK, Google ADK, LangChain, LangGraph, CrewAI, AutoGen, AG2, Pydantic AI,
Vercel AI SDK, MCP).

⚠️ Proprietary — not open source. This extension is licensed for use only
(install it and reference the task in your pipelines). Copying, forking,
modifying, or redistributing the source is not permitted. See LICENSE.

Demo

A sixty-second walkthrough: install it, gate it, run the first scan, and the
exit code CI reads.

▶ Watch the demo · assets/trustabl-demo.mp4

Evaluating this against other tools? docs/EVALUATION.md
covers how to trial it and how to read what it reports.

Capabilities

• Downloads the official trustabl release binary (Linux / Windows / macOS agents).
• Scans your sources for tools, agents, subagents, and MCP servers.
• Computes a readiness score and its inverse risk score.
• Fails the build on a risk-score or severity threshold (both optional).
• Publishes trustabl.json + trustabl.sarif as a pipeline artifact.
• Uploads a markdown run summary and prints a console report.
• Exposes output variables (readiness, risk, severity, findings count, exit code).
• Optionally runs AI enrichment on findings (explanations + suggested fixes), published as enriched.json.

Usage

steps:
  - task: Trustabl@0
    inputs:
      severityThreshold: high      # fail on any high/critical finding (optional)

Zero-config (scans $(Build.SourcesDirectory), fails only if trustabl itself
flags medium+):

steps:
  - task: Trustabl@0

Full example (annotated)

Every input shown with its default and purpose:

steps:
  - task: Trustabl@0
    name: trustabl                        # ref name → read outputs as $(trustabl.<var>)
    inputs:
      target: $(Build.SourcesDirectory)   # path or GitHub URL to scan
      version: latest                     # trustabl release tag (e.g. v0.5.0) or 'latest'
      detectors: ''                       # subset: claude_sdk,openai_sdk,google_adk — empty = all
      strict: false                       # --strict: fail on ANY finding, regardless of severity
      riskScoreThreshold: '0'             # fail when risk (100 - readiness) >= N (1-100); 0 = off
      severityThreshold: none             # fail at >= none | low | medium | high | critical
      publishArtifact: true               # upload trustabl.json + trustabl.sarif as an artifact
      artifactName: trustabl-scan-results # name of that artifact
      sarifFile: trustabl.sarif           # SARIF output path
      jsonFile: trustabl.json             # JSON ScanResult output path
      rulesRef: ''                        # pin an agent-reliability-rules git ref (empty = default)
      rulesRepo: ''                       # override agent-reliability-rules source repo (empty = default)
      githubToken: $(GITHUB_TOKEN)        # optional secret to dodge the GitHub API rate limit
      enrich: false                       # run AI enrichment on findings (explanations + suggested fixes)
      llmProvider: anthropic              # LLM provider for enrichment (currently only 'anthropic')
      llmKey: $(ANTHROPIC_API_KEY)        # API key for the LLM provider (BYOK); required when enrich is true
      enrichModel: ''                     # Claude model for enrichment (empty = trustabl binary default)
      enrichRules: ''                     # comma-separated rule IDs to enrich; empty = all findings

Consuming outputs

Give the step a name, then read $(<name>.<var>) in later steps:

steps:
  - task: Trustabl@0
    name: trustabl
    inputs:
      riskScoreThreshold: "0"      # observe, don't gate
  - script: echo "readiness=$(trustabl.readinessScore) risk=$(trustabl.riskScore) findings=$(trustabl.findingsCount)"

Enrich

Optionally run AI enrichment on findings — generates plain-language explanations and
suggested fixes for each finding, written to enriched.json and published alongside
the scan results (when publishArtifact is true).

Requires your own LLM API key (BYOK — bring your own key). Currently supports Anthropic.

steps:
  - task: Trustabl@0
    inputs:
      enrich: true
      llmKey: $(ANTHROPIC_API_KEY)   # pipeline secret variable

Narrow enrichment to specific rules, or pin a non-default model:

steps:
  - task: Trustabl@0
    inputs:
      enrich: true
      llmKey: $(ANTHROPIC_API_KEY)
      enrichModel: claude-sonnet-4-6
      enrichRules: ADK-201,ADK-105

Enrich is best-effort: if it fails (bad key, rate limit, etc.) it logs a warning and the
scan/gate results are unaffected.

Enrich only generates explanations and suggested fixes — it does not modify source files
or open pull requests.

Inputs

Output variables

Building from source

cd Trustabl
npm install
npm run typecheck   # tsc --noEmit
npm run build       # tsc → emits index.js next to task.json

Package the extension (needs tfx-cli):

npm i -g tfx-cli
tfx extension create --manifest-globs vss-extension.json

Upload the resulting .vsix to the Visual Studio Marketplace, then install it on
your Azure DevOps organization.