Project Nexus · ShadowScan

ShadowScan is an open-source tool that discovers evidence of AI agents and related integrations, then reconciles it against your approved agent registry.

It inspects six surfaces: code repositories, identity providers, LLM gateway logs,
low-code platforms, SaaS apps, and cloud accounts. It fingerprints frameworks and
model providers, scores findings, and reconciles discoveries against your approved
registry of
Agent Cards. Static code signals identify candidates; trusted runtime evidence is needed to establish execution.
Counts and severity labels need an analyst review before they drive enforcement.

Example findings (totals vary as signatures evolve):

$ shadowscan scan -c examples/shadowscan.offline.yaml --max-rows 5

╭──────────────────────────────── ShadowScan ────────────────────────────────╮
│ 99 findings  •  95 shadow (inventory: 3 registered agents)                  │
│ critical 12  high 52  medium 35  •  code 11 identity 19 cloud 27 …          │
╰────────────────────────────────────────────────────────────────────────────╯
 CRITICAL 100  SHADOW  saas      bot-app      GitHub App installed: claude
 CRITICAL 95  SHADOW  code      mcp-server   MCP configuration: .mcp.json (inline GitHub PAT, Zapier remote MCP, docker/postgres)
 CRITICAL 90  SHADOW  code      secret       LLM provider credential in services/research-agent/app/config.py
 HIGH     73  SHADOW  lowcode   agent        Copilot Studio agent: HR Helper
 MEDIUM   33  ops-provisioning-04  cloud  agent  Bedrock Agent: ops-provisioning-04   ← registered via its card's resource binding; owner from the AWS resource tag

Why

Agents are no longer only Python scripts.
They are Copilot Studio bots built by HR, n8n flows with an AI Agent node, OAuth grants to meeting note-takers,
Bedrock Agents provisioned by Terraform, MCP servers wired into every
developer's editor, service principals with Mail.ReadWrite acting on behalf of nobody, and JWTs carrying an act claim.
Each surface has its own discovery API and its own vocabulary.
ShadowScan normalizes these observations into one finding model with evidence,
so investigators or auditors can ask: Who owns this AI Agent? What can it do, and is it
registered in the registry supplied for this scan?

Confidence is a heuristic evidence score, not a measured probability. Detection
quality depends on the repositories, providers, tenant permissions, and log
provenance in your environment. The bundled evaluation corpora are
author-written regression cases, including multi-file cases with documented
misses, not a field precision or recall estimate. See
evaluation and
rollout acceptance before using a risk
threshold as a production gate.

The assurance results preserve the baseline and
subsequent results on a frozen, independently AI-labeled corpus of 42 public
files (30 negatives). Read-only AWS and Slack canaries
validate named tenant controls when approved credentials are supplied; offline
replay does not establish live tenant acceptance.

Deployment evidence can be checked with the offline
acceptance verifier. It requires current source and
signature identities, declared human-reviewed holdout evidence, and live tenant
receipts for supported live deployment scopes. It validates supplied evidence;
it cannot authenticate reviewer independence or manufacture tenant acceptance.
The release-evidence workflow builds a candidate
wheel and retains hashes, a runtime dependency SBOM, and provenance after the
selected commit passes CI and CodeQL. Artifact provenance does not establish deployment
acceptance, and the workflow does not publish a release.

Surfaces & connectors

ShadowScan ships 27 connectors across the six surfaces below.

Connectors support live API collection, offline JSON/CSV/log exports,
or both; see the connector guide for the supported modes and provider scope.
Offline analysis can run in CI, on an analyst's laptop, or against a SIEM export.

Frameworks & products recognised

215 signatures / 1001 signals, YAML-defined with explicit opt-in overrides:

• Orchestrators – LangChain, LangGraph, Deep Agents, LlamaIndex, CrewAI, Google ADK, AWS Strands Agents, Microsoft Agent Framework, Semantic Kernel, AutoGen/AG2, Hugging Face smolagents, OpenAI Agents SDK, OpenAI Swarm, Claude Agent SDK, Pydantic AI, Vercel AI SDK, Mastra, Haystack, DSPy, Agno, Letta, MetaGPT, CAMEL, Griptape, Composio, Langroid, AgentScope, Swarms, AutoGPT, BabyAGI, BeeAI, Atomic Agents, Julep, Marvin, Mirascope, Qwen-Agent, NVIDIA NeMo Agent Toolkit, Dapr Agents, PraisonAI, SWE-agent, GPT Engineer, Open Interpreter, Chainlit, Prompt flow, Guardrails AI / NeMo Guardrails / LLM Guard, LangChain4j, Spring AI, Rig, LangChainGo, Genkit, Eino, M365 Agents SDK, Bot Framework, Teams AI, Cloudflare Agents, Inngest AgentKit, VoltAgent, CopilotKit/AG-UI, Rasa, Botpress, Browser Use, Stagehand, OpenHands, Nova Act, Anthropic computer use
• Protocols – MCP (all client config locations, servers, registries, remote MCP hosts), A2A agent cards, ACP, tool/function-calling request shapes, ChatGPT plugin/GPT Action manifests
• Coding agents – Claude Code, GitHub Copilot coding agent, Cursor, Windsurf, Cline, Roo, OpenAI Codex, Gemini CLI/Jules, Amazon Q/Kiro, Goose, Aider, Continue, Cody/Amp, Junie, AGENTS.md, PR review bots (CodeRabbit, Sweep, Ellipsis, Greptile, Qodo…)
• Platforms/gateways – LiteLLM, Portkey, Kong AI Gateway, Helicone, OpenAI AgentKit, Dify, Flowise, Langflow, n8n, Make, Zapier, Workato, Copilot Studio, Power Platform AI connectors, M365 declarative agents, Agentforce, Now Assist, IBM watsonx Orchestrate, Retool, Open WebUI/LibreChat/AnythingLLM, Coze/Relevance/Lindy/Vellum…
• Model providers – OpenAI, Anthropic, Gemini API, Vertex AI, Bedrock, Azure OpenAI, Mistral, Cohere, Groq, Together, Fireworks, OpenRouter, Ollama, vLLM, SGLang, llama.cpp, LM Studio, LocalAI, Llama Stack, Meta Llama API, Hugging Face, xAI, DeepSeek, Alibaba DashScope/Qwen, Moonshot/Kimi, AI21, Perplexity, Replicate, Cerebras, SambaNova, NVIDIA NIM, OCI Generative AI, watsonx, Databricks, Cloudflare Workers AI, Snowflake Cortex (deps, imports, endpoints, env vars, user agents, model IDs, key formats)
• Observability/memory/sandboxes – LangSmith, Langfuse, Phoenix, AgentOps, Traceloop, Weave, Braintrust…, Mem0, Zep, vector stores, E2B, Daytona, web search/scrape tool providers
• AI SaaS as OAuth apps – ChatGPT, Claude, Gemini, Copilot, Perplexity, Mistral Le Chat, DeepSeek, Qwen, Kimi, Grok, Poe, Character.ai, Glean, meeting note-takers (Otter, Fireflies, Read.ai, Fathom, tl;dv, Gong…), writing/coding/automation/research/media products, browser AI extensions
• Permission policies – privileged, data-access and LLM-access scope classes across Graph, Google, Okta, Slack, GitHub, GitLab, Salesforce, Atlassian, Zoom, Notion, HubSpot, AWS IAM, Azure RBAC, GCP IAM, OCI

shadowscan signatures list shows everything; shadowscan signatures test <value>
tells you what a package, host, user agent, model id, scope or file path maps to.

Install

Select the full 40-character commit SHA after reviewing its changes and CI
results. Set SHADOWSCAN_REVISION to that SHA; do not use a moving branch or an
unpublished tag in a deployment job. Install the core scanner or the cloud
extra in a clean virtual environment:

SHADOWSCAN_REVISION="REPLACE_WITH_REVIEWED_40_CHARACTER_SHA"
python -m pip install "git+https://github.com/aisecnomad/Project-Nexus.git@${SHADOWSCAN_REVISION}"

For cloud collection, install the extra from the same reviewed revision:

SHADOWSCAN_REVISION="REPLACE_WITH_REVIEWED_40_CHARACTER_SHA"
python -m pip install "project-nexus-shadowscan[cloud] @ git+https://github.com/aisecnomad/Project-Nexus.git@${SHADOWSCAN_REVISION}"

The distribution is named project-nexus-shadowscan; the command and Python
imports remain shadowscan. The unrelated shadowscan package on PyPI is not
this project. The new distribution name is not a claim of a published package
or a reserved namespace: install the reviewed source or built wheel below.

The current 0.1.1 source version is an unreleased candidate; the version
string does not imply a published or signed artifact. These VCS installs resolve
transitive dependencies at install time. For deployment, use the locked install
below. Python 3.11, 3.12, or 3.13 is required and covered by CI. Core dependencies
include click, rich, PyYAML, requests, urllib3,
PyJWT[crypto] and regex. Cloud SDKs are optional extras; every cloud connector
also accepts an offline record dump.

For deployment on Linux x86_64 with Python 3.11 to 3.13, the checked-in
requirements.lock pins and hashes the core and all cloud runtime dependencies.
Build and retain a wheel from the selected commit; see
locked installs and release evidence.
The consumer GitHub Action example requires
the repository variable SHADOWSCAN_REVISION to hold that reviewed full SHA;
it fails until the variable is set.

Quick start

# 1. Scan a checkout (or your whole ~/src) — no credentials needed
shadowscan code . --inventory agent-card.yaml

# 2. Try every fixture-backed connector offline (demo; 25 of 27 connectors ship fixtures)
shadowscan scan -c examples/shadowscan.offline.yaml --format html -o report.html

# 3. Real estate: one config, live connectors, secrets from the environment
shadowscan scan -c shadowscan.yaml --format sarif -o shadowscan.sarif

# 4. Single connector, ad-hoc
shadowscan run identity.entra --set tenant_id=$AZURE_TENANT_ID
shadowscan run cloud.aws --set regions=us-east-1,eu-west-1 --dump-records ./exports
# Read exports/manifest.json and use the exported filename for this instance:
shadowscan run cloud.aws --input ./exports/0001-cloud_aws.jsonl   # re-analyse later, offline

# 5. Logs and tokens
shadowscan gateway litellm-spend.jsonl bedrock-invocations/ egress-proxy.log
shadowscan jwt --file ./token.jwt --jwks-url https://acme.okta.com/oauth2/default/v1/keys

# 6. Register what you found
shadowscan inventory stubs report.json -o inventory/pending/    # capability-card stubs for shadow agents
shadowscan diff last-week.json today.json                        # what is new / resolved/changed

Steps 1 and 2 need a repository checkout: agent-card.yaml, examples/ and
the sample exports under tests/fixtures/ are not shipped in the wheel. Steps
5 and 6 use your own logs, token, and an earlier JSON report
(--format json -o report.json); sample gateway logs live under
tests/fixtures/gateway/ in a checkout.

Configuration

# shadowscan.yaml
inventory: [./inventory]              # Agent Cards, agents.yaml or CSV
signatures: [./custom-signatures]     # optional: extra packs; overrides require opt-in
options:
  plugins: []                        # exact names of reviewed third-party connectors
  allow_signature_override: false
  allow_private_origin: false        # opt in only for trusted private HTTPS APIs
  allow_credential_mixing: false     # separate repository scans from live tenant access
  allow_instance_credentials: false # cloud metadata credentials require explicit opt-in
  connector_timeout_seconds: 120    # soft deadline; also enforce a host job timeout
  parallel: 4                        # worker threads; use 1-2 for CPU-bound offline scans
  min_confidence: 0.3
  dump_records: ./exports             # sanitized records for offline re-runs; excludes JWTs
connectors:
  - name: identity.entra
    tenant_id: ${AZURE_TENANT_ID}
    client_id: ${AZURE_CLIENT_ID}
    client_secret: ${AZURE_CLIENT_SECRET}
  - name: identity.google-workspace
    service_account_file: ./sa.json
    admin_email: [email protected]
  - name: gateway.logs
    label: litellm-prod                # names this entry in --only, progress and exports
    input: ./exports/litellm-spend.jsonl
  - name: lowcode.power-platform
    tenant_id: ${AZURE_TENANT_ID}
    client_id: ${AZURE_CLIENT_ID}
    client_secret: ${AZURE_CLIENT_SECRET}
  - name: saas.slack
    enabled: false                     # keep the entry, skip it on this run
    token: ${SLACK_TOKEN}
  - name: cloud.aws
    regions: [us-east-1, eu-west-1]
    cloudtrail_days: 7

shadowscan connectors lists every connector with its configuration keys and
required extras; shadowscan connectors --json also includes each connector's
offline export formats. Every entry also accepts enabled (default true) and
label (a distinct id when a connector runs more than once). See
docs/connectors.md for entry keys, credentials and
least-privilege scopes per connector. Run repository scans in
a separate job/configuration from live tenant collection. Mixing these credential
boundaries requires an explicit allow_credential_mixing exception; keep the
separation for untrusted repositories. Cloud instance-metadata credentials require
allow_instance_credentials: true; use an explicit audit identity by default.

Use --incremental to reuse completed scans of unchanged local checkouts and
static cloud exports. Live APIs and gateway logs are refreshed on every run.
Configure gateway.logs.correlation_bindings to link a code resource to a
specific gateway caller and scope; matching timestamped framework fingerprints
then appear in metadata.runtime_activity, including the observation window
and any production label claimed in the logs. Treat caller and environment
fields according to the export's provenance; ShadowScan does not authenticate
the source of an imported log. See scan state and runtime correlation
for configuration, limitations, and migration guidance.

Oversize files that the scanner would inspect and symbolic links that leave the
scan root make coverage incomplete (exit 3) by default. --strict-coverage
(strict_coverage: true) records them as errors instead of warnings; explicitly
declared oversize_skip_globs remain warnings. An in-root link stays complete
when its own name is never read (a lockfile or an image) or when it is a source
file whose target is analyzed in the same project with the same test
classification; directory links are incomplete because their alias paths are
not scanned. Evidence found only in test or fixture code cannot establish an
agent unless --include-tests is set.

The CLI exits 3 for incomplete scans, 2 for a completed scan that reaches
--fail-on, and 0 for a completed scan that passes. SARIF records incomplete
scans as unsuccessful, while preserving findings from successfully assessed inputs.
Enable --fail-on only after a frozen, independently adjudicated holdout
and read-only tenant canary
establish an acceptable threshold for that environment. A complete static scan
does not prove that an agent executed or that every eligible resource was collected.
The CLI normally exits promptly after a connector deadline even when a blocked
worker cannot be joined. A filesystem publication already in progress can still
delay timeout handling; enforce a host job timeout for hard limits.
Confidence thresholds must be finite numbers from 0 to 1; invalid CLI or YAML
values stop the scan before the risk gate runs.

Git history enrichment is disabled by default. Set connector use_git: true
only for a reviewed local checkout when author/history metadata is needed;
metadata commands cannot fetch missing objects. Every explicit --only selector
must match an enabled connector name or label. Unsupported records and saved API
error responses cannot establish an empty, successful inventory.

After upgrading from reports without the v2 finding-identity schema, regenerate
your comparison baseline. Findings now keep their identity when inferred classification changes;
legacy baselines cannot establish resolution under the new identity schema.
See deployment and migration for the rollout checks.

What a finding looks like

{
  "id": "ss-3f9c1e2a7b4d8c10",
  "surface": "cloud", "connector": "cloud.aws", "kind": "agent",
  "title": "Bedrock AgentCore runtime: strands_support_agent",
  "resource": "arn:aws:bedrock-agentcore:us-east-1:123456789012:runtime/rt1",
  "account": "123456789012", "region": "us-east-1", "owner": null,
  "frameworks": ["cloud.aws-bedrock-agents"], "model_providers": ["provider.openai"],
  "capabilities": ["tool-use"], "tags": ["plaintext-credential", "secret-in-env"],
  "confidence": 1.0, "likelihood": "confirmed",
  "shadow": true, "registry_match": null,
  "risk": {"score": 90, "level": "critical", "factors": [
      {"id": "shadow", "description": "not present in the sanctioned agent inventory", "weight": 25},
      {"id": "tag:plaintext-credential", "description": "plaintext credential in environment/configuration", "weight": 25},
      {"id": "no-owner", "description": "no identifiable owner", "weight": 10}, "..."]},
  "evidence": [
      {"signal": "aws:agentcore-runtime", "description": "AgentCore runtime 'strands_support_agent' (READY) role arn:aws:iam::…", "weight": 0.97},
      {"signal": "secret: provider.openai", "description": "Plaintext OpenAI API key in environment variable OPENAI_API_KEY: sk-p…KLMN", "weight": 0.6}],
  "metadata": {"status": "READY", "protocol": "HTTP", "network": "PUBLIC", "related": ["ss-…"]}
}

• confidence combines evidence weights with noisy-OR. It groups correlated source evidence first, so repeated matches cannot inflate the score. It is a heuristic evidence score, not a calibrated probability or proof that an agent executed.
• potential_capabilities in static finding metadata records framework features supported only by availability evidence, such as an import or dependency. These are excluded from capability risk factors until stronger source evidence supports them.
• risk is additive and explainable: kind, capabilities (code-exec, autonomous, SaaS actions…), permission classes, credential exposure, exposure/auditability tags, registration status, ownership — scaled by confidence. The listed factors always add up to score; confidence scaling and the 0–100 bounds appear as factors.
• danger_score is the same model without the governance factors (inventory registration and ownership): what the agent can do, independent of whether anyone approved it. Set options.risk_basis: danger to base level and --fail-on on it, and options.risk_weights to tune weights (see Risk policy).
• shadow is true unless exactly one inventory entry matches an explicit resource pattern and its configured scope restrictions; names only suggest entries for review. An approved entry lends its owner to the finding.
• related links findings across surfaces (the Terraform that provisions an agent ↔ the agent in the account ↔ the role calling Bedrock ↔ the CloudTrail caller).

Outputs: table (terminal), json, sarif (GitHub code scanning; code
findings carry file: line locations), csv, markdown, html (self-contained,
filterable, with evidence drill-down). In csv output a literal ' is
inserted at the start of a value, and after each ,, ;, tab, | or line
break inside it, where the text that follows begins with =, +, - or @
(also after whitespace or quotes) or with a tab or carriage return.
Spreadsheets then read such cells as text (common for @team owners)
whichever delimiter they split on. A value that begins with a line feed is
marked too; other tabs and line breaks inside a value are left alone. Strip
those markers when consuming the file programmatically, or use json.
markdown output defangs links (hxxps://, www[.]) and writes @ as [@]
in untrusted text, so a report pasted into an issue or pull request creates no
links, @-mentions or e-mail links; code spans keep identifiers verbatim.

Risk policy

options:
  risk_basis: danger          # combined (default) | danger: level from capabilities, not registration
  risk_weights:               # integers -100..100; unknown groups, kinds or governance keys are rejected
    capabilities: {code-exec: 25}
    tags: {meeting-bot: 20}
    providers: {provider.deepseek: 20}
    kinds: {agent: 20}
    governance: {shadow: 15, no-owner: 5, registered: -10}

Sanctioned inventory

Drop your Agent Cards in a directory. The card's metadata.agent_id
identifies the registration; explicit discovery.resources bind it to concrete resources:

metadata:
  agent_id: "ops-provisioning-04"
  owner_team: "Platform-Engineering"
discovery:
  resources:
    - "arn:aws:bedrock:*:123456789012:agent/AGENT1"
    - "github: acme/infra-agents"
  names: ["ops provisioning agent"]

Simple agents.yaml lists and CSV work too. shadowscan inventory stubs
turns shadow findings into card skeletons for review. See
docs/inventory.md.

Extending

• Signatures are YAML; add a pack directory with --signatures / signatures:
  to add products. Replacing built-in signatures requires explicit opt-in. Schema and authoring guide in
  docs/signatures.md.
• Connectors implement collect() (live) and analyze() (records → findings)
  and register through the shadowscan.connectors entry-point group. Execution
  requires an exact-name options.plugins allowlist entry. See
  docs/architecture.md.

Development

pip install -e ".[cloud,dev]"
python -m shadowscan.signatures.validate
ruff check shadowscan tests tools
mypy shadowscan tools/evaluation tools/canaries tools/acceptance tools/release
pip-audit --progress-spinner off
pytest -q --cov=shadowscan --cov-fail-under=80
shadowscan scan -c examples/shadowscan.offline.yaml

Install the cloud extra for the same connector coverage as CI. Tests that
require missing optional SDKs can be skipped, so a core-only run does not validate all
connectors. See CONTRIBUTING.md.

Community and contributing

Contributions are welcome from developers, security practitioners, technical
writers and people testing the scanner against their own authorized data.
A small documentation fix or a reproducible false-positive report is useful.

Use synthetic, minimal examples in public reports. Scan results can contain
credentials, personal data, and sensitive inventory even after redaction.

Safety notes

• Known credential formats, sensitive configuration fields, and credential-bearing URLs are redacted before findings or sanitized record exports are persisted. Redaction cannot identify every arbitrary secret; reports still contain security-sensitive inventory data.
• Secret stores (Secrets Manager, Key Vault, Secret Manager, OCI Vault) are read for names only.
• JWTs are never persisted; findings reference a truncated hash.
• Built-in collectors inspect provider resources using read operations. Scope the audit identity to the documented read permissions and review any enabled third-party plugin separately.

Deployment behavior, migration options, and limits are documented in
SECURITY.md and docs/production.md.

License

Apache-2.0. See LICENSE and NOTICE.