awesome-ai-security-engineering
Health Uyari
- License — License: MIT
- Description — Repository has a description
- Active repo — Last push 0 days ago
- Low visibility — Only 6 GitHub stars
Code Gecti
- Code scan — Scanned 12 files during light audit, no dangerous patterns found
Permissions Gecti
- Permissions — No dangerous permissions requested
Bu listing icin henuz AI raporu yok.
Curated, vendor-neutral open-source tools, standards, benchmarks and labs for AI Security Engineering, organized by lifecycle (Discover, Design, Build, Break, Defend). From the AI Security Engineers Community.
Awesome AI Security Engineering
Open-source tools, research, standards, and best practices for securing software that AI writes, powers, and operates, curated by the AI Security Engineers Community.
This repository is created and maintained by the AI Security Engineers Community. It is vendor-neutral: every entry is judged against the same inclusion criteria, and anything affiliated with the maintainers' sponsor is labeled.
Why it matters
AI changes software in three ways at once. AI writes software: coding assistants and agents produce code faster than people can review it. AI lives inside software: models, retrieval, agent memory, MCP servers and skills become components with their own attack surface. AI operates software: agents call APIs, use credentials and change real systems, so deterministic permissions end up in the hands of probabilistic systems.
Developers, AppSec, cloud/IAM, AI/ML, SOC and GRC teams each own a piece. Nobody necessarily owns the security of the whole AI-native system. That ownership gap is what AI Security Engineering addresses.
What is AI Security Engineering?
AI Security Engineering is the discipline of designing, building, testing, and defending AI-native software throughout its lifecycle, combining software security, AI engineering, and adversarial thinking to manage risks introduced by AI-assisted development, AI-powered applications, and autonomous agents.
It is an engineering role at the intersection of software engineering, security engineering and AI engineering, and it does not replace existing teams. The discipline is still forming, so definitions here are proposals for the community to refine. Read more in foundations.
The lifecycle
Five complementary, iterative stages organize this list. Each page explains the stage and lists the matching resources.
| Stage | Question it answers | Page |
|---|---|---|
| Discover | What AI exists in our environment, and what can it reach? | discover |
| Design | What are the trust boundaries, identities and permissions? | design |
| Build | How do we ship AI-assisted and AI-powered software securely? | build |
| Break | How does the system fail under adversarial conditions? | break |
| Defend | How do we detect, contain and respond at runtime? | defend |
Contents
- Resources by lifecycle stage (below)
- Resources by technical domain
- Full catalogue with license and verification details
- Research and benchmarks, including a VulnBench write-up
- Best practices
- Learning and labs
- Foundations: initial vision
- Foundations: definition, responsibilities, frameworks
- Community
- Contributing
Resources
Each resource appears once here, under its primary stage. Many span several stages; the stage pages and topic index cross-reference the same canonical entries. Licenses shown are the verified license of the listed artifact; read the catalogue for caveats such as mixed licensing or commercial add-ons.
Discover
| Resource | What it is | Type | License |
|---|---|---|---|
| agent-bom | Inventories local agent configs, MCP servers, skills and models, and emits CycloneDX/SPDX/SARIF. | Open-Source Tool | Apache-2.0 |
| CycloneDX | Bill-of-materials standard (ECMA-424) with ML-BOM support for models and datasets. | Open Standard / Protocol | Apache-2.0 |
| Snyk Agent Scan | Discovers agent configurations, MCP servers and skills; deeper security analysis uses a Snyk-hosted API (open core). | Open-Source Tool | Apache-2.0 |
Design
| Resource | What it is | Type | License |
|---|---|---|---|
| CoSAI Secure Design Patterns for Agentic Systems | OASIS-hosted whitepapers and design patterns for MCP security and agentic identity and access management. | Reference Implementation | CC-BY-4.0 |
| Microsoft Agent Governance Toolkit | Multi-language SDKs for agent policy enforcement, an MCP security gateway, agent identity and audit logging. | Open-Source Tool | MIT |
| MITRE ATLAS | Knowledge base of adversary tactics, techniques and case studies against AI-enabled systems. | Open Security Framework | Apache-2.0 |
| Model Context Protocol (MCP) | Open protocol connecting AI applications to tools and data, with an OAuth-based authorization model. | Open Standard / Protocol | Apache-2.0 |
| Open Policy Agent (OPA) | General-purpose policy engine (Rego) for deterministic authorization and policy-as-code. | Open-Source Tool | Apache-2.0 |
| OWASP GenAI Security Project | Umbrella OWASP project for LLM and agentic AI security guidance, cheat sheets and initiatives. | Open Security Framework | CC-BY-SA-4.0 |
| OWASP Top 10 for Agentic Applications 2026 | Peer-reviewed list of the ten most critical risks for autonomous, tool-using AI agents. | Open Security Framework | CC-BY-SA-4.0 |
Build
| Resource | What it is | Type | License |
|---|---|---|---|
| Cisco AI Defense Skill Scanner | Scanner for agent skills using YAML/YARA rules and dataflow analysis, with SARIF output. | Open-Source Tool | Apache-2.0 |
| Eclipse Enclave | Runs coding agents in per-session Docker containers with a gateway that enforces network allowlists. | Open-Source Tool | MIT |
| Fickling | Static analyzer, decompiler and safe loader hooks for Python pickle files, including PyTorch models. | Open-Source Tool | LGPL-3.0-or-later |
| NeMo Guardrails | Toolkit for adding programmable guardrails (input, output, dialog, retrieval) to LLM applications. | Open-Source Tool | Apache-2.0 |
| Promptfoo | CLI and library for LLM evaluations and red-team testing, usable in CI. | Open-Source Tool | MIT |
| sandbox-runtime | Filesystem and network sandbox for running agent processes with restricted access. | Open-Source Tool | Apache-2.0 |
| Sigstore model-transparency (model-signing) | Python library and CLI for signing and verifying ML model artifacts with Sigstore. | Open-Source Tool | Apache-2.0 |
| Snyk VulnBench | Benchmark and study of how repeatable LLM-based vulnerability findings are, versus a deterministic SAST reference. | Benchmark | Apache-2.0 |
Break
| Resource | What it is | Type | License |
|---|---|---|---|
| AgentDojo | Dynamic benchmark environment for prompt-injection attacks and defenses against tool-using agents. | Benchmark | MIT |
| Cybench | CTF-based benchmark of 40 tasks for evaluating the offensive cyber capabilities of LLM agents. | Benchmark | Apache-2.0 |
| CyberSecEval (Purple Llama) | Meta's cybersecurity benchmark suite for LLMs, including prompt injection, insecure code and autopatch tests. | Benchmark | MIT |
| DeepTeam | Open-source LLM red-teaming framework with vulnerability and attack modules and mappings to OWASP and NIST. | Open-Source Tool | Apache-2.0 |
| garak | LLM vulnerability scanner that probes models and applications with many attack and failure modes. | Open-Source Tool | Apache-2.0 |
| Inspect AI | Framework for reproducible LLM and agent evaluations with sandboxed tool execution. | Open-Source Tool | MIT |
| MCPSecBench | Security benchmark for Model Context Protocol servers, clients and hosts. | Benchmark | MIT |
| OWASP FinBot Agentic AI CTF | Multi-agent financial-services app with agentic attack challenges (goal manipulation, prompt injection). | Learning / Lab | Apache-2.0 |
| PyRIT | Python framework for automated and human-assisted red teaming of generative AI systems. | Open-Source Tool | MIT |
Defend
| Resource | What it is | Type | License |
|---|---|---|---|
| Guardrails AI | Python framework for validating and structuring LLM inputs and outputs with pluggable validators. | Open-Source Tool | Apache-2.0 |
| OpenTelemetry GenAI Semantic Conventions | Vendor-neutral span, metric and attribute conventions for tracing GenAI calls, agents and tool use. | Open Standard / Protocol | Apache-2.0 |
Inclusion criteria
Only open-source software, qualifying open standards, and open research with reproducible artifacts are listed. A public website or a free tier is not enough.
- Software: public source, an identifiable open-source license (OSI-approved preferred), and meaningful functionality in the open-source version. Open-core projects qualify only for capabilities that are genuinely open.
- Standards and frameworks: publicly accessible specification, documented implementation rights, and a transparent development process. Formal standards, community frameworks and guidance are labeled differently.
- Research: open artifacts such as code, datasets, benchmark harnesses or reproducible experiments. A free paper alone does not qualify.
- Quality: engineering relevance, technical credibility, documentation, maintenance, distinctiveness. Stars are not a criterion.
- Neutrality: no placement based on sponsorship or brand, no proprietary products, and no projects maintained by direct competitors of the community's sponsor.
Full rules, including exclusions, are in CONTRIBUTING.md. Candidates that were evaluated and rejected are recorded in the research inventory.
Contributing
Suggest a resource with the new resource issue form or open a pull request that edits data/resources.yml and runs python scripts/generate.py. Broken links, license concerns and taxonomy ideas have their own forms. See CONTRIBUTING.md.
Community
The AI Security Engineers Community runs local chapters and meetups for practitioners who build with AI and secure it. See community.
License
Repository content and scripts are released under the MIT License. Listed projects keep their own licenses; check each before use.
Yorumlar (0)
Yorum birakmak icin giris yap.
Yorum birakSonuc bulunamadi