RAPTOR — Open-Source Multimodal AI Framework & Agent Harness

RAPTOR Logo

Open-Source Multimodal AI Framework & Agent Harness
An enterprise AI runtime for building agentic applications over video, audio, images, documents, and organizational knowledge

About • Features • Installation • Quick Start • Documentation • Contributing • Website

RAPTOR is an enterprise AI runtime for building agentic applications over video, audio, images, documents, and organizational knowledge — providing MCP tools, A2A orchestration, persistent memory, hybrid RAG, GraphRAG, evaluation, guardrails, and model lifecycle services. Developed by the DHT Taiwan Team at DHT Solutions.

🚀 Current Release

Aigle 0.4 - Community Beta (August 2026)

This release continues the open-source RAPTOR framework, codenamed "Aigle". Release 0.4 grows the platform to 27 independently deployable Docker Compose modules (three earlier modules — hybrid-search, graph-database, graph-service — are retired and kept only for rollback) driven by the same single build system (Aigle/0.4/deployment/modules/build.py), and delivers the v0.4 roadmap: MCP interfaces across core services, persistent multimodal memory, and a per-user isolated multi-model database that replaces the previous Qdrant/OpenSearch/Neo4j trio.

Highlights:

🔌 MCP Server — Raptor's search, RAG, asset, and memory APIs exposed as Model Context Protocol tools/resources for LLM clients (module 27)
🧠 Memory Service — persistent per-user/per-session memory (facts, preferences, conversation history, multimedia) backed by MemVID, layered underneath the existing Redis short-term chat cache (module 26)
🗄️ Personal DB Service — one physically isolated ArcadeDB database per user, replacing the old per-user Qdrant + OpenSearch + Neo4j trio with unified hybrid BM25 + vector + graph + temporal-fact search (modules 24/25)
🛡️ Guardrail Service — LLM input/output content moderation with pluggable guard models (Llama Guard 3 / Granite Guardian / GPT-OSS-Safeguard) and a policy engine; ships disabled by default (module 23)
📊 Benchmark Service — user-defined scoring schemas (keyword match, LLM-as-judge, cosine similarity) to regression-test chat/RAG/search/classification pipelines (module 22)
🎬 Video Search 2.0, GraphRAG, A2A, branch isolation — carried over from Aigle 0.3 (see Features below)

See Aigle/0.4/README.md for the module reference, Aigle/0.4/BUILD.md for the build guide, Aigle/0.4/API_REFERENCE.md for the API reference, and Aigle/0.4/MCP_REFERENCE.md for the new MCP reference.

🧪 Evaluation and Testing API (Aigle 0.4)

To help developers get started with the RAPTOR framework quickly and easily, we've deployed a test run API on DHT's development infrastructure. This evaluation API allows developers to:

Test and evaluate RAPTOR capabilities without setting up infrastructure
Develop AI applications using the RAPTOR framework with zero deployment overhead
Utilize DHT resources for testing and development purposes
Prototype faster by accessing pre-configured AI services

This is an excellent way to explore RAPTOR's features, build proof-of-concepts, and validate your use cases before deploying your own infrastructure.

🔗 Access the Evaluation API:
http://raptor_open_0_4_api.dhtsolution.com:8012/

For detailed API documentation, usage examples, and access instructions, see Aigle/0.4/API_REFERENCE.md or visit the link above.

Note: This is a development environment intended for evaluation and testing purposes. For production deployments, please refer to the Installation and Development sections below.

🎯 About RAPTOR

RAPTOR is an open-source multimodal AI framework and agent harness — an enterprise AI runtime for building agentic applications over video, audio, images, documents, and organizational knowledge. Rather than a single content-search product, RAPTOR is a platform of composable services that an agent (or a human developer) can call directly: MCP tools and A2A orchestration for agent interoperability, persistent multimodal memory, hybrid BM25/vector RAG, GraphRAG and temporal knowledge graphs, pipeline evaluation and benchmarking, LLM guardrails, and model lifecycle management — all exposed as REST, MCP, and A2A interfaces over the same per-user isolated index.

Business Value Proposition

85% reduction in manual content tagging and metadata generation
10x faster content discovery through semantic search
60% improvement in content reuse and operational efficiency
Real-time insights from video, audio, and document content
Agent-ready by default — every capability above is callable by an LLM agent (MCP), another agent framework (A2A), or a human developer (REST), not locked behind a single interface

Strategic Differentiators

Agent-Native Interfaces: MCP tools and A2A orchestration (agent cards, JSON-RPC, five spec-compliant sub-agents) expose the platform to LLM agents and other agent frameworks, not just human API callers
AI-Native Architecture: Built from the ground up around LLM orchestration and vector search
Multi-Modal Understanding: Unified analysis across video, audio, image, and text
Semantic Intelligence: Hybrid BM25 + vector search plus GraphRAG and temporal knowledge graphs for context-aware retrieval, not just keyword matching
Composable Enterprise Services: Persistent memory, LLM guardrails, pipeline benchmarking, and model lifecycle management as first-class, independently deployable services
Open + Enterprise Model: Open-source core with premium enterprise features; Docker Compose today, with a Kubernetes-native production path planned for v1.0

✨ Features

Version Aigle 0.4

New in this release:

MCP Interfaces: Model Context Protocol tools/resources over search, RAG, asset, and memory APIs, plus MCP Prompts (module 27)
Memory Service: persistent per-user/per-session facts, preferences, conversation history, and multimedia memory on MemVID, with hybrid semantic + BM25 retrieval, layered under the existing Redis short-term chat cache (module 26)
Personal DB Service: one physically isolated ArcadeDB database per user — unified vector, BM25, and graph storage replacing the old per-user Qdrant + OpenSearch + Neo4j trio (modules 24/25)
Guardrail Service: LLM input/output safety checks (Llama Guard 3 / Granite Guardian / GPT-OSS-Safeguard) with a policy engine and violation audit logging; ships disabled by default (module 23)
Benchmark Service: schema-driven scoring (keyword match, exact/regex match, cosine similarity, LLM-as-judge) with run history and pairwise comparison (module 22)

Carried over from Aigle 0.3:

Modular Deployment System: one build entry point (build.py) — start, stop, rebuild, or inspect any subset of the platform
Hybrid Search & GraphRAG: now served per-user through Module 25's ArcadeDB index (BM25 + vector + graph + temporal facts), replacing the original OpenSearch/Qdrant/Neo4j-backed implementation
A2A Agent Orchestration: JSON-RPC agent-to-agent discovery and multi-agent RAG pipelines
Video-centric Search API: results aggregated per video with the most relevant time segments
Branch-based Multi-tenancy: full branch_id isolation across upload, processing, indexing, and retrieval
Demo Web Frontend: React + Vite UI for upload, video search, and history management

Carried over from Aigle 0.2:

Core Capabilities

Multi-Modal Content Analysis: Process and understand video, audio, images, and text
Semantic Search Engine: Context-aware search using vector embeddings
AI-Powered Metadata Generation: Automated tagging and content classification
LLM Orchestration: Flexible integration with multiple language models
Vector Database Integration: High-performance similarity search and retrieval
Model fine-tuning & training: Fine-tuning workflows, training pipelines, and evaluation for domain-specific models

Intelligence Features

Content Understanding: Extract insights from unstructured media
Entity Recognition: Identify people, places, objects, and concepts
Sentiment Analysis: Understand emotional context in content
Topic Modeling: Automatic categorization and clustering
Temporal Analysis: Track content evolution over time

Enterprise Ready

Scalable Architecture: Kubernetes-native deployment
API-First Design: RESTful APIs for seamless integration
Security: Enterprise-grade authentication and authorization
Monitoring: Built-in observability and logging
Extensible: Plugin architecture for custom processors

For detailed release notes, see CHANGELOG.md.

📦 Installation

Prerequisites (full details, host sizing, and port matrix: Aigle/0.4/BUILD.md §1):

Software: Docker Engine 24.0+, Docker Compose v2.20+, Python 3.10+, NVIDIA Container Toolkit on GPU hosts
GPU: NVIDIA driver supporting CUDA 12.8+ (media stack targets sm_120 / Blackwell); 24 GB+ VRAM, 36 GB+ recommended for video processing (InternVL)
Inference servers: an Ollama server (port 11434) reachable from the cluster — it may be an existing external server; optionally a vLLM OpenAI-compatible server as an alternative LLM backend (vLLM is not bundled)
Storage: an NFSv4 export (module 01 provides a containerized NFS server, or use a native one) mounted by object-storage, AI-ML services, and media workers
Network: all hosts routable with static IPs, synchronized clocks, and firewall openings per the BUILD.md port matrix; single-host evaluation and multi-host production topologies are both supported

# Clone the repository
git clone https://github.com/DHT-AI-Studio/RAPTOR.git
cd RAPTOR/Aigle/0.4

# Configure: copy the templates and fill in hosts, credentials, and model names
cd deployment/modules
cp .env.example .env
for m in */; do [ -f "$m/.env.example" ] && cp "$m/.env.example" "$m/.env"; done
cd ../..

See Aigle/0.4/BUILD.md for the full configuration reference.

Development

cd Aigle/0.4

# Build the shared GPU base image first (used by modules 09-12)
bash deploy.sh -m 08 --build

# Start all modules in dependency order (or --cpu-only to skip GPU modules)
bash deploy.sh

# Inspect
bash deploy.sh --status              # running / stopped status per module
bash deploy.sh -m <id> --logs        # follow a module's logs

Quick Start

Deploy the platform (see Development above), then open the API Gateway docs:
```
curl -s http://<host_ip>:8012/docs
```

curl -X POST "http://<host_ip>:8012/api/0.4/sso/login" \
  -H "Content-Type: application/json" \
  -d '{"username": "<user>", "password": "<password>"}'

Upload a media file for automatic AI processing (transcription, OCR, frame description, summary, embeddings, knowledge graph):

curl -X POST "http://<host_ip>:8012/api/0.4/asset/fileupload_analysis" \
  -H "Authorization: Bearer <token>" \
  -F "file=@/path/to/video.mp4"

Search — video-centric, multi-recall (BM25 + Vector + GraphRAG + TKG) with cross-encoder re-ranking:

curl -X POST "http://<host_ip>:8012/api/0.4/search/video_search" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"query": "OpenAI announcement", "top_k": 5}'

Ask questions over your content with RAG chat:

curl -X POST "http://<host_ip>:8012/api/0.4/chat/chat" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"message": "Summarize the uploaded video"}'

Or use the demo frontend instead of raw APIs:

cd Aigle/0.4/raptor-demo-frontend
cp .env.example .env        # set API_TARGET / DEMO_PORT
docker compose up -d --build

For the complete endpoint list, request/response schemas, and Python client examples, see Aigle/0.4/API_REFERENCE.md and Aigle/0.4/raptor_client.py.

📚 Documentation

Available Documentation

🧩 Module Reference (0.4) - 27-module architecture, deprecated/in-development modules, service ports
🛠️ Setup, Build & Configuration Guide (0.4) - Prerequisites, .env configuration, single/multi-host deployment, source maintenance
📝 API Reference (0.4) - Complete endpoint documentation with examples
🔌 MCP Reference (0.4) - Model Context Protocol tool catalog and client-authentication guide
🤝 A2A Reference (0.4) - Agent-to-Agent protocol guide: orchestrator JSON-RPC, five spec-compliant specialist sub-agents, peer registration
🚀 Quick Start Guide - Get started in minutes
📖 System Design & Architecture - High-level system design
🔧 Technical Implementation Guide - Detailed implementation
📋 CHANGELOG - Version history and updates

Additional Resources

GitHub Wiki (Coming soon) - Tutorials, guides, and best practices
API Reference (Coming soon) - Complete API documentation
Video Tutorials (Coming soon) - Step-by-step video guides
Examples Repository (Coming soon) - Sample projects and use cases

🛠️ Built With

RAPTOR leverages cutting-edge technologies:

AI & Machine Learning:

🤖 Large Language Models (LLM) - Multi-provider support
🧠 LangChain - LLM orchestration framework
🔍 Qdrant - High-performance vector database
📊 MLflow - ML lifecycle management
🎯 Sentence Transformers - Text embeddings

Backend & Infrastructure:

⚡ FastAPI - Modern Python web framework
🐍 Python 3.8+ - Core programming language
🐳 Docker & Docker Compose - Containerization
☸️ Kubernetes - Container orchestration (v1.0+)
📨 Apache Kafka - Event streaming platform
💾 Redis Cluster - High-performance caching

Processing & Analysis:

🎥 FFmpeg - Video/audio processing
🔊 Whisper - Speech recognition
🖼️ OpenCV - Computer vision
📄 PyPDF2, python-docx - Document processing
🎵 Librosa - Audio analysis

Observability:

📊 Prometheus - Metrics collection
📈 Grafana - Metrics visualization
🔍 ELK Stack - Logging (roadmap)

🌐 Community & Support

We value your feedback and encourage community participation!

🐛 Reporting Issues & Feature Requests

Please use GitHub Issues to:

🐛 Report bugs
✨ Request new features
❓ Ask questions
💡 Share suggestions

Before opening an issue:

Check existing issues to avoid duplicates
Use issue templates when available
Provide detailed information and steps to reproduce

📱 Stay Connected

Join our community on multiple platforms:

Follow us for updates, announcements, and community discussions!

Coming soon: Discord server, LinkedIn group, and monthly community calls!

💬 Get Help

📖 Check our Documentation first
🔍 Browse closed issues for solutions
💭 Start a Discussion for questions
📧 Email us through dhtsolution.com

We'll post updates, respond to questions, and collaborate with users across these platforms!

🤝 Contributing

We welcome contributions from the community! Please read our CONTRIBUTING.md guide to get started.

How to Contribute

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

Please read our Code of Conduct before contributing.

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Copyright 2025 DHT Taiwan Team

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

🙏 Acknowledgments

RAPTOR is developed and maintained by the DHT Taiwan Team.

About DHT Solutions

DHT Solutions is a technology company specializing in AI and software development solutions. Learn more at https://dhtsolution.com/.

👥 Development Team

Meet the talented developers behind RAPTOR:

_titanh	_Cing-dht	_fungdht	_GeorgeDHT	_{NelsonYou1026}
_tianyu0223	_Robertdht	_QuinnChueh	_{Matthew20040407}	_{minnie-dhtsolution}
_lunar8386	_Joe-DHT	_{benjamin-dhtsolution}

🗺️ Future Development Roadmap

The following features are planned for upcoming releases to transform RAPTOR into a production-ready, enterprise-grade platform:

1. Advanced Video Understanding (v0.3 — ✅ delivered)

Implement temporal reasoning models for event sequences
Add action recognition and activity detection
Build scene relationship graphs
Create timeline-based navigation interface
Note: Docker Compose deployment won't scale to production needs

2. Content Moderation & Compliance (v0.5)

Train content moderation models (NSFW, violence, hate speech)
Implement automated flagging system
Build GDPR/CCPA compliance workflows
Create comprehensive audit reporting

3. Graph database & GraphRAG (v0.3 — ✅ delivered)

Graph-native storage and retrieval augmented generation on graph structure

4. Agents & interoperability (v0.3 — ✅ delivered)

Multi-agent workflows, agent-to-agent coordination, JSON-RPC communication, and self-learning / self-tuning agent behavior

5. Temporal model & temporal knowledge graph (v0.3 — ✅ delivered)

Time-aware models and knowledge graphs for evolving facts and sequences

6. BM25 RAG & BM25 search (v0.3 — ✅ delivered)

Hybrid and keyword-first retrieval with BM25 alongside semantic RAG

7. Contextual embedding (v0.3 — ✅ delivered)

Embeddings that preserve richer context for retrieval and downstream reasoning

8. AI LLM Interface - MCP Integration (v0.4 — ✅ delivered)

Implement Model Context Protocol (MCP) interfaces for core services:

Document Processing
Video Analysis
Audio Processing
Image Analysis
Semantic Search
Vector Database queries
Model Management
MCP Prompts

9. Memory Services (v0.4 — ✅ delivered)

Persistent memory: Stores complete per-user and per-session histories beyond Redis limits
Two-tier architecture: Uses Redis for speed and MemVID for durable storage
Semantic retrieval: Combines BM25, vector search, timelines, and context merging
Multimodal memory: Indexes video, audio, images, and documents using searchable embeddings
Service APIs: Support memory CRUD, search, export, statistics, and secure deletion
Application integration: Enables cross-session continuity, preference recall, entity tracking, and knowledge reuse

10. Personal Database Service (v0.4 — ✅ delivered)

ArcadeDB replaces Neo4j, Qdrant, and OpenSearch, providing unified multi-model storage and efficient retrieval
Physically isolated per-user databases guarantee independent data privacy, seamless backups, exports, and deletions
ArcadeDB manages lifecycles securely through gateway authentication
Kafka asynchronously routes worker outputs into user-specific ArcadeDB databases through dedicated event topics
Redis actively prevents duplicate indexing while Module 07 generates necessary embeddings for records
Unified native SQL efficiently combines vector similarity, BM25 text, and complex graph traversals
The platform natively supports comprehensive temporal knowledge graphs and advanced GraphRAG query execution

11. Real-time audio processing (v0.5)

Low-latency audio ingestion, analysis, and pipeline integration

12. Guardrail Service (v0.4 — ✅ delivered)

Guardrail (module 23) sits in front of LLM inference, ready to check inputs/outputs against configurable safety policies
Ships with pluggable guard-model classification (Llama Guard 3 / Granite Guardian / GPT-OSS-Safeguard) plus a separate regex/detector-based policy engine and LLM-judged policy checks
Administrators can define, upload, and activate custom safety policies stored securely within PostgreSQL, with violation audit logging
Redis-backed toggles let administrators enable/disable the whole system, or individual policies, without redeployment
Disabled by default: today's production traffic (Modules 07/13) only exercises the policy-free guard-model check — no policy has been activated yet

13. gRPC API Interface (v0.5)

New gRPC interface optimizing backend communication and media streaming operations
A strictly versioned protobuf contract natively handles search, asset transfers, job tracking, and analysis
StreamVideoSegment dynamically transcodes media into fragmented MP4 formats for immediate client playback capabilities
The abstract client transparently switches between traditional REST and new gRPC transports using configurations
Browser clients receive seamless WebSocket job progress updates relayed directly from backend gRPC streams
Advanced gRPC interceptors enforce authentication, propagate backpressure, and guarantee constant memory during large transfers

14. Kubernetes Production Deployment (v1.0)

Production-ready Kubernetes deployment with Helm charts
Automated horizontal and vertical scaling
Service mesh integration for resilience
Multi-environment support (dev/staging/prod)
Critical: Docker Compose is not suitable for production scale

15. Centralized Logging & Observability (v1.0)

Deploy ELK Stack (Elasticsearch, Logstash, Kibana)
Configure log shipping from all 30+ services
Centralized log aggregation and retention
Advanced log search and analytics
Distributed tracing integration

Release Timeline

Version	Target	Focus	Key Features
v0.3	June 2026 ✅	AI enhancement & retrieval	Advanced video, Graph DB & GraphRAG, Agents/JSON-RPC, temporal KG, BM25, contextual embeddings — Delivered
v0.4	Aug 2026 ✅	LLM interoperability & memory	MCP integration across core services, persistent session-based multimodal memory, per-user isolated multi-model databases with hybrid/graph/temporal search, Guardrail content moderation, Benchmark scoring service — Delivered
v0.5	Sep 2026	Interfaces, media & compliance	gRPC API interface, content moderation, Guardrail services integration, GDPR/CCPA, real-time audio processing
v1.0	Q4 2026	Production ready	Kubernetes, ELK Stack, 99.9% SLA

Current Status: Aigle 0.4 (Community Beta) - August 2026 ✅
Next Milestone: v0.5 (Sep 2026)
Production Target: v1.0 in Q4 2026

Made with ❤️ by the DHT Taiwan Team

For business inquiries: https://dhtsolution.com/