🌟 Key Highlights

• ⚡ Native C++ Vector Search: Powered by sqlite-vec and better-sqlite3, querying 100,000+ vector chunks in under 60ms with zero memory leaks.
• 🔍 Hybrid Retrieval (RRF): Fuses Full-Text Search (FTS5 BM25) with Approximate Nearest Neighbors (ANN vector search) using dynamic Reciprocal Rank Fusion.
• 🎯 Batched Neural Reranker: Integrates local cross-encoder models (ms-marco-MiniLM-L-6-v2) to re-score candidate chunks for maximum precision.
• 🤖 Native MCP Server: Built on @modelcontextprotocol/sdk to serve live retrieval tools directly to AI coding agents (Claude Desktop, Cursor, Antigravity, OpenAI Codex).
• 📥 Universal Ingestion: Ingests YouTube transcripts, dynamic web pages, PDF documents, DOCX files, Markdown trees, and RSS feeds.
• 🔒 100% Offline & Private: Zero cloud API keys required. All embeddings (all-MiniLM-L6-v2), parsing, and storage remain strictly on your local machine.

🏗️ Architecture

flowchart TD
    subgraph Ingestion ["1. Universal Ingestion"]
        YT["YouTube Captions / yt-dlp"]
        DOC["PDFs / DOCX / Markdown"]
        WEB["Web URLs / RSS Feeds"]
    end

    subgraph Processing ["2. Chunking & Embeddings"]
        CH["Sliding Window Chunker"]
        EMB["HuggingFace Transformers (all-MiniLM-L6-v2)"]
    end

    subgraph Storage ["3. Embedded Storage (SQLite)"]
        FTS[("FTS5 Full-Text Index")]
        VEC[("sqlite-vec C++ Native Index")]
    end

    subgraph Retrieval ["4. Hybrid Retrieval Engine"]
        BM25["BM25 Keyword Scoring"]
        ANN["Cosine Similarity ANN"]
        RRF["Reciprocal Rank Fusion (RRF)"]
        RERANK["Cross-Encoder Reranker (ms-marco-MiniLM)"]
    end

    subgraph Consumers ["5. AI Agents & Consumers"]
        MCP["Model Context Protocol (MCP)"]
        CLI["CLI Interface"]
        SYN["Local AI Synthesis (Ollama / Anki / Notes)"]
    end

    Ingestion --> CH --> EMB
    EMB --> VEC
    CH --> FTS
    FTS --> BM25 --> RRF
    VEC --> ANN --> RRF
    RRF --> RERANK
    RERANK --> MCP
    RERANK --> CLI
    RERANK --> SYN

📥 Ingestion Support Matrix

📦 Installation

Install globally via npm:

npm install -g vidilearn

Or run directly without global installation using npx:

npx vidilearn --help

🚀 Quick Start

1. Ingest Knowledge

Ingest a technical YouTube lecture, article, or local documentation:

# Ingest YouTube video transcripts
vidilearn ingest https://www.youtube.com/watch?v=sal78ACtGTc

# Ingest an article or documentation page
vidilearn ingest https://developer.mozilla.org/en-US/docs/Web/API

# Ingest local Markdown or PDF folder
vidilearn ingest ./docs/architecture.pdf

2. Query with Hybrid Search

Search across all ingested materials with semantic hybrid retrieval:

vidilearn search "agentic workflows and tool-calling patterns" --hybrid

3. Generate Study Notes & Flashcards

Synthesize Cornell notes, flashcards (Anki TSV), and Obsidian markdown:

vidilearn study https://www.youtube.com/watch?v=sal78ACtGTc

4. Extract Hook Timestamps & Clips

Detect high-density segments and pivotal topic transitions:

vidilearn clips https://www.youtube.com/watch?v=sal78ACtGTc

🤖 Model Context Protocol (MCP) Integration

Vidilearn includes a native Model Context Protocol (MCP) server over stdio, allowing coding assistants to query your local knowledge base during generation.

Claude Desktop Configuration

Add Vidilearn to your claude_desktop_config.json:

{
  "mcpServers": {
    "vidilearn": {
      "command": "npx",
      "args": ["-y", "vidilearn", "mcp-server"]
    }
  }
}

Cursor / Antigravity / AI Agent Configuration

Add to your agent settings or MCP manifest:

{
  "name": "vidilearn",
  "command": "vidilearn",
  "args": ["mcp-server"]
}

Exposed MCP Tools:

• search_knowledge: Execute high-precision semantic hybrid search against your indexed chunks.
• ingest_resource: Dynamically ingest URLs or local files into the active agent context.
• get_document_metadata: Inspect document titles, hashes, and source provenance.

📊 Performance Benchmarks

Benchmarked against 100,000 synthetic chunks (~205 MB SQLite Database):

📋 Command Reference

💻 Programmatic API

Vidilearn can be imported directly into Node.js / TypeScript applications:

import { IngestionEngine, HybridSearcher } from 'vidilearn';

// 1. Ingest content programmatically
const engine = new IngestionEngine();
await engine.ingest('https://www.youtube.com/watch?v=sal78ACtGTc');

// 2. Perform hybrid retrieval
const searcher = new HybridSearcher();
const results = await searcher.query('How does reciprocal rank fusion work?', {
  hybrid: true,
  limit: 5
});

console.log(results);

🛡️ Security & Privacy

• Zero Remote Telemetry: Vidilearn sends zero tracking, analytics, or behavioral telemetry.
• Local Storage: All vector embeddings and indexes remain in a local SQLite file on your disk.
• No API Keys Needed: Does not require OpenAI, Anthropic, or external embedding API keys to operate.
• Memory-Safe Concurrency: Features built-in LRU caching and single-flight request coalescing to prevent thundering herd locks on Node.js worker threads.

🤝 Contributing

Contributions are warmly welcomed! Please read our CONTRIBUTING.md guide and code of conduct before opening pull requests.

# Clone the repository
git clone https://github.com/Alfo-Tech-Lab/vidilearn.git

# Install dependencies
npm install

# Run tests
npm test

📄 License

Distributed under the MIT License. See LICENSE for more information.