DigCrawl

Multi-platform content retrieval for AI agents.

Retrieve pages, text, images, and video from public and user-authorized signed-in sources.

中文 · Quick start · Sources · Collect & watch · Docs · Releases

DigCrawl Skill: Run It in Your Agent

Paste into your agent chat (Codex, Grok Build, Claude Code, or similar):

Install the DigCrawl Skill for me: https://github.com/originaleric/digcrawl

Follow the instructions to restart your agent or open a new chat, then send a link:

Read and summarize this link: <URL>

Quick Start

Install with npx (macOS / Linux / Windows; Node.js 20.19+):

npx digcrawl-skill install

Installs the program, complete Skill, and default profile. Configures Codex and installs the Claude Code Skill when detected.

Or use the install script (macOS / Linux; no Node.js or Go required):

curl -fsSL https://raw.githubusercontent.com/originaleric/digcrawl/main/scripts/install.sh | sh

Both methods verify downloads. Restart your agent and open a new chat after installation.

Choose an agent, install locations, and alternatives

npx digcrawl-skill install codex
npx digcrawl-skill install claude-code
npx digcrawl-skill status

• Configures Codex automatically and installs the Claude Code Skill when detected. See MCP setup for other hosts.
• Installs to ~/.local/bin; add it to PATH if the command is not found.
• Windows users can also download a Release. See installation for version pinning and source builds.

What You Can Do

Use case	Capability
Content retrieval	Read pages, articles, posts, and source images
Video understanding	Retrieve available captions; optionally transcribe speech and recognize on-screen text
Ongoing collection	Process URL batches, monitor updates, and notify
Agent reading	Supply source content for summaries, comparisons, and research

Supported Sources

• Posts and communities: WeChat Official Accounts, Xiaohongshu, Weibo, Zhihu, X, Reddit, Threads, Instagram.
• Video and content platforms: Douyin, Toutiao, Bilibili, Kuaishou, YouTube.
• Web and feeds: Web pages, RSS, Atom, sitemaps, platform lists, and profiles.

See source details for platform capabilities and login requirements.

Collect, Export, and Watch

Save retrieved content as Markdown, HTML, JSON, and optional media files, or monitor sources continuously with Watchlist.

Common CLI commands: pages, batches, video, and watchlists

# Single pages and batches
digcrawl collect "https://mp.weixin.qq.com/s/..." --format md,html,json
digcrawl collect --file urls.txt --out ./collect-output

# Analyze local video using the configured processors
digcrawl video ./meeting.mp4

# Run once or monitor continuously
digcrawl watchlist run --once --config config/config.yaml
digcrawl watchlist serve --config config/config.yaml

Transcription and OCR require the relevant dependencies; see video analysis and Collect.

Use with AI Agents

Original pages beside actual retrieval results (2026-10-17). WeChat uses a source excerpt; video examples did not run speech transcription.

Article · WeChat

Read this agent article and extract its text and sections. https://mp.weixin.qq.com/s/mbU0bE6BV5YfAsTmoY4H3A

Source · Retrieved the article text, sections, and 7 image references.

View returned field excerpts

{
  "status": "success",
  "title": "别再只卷Prompt 了，真正拉开Agent差距的是Context Engineering",
  "author": "腾讯技术工程",
  "media_references": {
    "image": 7
  },
  "excerpt": "Prompt Engineering → ReAct → Context Engineering\nContext 的七类构成要素\n长任务的三大核心技术"
}

Verification data

Image Note · Xiaohongshu

Read this agent image note and extract its text and images. <paste the full Xiaohongshu share link>

Source · Original screenshot · Used an authorized local session to retrieve the note text and 2 image references. Use the full share link.

View returned field excerpts

{
  "status": "success",
  "title": "2026 Agent新物种地图｜20款近期高热 Agent",
  "author": "略懂AI的Asher",
  "media_references": {
    "image": 2
  },
  "excerpt": "能力范式转向；组织形态重组；负责周期跨越",
  "image_observation": [
    "Personal",
    "Coding",
    "Workflow",
    "Research",
    "Creative"
  ]
}

Verification data

Video

Bilibili · Agent Fundamentals

Read this agent tutorial and extract its title, description, and parts. https://www.bilibili.com/video/BV1z3ZFYUE6K/

Source · Original screenshot · Retrieved video metadata and media references; transcription was not requested.

View returned field excerpts

{
  "status": "success",
  "title": "李宏毅：一堂课搞懂AI Agent原理【生成式AI的机器学习(2025)】大模型LLM",
  "author": "AI算法工程师笔记-",
  "media_references": {
    "image": 1,
    "video": 2
  },
  "excerpt": "分P1: 第二讲：一堂课搞懂AI Agent原理\n分P2: 大模型路线图-\n分P4: ML2025Agent"
}

Verification data

X · Operator Demo

Read this Operator video post and extract its text and video information. https://x.com/OpenAI/status/1882509286439637448

Source · Original screenshot · Retrieved the post text, video reference, and cover reference; no transcription was requested.

View returned field excerpts

{
  "status": "success",
  "title": "A research preview of Operator, an agent that can use its own browser to perform...",
  "author": "OpenAI",
  "media_references": {
    "video": 1,
    "image": 1
  },
  "excerpt": "A research preview of Operator, an agent that can use its own browser to perform tasks for you."
}

Verification data

YouTube · Agent Building Advice

Read this agent video and extract its description and chapters. https://www.youtube.com/watch?v=LP5OCa20Zpg

Source · Original video screenshot · Retrieved the description, chapters, and cover after sign-in; caption text was not retrieved.

View fields before and after sign-in

{
  "before_sign_in": {
    "status": "error",
    "error": {
      "code": "LOGIN_REQUIRED",
      "message": "Sign in to confirm you’re not a bot"
    }
  },
  "after_sign_in": {
    "status": "success",
    "title": "Tips for building AI agents",
    "author": "Anthropic",
    "media_references": {
      "image": 1
    },
    "captions_available": true,
    "warnings": [
      "YouTube captions were advertised but could not be read"
    ],
    "excerpt": "00:26 Defining AI agents and workflows\n10:47 Coding agents: Potential and challenges\n16:26 Advice for developers exploring agents"
  }
}

After sign-in evidence · Before sign-in evidence

YouTube Sign-In

On LOGIN_REQUIRED, ask your agent to open a dedicated, ordinary Chrome window:

Open the DigCrawl sign-in window for YouTube.

Sign in to Google yourself, return to YouTube, then tell the agent “I’ve finished signing in.” The agent saves the local session and reads the original video again.

YouTube manual sign-in and session details

digcrawl browser login --config "<profile path used by MCP>" --platform youtube

Sign in through the dedicated Chrome window and return to YouTube, then press Enter in the original terminal. DigCrawl closes the window and synchronizes the session. On Windows, close all dedicated Chrome windows before pressing Enter.

Use the same profile as MCP; your everyday browser session is not copied automatically. Credentials and cookies stay on your machine. Preserve any Google sign-in rejection; see sign-in guidance.

When Sign-In Is Required · Xiaohongshu Example

On LOGIN_REQUIRED, ask your agent to open DigCrawl’s dedicated sign-in window:

Open the DigCrawl sign-in window for Xiaohongshu.

Sign in yourself, then tell the agent “I’ve finished signing in.” The agent saves the local session and reads the original link again.

Manual sign-in and session details

digcrawl browser login --config "<profile path used by MCP>" --platform xiaohongshu

When running the command yourself, press Enter in that terminal after signing in. Use the same profile as MCP; your everyday browser session is not copied automatically. Credentials and cookies stay on your machine.

See sign-in guidance for other platforms and troubleshooting.

Manual Skill and MCP configuration

Install the complete Skill directory, then configure the local server for your host:

• Codex example
• Claude Code example
• Other MCP hosts

Use absolute program and profile paths, and copy the whole Skill rather than only SKILL.md. See setup for profile initialization.

Local-First Security

• Runs on your machine; HTTP Gateway listens on localhost by default.
• The default Skill needs no API key and does not expose cookies to the agent.
• Signed-in sources use user-authorized local sessions; users handle login and captchas.
• Access failures and rate limits are reported explicitly; error pages are not treated as content.

Browser recovery and diagnostics

Follow the returned error guidance or see browser troubleshooting.

Check the local environment

digcrawl doctor --config "$HOME/.config/digcrawl/digcrawl-skill.yaml"

See the MCP documentation for upgrade reconnection and Bridge lifecycle details.

Architecture and Interfaces

Interface	Use
CLI / Watchlist	Local retrieval, batches, and scheduled monitoring
Skill / MCP	Content reading tools for AI agents
HTTP / stdio / Go SDK	Integration with applications and data workflows

See Runtime, Transport, and Protocol for internal modules and tool contracts.

Documentation

For users

• Installation and usage · Product guide · Browser troubleshooting
• Agent Skill · MCP setup

For developers

• HTTP / stdio · Go SDK · Configuration examples
• Collect · Runtime · Protocol

Contributing

Issues and PRs are welcome. Include your use case, reproduction steps, or tests. Never commit credentials, cookies, browser data, or private content.

Open Source Notes

Respect source terms, permissions, and applicable law. Do not bypass login, captchas, or access restrictions.

License

MIT License