scrapfly-cli

The agentic CLI for web data and browser automation.
Scrape, extract, and drive a cloud browser over CDP from any LLM tool-use loop. JSON-first, pipe-friendly, one static binary.

scrapfly is the official CLI for the Scrapfly
platform. It is built agent-first: every verb returns a stable JSON
envelope, every product is a tool an LLM can call, and an autonomous
agent is baked in.

• Agentic by design: scrapfly agent "<task>" runs a Playwright-MCP
  style loop against a cloud browser. Providers ship out of the box for
  Anthropic, OpenAI, Gemini, and any OpenAI-compatible endpoint (Ollama,
  vLLM). For any other tool-use loop, install the scrapfly-cli agent
  skill — see
  For LLM agents below.
• Full product surface: Web Scraping, Screenshot, Extraction, Crawler,
  and the CDP-driven Browser, each with complete SDK-parity flags.
• Pipe-friendly: stable envelope ({success, product, data|error}),
  --content-only/--data-only for single-line piping, ndjson for
  batch runs, --pretty for humans.
• Composable: multi-call browser sessions via a Unix-socket daemon
  (cookies + AXTree refs persist across invocations), WARC/HAR in and
  out, scrapfly browser outputs a CDP URL you can hand to Playwright,
  Puppeteer, or browser-use.

Install

One-liner (macOS / Linux):

curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh

Pinned version or custom prefix:

curl -fsSL https://scrapfly.io/scrapfly-cli/install | sh -s -- --version v0.1.0 --prefix ~/.local/bin

Or grab a release tarball directly from
Releases - artifacts
follow scrapfly-{os}-{arch}.tar.gz (scrapfly-windows-amd64.zip on
Windows). Supported: darwin-universal, darwin-amd64, darwin-arm64,
linux-amd64, linux-arm64, windows-amd64.

From source (Go 1.24+):

go install github.com/scrapfly/scrapfly-cli/cmd/scrapfly@latest

Or from a package manager:

# npm / pnpm / yarn
npm install -D scrapfly-cli
npx scrapfly version

macOS Gatekeeper

The released macOS binary is not yet signed with an Apple Developer ID, so
first-run may hit "Apple could not verify 'scrapfly' is free of malware".

• install.sh strips the quarantine bit automatically.
• If you downloaded the tarball manually, run once:

  xattr -d com.apple.quarantine /path/to/scrapfly
  

  (or right-click → Open in Finder the first time).

Signing + notarization is on the roadmap.

Self-update

Once installed, keep the binary current without re-running the installer:

scrapfly update           # upgrade in place to the latest GitHub release
scrapfly update --check   # just print whether a newer version exists
scrapfly update --version v0.3.0   # pin to a specific release

The binary always comes from the GitHub release + checksums.txt; the
release-notes feed is only used for the human-readable changelog.

Authentication

export SCRAPFLY_API_KEY=scp-live-...
# or persist it
scrapfly config set-key scp-live-...

Resolution order: --api-key flag > SCRAPFLY_API_KEY env >
~/.scrapfly/config.json.

Quick start

# Scrape a JS-heavy page with anti-bot + markdown output
scrapfly scrape https://web-scraping.dev/products --render-js --asp --format markdown

# Pipe scrape into extract (two-step: fetch + AI extraction)
scrapfly scrape https://web-scraping.dev/product/1 --render-js --proxified \
  | scrapfly extract --content-type text/html --prompt "name, price, sku"

# Screenshot auto-named into a directory
scrapfly -O ./shots screenshot https://example.com --resolution 1920x1080

# Crawl synchronously, then extract the WARC you downloaded
scrapfly crawl run https://example.com --max-pages 20 --content-format markdown
scrapfly crawl artifact <uuid> --type warc -o crawl.warc
scrapfly crawl parse warc-list crawl.warc --pretty

Browser & Agent

scrapfly browser gives you a CDP WebSocket URL you can hand to
Playwright, Puppeteer, browser-use,
or drive directly through the built-in session daemon:

scrapfly browser --session demo start &
scrapfly browser navigate https://web-scraping.dev/login
scrapfly browser fill 'input[name=username]' user123
scrapfly browser fill 'input[name=password]' password
scrapfly browser click 'button[type=submit]'
scrapfly browser close

On Scrapfly's custom browser, fill / click / wait / slide
automatically go through the Antibot CDP domain (human-like timing,
slider-captcha support) and content uses Page.getRenderedContent
(HTML with iframes inlined).

For fully autonomous runs:

ANTHROPIC_API_KEY=... scrapfly agent "Find the first product name and price" \
  --url https://web-scraping.dev/products \
  --schema '{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"string"}},"required":["name","price"]}'

Providers: Anthropic (default), OpenAI, Google Gemini, and any
OpenAI-compatible endpoint (Ollama, vLLM, …).

Product coverage

Every documented SDK field is exposed as a flag. See the
scrapfly-cli agent skill
for the complete map.

For LLM agents

The agent skill that teaches Claude Code, Cursor, Codex, Copilot, Cline,
Windsurf, and 50+ other agents how to drive this CLI
lives in the central scrapfly/skills
repo. Install it via the open skills
CLI:

# Just the CLI skill
npx skills add scrapfly/skills --skill scrapfly-cli

# Or every Scrapfly skill (SDK + CLI + MCP agent rules)
npx skills add scrapfly/skills --all

That ships a compact guide covering the six usage paths, auth, the JSON
envelope contract, and every CLI verb grouped by intent. Or point your own
tool-use loop at the raw markdown:
scrapfly/skills/scrapfly-cli/SKILL.md.

GitHub Actions

Use this repo as a reusable action to install the CLI on a runner:

- uses: scrapfly/[email protected]
  with:
    api-key: ${{ secrets.SCRAPFLY_API_KEY }}
- run: scrapfly scrape https://example.com --format markdown

Inputs: version (default latest), api-key (exported to SCRAPFLY_API_KEY
with log masking), prefix (default $HOME/.local/bin).

MCP server

For Cursor, Claude Desktop, Claude Code, and any other MCP-compatible
client:

{
  "mcpServers": {
    "scrapfly": {
      "command": "scrapfly",
      "args": ["mcp", "serve"],
      "env": { "SCRAPFLY_API_KEY": "scp-live-..." }
    }
  }
}

Exposes scrape, screenshot, extract, crawl_run, and selector as
tools. Schemas are generated from the flag definitions so the client sees
exactly the surface the CLI does.

Shell completion

scrapfly completion zsh  > "${fpath[1]}/_scrapfly"       # zsh
scrapfly completion bash > ~/.bash_completion.d/scrapfly # bash
scrapfly completion fish > ~/.config/fish/completions/scrapfly.fish

Development

make install   # go mod download
make dev       # build dist/scrapfly
make test      # go test ./...
make lint      # go vet ./...
make fmt       # gofmt -w .

# cut a release (tags + pushes; CI publishes binaries)
make release VERSION=0.2.0 NEXT_VERSION=0.2.1

Links

• Scrapfly docs: https://scrapfly.io/docs
• Official SDKs (Python, TypeScript, Go, Rust): https://scrapfly.io/docs/sdk
• Issues: https://github.com/scrapfly/scrapfly-cli/issues

License

MIT. See LICENSE.