bulkgrid

mcp
Guvenlik Denetimi
Uyari
Health Uyari
  • No license — Repository has no license file
  • Description — Repository has a description
  • Active repo — Last push 0 days ago
  • Community trust — 66 GitHub stars
Code Uyari
  • process.env — Environment variable access in examples/map.mjs
Permissions Gecti
  • Permissions — No dangerous permissions requested

Bu listing icin henuz AI raporu yok.

SUMMARY

Bulkgrid - Better AI Answers From Sources You Trust

README.md

Bulkgrid logo

Bulkgrid

Search the web. Map websites. Crawl content. Build your index.

Web data and trusted knowledge for your applications and AI agents.

Website · Documentation · Dashboard · TypeScript · Python

One platform, from discovery to retrieval

Bulkgrid turns web pages and repository content into useful data. Discover information with live search, map a website's URLs, collect page content, extract structured fields, and maintain a searchable index for your apps and agents.

Capability What you can do TypeScript SDK
Search Find web pages, images, or news. Optionally crawl their source pages in the same run. client.search()
Map Discover URLs from a website's sitemaps, with keyword filtering and a result limit. client.map()
Crawl Fetch specific URLs as Markdown, HTML, text, or screenshots. client.crawl()
Deep Crawl Follow links within page, depth, and path boundaries. client.deepCrawl()
Extract Turn page content into structured data using a query and JSON schema. client.extract()
Index Search indexed website and GitHub content, curate collections, and refresh sources. client.index.search()

Use these capabilities independently or combine them: Search or Map → Crawl or Extract, or Sources → Index → Collections → Retrieval. Map returns sitemap URLs; it does not fetch page content. Live search returns ranked results immediately. Map, Crawl, Deep Crawl, and Extract return runs that you can inspect for progress and results.

Get started

Create a workspace in the dashboard, then create an API key in Settings → API Keys. Keep the key in your backend environment.

TypeScript / JavaScript

Requires Node.js 22 or newer.

npm install @bulkgrid/sdk
export BULKGRID_API_KEY='bg_live_your_api_key'

Search the live web:

import { BulkgridClient } from "@bulkgrid/sdk";

const client = new BulkgridClient({
  apiKey: process.env.BULKGRID_API_KEY ?? "",
});

const search = await client.search({
  query: "open source web crawlers",
  limit: 5,
});

console.log(search.results);
console.log(search.run.id); // Reopen saved results with client.runs.get(id)

Set type: 'images' or type: 'news' for other search modes. Add crawl: true to collect the results' source pages in the same run.

Map a website or crawl a known page:

const map = await client.map({
  url: "https://example.com",
  config: { search: "docs", limit: 100 },
});

const crawl = await client.crawl({
  urls: ["https://example.com"],
  options: { formats: ["markdown"] },
});

console.log(map.id, crawl.id);

Use client.runs.get(id) to check progress. A completed map exposes its URLs in result_data.links; crawled pages are available through client.runs.results(id) and client.runs.getResultContent(...). See Runs and results for polling, pagination, retry, and cancellation.

Search your index:

const evidence = await client.index.search({
  query: "How does authentication work?",
  limit: 5,
});

for (const result of evidence.results) {
  console.log(result.text_content, result.url);
}

Index retrieval requires indexed content in collections accessible to your key. Manage that content with client.index.sources and client.index.collections. Public website and GitHub sources can be reused across workspaces; private sources stay within their workspace. See Sources and Collections.

Search requires search:query. Map and Crawl require runs:write; inspecting runs requires runs:read, and fetching page results requires results:read. See API-key scopes for extraction and source-management permissions, and the SDK guide for full examples.

Python

The Python SDK includes synchronous and asynchronous clients for Python 3.10+.

pip install bulkgrid
import os
from bulkgrid import BulkgridClient

with BulkgridClient(api_key=os.environ['BULKGRID_API_KEY']) as client:
    search = client.search({'query': 'open source web crawlers', 'limit': 5})
    print(search['results'])

    evidence = client.index.search({'query': 'authentication setup', 'limit': 5})
    print(evidence['results'])

Python uses methods such as deep_crawl() and get_result_content() while preserving API field names in dictionaries. See the Python SDK README for async usage, configuration, and local installation. Repository changes may appear before their corresponding npm or PyPI release.

Upgrading from an earlier SDK

The 0.4 API separates live discovery from indexed retrieval:

Earlier interface Current interface
client.search() for indexed content client.index.search()
client.sources / client.collections client.index.sources / client.index.collections
/api/v1/sources/* / /api/v1/collections/* /api/v1/index/sources/* / /api/v1/index/collections/*
/api/v1/search for indexed content /api/v1/index/search

client.search() and /api/v1/search now perform live web search. Earlier source and collection routes have no compatibility aliases. See the SDK changelog before upgrading.

Connect your AI agent

Add the remote MCP server in your client's settings:

https://bulkgrid.com/api/v1/mcp

Complete browser sign-in, choose your workspace, and select the collections the connection can access. MCP retrieval searches indexed content and returns passages with source URLs.

Claude and ChatGPT use the same MCP service. The updated integration offers these capabilities when the supporting service version is deployed:

App capability Required consent
Search indexed content, list sources and collections, read documents Default retrieval access
Crawl, Deep Crawl, Map, Extract, and read run results Optional Crawl, map and extract
Add or refresh index sources and manage collection membership Optional Manage the index

Both optional choices are off by default. Crawling and extraction may consume credits, and sources may refresh automatically. Index management also grants run permissions used by Crawl and Map. Monitoring remains available through scoped API keys, outside OAuth app access.

For clients using mcpServers configuration:

{
  "mcpServers": {
    "bulkgrid": {
      "url": "https://bulkgrid.com/api/v1/mcp"
    }
  }
}

Try: “Use Bulkgrid to find how authentication works in our documentation. Search the relevant collection and cite the source URLs.”

For an existing connection, revoke its access in Settings → Connected Apps, then reconnect and approve the desired optional capabilities. Updating the plugin or refreshing a token does not add permissions. See MCP setup for instructions and agent plugins for deployment and client acceptance status.

CLI and agent skills

The skills target CLI 0.6.0 or newer for web and index search, crawling, extraction, document parsing, and source management. Browser login is the default; an API key is optional and takes precedence when BULKGRID_API_KEY is set.

npx @bulkgrid/cli --version
npx @bulkgrid/cli login
npx @bulkgrid/cli search 'solar panel recycling' --limit 5
npx @bulkgrid/cli crawl https://example.com --options '{"formats":["markdown"]}'
npx @bulkgrid/cli parse ./report.pdf
npx @bulkgrid/cli index search 'authentication setup' --limit 5
npx @bulkgrid/cli sources find 'example.com'

Approve Crawl, map, extract and parse for web operations and document parsing, and Manage the index for source management. Use status to inspect the active login. Existing grants need logout/login to approve added scopes on the same deployment.

The prepared setup skills installer copies all six bundled skills into the project's .agents/skills without authentication or network access. Use --global or --directory for another destination. Existing edits are preserved unless you pass --force; --dry-run previews changes. This checkout's CLI implementation is still 0.5.0 and must be synced to 0.6.0 before releasing these updated instructions. See the CLI README for the checkout's implementation and installer details.

Skill Purpose
bulkgrid-index Search indexed sources and collections, read document passages, and cite evidence.
bulkgrid-search Search the web for pages, images, and news.
bulkgrid-crawl Crawl websites, map URLs, collect bounded content, and retrieve outputs.
bulkgrid-extract Parse web pages into structured fields, design schemas, and validate extraction results.
bulkgrid-parse Convert local documents and images into Markdown and retrieve parse results.
bulkgrid-sources Inspect sources, manage ingestion, and organize collection access.

All six skills are bundled in the CLI package and both plugin bundles, generated from the same sources. They prefer CLI commands when a terminal is available and retain MCP/SDK fallback guidance. Skills do not grant permissions. bulkgrid search searches the live web; bulkgrid index search and MCP search retrieve indexed passages.

For parsing, use bulkgrid-parse with CLI 0.6.0 for local documents and images, Crawl for readable web pages, and Extract for schema-based JSON from URLs. See the skill coverage review for workflows and capability boundaries.

In this repository

Component Location
TypeScript SDK packages/sdk
Python SDK packages/sdk-python
CLI and MCP client setup packages/cli
Agent plugins AGENT_PLUGINS.md
Shared agent skills skills
Documentation docs
OpenAPI specification openapi/bulkgrid.openapi.json
Runnable examples examples

These tools connect to the hosted Bulkgrid service. The API backend, MCP service, crawling and indexing engines, database, and billing are maintained in the private platform repository. This repository contains the public developer tools, not a self-hostable backend.

Development and contributions

npm ci
npm run check
node packages/cli/dist/cli.js --help

Use Node.js 22+ and Python 3.10+ to validate all packages. See CONTRIBUTING.md for checks and MAINTAINERS.md for the platform export workflow. API contracts and SDK types follow the platform schemas; plugins, shared skills, and public tooling are maintained here.

License

Agent plugin resources and shared skills use Apache-2.0. Bulkgrid branding retains its trademark rights. SDK and CLI code remains UNLICENSED; the Python SDK declares a proprietary license. The documentation template's existing license is preserved in docs/LICENSE.

Documentation · Dashboard · X · Support

Yorumlar (0)

Sonuc bulunamadi