1.0.0 • Published yesterdayCLI

shantycrawl-mcp

Licence

MIT

Version

1.0.0

Deps

Size

69 kB

Vulns

Weekly

1.0K

Summary Dependency Versions

ShantyCrawl MCP

Firecrawl with a chill token footprint.

Drop-in replacement for firecrawl-mcp that loads 6 tools instead of 28 on session start — saving ~18,000 tokens before you've even sent your first message.

_{Why · Quick Start · Configuration · Tool Architecture · How It Works · Local Dev · Contributing · License}

Why ShantyCrawl MCP

Most full-featured MCP servers register every tool on every session — including the ones you'll rarely touch — with full JSON schemas re-sent on each tool-list fetch. ShantyCrawl loads 6 core tools up front and lazy-loads the rest on demand, which measured out to ~18,000 fewer tokens on a fresh session in our testing:

	`firecrawl-mcp` (official)	`shantycrawl-mcp`
Tools on session start	28	6
Initial schema footprint	Full schemas for all 28 tools	Minimal schemas for 6 tools
Advanced tools (research, monitoring, agent, etc.)	Always loaded	Loaded on demand via `tool_enable`
Measured token cost, fresh session	Baseline	~18k tokens lighter
Runtime dependencies	—	Zero third-party wrappers, native `fetch`

Is This For You?

Good fit — you use Firecrawl mostly for scrape/crawl/search, run long or multi-tool agent sessions, or are tight on context budget.
Less critical — you're already using tool_enable-style dynamic loading elsewhere, or you live almost entirely in the monitoring/research toolset and rarely call the core 6 — in that case the savings are smaller, since you'll be loading most of the schema anyway.

Quick Start

Run instantly via npx — no installation required:

FIRECRAWL_API_URL=http://localhost:3002 npx shantycrawl-mcp

Environment variables:

Variable	Required	Default	Description
`FIRECRAWL_API_URL`	No	`http://localhost:3002`	URL of your Firecrawl instance (self-hosted or cloud)
`FIRECRAWL_API_KEY`	Only for Firecrawl Cloud	—	API key, required if `FIRECRAWL_API_URL` points to a hosted/cloud instance

Requirements: Node.js 18+ and a reachable Firecrawl instance (self-hosted or cloud).

Configuration

Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "shantycrawl": {
      "command": "npx",
      "args": ["shantycrawl-mcp"],
      "env": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

OpenCode

Add this to your workspace settings:

{
  "mcp": {
    "shantycrawl": {
      "type": "local",
      "command": ["npx", "shantycrawl-mcp"],
      "environment": {
        "FIRECRAWL_API_URL": "http://localhost:3002"
      }
    }
  }
}

Tool Architecture

Always Available (6 Core Tools)

Tool	Description
`scrape`	Extract clean markdown from any URL
`crawl`	Crawl a target website recursively
`search`	Execute web searches via Firecrawl
`tool_enable`	Dynamically inject an advanced tool into the active session
`tool_disable`	Unload an advanced tool to free up context
`check_crawl_status`	Check the status of an async `crawl` job

Lazy-Loaded Tools (22 Advanced)

Activate any tool instantly during a session — e.g. tool_enable map unblocks the mapping capability.

Category	Tools
Discovery	`map`, `extract`, `parse`
AI Agent	`agent`, `agent_status`
Browser Interaction	`interact`, `interact_stop`
Research & Academic	`research_search_papers`, `research_inspect_paper`, `research_read_paper`, `research_related_papers`, `research_search_github`
Monitoring	`monitor_create`, `monitor_check`, `monitor_checks`, `monitor_delete`, `monitor_get`, `monitor_list`, `monitor_run`, `monitor_update`
Feedback	`search_feedback`, `feedback`

How Dynamic Loading Works

The client sees a minimal 6-tool schema on connect.
When the AI needs an advanced capability, it calls tool_enable <tool_name>.
The server fires a notifications/tools/list_changed event, and the client's available tools update instantly — no reconnect required.
Call tool_disable <tool_name> once you're done to free the context back up.

Known Issues

Some agents don't call tool_enable on their own. A compelling tool description isn't always enough — if another skill or system prompt has already taught the agent a different path to the same outcome (e.g. calling the Firecrawl API directly via curl/raw HTTP), it can follow that instead of discovering tool_enable. This shows up most with coding agents that have built-in or third-party Firecrawl skills already loaded.

Fix: add an explicit instruction to your agent's system prompt or AGENTS.md/CLAUDE.md:

This project uses the shantycrawl-mcp server. Advanced tools (map, extract, parse,
agent, interact, research_*, monitor_*) are not available until activated with
tool_enable("<tool_name>"). Always use the MCP server's tools directly — never call
the Firecrawl API via curl, raw HTTP, or other bash-based workarounds.

We also ship a ready-made SKILL.md that encodes this protocol — drop it into your skills directory if your agent supports skill loading, and pair it with the snippet above for the strongest guarantee.

Local Development

git clone https://github.com/schlemperdev/shantycrawl-mcp.git
cd shantycrawl-mcp
npm install
npm run build
npm start