@hasna/crawl

@hasna/crawl

AI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.

Install

npm install -g @hasna/crawl

CLI Usage

crawl --help

• crawl crawl <url>
• crawl list
• crawl stats
• crawl search <query>
• crawl sitemap <url>
• crawl map <url>
• crawl export

MCP Server

crawl-mcp

30 tools available.

HTTP mode

Long-lived Streamable HTTP transport for shared agent sessions (binds 127.0.0.1 only):

crawl-mcp --http              # default port 8812
crawl-mcp --http --port 8812
MCP_HTTP=1 MCP_HTTP_PORT=8812 crawl-mcp

• GET /health → {"status":"ok","name":"crawl"}
• POST /mcp — Streamable HTTP MCP endpoint (also mounted on crawl-serve)

Stdio remains the default transport for gradual rollout.

REST API

crawl-serve

Cloud Sync

This package supports cloud sync via @hasna/cloud:

cloud setup
cloud sync push --service crawl
cloud sync pull --service crawl

Data Directory

Data is stored in ~/.hasna/crawl/.

License

Apache-2.0 -- see LICENSE