Keyword: crawler
- firecrawl-cliReleased2d agoVersion1.19.21
Command-line interface for Firecrawl. Scrape, crawl, and extract data from any website directly from your terminal.
- fiftyone.devicedetection.onpremiseReleased3d agoVersion4.5.59
Device detection on-premise services for the 51Degrees Pipeline API
- smart-web-mcpReleasedyesterdayVersion0.37.0
Local MCP for web search, direct-URL fetch, compact normalized output, site crawling, and docs export.
- @hasna/crawlReleasedyesterdayVersion0.4.15
AI-powered web crawler — self-hosted Firecrawl alternative. Crawl, extract, render JS, search. CLI + MCP + REST API + Dashboard.
- @cosmocoder/mcp-web-docsReleased6d agoVersion2.0.5
MCP server for crawling and indexing web documentation - works with any website
- cloudflare-bot-directoryReleased9h agoVersion1.0.33
CloudFlare Radar verified bots directory - 500+ web crawlers, search engine bots, and user agents as JSON for bot detection and traffic filtering
- obscura-nodeReleasedyesterdayVersion1.1.0
Node.js wrapper for Obscura — the Rust-powered headless browser for AI agents and web scraping. Drop-in replacement for headless Chrome with Puppeteer and Playwright. Ultra-lightweight (~30 MB), instant startup, built-in stealth mode.
- @geosuite/ai-crawler-botsReleasedyesterdayVersion0.6.2
Curated, sourced list of AI crawler / training bot user agents, plus a small CLI to test whether a URL is reachable to each bot.
- crawl4ai-cloudReleased21h agoVersion1.4.0
Lightweight cloud SDK for Crawl4AI - mirrors the OSS API
- turbo-surfReleased2d agoVersion0.2.7
Native-speed, browserless web crawler + MCP server for AI agents. A standalone Rust engine (turbo-dom + a V8 JS-render tier) — fetch + parse + extract + run page JS with no headless browser. This npm package is a thin launcher that spawns the native binar
- @geosuite/sitemap-builderReleased5d agoVersion0.3.3
Crawl a website and emit a valid sitemap.xml — for sites that ship without one or whose CMS pipeline forgot to.
- @waqashanifkhan/crawlerReleased4h agoVersion1.2.0
A crawler and actionable audit utility for SEO and GEO website analysis.
- @synoppy/sdkReleased5d agoVersion1.1.0
Official TypeScript/JavaScript SDK for the Synoppy web-data API — scrape, screenshot, crawl, map, extract, classify, enrich, and images on one key.
- @synoppy/mcpReleased5d agoVersion1.1.0
MCP server for the Synoppy web-data API — read, screenshot, crawl, map, search, extract, classify, enrich, and images as tools for any MCP client.
- @writechoice/mint-scopeReleased2d agoVersion0.1.12
CLI tool for scoping documentation sites for migration to Mintlify with support for repeated --scope-prefix filters
- @easyasstudio/xhs-cliReleased1 week agoVersion1.6.0
小红书创作者 CLI(fork: 配置目录统一到 ~/.config/xhs-cli,与 xianyu-auto skill 约定一致)
- krawlrReleased4 years agoVersion1.1.3
An event-driven web scraping library with polling functionality built on top of Puppeteer.
- @deepcrawl/oreoReleased1h agoVersion2.17.0
deepcrawl cli