Keyword: robots.txt
- super-sitemapReleasedyesterdayVersion2.0.4
Sitemap library for TanStack Start and SvelteKit, focused on ease of use and making it impossible to forget to add your paths.
- claude-seo-skillsReleased5d agoVersion1.6.1
A collection of Claude Agent Skills for SEO audits (Core Web Vitals, sitemaps, robots.txt, schema, links, headings, images, redirects, metadata, and more), with a CLI to install them into your Claude skills directory.
- @geosuite/ai-crawler-botsReleasedyesterdayVersion0.6.2
Curated, sourced list of AI crawler / training bot user agents, plus a small CLI to test whether a URL is reachable to each bot.
- @casoon/astro-site-filesReleased6d agoVersion0.4.2
Astro integration that generates robots.txt, llms.txt, sitemap.xml, security.txt, and humans.txt at build time from typed configuration.
- @particle-academy/fancy-x-filesReleasedyesterdayVersion0.1.2
Zero-dependency, isomorphic manager for the well-known files every modern web app owes bots and agents — robots.txt, security.txt, llms.txt, humans.txt, sitemap.xml, ai.txt. The Node/TypeScript mirror of the PHP particle-academy/fancy-x-files. Includes a
- smart-seo-liteReleased5h agoVersion3.0.0
Add SEO to your React/Next.js app in 3 lines of code. v3: SEOProvider context, Social Preview widget, generateMetaTags for App Router, Core Web Vitals monitoring, SEOLink, and more.
- @assertkit/bot-analyzerReleased3d agoVersion0.3.1
Pure, dependency-free engine for bot detection, posture analysis, and remediation snippet generation. Classify User-Agents, score a site's robots.txt + meta + X-Robots-Tag posture, and generate ready-to-paste robots.txt / meta / X-Robots-Tag / Edge-middle
- @veris.works/vibecodecheckReleased3h agoVersion1.0.2
Pre-launch site check for vibe-coded MVPs — SEO, AEO, GEO, AI crawler, security in one scan
- rsl-licensingReleased6h agoVersion0.1.0
Build, validate, and embed RSL (Really Simple Licensing) AI-licensing terms for JS/TS sites — robots.txt, HTTP Link, HTML, RSS, .well-known.
- cockroach-crawlerReleasedyesterdayVersion0.1.1
Public-web crawler for agent workflows with robots.txt, sitemaps, depth limits, filters, markdown extraction, and JSONL output.
- ajnas-agent-crawlerReleasedyesterdayVersion0.1.0
Fast, respectful web crawler for agent workflows with robots.txt, sitemap discovery, markdown extraction, and JSONL output.
- @hashmivue/web-downloader-cliReleased3d agoVersion1.0.3
A terminal tool for downloading complete web pages with all assets for offline viewing
- @knownagents/sdkReleased4d agoVersion2.1.0
The official Node.js SDK for Known Agents: AI agent analytics (bot traffic), LLM referral tracking, automatic robots.txt, agent verification
- astro-ai-robots-txtReleased4d agoVersion0.1.5
Block AI scrapers in the robots.txt for your Astro site
- grunt-html-snapshotsReleased3d agoVersion10.1.1
The grunt task for html-snapshots
- @peac/mappings-content-signalsReleased16h agoVersion0.16.0
Content use policy signal parsing for PEAC (robots.txt, tdmrep.json, Content-Usage)