Description
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use?domain=example.com.
This is a ready-to-deploy endpoint for web content retrieval, delivered as ready-to-run Node.js source.
Accepts 1 query parameter as documented in the included endpoint.json. It has no external API dependencies — it works standalone once deployed.
Package includes: handler.js (the request handler), endpoint.json (metadata and pricing schema), and a README with setup notes. Drop it into an Express (or similar) server and you have a working API endpoint in minutes.
Reviews
There are no reviews yet.