Description
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use?domain=example.com.
Requires your own data source. This endpoint fetches from an upstream provider at runtime. You supply and pay for that access — no API key, feed or usage quota is included in this purchase.
This is a ready-to-deploy endpoint for web content retrieval, delivered as ready-to-run Node.js source.
Accepts 1 query parameter as documented in the included endpoint.json. Any environment variables or third-party API keys you must supply yourself are listed in the included README. Endpoints that return live or third-party data query that upstream source at runtime; no data subscription, API key or usage quota is included in this purchase.
Package includes: handler.js (the request handler), endpoint.json (metadata and pricing schema), and a README with setup notes. Drop it into an Express (or similar) server and you have a working API endpoint in minutes.
Reviews
There are no reviews yet.