Search
@lydio/robotstxt
A structured and fluent API for generating robots.txt files in JavaScript. Supports multiple user-agents, custom rules, crawl delays, and multiple sitemaps with an expressive syntax.
v1.0.1 URL:
https://unpkg.com/@lydio/robotstxt@1.0.1/src/index.mjs
Open
Browse Files
robots.txt
robots
sitemap
seo
search-engines
crawler
crawl-rules
fluent-api
javascript
web
lydio
automation
web-crawling
web-scraping
site-indexing
robots-txt-generator