robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

v3.0.1 URL: https://unpkg.com/robots-parser@3.0.1/index.js

Open Browse Files

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

generate-robotstxt

Awesome generator robots.txt

v8.0.3 URL: https://unpkg.com/generate-robotstxt@8.0.3/dist/index.js

Open Browse Files

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot robots

sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

v4.0.2 URL: https://unpkg.com/sitemapper@4.0.2/lib/assets/sitemapper.js

Open Browse Files

parse sitemap xml robots.txt sitemaps crawlers webcrawler

express-robots-txt

Express middleware to serve and generate robots.txt

v1.0.0 URL: https://unpkg.com/express-robots-txt@1.0.0/esm/index.mjs

Open Browse Files

express robots robots.txt robots

robotstxt-webpack-plugin

A webpack plugin to output a robots.txt file

v8.0.0 URL: https://unpkg.com/robotstxt-webpack-plugin@8.0.0/dist/cjs.js

Open Browse Files

robots.txt robots robotstxt generate robots-txt user-agent webpack webpack-plugin plugin