robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

v3.0.1 URL: https://unpkg.com/robots-parser@3.0.1/index.js

Open Browse Files

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

generate-robotstxt

Awesome generator robots.txt

v8.0.3 URL: https://unpkg.com/generate-robotstxt@8.0.3/dist/index.js

Open Browse Files

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot robots

sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

v4.0.2 URL: https://unpkg.com/sitemapper@4.0.2/lib/assets/sitemapper.js

Open Browse Files

parse sitemap xml robots.txt sitemaps crawlers webcrawler

robot-directives

Parse robot directives within HTML meta and/or HTTP headers.

v0.4.0 URL: https://unpkg.com/robot-directives@0.4.0/lib/index.js

Open Browse Files

crawlers header html http meta metadata nofollow noindex robots robots.txt seo spiders

express-robots-txt

Express middleware to serve and generate robots.txt

v1.0.0 URL: https://unpkg.com/express-robots-txt@1.0.0/esm/index.mjs

Open Browse Files

express robots robots.txt robots