robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

v3.0.1 URL: https://unpkg.com/robots-parser@3.0.1/index.js

Open Browse Files

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.

v0.0.5-dev URL: https://unpkg.com/robots-txt-component@0.0.5-dev/src/component.js

Open Browse Files

robots.txt robots robots-parser robots-exclusion-standard javascript js node-js robots-txt crawler crawlers