robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.
OpenBrowse Files
robots.txtrobotsrobots-parserrobots-exclusion-standardjavascriptjsnode-jsrobots-txtcrawlercrawlers