robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

Browse on unpkg