robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

v3.0.1 URL: https://unpkg.com/robots-parser@3.0.1/index.js

Open Browse Files

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

simplecrawler

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

v1.1.9 URL: https://unpkg.com/simplecrawler@1.1.9/lib/index.js

Open Browse Files

simple crawler spider cache queue simplecrawler eventemitter

isbot-fast

JavaScript module detecting bots/crawlers/spiders via user-agent

v1.2.0 URL: https://unpkg.com/isbot-fast@1.2.0/index.js

Open Browse Files

shift-parser

ECMAScript parser that produces a Shift format AST

v8.0.0 URL: https://unpkg.com/shift-parser@8.0.0/src/index.js

Open Browse Files

Shift AST node parser SpiderMonkey Parser API parse spider monkey abstract syntax tree

es6-crawler-detect

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

v4.0.2 URL: https://unpkg.com/es6-crawler-detect@4.0.2/dist/index.js

Open Browse Files

es6-javascript crawler bots spider detection