robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

v3.0.1 URL: https://unpkg.com/robots-parser@3.0.1/index.js

Open Browse Files

robots.txt parser user-agent scraper spider bot robots-exclusion-standard

simplecrawler

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

v1.1.9 URL: https://unpkg.com/simplecrawler@1.1.9/lib/index.js

Open Browse Files

simple crawler spider cache queue simplecrawler eventemitter

isbot-fast

JavaScript module detecting bots/crawlers/spiders via user-agent

v1.2.0 URL: https://unpkg.com/isbot-fast@1.2.0/index.js

Open Browse Files

shift-parser

ECMAScript parser that produces a Shift format AST

v8.0.0 URL: https://unpkg.com/shift-parser@8.0.0/src/index.js

Open Browse Files

Shift AST node parser SpiderMonkey Parser API parse spider monkey abstract syntax tree

spider-detector

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

v2.1.0 URL: https://unpkg.com/spider-detector@2.1.0/index.js

Open Browse Files

crawler detector spider bot middleware single page app expressjs