spiderhunt

Very straightforward web crawler. Uses EventEmitter. Based on Simplecrawler but using a distributed queue system.

v0.1.2 URL: https://unpkg.com/spiderhunt@0.1.2/lib/index.js

Open Browse Files

simple crawler spider cache queue simplecrawler eventemitter spiderhunt distributed redis rabbitmq rabbit scalable