simplecrawler

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

v1.1.9 URL: https://unpkg.com/simplecrawler@1.1.9/lib/index.js

Open Browse Files

simple crawler spider cache queue simplecrawler eventemitter

semantic-crawler

Priority based Semantic Web Crawler.

v0.0.2 URL: https://unpkg.com/semantic-crawler@0.0.2/semantic-crawler.js

Open Browse Files

semantic-crawler pdf-crawler text-crawler priority priority-crawler scraper crawling spider scraper scraping simplecrawler crawler osmosis js-crawler supercrawler node-web-crawler web-crawler

web-spider

web-spider is a simple and fast web spider written with Nodejs!

v0.3.1 URL: https://unpkg.com/web-spider@0.3.1/index.js

Open Browse Files

crawler spider phantomjs superagent jquery simplecrawler simplespider cheerio

spiderhunt

Very straightforward web crawler. Uses EventEmitter. Based on Simplecrawler but using a distributed queue system.

v0.1.2 URL: https://unpkg.com/spiderhunt@0.1.2/lib/index.js

Open Browse Files

simple crawler spider cache queue simplecrawler eventemitter spiderhunt distributed redis rabbitmq rabbit scalable

ispider

iSpider is a simple and fast web spider written with Nodejs!

v0.0.4 URL: https://unpkg.com/ispider@0.0.4/lib/index.js

Open Browse Files

crawler spider phantomjs superagent jquery simplecrawler simplespider cheerio