crawlbug

A NodeJS web crawler that can be deployed to multiple machines and writes page data to a Firebase database.