wwweb

An autonomous webcrawler for indexing robots.txt files.