isbot

🤖/👨‍🦰 Recognise bots/crawlers/spiders using the user agent string.

v5.1.29 URL: https://unpkg.com/isbot@5.1.29/index.js

Open Browse Files

bot crawlers spiders googlebot useragent user agent parsing 🤖

@crawlee/utils

A set of shared utilities that can be used by crawlers

v3.14.1 URL: https://unpkg.com/@crawlee/utils@3.14.1/index.js

Open Browse Files

apify crawlers crawling utilities utils

sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

v4.0.2 URL: https://unpkg.com/sitemapper@4.0.2/lib/assets/sitemapper.js

Open Browse Files

parse sitemap xml robots.txt sitemaps crawlers webcrawler

robot-directives

Parse robot directives within HTML meta and/or HTTP headers.

v0.4.0 URL: https://unpkg.com/robot-directives@0.4.0/lib/index.js

Open Browse Files

crawlers header html http meta metadata nofollow noindex robots robots.txt seo spiders

@microlink/ua

A simple redis primitives to incr() and top() user agents

v1.2.5 URL: https://unpkg.com/@microlink/ua@1.2.5/src/index.js

Open Browse Files

bot crawlers googlebot redis spiders user-agent useragent