isbot

🤖/👨‍🦰 Recognise bots/crawlers/spiders using the user agent string.

v5.1.32 URL: https://unpkg.com/isbot@5.1.32/index.js

Open Browse Files

bot crawlers spiders googlebot useragent user agent parsing 🤖

@crawlee/utils

A set of shared utilities that can be used by crawlers

v3.15.2 URL: https://unpkg.com/@crawlee/utils@3.15.2/index.js

Open Browse Files

apify crawlers crawling utilities utils

sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

v4.0.2 URL: https://unpkg.com/sitemapper@4.0.2/lib/assets/sitemapper.js

Open Browse Files

parse sitemap xml robots.txt sitemaps crawlers webcrawler

robot-directives

Parse robot directives within HTML meta and/or HTTP headers.

v0.4.0 URL: https://unpkg.com/robot-directives@0.4.0/lib/index.js

Open Browse Files

crawlers header html http meta metadata nofollow noindex robots robots.txt seo spiders

parallaxapis-sdk-ts

ParallaxAPIs SDK

v1.0.10 URL: https://unpkg.com/parallaxapis-sdk-ts@1.0.10/dist/index.js

Open Browse Files

perimeterx web scraping datadome crawlers