Search

robotstxt-util

RFC 9309 spec compliant robots.txt builder and parser. 🦾

v4.1.0 URL: https://unpkg.com/robotstxt-util@4.1.0/dist/index.js

OpenBrowse Files

robotsrobotstxtseosearch-engine-optimizationrobotstxt-parserrobotstxt-builderrfc5234

astro-robots-txt

Generate a robots.txt for Astro

v1.0.0 URL: https://unpkg.com/astro-robots-txt@1.0.0/index.js

OpenBrowse Files

astro-integrationastroastro-componentseorobots.txtrobotstxtrobots-txt

generate-robotstxt

Awesome generator robots.txt

v8.0.3 URL: https://unpkg.com/generate-robotstxt@8.0.3/dist/index.js

OpenBrowse Files

robotstxtrobots.txtgeneraterobots-txtuser-agentpoliceallowdisallowcrawl-delayrobotrobots

robotstxt-webpack-plugin

A webpack plugin to output a robots.txt file

v8.0.0 URL: https://unpkg.com/robotstxt-webpack-plugin@8.0.0/dist/cjs.js

OpenBrowse Files

robots.txtrobotsrobotstxtgeneraterobots-txtuser-agentwebpackwebpack-pluginplugin

robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

v2.0.3 URL: https://unpkg.com/robots-txt-parser@2.0.3/src/index.js

OpenBrowse Files

robotstxtrobots.txtparsercrawlerspiderbotrobotstxtscraper