express-robots-txt

Express middleware to serve and generate robots.txt

v1.0.0 URL: https://unpkg.com/express-robots-txt@1.0.0/esm/index.mjs

Open Browse Files

express robots robots.txt robots

robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

v2.0.3 URL: https://unpkg.com/robots-txt-parser@2.0.3/src/index.js

Open Browse Files

robots txt robots.txt parser crawler spider bot robotstxt scraper

@itsmatteomanf/astro-robots-txt

Generate a robots.txt file for your Astro site

v0.2.0 URL: https://unpkg.com/@itsmatteomanf/astro-robots-txt@0.2.0/dist/index.js

Open Browse Files

astro-integration astro-component withastro astro robots robots.txt robots-txt

generate-robotstxt

Awesome generator robots.txt

v8.0.3 URL: https://unpkg.com/generate-robotstxt@8.0.3/dist/index.js

Open Browse Files

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot robots

grunt-robots-txt

UNMAINTAINED Grunt plugin which generates a robots.txt file

v1.0.1 URL: https://unpkg.com/grunt-robots-txt@1.0.1/grunt.js

Open Browse Files

gruntplugin robots yahoo google bing seo txt