generate-robotstxt

Awesome generator robots.txt

v8.0.3 URL: https://unpkg.com/generate-robotstxt@8.0.3/dist/index.js

Open Browse Files

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot robots

probot

A framework for building GitHub Apps to automate and improve your workflow

v13.4.4 URL: https://unpkg.com/probot@13.4.4/lib/index.js

Open Browse Files

probot github-apps github automation robots workflow

express-robots-txt

Express middleware to serve and generate robots.txt

v1.0.0 URL: https://unpkg.com/express-robots-txt@1.0.0/esm/index.mjs

Open Browse Files

express robots robots.txt robots

robot-directives

Parse robot directives within HTML meta and/or HTTP headers.

v0.4.0 URL: https://unpkg.com/robot-directives@0.4.0/lib/index.js

Open Browse Files

crawlers header html http meta metadata nofollow noindex robots robots.txt seo spiders

robotstxt-webpack-plugin

A webpack plugin to output a robots.txt file

v8.0.0 URL: https://unpkg.com/robotstxt-webpack-plugin@8.0.0/dist/cjs.js

Open Browse Files

robots.txt robots robotstxt generate robots-txt user-agent webpack webpack-plugin plugin