astro-robots-txt

Generate a robots.txt for Astro

v1.0.0 URL: https://unpkg.com/astro-robots-txt@1.0.0/index.js

Open Browse Files

astro-integration astro astro-component seo robots.txt robotstxt robots-txt

@itsmatteomanf/astro-robots-txt

Generate a robots.txt file for your Astro site

v0.1.2 URL: https://unpkg.com/@itsmatteomanf/astro-robots-txt@0.1.2/index.js

Open Browse Files

astro-integration astro-component withastro astro robots robots.txt robots-txt

generate-robotstxt

Awesome generator robots.txt

v8.0.3 URL: https://unpkg.com/generate-robotstxt@8.0.3/dist/index.js

Open Browse Files

robotstxt robots.txt generate robots-txt user-agent police allow disallow crawl-delay robot robots

robotstxt-webpack-plugin

A webpack plugin to output a robots.txt file

v8.0.0 URL: https://unpkg.com/robotstxt-webpack-plugin@8.0.0/dist/cjs.js

Open Browse Files

robots.txt robots robotstxt generate robots-txt user-agent webpack webpack-plugin plugin

parse-robots

Promise based parser for robots.txt files.

v3.2.0 URL: https://unpkg.com/parse-robots@3.2.0/index.js

Open Browse Files

crawl web robots.txt seo robotstxt robots robots-txt sitemap