Visual robots.txt File Generator

Create custom robots.txt rules, block AI scrapers, and link your XML sitemap.

Visual robots.txt File Generator

Visual robots.txt generator. Create crawl rules, block AI training bots (GPTBot, ClaudeBot), and link XML sitemaps.

How to use this tool

  1. Set default crawl permissions and list disallowed directories.
  2. Toggle AI Bot Protection to disallow scrapers like GPTBot and ClaudeBot.
  3. Copy or download your robots.txt file ready to deploy to your root domain.

Frequently Asked Questions

Where does the robots.txt file go on a website?

It must always reside in the top-level root directory (e.g., https://example.com/robots.txt) for web crawlers to discover it.

Related tools

XML Sitemap GeneratorLLMs.txt Standard File GeneratorRedirect Chain & Status Code Tracer

Google SERP Simulator

Accurate pixel width calculation and mobile/desktop preview to prevent snippet truncation in Google.

Schema JSON-LD & AI Standards

Generate valid JSON-LD schemas and create standard robots.txt and llms.txt files for AI crawlers.

100% Client-Side Privacy

Instant analysis, generation, and inspection inside your browser with zero data sent to external servers.