Visual robots.txt File Generator & AI Bot Blocker
Generate customized robots.txt files in seconds. Configure search engine access, block intrusive AI scrapers, specify crawl delays, and link your XML sitemap.
如何使用此工具
- Set default crawl permissions and list disallowed directories.
- Toggle AI Bot Protection to disallow scrapers like GPTBot and ClaudeBot.
- Copy or download your robots.txt file ready to deploy to your root domain.
常见问题解答
Where does the robots.txt file go on a website?
It must always reside in the top-level root directory (e.g., https://example.com/robots.txt) for web crawlers to discover it.
相关工具
Google SERP 模拟器
精准计算桌面端与移动端的像素宽度限制，避免标题和描述在搜索结果中被截断。
Schema 结构化数据与 AI 标准
一键生成合规的 JSON-LD 数据标记，并创建针对 AI 爬虫的 robots.txt 和 llms.txt 文件。
100% 浏览器本地隐私保护
所有分析、生成和检测完全在您浏览器本地运行，绝不向任何外部服务器上传数据。