Robots.txt Generator

Create custom robots.txt files to control search engine crawling and indexing of your website.

Back to All Tools

Configure Your Robots.txt

Allow All Crawlers

Block Specific Directories

Sitemap URL

Crawl Delay (seconds)

Robots.txt Guide

What is Robots.txt?

Robots.txt is a text file in your site's root directory that instructs search engine crawlers which pages they should or shouldn't access. It's a critical SEO tool for controlling how search engines interact with your site.

Common Directives

User-agent: Specifies which crawler the rules apply to (* = all)
Disallow: Paths that should not be crawled
Allow: Overrides Disallow for specific paths
Sitemap: Location of your XML sitemap
Crawl-delay: Seconds between requests

Best Practices

  • Always place in root directory
  • Include sitemap reference
  • Block admin and private areas
  • Don't block CSS/JS (affects rendering)
  • Test with Google Search Console
  • Keep it simple and clear

Common Mistakes

  • Blocking important content
  • Forgetting sitemap reference
  • Typos in paths
  • Blocking CSS/JavaScript
  • Not testing after deployment

Example Use Cases

Block duplicate content: Disallow: /print/
Block search results: Disallow: /*?s=
Block specific bots: User-agent: BadBot + Disallow: /
Allow everything: User-agent: * + Disallow: (empty)

FAQ

Is robots.txt mandatory?

No, but highly recommended for better crawl control.

Can I block sensitive data?

Robots.txt is not security—use actual authentication for sensitive content.

How to block all bots?

User-agent: * and Disallow: /