Robots.txt Analyzer Tool
Analyze and optimize your robots.txt file with our free tool. Check for errors, syntax issues, security concerns, and get recommendations for improving your site's crawling directives.
- Robots.txt Analyzer
- About Robots.txt
Quick Robots.txt Tips
Basic Structure
Always start with a User-agent directive followed by Allow/Disallow rules for that agent.
Security Warning
Never use robots.txt to hide sensitive information - it's publicly accessible!
SEO Impact
Blocking CSS/JS can negatively impact how search engines render and rank your pages.
Add Sitemaps
Include your sitemap URL to help search engines discover your content efficiently.
What is robots.txt?
The robots.txt file is a web standard used to communicate with web crawlers and other web robots. It tells these automated visitors which pages or sections of your website should not be processed or scanned.
Why is robots.txt important?
- Crawl Budget Management: Helps search engines focus their crawling efforts on important content
- Server Resource Optimization: Prevents crawlers from overloading your server
- Privacy Protection: Keeps private or sensitive areas of your site from being indexed
- SEO Impact: Properly configured robots.txt files can improve your overall SEO performance
Common Robots.txt Directives
| Directive | Description | Example |
|---|---|---|
| User-agent | Specifies which web crawler the rules apply to | User-agent: Googlebot |
| Disallow | Prevents crawling of specified pages or directories | Disallow: /admin/ |
| Allow | Explicitly allows crawling of specified pages (overrides Disallow) | Allow: /admin/public/ |
| Sitemap | Indicates the location of your XML sitemap | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Suggests a delay between crawler requests (in seconds) | Crawl-delay: 10 |
Common Search Engine User Agents
| Bot Name | User-agent String | Search Engine |
|---|---|---|
| Googlebot (Standard) | Googlebot |
|
| Googlebot Images | Googlebot-Image |
Google Images |
| Bingbot | Bingbot |
Microsoft Bing |
| Yahoo! Slurp | Slurp |
Yahoo! |
| Baiduspider | Baiduspider |
Baidu |
| Yandex Bot | YandexBot |
Yandex |
| DuckDuckGo | DuckDuckBot |
DuckDuckGo |
| All Crawlers | * |
Universal Wildcard |
Best Practices for Robots.txt
- Place your robots.txt file at the root of your domain (e.g., example.com/robots.txt)
- Only use robots.txt to block resources that don't need to be crawled
- Don't use robots.txt to hide sensitive content (use password protection instead)
- Include your sitemap URL in the robots.txt file
- Be specific with user-agent directives when targeting specific crawlers
- Test your robots.txt file after making changes
- Use case-sensitive paths in your directives
Common Mistakes to Avoid
- Blocking all crawlers with
User-agent: * Disallow: /(prevents indexing) - Blocking CSS and JavaScript files (prevents proper rendering)
- Using robots.txt to block private content (not secure)
- Syntax errors that make your directives unreadable
- Not using the correct path syntax (absolute vs. relative paths)
- Forgetting to update robots.txt after site restructuring
Modern SEO Considerations for Robots.txt
Mobile-First Indexing
With Google's mobile-first indexing, make sure your robots.txt doesn't block mobile versions of your site. Both desktop and mobile crawlers should have access to CSS, JavaScript, and images for proper rendering.
# DON'T block these resources
User-agent: *
Allow: /css/
Allow: /js/
Allow: /images/
JavaScript Rendering
Modern search engines render JavaScript, so don't block access to JS files. This ensures crawlers can see your site as users do and properly index dynamic content.
Bad Practice:
User-agent: *
Disallow: *.js$
Multiple Sitemaps
Large sites often benefit from multiple specialized sitemaps. You can list all of them in your robots.txt file to improve crawling efficiency.
Sitemap: https://example.com/sitemap-main.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-blog.xml
Crawl Budget Optimization
For large sites, focusing crawl budget on important pages is crucial. Use robots.txt to prevent crawling of low-value pages like filtered product results or paginated archives.
User-agent: *
Disallow: /products/filter/
Disallow: /archive/page/