Readiness check · Enable lane · weight 5
robots.txt published
/robots.txt returns a plain-text file.
- Observed on
- 76%
- Top 1K sites
- 76%
- Top 1M band
- 81%
- Not measured
- 2,191
of 37,904 measured sites
241 measured
5,385 measured
blocked, challenged or not applicable
Why it matters
robots.txt is where crawlers look first for crawl rules and sitemap locations. Without it, they assume everything is open.
How to fix it
Publish /robots.txt with any crawl rules you need and a Sitemap: line pointing to your XML sitemap.
How we check
/robots.txt returns a plain-text file.
Observed: robots.txt found.
Not observed: No robots.txt found.
Limit: Only the standard location was requested, once, at the time of the check.
By popularity band
| Chrome UX Report band | Observed | Measured |
|---|---|---|
| Top 1K | 76% | 241 |
| Top 5K | 76% | 1,624 |
| Top 10K | 75% | 2,418 |
| Top 50K | 74% | 18,020 |
| Top 100K | 78% | 5,037 |
| Top 500K | 79% | 5,179 |
| Top 1M | 81% | 5,385 |
Categories most likely to do this
Categories least likely
By stack
- Shopify>99%
- Next.js on Fastly99%
- PrestaShop98%
- WordPress on Cloudflare97%
- WordPress on Amazon CloudFront97%
- WordPress96%
- Astro96%
- Webflow95%
- Magento / Adobe Commerce95%
- Adobe Experience Manager95%
- Next.js on Cloudflare95%
- Drupal on Cloudflare95%
- Next.js on Amazon CloudFront95%
- Drupal94%
- Next.js on Akamai94%
- Next.js on Google edge94%
- Next.js93%
- Adobe Experience Manager on Amazon CloudFront93%
- Next.js on Vercel92%
- Nuxt on Cloudflare92%
- Angular91%
- Nuxt91%
- Nuxt on Amazon CloudFront89%
- Sitecore89%
- Blogger88%
By provider
- Cloudflare93%
- Azure Front Door/CDN91%
- Fastly91%
- Akamai90%
- Google edge90%
- Amazon CloudFront88%
- Vercel82%
- Imperva72%
Popular sites where we observed this
Sources and guides
Guide: Robots.txt Guide: Control Crawling Without Hiding MistakesCheck your own site