robots.txt does not make content private
The file is public and controls whether compliant crawlers request URLs. A blocked URL may still appear in search results without a description if other pages link to it. Protect private content with authentication or access controls.
Inspect the live file
Open robots.txt on the affected hostname
- Open https://your-domain.example/robots.txt in a private browser window.
- Confirm it returns plain text with a successful response, not HTML, a login page or a redirect loop.
- Check the exact hostname: root, www and a subdomain can serve different files.
- Save the rules and record the exact URL that is not being crawled.
Understand the rules
Match user-agent and path
- User-agent: *
- The following group applies to crawlers without a more specific matching group.
- Disallow: /
- Blocks crawling of the entire host for that user-agent.
- Disallow: /private/
- Blocks matching paths under that directory, but does not add password protection.
- Allow
- Can permit a more specific path inside a broader blocked area for crawlers that support it.
- Empty Disallow
- Allows crawling; it is equivalent to having no crawl restriction for that group.
- Sitemap
- Points crawlers to a sitemap but does not override a blocking rule.
Test Google’s view
Use URL Inspection in Search Console
- Open the verified Search Console property for the exact hostname.
- Inspect the complete affected URL.
- Review crawl permission, indexing status and the last crawl result.
- After correcting the rule, run a live test and request indexing when appropriate.
Crawl versus index
Use the correct control for the goal
- Allow a public page into search
- Allow crawling, return HTTP 200, provide indexable content and avoid a noindex directive.
- Keep a public page out of search
- Allow crawling and add a supported noindex meta tag or HTTP header so the crawler can see it.
- Reduce crawling of duplicate paths
- Use robots.txt only after canonical URLs and navigation are designed correctly.
- Protect private information
- Require authentication or remove public access; do not rely on robots.txt or noindex.
Unexpected blocking
Check who generated the file
- A WordPress SEO or security plugin.
- A staging or maintenance setting left enabled after launch.
- A physical robots.txt file overriding the application’s virtual response.
- A CDN, proxy or cached old version.
- Different configuration on www, root or another subdomain.
Still unsure why a page is blocked?
Send the exact URL, robots.txt URL and contents, intended result, Search Console message, last known change and whether a plugin, CDN or staging mode is involved. Do not send Search Console or hosting passwords.
Ask the Website team