Search visibility controls
Use the right control for the job. Crawling, indexing and privacy are different: a rule that limits a crawler does not necessarily remove a page from search results.
Control crawlingUse robots.txt to guide compliant crawlers away from selected paths.
Control indexingUse a noindex directive on a page that crawlers are allowed to access.
Protect private contentUse authentication. Search directives are not access controls.
Start here
First decide whether you want to reduce crawling, remove a page from search results, avoid duplicate URLs or make content genuinely private.
Reduce crawler visitsAdd an appropriate Disallow rule to robots.txt.
Keep a page out of searchAdd a robots meta tag or HTTP header with noindex.
Consolidate duplicate pagesUse a canonical URL and consistent internal links.
Restrict accessRequire a login or password and avoid exposing private files publicly.
Crawling
A robots.txt file belongs at the root of the hostname, for example https://example.com/robots.txt. Rules apply only to that protocol and hostname, so a subdomain may need its own file.
Open the live fileConfirm the server returns plain text with a successful response.
Read the matching user-agent groupThe most specific applicable group determines which rules a compliant crawler follows.
Test the exact pathSmall differences in slashes, case or URL parameters can change the result.
For a focused test procedure, see Check whether robots.txt blocks a URL.
Indexing
Add <meta name="robots" content="noindex,follow"> inside the page’s <head>, or send an equivalent X-Robots-Tag HTTP header for non-HTML files.
The crawler normally needs to fetch the page to see its noindex instruction. A blocked URL can remain known from links even when its content is not crawled.
Private content
Never rely on robots.txt, noindex or an unlinked URL to protect customer data, staging systems, backups or confidential documents. Use authentication, appropriate file permissions and server access controls.
Verify the outcome
Inspect the responseCheck the status code, final URL, HTML meta tags and HTTP headers.
Use URL InspectionGoogle Search Console shows whether Google can access and index a URL.
Review internal linksAvoid continuing to promote a page that you intend to retire.
Changes are not instantaneous. Search engines must revisit the URL before their records change; removal tools can accelerate urgent temporary removals but do not replace a lasting site directive.
WordPress
WordPress can request that search engines avoid a site, but the setting is advisory and is not suitable for private staging. Confirm the generated directives and use password protection for non-public work.
Need help?
Include the exact affected URL, the result you want, when the directive changed, and a copy of the current robots.txt, meta tag or HTTP header. Tell us whether the site uses WordPress and whether it is hosted by UKC.
Still not behaving as expected?Open a support ticket with those details and any Search Console message. Never send a password or API key.
