What robots.txt does—and does not do#
A robots.txt file is a publicly available set of crawl rules, normally served at the root of a host. It tells compliant automated clients which URL paths they may access. The Robots Exclusion Protocol in RFC 9309 defines groups selected by user-agent and rules using allow and disallow path patterns.
The limit is as important as the syntax: RFC 9309 says these rules are not authorization. Google likewise says its robots.txt guide is primarily for managing crawler traffic, not for keeping a web page out of Search or keeping information secret. A disallowed public URL can still be discovered from links, and a noncompliant client can ignore the file. Put private pages behind authentication and authorization; use robots rules as a stated preference for crawl behavior.
For AI-related traffic, the practical goal is policy clarity. Decide what each documented crawler purpose may request, publish accurate rules, and validate the delivered file and observed enforcement after deployment.
Design policy by purpose, not by hype#
“AI crawler” is not a single technical category. A crawler may support web search, potential model training, ad review, user-driven browsing, monitoring, or another task. Conflating them leads to a broad rule that makes no clear business or engineering decision.
OpenAI’s crawler overview documents separate user agents including OAI-SearchBot, GPTBot, OAI-AdsBot, and ChatGPT-User, with different stated roles. That documentation is operator-specific, not a universal contract. Read each provider’s official documentation before adding a named agent group, and review it when the provider updates its guidance.
Write a short policy record before editing the file:
| Question | Example decision |
|---|---|
| Which public routes are in scope? | Documentation and product pages; not internal search |
| What purpose is permitted? | Search discovery allowed; training reviewed separately |
| Is the agent identity documented? | Use only the exact published product token |
| What protects private content? | Login and application authorization, not robots rules |
| How will the decision be checked? | Fetch /robots.txt, inspect edge events, test policy rules |
This record turns a one-line directive into something another engineer can review later.
Use small, readable groups#
A simple file may contain a default group and a targeted exception. The exact choice belongs to your policy, but the structure should remain easy to inspect:
User-agent: ExampleSearchBot
Allow: /docs/
Allow: /learn/
Disallow: /internal-search/
User-agent: *
Disallow: /private-preview/
Sitemap: https://example.com/sitemap.xml
Do not copy this example with ExampleSearchBot into production. Replace it only with a current documented product token. Keep comments about intent close to unusual rules. Avoid a massive list of guessed bot names; it creates drift and false confidence.
RFC 9309 specifies that crawler product tokens are limited to letters, underscores, and hyphens, and that a crawler selects relevant groups based on its product token. It also defines path matching behavior. Real crawlers can have implementation differences, so the safest configuration is explicit, simple paths and post-release verification rather than clever wildcard constructions.
A deployment checklist#
Before release, run this checklist:
- Serve
https://host/robots.txtwith a successful HTTP response on the canonical host and protocol. - Confirm redirects do not lead to an HTML error page, login page, or different host unexpectedly.
- Ensure the rule applies to the intended path prefix and does not hide scripts, styles, or data needed for a route you expect to be understood.
- Keep the sitemap URL current if you publish one.
- Check whether CDN, WAF, and application rules independently block the same client; robots policy and network policy are separate layers.
- Version-control the change and include the purpose and owner in the review.
- Fetch the deployed artifact from outside your build environment and retain the response as evidence.
Google cautions that blocking resource files can make pages harder for its crawler to analyze when those resources materially affect the page. That caution is in its robots.txt documentation; it is useful as a general reminder to assess dependencies rather than blanket-disallowing assets.
Monitor observed behavior carefully#
After publishing a change, look for the raw facts: requests for /robots.txt, requests to affected paths, status codes, and edge policy outcomes. A crawler that stops requesting a route may be following the directive, may have exhausted a crawl queue, may have changed schedule, or may never have selected the route. Do not turn an absence of traffic into a claim about indexing or visibility.
For a request that claims a known AI identity, verify the identity before placing it in a provider report. See how to verify AI crawler traffic. When a page is JavaScript-dependent, compare the raw response with a rendered browser observation as part of an accessibility audit; that shows the artifacts you observed, not a guarantee of a specific bot’s renderer.
CrawlReady’s crawlability score documentation explains the related site checks, and a scan can help identify visible delivery issues.
Common mistakes#
Using robots.txt as a privacy control. Sensitive material remains sensitive even when disallowed. Require authentication and prevent exposure at the origin.
Blocking a route to remove it from results. Crawling and indexing are different controls. Google explicitly explains that a blocked URL can still appear in results; use the appropriate indexing or access mechanism for your goal.
Allowing an agent and assuming a business result. An allow rule is not a request, index entry, citation, referral, ranking, or conversion guarantee.
Treating a user-agent label as identity. Verify before reporting or granting special network treatment.
FAQ#
Can robots.txt block AI crawlers from my website?#
robots.txt can express a crawl preference to compliant crawlers, but it is not an authorization control. Use authentication, authorization, and network controls for private material; then monitor observed requests and policy enforcement separately.
Should I use one rule for every AI bot?#
No. Use documented user-agent groups and make decisions by the bot's stated purpose. Search, training, advertising validation, and user-directed requests can have different operator documentation and different policy implications.
Does allowing an AI crawler in robots.txt guarantee visibility?#
No. An allow rule only removes that robots.txt restriction for a compliant crawler. It does not guarantee fetching, indexing, retrieval, citation, ranking, referral traffic, or conversion.