Verify identity before interpreting traffic#
A request that claims to be an AI crawler is not necessarily one. Anyone can send User-Agent: GPTBot, OAI-SearchBot, or another public label. If that string is counted as provider traffic, the chart becomes vulnerable to spoofing and an allowlist can become a security mistake.
Verification means connecting a request to evidence the operator or a trusted intermediary publishes. The evidence can be a documented IP range, reverse-DNS procedure, cryptographic assertion, or an edge product’s verified classification. It should result in a specific conclusion—verified, failed, unavailable, or unknown—not a vague “looks legitimate.”
OpenAI’s crawler documentation publishes distinct bot purposes and IP-list links. Cloudflare’s verified-bot documentation describes verification approaches including Web Bot Auth, published IP data with stable user-agent behavior, and reverse DNS. Follow the current operator instructions for the bot you are checking; do not invent a common verification contract for unrelated bots.
A practical verification sequence#
Start from the complete request record at the edge or origin. Preserve the source address, claimed user-agent, timestamp, TLS or proxy context when available, requested URL, and response outcome. Do not make an allow/deny decision from an analytics export that has already discarded network evidence.
Then apply this sequence:
- Identify the claim. Extract the product token from the user-agent and link it to a known operator document. A familiar-looking name is still only a claim.
- Use the official verification route. Fetch current published ranges or execute the documented reverse-DNS and forward-resolution checks. If you use a CDN classification, record that provider and the classification version or time.
- Check consistency. The claimed agent, verified network identity, and documented purpose should not contradict each other. A passing IP match with a mismatched claimed bot deserves review.
- Record an immutable result. Store the result, method, time checked, and a compact evidence reference with the request event.
- Apply access policy separately. Verification proves identity to a defined degree; it does not decide whether a route should be served.
If the official method cannot be completed—because an address is missing, DNS does not resolve as required, or the operator has no public method—use unverified. Do not upgrade a request because it appears well behaved.
Keep classifications precise#
The label “AI bot” hides different kinds of activity. OpenAI, for example, documents OAI-SearchBot for search, GPTBot for potential training use, OAI-AdsBot for ad landing-page validation, and ChatGPT-User for certain user-initiated actions. The same operator name does not imply the same request purpose or the same appropriate policy. Read the exact entries in the OpenAI crawler overview before making a rule.
Use a small classification model:
| Field | Example values | Why separate it? |
|---|---|---|
| Claimed agent | OAI-SearchBot, unknown | What the request says |
| Verified operator | OpenAI, none | Who verification supports |
| Activity category | search, training, user-directed | What official documentation says it is for |
| Verification state | verified, failed, unavailable | Strength of identity evidence |
| Access decision | allow, block, challenge, rate-limit | Your independent site policy |
This separation prevents two common errors: calling all verified requests “search traffic,” and treating a block as proof that the client was malicious.
Verify at the right layer#
An edge service is often the best place to evaluate identity because it sees the original client address before an application proxy rewrites it. If your architecture has multiple proxies, document which header is trusted and who is permitted to set it. A malicious client can supply common forwarding headers; only infrastructure you control should establish client identity.
Do not build a per-request external lookup that slows page delivery. Refresh published verification data on a defined schedule, cache it safely, and capture the data version used for a decision. For reverse DNS, cache carefully and validate the full official procedure rather than only checking that a hostname contains a recognizable word.
CrawlReady’s analytics overview explains reporting concepts, while alerts helps convert repeat failures into a review signal.
Investigate failures with artifacts#
A failed verification is not an incident by itself. First distinguish a spoofed request from a verification-data outage, an address-family mismatch, or an edge configuration change. Review the raw request, the verification lookup result, and the policy action together.
For a verified request that receives an unexpected block, collect these observed artifacts: request time, route, response status, WAF or policy rule ID, redirect chain, and deployment release. If the route is JavaScript-heavy, compare the server response with a browser-rendered observation. Call them raw and rendered artifacts; neither artifact demonstrates how every external client will behave.
A concise incident note should say: “At this time, this verified identity requested this route; the edge returned this status because of this policy.” It should not say the provider could not index, cite, or use the content unless the provider supplies evidence for that separate claim.
Choose policies after verification#
Verification lets you apply deliberate policy. You may allow public documentation, rate-limit expensive route families, require ordinary authentication for private content, or block a documented purpose that does not fit your policy. robots.txt is another signal to configure and monitor, but it is not access control; see AI crawlers and robots.txt for the boundary.
Recheck verification rules whenever an operator changes documentation, your CDN changes classification, or you change proxy topology. Use a site scan to investigate delivery and content artifacts, not to infer third-party crawler behavior.
FAQ#
Is a crawler user-agent string enough to verify an AI bot?#
No. User-agent strings are easy to copy. Treat them as a lead for verification, then use the bot operator's documented method or a trusted edge verification result before reporting provider-specific traffic.
What should I save when an AI crawler is verified?#
Save the timestamp, requested path, response status, normalized identity, verification method and result, policy decision, and a privacy-safe correlation ID. These fields let a later reviewer reproduce the access conclusion without retaining unnecessary data.
Should I allow every verified AI crawler?#
No. Verification answers who made the request, not whether your site should allow it. Decide access separately for each documented purpose, route, rate, and business requirement, then enforce the decision consistently.