What this measurement can establish#
Cloudflare Workers AI crawler analytics is an edge log of requests your Worker actually handled. It can establish that a request with a particular user-agent, path, time, response status, and selected request properties reached that edge. It cannot establish what a model read, whether content entered an index, or whether an answer engine later cited a page. Start with that narrow definition; it prevents a traffic dashboard from becoming a claim about visibility.
Workers Analytics Engine is designed for custom, high-cardinality analytics written from Workers and queried with SQL. A useful event has stable dimensions—site, route pattern, bot family, verification state, status class—and numeric values such as one request or response milliseconds. Avoid putting full URLs, IP addresses, cookies, query strings, authorization headers, or page text into the event by default.
Build an event that remains interpretable#
Use a request-time classifier only to decide what to observe. Normalize the path to a route template or a bounded pathname; record a coarse status class such as 2xx, 3xx, 4xx, or 5xx; and record a deployment version so changed logic is visible in reports. Preserve the original user-agent in secure short-retention request logs only when operationally necessary, then derive a controlled claimedBotFamily for analytics.
| Field | Example | Why it exists |
|---|---|---|
eventTime | request timestamp | supports time windows and incident review |
route | /guides/:slug | identifies affected content areas without query data |
claimedBotFamily | openai-search | a classifier result, not identity proof |
verification | unverified | prevents a user-agent claim being treated as fact |
statusClass | 2xx | separates successful fetches from failures |
contentVariant | html | records the artifact actually served |
Cloudflare says a Verified bot is one it has confirmed is transparent and non-abusive; its classification includes Search, Agent, and Training behaviors. That is useful platform context, but it does not turn every matching request into a verified request. Read Cloudflare’s definition before using a managed signal in policy decisions.
Keep logging off the response path#
Return the origin response first and make nonessential logging failure-tolerant. Cloudflare documents ctx.waitUntil() for background work and notes it can be called multiple times; a rejected background promise does not stop other waitUntil() work. It also notes that Workers Analytics Engine writes do not need waitUntil(). Follow the current API for the binding you use and test both success and failure paths.
A practical checklist:
- Fetch or generate the normal response without changing its body or cache policy.
- Derive only bounded event fields from the request and response.
- Write one event; do not synchronously call a third-party analytics endpoint per crawler request.
- Catch logging errors, increment an internal error counter, and still return the normal response.
- Sample or aggregate broad non-bot traffic so crawler analysis does not create an uncontrolled dataset.
See Cloudflare’s Context API for the lifecycle semantics. A production test should demonstrate that a deliberately unavailable analytics dependency does not change the page status or body.
Query evidence, then investigate changes#
Build daily reports around observed counts: requests by claimed family, verified state, route, status class, and response artifact. Compare a baseline period to a release period, then open a sample of raw edge logs for any unexpected change. A sudden zero is often a deployment, routing, firewall, caching, or classifier problem; it is not evidence that a crawler stopped using the site.
For a high-value route, save a small evidence packet: query window and SQL version, event totals, representative request IDs where permitted, response status, headers relevant to caching, and the rendered or raw content artifact you inspected. CrawlReady’s analytics overview explains the product-side distinction between detecting an inbound request and reporting it. Use the Cloudflare Workers integration if you are instrumenting CrawlReady rather than building a separate event pipeline.
Scope and limits. This procedure creates an audit trail, not a promise of crawler access, inclusion, indexing, citations, referral traffic, rankings, or conversions. User-agent text is easy to spoof. Treat the records as observed request artifacts unless the requester is verified by the relevant operator method. Keep the raw event, the verification result, and the interpretation separate so a later reviewer can reproduce the conclusion.
Evidence checklist#
Before sharing a chart, confirm the query names the dataset and time zone, excludes internal test traffic, and states whether counts are sampled. Capture one known test request end to end: request path, classifier output, analytics event, and dashboard row. Review retention, access controls, and deletion requirements with the team that owns logs. If a policy blocks or challenges traffic, record the policy ID and result rather than describing the bot’s intent. This turns an attractive dashboard into operational evidence.
FAQ#
What does a Workers crawler event prove?#
It proves that the Worker observed a request with recorded properties at the edge. It does not prove indexing, model training, citation, ranking, or a human visit.
Should a matching user-agent be called verified?#
No. A matching user-agent is a claim. Record it as observed until it passes the relevant operator or platform verification method.
Does Workers Analytics Engine require ctx.waitUntil?#
Cloudflare says Workers Analytics Engine writes do not need ctx.waitUntil. Use ctx.waitUntil for appropriate background promises and keep analytics failure off the response path.