CrawlReady Analytics records every AI crawler visit to your site and organizes the data so you can answer four questions:
- Which bots visit your site? See every crawler by name, category, and verification status.
- Which pages do they access? Identify the content AI systems request most often.
- How does traffic trend over time? Track daily and hourly visit volume across days, weeks, or months.
- Are traffic patterns changing? Compare periods to spot new crawlers, shifting interest, or drops in activity.
The Dashboard is where you read these metrics. This page explains how data gets there.
Bot categories#
Every detected crawler is assigned to one of four categories:
| Category | Description | Examples |
|---|---|---|
ai-training | Crawlers collecting page content to train large language models. | GPTBot, ClaudeBot, CCBot |
ai-search | Crawlers fetching content in real time to answer user queries in AI-powered search engines. | PerplexityBot, YouBot |
seo | Traditional search engine crawlers indexing pages for organic search results. | Googlebot, Bingbot |
unknown | Bots that match crawler patterns but are not in the CrawlReady known-bot registry. | Any unrecognized User-Agent that exhibits crawler behavior. |
Verified vs. unverified bots#
A bot is marked verified when CrawlReady confirms its identity against a known IP range or User-Agent signature published by the bot operator. Unverified bots are classified by name pattern only -- they claim to be a particular crawler, but CrawlReady cannot confirm the claim. Both verified and unverified bots appear in the dashboard; a badge indicates verification status.
Data flow#
Your integration -- whether Next.js middleware, Express middleware, Cloudflare Workers, Vercel Edge Middleware, or a Script Tag -- inspects each inbound request and matches the User-Agent against the CrawlReady bot registry. When a crawler is detected, the integration sends a non-blocking beacon to the CrawlReady ingest API with the bot name, requested path, and timestamp. The dashboard then queries the stored visits to produce the charts and tables described on the Dashboard page.
Frequently Asked Questions
What four questions does CrawlReady Analytics answer?
Which bots visit your site, which pages they access, how traffic trends over time, and whether traffic patterns are changing — comparing periods to spot new crawlers, shifting interest, or drops in activity.
What are the four AI crawler bot categories?
ai-training (crawlers collecting content to train large language models, like GPTBot, ClaudeBot, and CCBot), ai-search (crawlers fetching content in real time for AI-powered search engines, like PerplexityBot and YouBot), seo (traditional search engine crawlers like Googlebot and Bingbot), and unknown (bots that match crawler patterns but aren't in the CrawlReady known-bot registry).
What's the difference between a verified and unverified bot?
A bot is marked verified when CrawlReady confirms its identity against a known IP range or User-Agent signature published by the bot operator. Unverified bots are classified by name pattern only — they claim to be a particular crawler, but CrawlReady can't confirm the claim. Both types appear in the dashboard with a badge indicating verification status.
How does crawler visit data get into the dashboard?
Your integration — Next.js middleware, Express middleware, Cloudflare Workers, Vercel Edge Middleware, or a script tag — inspects each inbound request, matches the User-Agent against the CrawlReady bot registry, and sends a non-blocking beacon to the ingest API with the bot name, requested path, and timestamp when a crawler is detected.
