CrawlReady Analytics records every AI crawler visit to your site and organizes the data so you can answer four questions:
- Which bots visit your site? See every crawler by name, category, and verification status.
- Which pages do they access? Identify the content AI systems request most often.
- How does traffic trend over time? Track daily and hourly visit volume across days, weeks, or months.
- Are traffic patterns changing? Compare periods to spot new crawlers, shifting interest, or drops in activity.
The Dashboard is where you read these metrics. This page explains how data gets there.
Bot categories
Every detected crawler is assigned to one of four categories:
| Category | Description | Examples |
|---|---|---|
ai-training | Crawlers collecting page content to train large language models. | GPTBot, ClaudeBot, CCBot |
ai-search | Crawlers fetching content in real time to answer user queries in AI-powered search engines. | PerplexityBot, YouBot |
seo | Traditional search engine crawlers indexing pages for organic search results. | Googlebot, Bingbot |
unknown | Bots that match crawler patterns but are not in the CrawlReady known-bot registry. | Any unrecognized User-Agent that exhibits crawler behavior. |
Verified vs. unverified bots
A bot is marked verified when CrawlReady confirms its identity against a known IP range or User-Agent signature published by the bot operator. Unverified bots are classified by name pattern only -- they claim to be a particular crawler, but CrawlReady cannot confirm the claim. Both verified and unverified bots appear in the dashboard; a badge indicates verification status.
Data flow
Your integration -- whether Next.js middleware, Express middleware, Cloudflare Workers, Vercel Edge Middleware, or a Script Tag -- inspects each inbound request and matches the User-Agent against the CrawlReady bot registry. When a crawler is detected, the integration sends a non-blocking beacon to the CrawlReady ingest API with the bot name, requested path, and timestamp. The dashboard then queries the stored visits to produce the charts and tables described on the Dashboard page.