AI Readiness Score#
A single 0–100 score that measures how ready your site is for AI crawlers and agents. It combines three sub-scores: Crawlability (50% weight), Agent Readiness (25%), and Agent Interaction (25%). Floor rule: the overall score cannot exceed 60 if any sub-score falls below 20, regardless of how well the other two perform.
Site Key#
A unique token in the format cr_live_* that identifies your site for analytics ingest. Generated automatically when you register a domain. You pass the site key to the middleware — it's included in every ingest event sent to /api/v1/ingest. The site key is not a secret, but it is unique per domain and rate-limited. It is different from an API key.
API Key#
A Bearer token used for authenticated API access — running scans, managing sites, listing results. Created from the dashboard or via POST /api/v1/api-keys. The full secret is shown exactly once at creation time; store it immediately. API keys are scoped to your user or organization and subject to tier-based rate limits. Different from a site key: API keys authenticate you, site keys identify your site.
Integration Types#
Middleware runs server-side on every inbound request before your application handles it. It reads the User-Agent header, detects AI crawlers, search engines and SEO tools, and fires an ingest event asynchronously without blocking the response. Because it runs at the edge, it captures crawlers that never execute JavaScript — the bots most likely to be indexing your content for AI answers. Coverage is approximately 100% of bot traffic.
Script tag runs client-side after the page loads in a browser. It only fires when JavaScript executes successfully, which means any crawler that does not run a JS engine (the majority of AI crawlers) is invisible to it. Use the script tag only when you cannot deploy server-side middleware.
Indexing (indexNowKey) is a middleware capability, not a separate integration — set the option and the same middleware also answers Bing's IndexNow key-file check, so new and changed pages get pushed within minutes. Free on every plan. See Get Indexed by Bing, ChatGPT & Copilot.
Optimized serving (optimize) is the paid tier of the same middleware: eligible AI crawlers receive a fully rendered, AI-readable version of the page from the CrawlReady edge, while humans keep hitting your origin untouched. See the @crawlready/middleware README for the full configuration and fail-safe behavior.
Middleware Configuration#
All four middleware adapters (@crawlready/middleware/next, /express, /cloudflare, and /next for Vercel Edge) accept the same configuration options:
| Option | Type | Default | Description |
|---|---|---|---|
siteKey | string | required | Your site key from the CrawlReady dashboard (format: cr_live_*) |
botFilter | 'standard' | 'ai-only' | 'all' | 'standard' | 'standard' tracks AI crawlers plus search engines and SEO tools (Googlebot, Bingbot, AhrefsBot, …); 'ai-only' tracks AI crawlers only; 'all' tracks all 86+ known bots including social media |
endpoint | string | CrawlReady ingest URL | Override the ingest endpoint (useful for testing) |
onBotDetected | function | — | Called synchronously before each ingest event; return false to suppress it |
indexNowKey | string | — | Serve the IndexNow key file at /{key}.txt — proves domain ownership to Bing and the other IndexNow engines. Free on every plan, independent of optimize. |
optimize | boolean | false | Serve optimized content to eligible AI crawlers. Requires a paid plan. Also stops the SDK's own analytics beacon — the edge emits a richer event instead. |
edgeSecret | string | — | Signs optimized-serving requests. Server-only — never inline in client code. Required when optimize is true. |
edgeEndpoint | string | https://edge.crawlready.app | Override the edge origin. |
serveTimeoutMs | number | 1500 | Timeout for the edge serve request, kept below the ~2s at which crawlers give up. |
excludePaths | string[] | [] | Glob patterns (*, **) never sent to the edge for optimized serving. |
onServe | function | — | Observability hook called after each serve attempt. Thrown errors are swallowed. |
Frequently Asked Questions
What is the AI Readiness Score?
A single 0-100 score that measures how ready a site is for AI crawlers and agents. It combines three sub-scores — Crawlability (50% weight), Agent Readiness (25%), and Agent Interaction (25%) — with a floor rule: the overall score cannot exceed 60 if any sub-score falls below 20, regardless of how well the other two perform.
What is a site key?
A unique token in the format cr_live_* that identifies your site for analytics ingest. It's generated automatically when you register a domain and gets passed to the middleware, which includes it in every ingest event sent to /api/v1/ingest. A site key is not a secret, but it is unique per domain and rate-limited — it's different from an API key.
What is the difference between a site key and an API key?
A site key identifies your site and is included in ingest events; it is not a secret. An API key is a Bearer token that authenticates you as a user for API access — running scans, managing sites, listing results — and is created from the dashboard or via POST /api/v1/api-keys, with the full secret shown only once at creation. In short: API keys authenticate you, site keys identify your site.
What is the difference between middleware and script tag integration?
Middleware runs server-side on every inbound request, reading the User-Agent header before the application handles it, so it captures crawlers that never execute JavaScript — roughly 100% of bot traffic. A script tag runs client-side after the page loads in a browser and only fires when JavaScript executes successfully, so any crawler without a JS engine, which is most AI crawlers, is invisible to it. The script tag should only be used when server-side middleware can't be deployed.
