Cloudflare Workers

Middleware Setup

Install the package:

npm install @crawlready/middleware
# or
pnpm add @crawlready/middleware

Wrap your existing Workers handler with withCrawlReady:

import { withCrawlReady } from '@crawlready/middleware/cloudflare';

const handler = {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    return new Response('Hello world');
  },
};

export default withCrawlReady(
  { siteKey: 'cr_live_xxxxx' },
  handler,
);

Set your site key as a Workers secret rather than hardcoding it:

wrangler secret put CRAWLREADY_SITE_KEY

Then reference it from your env object:

export default withCrawlReady(
  { siteKey: (env as { CRAWLREADY_SITE_KEY: string }).CRAWLREADY_SITE_KEY },
  handler,
);

withCrawlReady detects AI crawlers from the User-Agent header and uses ctx.waitUntil() to send the analytics beacon after the response is returned — the visitor never waits for it.

Configuration options

| Option | Type | Default | Description | |---|---|---|---| | siteKey | string | required | Your site key from the CrawlReady dashboard (format: cr_live_*) | | botFilter | 'ai-only' \| 'all' | 'ai-only' | Track only AI crawlers, or all 86+ known bots | | endpoint | string | CrawlReady ingest URL | Override the ingest endpoint (useful for testing) | | onBotDetected | function | — | Called before each ingest event; return false to suppress it |

Verify Your Integration

After deploying, use the Verify Integration button in your CrawlReady dashboard. It sends a synthetic GPTBot request to your site and confirms that a visit is recorded within 15 seconds.

If the check times out:

• Confirm the worker is deployed and your domain is routing through it.
• Confirm CRAWLREADY_SITE_KEY is set as a Workers secret (not just in wrangler.toml).
• Check the Workers logs in the Cloudflare dashboard for errors.

Going Further

• API Reference — trigger scans and retrieve scores programmatically
• Analytics Dashboard — understand your crawler traffic data
• Analytics Overview — what the metrics mean