Cloudflare Workers

Middleware Setup#

Install the package:

npm install @crawlready/middleware
# or
pnpm add @crawlready/middleware

Wrap your existing Workers handler with withCrawlReady:

import { withCrawlReady } from '@crawlready/middleware/cloudflare';

const handler = {
  async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise<Response> {
    return new Response('Hello world');
  },
};

export default withCrawlReady(
  { siteKey: 'cr_live_xxxxx' },
  handler,
);

Set your site key as a Workers secret rather than hardcoding it:

wrangler secret put CRAWLREADY_SITE_KEY

Then reference it from your env object:

export default withCrawlReady(
  { siteKey: (env as { CRAWLREADY_SITE_KEY: string }).CRAWLREADY_SITE_KEY },
  handler,
);

withCrawlReady detects AI crawlers from the User-Agent header and uses ctx.waitUntil() to send the analytics beacon after the response is returned — the visitor never waits for it.

For all configuration options, see Core Concepts — Middleware Configuration.

Instant indexing (optional)#

Add indexNowKey to publish new and changed pages to Bing — and through Bing's index, ChatGPT search and Microsoft Copilot — within minutes instead of waiting for a crawl. Free on every plan, no daily limit.

export default withCrawlReady(
  {
    siteKey: (env as { CRAWLREADY_SITE_KEY: string }).CRAWLREADY_SITE_KEY,
    indexNowKey: (env as { CRAWLREADY_INDEXNOW_KEY: string }).CRAWLREADY_INDEXNOW_KEY,
  },
  handler,
);

Get your key from Indexing → Settings in the dashboard and store it the same way as your site key — wrangler secret put CRAWLREADY_INDEXNOW_KEY. withCrawlReady answers the verification request itself once the key is set — nothing to deploy separately. See Get Indexed by Bing, ChatGPT & Copilot for the full setup.

Verify Your Integration#

After deploying, use the Verify Integration button in your CrawlReady dashboard. It sends a synthetic GPTBot request to your site and confirms that a visit is recorded within 15 seconds.

If the check times out:

• Confirm the worker is deployed and your domain is routing through it.
• Confirm CRAWLREADY_SITE_KEY is set as a Workers secret (not just in wrangler.toml).
• Check the Workers logs in the Cloudflare dashboard for errors.