Quickstart

1. Sign up and register your site#

Sign in at www.crawlready.app and register your domain. Registration calls POST /api/v1/sites and returns a site key.

curl -X POST https://www.crawlready.app/api/v1/sites \
  -H "Authorization: Bearer <your-api-key>" \
  -H "Content-Type: application/json" \
  -d '{ "domain": "example.com" }'

Response:

{
  "id": "...",
  "domain": "example.com",
  "site_key": "cr_live_abc123def456",
  "created_at": "2026-05-18T00:00:00.000Z"
}

Copy the site_key. You'll use it in the next step.

These steps use Next.js. For other platforms: Express | Cloudflare Workers | Vercel Edge | Script Tag

2. Install the middleware package#

npm install @crawlready/middleware
# or
pnpm add @crawlready/middleware

3. Add the middleware to your Next.js app#

Create or update middleware.ts at the root of your project:

// middleware.ts
import { withCrawlReady } from '@crawlready/middleware/next';

export default withCrawlReady({
  siteKey: process.env.CRAWLREADY_SITE_KEY!,
});

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

If you already have a Next.js middleware (for example, Clerk), compose them:

// middleware.ts
import { withCrawlReady } from '@crawlready/middleware/next';
import { clerkMiddleware } from '@clerk/nextjs/server';

export default withCrawlReady(
  { siteKey: process.env.CRAWLREADY_SITE_KEY! },
  clerkMiddleware(),
);

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

4. Set the environment variable#

# .env.local
CRAWLREADY_SITE_KEY=cr_live_abc123def456

5. (Optional) Turn on instant indexing#

Add indexNowKey to the same config object to publish new and changed pages to Bing — and through Bing's index, ChatGPT search and Microsoft Copilot — within minutes instead of waiting for a crawl. Free on every plan, no daily limit.

export default withCrawlReady({
  siteKey: process.env.CRAWLREADY_SITE_KEY!,
  indexNowKey: process.env.CRAWLREADY_INDEXNOW_KEY!,
});

Get your key from Indexing → Settings in the dashboard — the middleware answers the verification request itself once the key is set, nothing else to deploy. See Get Indexed by Bing, ChatGPT & Copilot for the full setup.

6. Verify#

Deploy your site. Check the analytics dashboard — visits from AI crawlers should appear within seconds of the first detected bot request.

To test locally, send a request with a known bot User-Agent:

curl http://localhost:3000/ \
  -H "User-Agent: GPTBot/1.0"

The middleware detects GPTBot, calls POST /api/v1/ingest with your site key, and your dashboard records the visit.

How the ingest call works#

The middleware sends a fire-and-forget POST to /api/v1/ingest. It never blocks your response. The payload looks like:

{
  "s": "cr_live_abc123def456",
  "p": "/blog/my-article",
  "b": "GPTBot",
  "t": 1747526400000,
  "src": "middleware"
}

Fields: s = site key, p = URL path, b = bot name, t = Unix timestamp in milliseconds, src = integration source. The endpoint always returns 204 No Content — errors are silent to avoid leaking information.