Next.js

Middleware Setup

Install the package:

npm install @crawlready/middleware
# or
pnpm add @crawlready/middleware

Create middleware.ts in your project root:

// middleware.ts
import { withCrawlReady } from '@crawlready/middleware/next';

export default withCrawlReady({
  siteKey: process.env.CRAWLREADY_SITE_KEY!,
});

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

Add your site key to .env.local:

CRAWLREADY_SITE_KEY=cr_live_xxxxx

For Vercel deployments, add CRAWLREADY_SITE_KEY in your project's Environment Variables settings.

Composing with existing middleware

If you already have middleware (e.g. Clerk), pass it as the second argument:

// middleware.ts
import { withCrawlReady } from '@crawlready/middleware/next';
import { clerkMiddleware } from '@clerk/nextjs/server';

export default withCrawlReady(
  { siteKey: process.env.CRAWLREADY_SITE_KEY! },
  clerkMiddleware(),
);

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

withCrawlReady detects AI bots, fires an analytics beacon via event.waitUntil() (so it never blocks the response), then hands off to your existing middleware.

Configuration options

| Option | Type | Default | Description | |---|---|---|---| | siteKey | string | required | Your site key from the CrawlReady dashboard (format: cr_live_*) | | botFilter | 'ai-only' \| 'all' | 'ai-only' | Track only AI crawlers, or all 86+ known bots | | endpoint | string | CrawlReady ingest URL | Override the ingest endpoint (useful for testing) | | onBotDetected | function | — | Called before each ingest event; return false to suppress it |

Verify Your Integration

After deploying, use the Verify Integration button in your CrawlReady dashboard. It sends a synthetic GPTBot request to your site and confirms that a visit is recorded within 15 seconds.

If the check times out:

• Confirm middleware.ts is in your project root (or src/ directory if you use that layout).
• Confirm the environment variable is set in your deployment environment, not just .env.local.
• Check that your matcher pattern covers the paths being tested.

Going Further

• API Reference — trigger scans and retrieve scores programmatically
• Analytics Dashboard — understand your crawler traffic data
• Analytics Overview — what the metrics mean