Express

Middleware Setup#

Install the package:

npm install @crawlready/middleware
# or
pnpm add @crawlready/middleware

Add the crawlready middleware to your Express app before your route handlers:

import express from 'express';
import { crawlready } from '@crawlready/middleware/express';

const app = express();

app.use(crawlready({
  siteKey: process.env.CRAWLREADY_SITE_KEY!,
}));

// Your routes follow here
app.get('/', (req, res) => {
  res.send('Hello world');
});

app.listen(3000);

Add your site key to your environment:

CRAWLREADY_SITE_KEY=cr_live_xxxxx

The middleware detects AI crawlers from the User-Agent header and fires an analytics beacon in the background. It calls next() immediately and never delays the response.

For all configuration options, see Core Concepts — Middleware Configuration.

Instant indexing (optional)#

Add indexNowKey to publish new and changed pages to Bing — and through Bing's index, ChatGPT search and Microsoft Copilot — within minutes instead of waiting for a crawl. Free on every plan, no daily limit.

app.use(crawlready({
  siteKey: process.env.CRAWLREADY_SITE_KEY!,
  indexNowKey: process.env.CRAWLREADY_INDEXNOW_KEY!,
}));

Get your key from Indexing → Settings in the dashboard. crawlready answers the verification request itself once the key is set — nothing to deploy separately. See Get Indexed by Bing, ChatGPT & Copilot for the full setup.

Verify Your Integration#

After deploying, use the Verify Integration button in your CrawlReady dashboard. It sends a synthetic GPTBot request to your site and confirms that a visit is recorded within 15 seconds.

If the check times out:

Confirm the middleware is registered before your route handlers.
Confirm CRAWLREADY_SITE_KEY is set in the environment where the process runs.
If your app is behind a reverse proxy, ensure the User-Agent header is forwarded through.

Frequently Asked Questions

Does the Express middleware delay the response to visitors?

No. The crawlready middleware calls next() immediately and fires the analytics beacon in the background, so it never delays the response.

Where in my Express app should the middleware go?

Add app.use(crawlready({ siteKey: ... })) before your route handlers, so it runs on every request before your routes process it.

Will CrawlReady still detect crawlers if my app is behind a reverse proxy?

It depends on the User-Agent header being forwarded through. If the Verify Integration check times out and your app sits behind a reverse proxy, confirm the proxy forwards the original User-Agent header to your Express process.

What should I check if the Verify Integration check times out?

Confirm the middleware is registered before your route handlers, confirm CRAWLREADY_SITE_KEY is set in the environment where the process runs, and if your app is behind a reverse proxy, ensure the User-Agent header is forwarded through.