How We Built the AI Crawler Checker

Introduction#

When we launched CrawlReady, we knew we needed a way to demonstrate the JavaScript rendering problem to potential users. That's why we built the AI Crawler Checker—a free tool that analyzes any website's compatibility with major AI crawlers.

In this post, I'll share the technical architecture behind the tool and some of the interesting challenges we solved.

The Core Problem#

To accurately assess AI crawler compatibility, we need to answer several questions:

1. Does the site rely on JavaScript for content?
2. What content is visible without JavaScript execution?
3. What structured data (Schema.org) is present?
4. Are there any crawler-blocking mechanisms?

Each of these requires different analysis techniques.

Architecture Overview#

The crawler checker consists of three main components:

1. Dual-fetch system - Fetches pages with and without JavaScript
2. Content analyzer - Compares rendered vs. raw HTML
3. Schema parser - Extracts and validates structured data

Dual-Fetch System#

We make two requests to each URL:

// Simple HTTP fetch (what most AI crawlers do)
const rawResponse = await fetch(url, {
  headers: {
    'User-Agent': 'PerplexityBot/1.0'
  }
});
const rawHtml = await rawResponse.text();

// Headless browser fetch (full JavaScript execution)
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle0' });
const renderedHtml = await page.content();

By comparing these two responses, we can determine how much content depends on JavaScript.

Content Analysis#

We extract text content from both versions and calculate a "content delta":

const rawText = extractTextContent(rawHtml);
const renderedText = extractTextContent(renderedHtml);

const contentDelta = {
  rawWordCount: rawText.split(/\s+/).length,
  renderedWordCount: renderedText.split(/\s+/).length,
  percentageIncrease: calculateIncrease(rawText, renderedText)
};

If the rendered version has significantly more content, the site relies heavily on JavaScript.

Schema.org Analysis#

We parse both JSON-LD and microdata formats:

// JSON-LD extraction
const jsonLdScripts = document.querySelectorAll(
  'script[type="application/ld+json"]'
);

// Microdata extraction
const microdataItems = document.querySelectorAll('[itemscope]');

We then validate the schemas against Schema.org specifications and check for AI-friendly properties.

Crawler Simulation#

Different AI crawlers have different capabilities. We simulate each one:

For each crawler, we adjust our simulation parameters accordingly.

Scoring Algorithm#

We calculate a 0-100 AI Readiness Score based on three dimensions:

• Crawlability (50%) - How much content is visible without JavaScript, structural clarity, noise ratio, and Schema.org presence
• Agent Readiness (25%) - Structured data quality, content negotiation, machine-actionable data, and standards adoption
• Agent Interaction (25%) - Semantic HTML, accessibility, navigation structure, and visual-semantic consistency

const aiReadinessScore = 
  crawlabilityScore * 0.5 +
  agentReadinessScore * 0.25 +
  agentInteractionScore * 0.25;

Each dimension consists of 4 checks (C1-C4, A1-A4, I1-I4) that evaluate specific aspects of your site's AI readiness. For the complete breakdown, see our scoring documentation.

Performance Optimizations#

Running headless browsers is expensive. We implemented several optimizations:

1. Request queuing - BullMQ manages concurrent browser instances
2. Caching - Results cached in Redis for 24 hours
3. Resource blocking - Skip images, fonts, and tracking scripts
4. Timeout management - Aggressive timeouts prevent hanging

Try It Yourself#

CrawlReady is free to start. Sign up to:

• Get your AI Readiness Score with detailed check-by-check analysis
• Track which AI crawlers visit your site
• Monitor your score over time
• Receive alerts when AI crawler traffic drops

If you're building something similar or have questions about the technical details, find me on Twitter @medartus.