Crawlability

What Crawlability measures#

Weight: 50% of the AI Readiness Score

Crawlability answers the question: can AI crawlers see your content?

CrawlReady fetches your page twice — once as a standard browser (JavaScript executed) and once as a bot that does not execute JavaScript. The Crawlability score is based entirely on what the bot-view fetch returns. If content only exists in the rendered DOM, it is invisible to crawlers like GPTBot and ClaudeBot.

The score has four checks.

C1 — Content Visibility#

Compares the amount of readable text in the bot-view response to the amount in the fully-rendered page.

Bot text / Rendered text	What the scanner checks
≥ 90%	Bot-view text is at least 90% of rendered text — nearly all content visible to crawlers
70–89%	Most content visible, but some sections depend on JavaScript
50–69%	Significant content missing from the bot-view response
20–49%	Majority of content is JavaScript-rendered only
< 20%	Content is effectively invisible to AI crawlers

If the bot fetch returns a non-200 status, or if the rendered page contains fewer than 50 characters of text, this check scores 0.

What the scanner sees:

The scanner fetches your page without executing JavaScript. Content that exists only after JavaScript runs is invisible to this check.

<!-- Scanner sees nothing — content loads via JavaScript -->
<div id="content"></div>
<script>fetch('/api/content').then(r => r.json()).then(d => render(d))</script>

<!-- Scanner sees the content — it's in the initial HTML -->
<div id="content">
  <h1>Your actual headline</h1>
  <p>Your actual content in the HTML response.</p>
</div>

Framework examples:

In a Next.js Server Component, the content is rendered on the server and included in the HTML response. The scanner sees everything:

// Server Component (default in App Router) — scanner sees all content
export default async function Page() {
  const data = await fetchContent();
  return (
    <main>
      <h1>{data.title}</h1>
      <p>{data.body}</p>
    </main>
  );
}

In a React SPA with client-side data fetching, the scanner sees an empty shell:

// Client component with useEffect — scanner sees only the loading state
'use client';
export default function Page() {
  const [data, setData] = useState(null);
  useEffect(() => { fetch('/api/content').then(r => r.json()).then(setData); }, []);
  if (!data) return <div>Loading...</div>;
  return <main><h1>{data.title}</h1></main>;
}

The scanner would see <div>Loading...</div> — none of the actual content.

C2 — Structural Clarity#

Checks that the bot-view HTML has clear, well-organized structure that AI systems can parse.

Signal	What the scanner checks
Exactly one `<h1>`	Bot-view HTML contains a single `<h1>` element
Heading hierarchy present	`<h1>` followed by at least one `<h2>`, with no skipped levels (e.g. no `<h1>` to `<h3>`)
At least 3 `<p>` elements with more than 20 characters of text	Bot-view has substantive paragraph content, not just stubs
At least one `<ul>`, `<ol>`, or `<table>`	Structured list or table content is present in the bot-view HTML
`<meta name="description">` with non-empty content	A non-empty meta description tag exists in `<head>`

What scores well vs. what doesn't:

<!-- Scores well: clear hierarchy, substantive content -->
<head>
  <meta name="description" content="A concise page description.">
</head>
<body>
  <h1>One clear page title</h1>
  <h2>First section</h2>
  <p>Substantive paragraph content here — more than 20 characters.</p>
  <h2>Second section</h2>
  <ul>
    <li>Structured list content</li>
  </ul>
</body>

<!-- Scores poorly: multiple h1s, skipped heading levels, no structure -->
<body>
  <h1>Title</h1>
  <h1>Subtitle</h1>
  <h3>Section</h3>
  <div>Content as divs, not paragraphs.</div>
</body>

C3 — Noise Ratio#

Measures the ratio of readable text tokens to total HTML tokens. High noise (scripts, inline styles, and markup) means crawlers spend more work extracting less signal.

The noise ratio is: 1 - (content tokens / total HTML tokens).

Noise ratio	What the scanner checks
< 60%	Content dominates the HTML payload — clean and efficient
60–74%	Moderate noise — typical for well-built sites
75–89%	Scripts, styles, and nav markup dominate — content is hard to extract
≥ 90%	Extremely noisy — content is buried under markup

What affects the ratio:

The noise ratio measures content tokens vs. total HTML tokens. High noise means scripts, styles, and markup dominate the payload.

Common sources of high noise:

Inline <script> blocks (vs. external .js files with src)
Large <style> blocks (vs. external .css files)
Unused CSS classes and data attributes in the HTML
Base64-encoded images or blobs inline in the markup

C4 — Schema.org Presence#

Checks whether the bot-view HTML contains Schema.org JSON-LD structured data.

Signal	What the scanner checks
Any JSON-LD `<script type="application/ld+json">` present	At least one JSON-LD script block exists in the bot-view HTML
JSON-LD has a valid `@type` property	The JSON-LD parses as valid JSON and includes an `@type` field
`@type` is a rich content type	Type is Product, FAQPage, HowTo, SoftwareApplication, Organization, Article, or WebPage
Multiple schemas or rich detail	Two or more Schema types present, or one type with more than 5 properties

What the scanner looks for:

The scanner checks bot-view HTML for <script type="application/ld+json"> blocks.

<!-- Scores well: valid JSON-LD with a rich content type and multiple properties -->
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Your Product",
  "description": "What it does.",
  "applicationCategory": "BusinessApplication",
  "offers": {
    "@type": "Offer",
    "price": "29",
    "priceCurrency": "USD"
  }
}
</script>

Rich content types (Product, FAQPage, HowTo, SoftwareApplication, Organization, Article, WebPage) score higher than generic types. Multiple schemas or a single schema with more than 5 properties score highest.

Frequently Asked Questions

What does the Crawlability score measure?

Crawlability measures whether AI crawlers can see a site's content. CrawlReady fetches the page twice — once as a standard browser with JavaScript executed, and once as a bot that does not execute JavaScript — and the Crawlability score is based entirely on what that bot-view fetch returns. Content that only exists in the rendered DOM is invisible to crawlers like GPTBot and ClaudeBot.

What are the four checks in the Crawlability score?

C1 Content Visibility compares the amount of readable text in the bot-view response to the fully-rendered page. C2 Structural Clarity checks heading hierarchy, paragraph content, and a meta description. C3 Noise Ratio measures readable text tokens against total HTML tokens. C4 Schema.org Presence checks for JSON-LD structured data in the bot-view HTML.

Why does a JavaScript-only site score near zero on Crawlability?

A site that relies entirely on client-side rendering sends AI crawlers an empty shell, since those crawlers fetch the page without executing JavaScript. The Crawlability score is based entirely on that bot-view fetch, so such a site typically scores near zero on this dimension regardless of how well everything else is set up.

What counts as good content visibility in the Content Visibility check?

C1 compares the amount of readable text in the bot-view response to the amount in the fully-rendered page. Bot-view text at 90% or more of rendered text means nearly all content is visible to crawlers, while under 20% means content is effectively invisible. The check scores 0 if the bot fetch returns a non-200 status or the rendered page has fewer than 50 characters of text.