What Crawlability measures
Weight: 40% of the AI Readiness Score
Crawlability answers the question: can AI crawlers see your content?
CrawlReady fetches your page twice — once as a standard browser (JavaScript executed) and once as a bot that does not execute JavaScript. The Crawlability score is based entirely on what the bot-view fetch returns. If content only exists in the rendered DOM, it is invisible to crawlers like GPTBot and ClaudeBot.
The score has four checks.
C1 — Content Visibility
Compares the amount of readable text in the bot-view response to the amount in the fully-rendered page.
| Bot text / Rendered text | What the scanner checks |
|---|---|
| ≥ 90% | Bot-view text is at least 90% of rendered text — nearly all content visible to crawlers |
| 70–89% | Most content visible, but some sections depend on JavaScript |
| 50–69% | Significant content missing from the bot-view response |
| 20–49% | Majority of content is JavaScript-rendered only |
| < 20% | Content is effectively invisible to AI crawlers |
If the bot fetch returns a non-200 status, or if the rendered page contains fewer than 50 characters of text, this check scores 0.
What the scanner sees:
The scanner fetches your page without executing JavaScript. Content that exists only after JavaScript runs is invisible to this check.
<!-- Scanner sees nothing — content loads via JavaScript -->
<div id="content"></div>
<script>fetch('/api/content').then(r => r.json()).then(d => render(d))</script>
<!-- Scanner sees the content — it's in the initial HTML -->
<div id="content">
<h1>Your actual headline</h1>
<p>Your actual content in the HTML response.</p>
</div>
Framework examples:
In a Next.js Server Component, the content is rendered on the server and included in the HTML response. The scanner sees everything:
// Server Component (default in App Router) — scanner sees all content
export default async function Page() {
const data = await fetchContent();
return (
<main>
<h1>{data.title}</h1>
<p>{data.body}</p>
</main>
);
}
In a React SPA with client-side data fetching, the scanner sees an empty shell:
// Client component with useEffect — scanner sees only the loading state
'use client';
export default function Page() {
const [data, setData] = useState(null);
useEffect(() => { fetch('/api/content').then(r => r.json()).then(setData); }, []);
if (!data) return <div>Loading...</div>;
return <main><h1>{data.title}</h1></main>;
}
The scanner would see <div>Loading...</div> — none of the actual content.
C2 — Structural Clarity
Checks that the bot-view HTML has clear, well-organized structure that AI systems can parse.
| Signal | What the scanner checks |
|---|---|
Exactly one <h1> | Bot-view HTML contains a single <h1> element |
| Heading hierarchy present | <h1> followed by at least one <h2>, with no skipped levels (e.g. no <h1> to <h3>) |
At least 3 <p> elements with more than 20 characters of text | Bot-view has substantive paragraph content, not just stubs |
At least one <ul>, <ol>, or <table> | Structured list or table content is present in the bot-view HTML |
<meta name="description"> with non-empty content | A non-empty meta description tag exists in <head> |
What scores well vs. what doesn't:
<!-- Scores well: clear hierarchy, substantive content -->
<head>
<meta name="description" content="A concise page description.">
</head>
<body>
<h1>One clear page title</h1>
<h2>First section</h2>
<p>Substantive paragraph content here — more than 20 characters.</p>
<h2>Second section</h2>
<ul>
<li>Structured list content</li>
</ul>
</body>
<!-- Scores poorly: multiple h1s, skipped heading levels, no structure -->
<body>
<h1>Title</h1>
<h1>Subtitle</h1>
<h3>Section</h3>
<div>Content as divs, not paragraphs.</div>
</body>
C3 — Noise Ratio
Measures the ratio of readable text tokens to total HTML tokens. High noise (scripts, inline styles, and markup) means crawlers spend more work extracting less signal.
The noise ratio is: 1 - (content tokens / total HTML tokens).
| Noise ratio | What the scanner checks |
|---|---|
| < 60% | Content dominates the HTML payload — clean and efficient |
| 60–74% | Moderate noise — typical for well-built sites |
| 75–89% | Scripts, styles, and nav markup dominate — content is hard to extract |
| ≥ 90% | Extremely noisy — content is buried under markup |
What affects the ratio:
The noise ratio measures content tokens vs. total HTML tokens. High noise means scripts, styles, and markup dominate the payload.
Common sources of high noise:
- Inline
<script>blocks (vs. external.jsfiles withsrc) - Large
<style>blocks (vs. external.cssfiles) - Unused CSS classes and data attributes in the HTML
- Base64-encoded images or blobs inline in the markup
C4 — Schema.org Presence
Checks whether the bot-view HTML contains Schema.org JSON-LD structured data.
| Signal | What the scanner checks |
|---|---|
Any JSON-LD <script type="application/ld+json"> present | At least one JSON-LD script block exists in the bot-view HTML |
JSON-LD has a valid @type property | The JSON-LD parses as valid JSON and includes an @type field |
@type is a rich content type | Type is Product, FAQPage, HowTo, SoftwareApplication, Organization, Article, or WebPage |
| Multiple schemas or rich detail | Two or more Schema types present, or one type with more than 5 properties |
What the scanner looks for:
The scanner checks bot-view HTML for <script type="application/ld+json"> blocks.
<!-- Scores well: valid JSON-LD with a rich content type and multiple properties -->
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "Your Product",
"description": "What it does.",
"applicationCategory": "BusinessApplication",
"offers": {
"@type": "Offer",
"price": "29",
"priceCurrency": "USD"
}
}
</script>
Rich content types (Product, FAQPage, HowTo, SoftwareApplication, Organization, Article, WebPage) score higher than generic types. Multiple schemas or a single schema with more than 5 properties score highest.