About CrawlReady — The Web Split in Two

I took a popular SaaS landing page — beautiful design, smooth animations, pricing table that converts like crazy — and I looked at it the way GPTBot does.

Not through a browser. Through raw HTML.

What I saw was <div id="root"></div> and three script tags. That's it. The entire pricing section, the feature comparison, the testimonials — all of it was invisible. GPTBot would never know this company sells anything at all.

I checked another site. Same thing. And another. Server-rendered this time, so the content was technically there — buried inside 14,000 tokens of navigation, cookie banners, analytics trackers, and chat widgets. The actual content? About 800 tokens. Ninety-four percent noise.

That's when I realized: the web has split in two. And most people have no idea it's happening.

The Numbers That Keep Me Up at Night

I'm a developer. I've built SPAs. I've built SSR apps. I've shipped React, Next.js, you name it. And like most developers, I assumed that if a human could see my content, everyone could see it.

Then I started paying attention to the data.

-38%

Organic Google referrals YoY

-60%

Small publisher search traffic

93%

AI search sessions end without a site visit

+527%

AI-sourced traffic surge

The citation happens at crawl time. If the crawler saw nothing, there's nothing to cite. Your content doesn't show up in ChatGPT. It doesn't show up in Perplexity. It doesn't show up in Claude. Not because your content isn't good enough — because the crawler literally couldn't see it.

And the sites that arecited? They get 20-40% more clicks. The sites that aren't? They lose 15-25%.

Why I Built This

I could have built a lot of things. I chose this because of three observations that, once I saw them, I couldn't unsee.

The problem is invisible by definition

You can't see what AI crawlers see. There's no Chrome DevTools for GPTBot. There's no "view source" for ClaudeBot. The problem is literally invisible — which means most people don't even know they have it. I wanted to make it visible. That's the diagnostic.

The fix shouldn't require trust you haven't earned

There are companies that will happily deploy an edge proxy on your domain and route your traffic through their infrastructure. Day one. Before you've seen any results. I don't want to ask for that kind of trust upfront. CrawlReady starts with a diagnostic — zero risk, zero integration, zero DNS changes. You see the problem first. Then, if you want to fix it, you add a 10-line middleware snippet to your infrastructure. You stay in control.

Transparency isn't optional — it's the product

If you're serving different content to bots than to humans, people will ask: "Is this cloaking?" Google themselves blessed this pattern for years — they called it "dynamic rendering." But I wanted to go further. Every CrawlReady-optimized site exposes a public endpoint where anyonecan verify what AI crawlers receive. A diff engine monitors for content divergence. You can't drift from the truth without getting caught.

The Bigger Picture

CrawlReady starts as a diagnostic. But the vision is bigger — the AI interface layer for the web. The infrastructure site owners use to manage their relationship with AI systems. That relationship is only going to get more complex.

AI Search

$1.5B market, 40% CAGR

Can crawlers see your content?

Agentic AI

$7.6B market, 40% CAGR

Can agents act on your content?

AI Compliance

EU AI Act — August 2, 2026

Can you prove transparency?

Traffic Defense

Happening right now

Can you stop losing traffic to competitors who ARE cited?

Same underlying architecture. Same customer. Different entry points depending on what keeps them up at night. Different doors. Same house.

The Honest Part

I'm building this solo. Nights and weekends. About 15-20 hours a week alongside my day job. That means everything takes 2-3x longer. It means I have to be ruthless about scope. I'm building one thing: making the AI visibility problem visible, and then fixable.

I've spent my career building for the web. I care about this medium. And I'm watching it bifurcate into two versions — one that humans experience, one that AI systems scrape — and most of the people who build websites have no idea it's happening.

The web has always been about access. About making information available. About the idea that if you publish something, people can find it. That social contract is breaking. Not because anyone intended it, but because the way we build websites evolved in one direction and the way AI systems consume them evolved in another.

I don't think this is anyone's fault. But I do think someone should help site owners see it and fix it. That's what CrawlReady is for.

— Marc-Etienne

Follow @medartus

The Web Split in Two.Most People Don't Know It.