What is ChatGPT search, mechanically?#
ChatGPT search lets ChatGPT retrieve current information from the live web to answer a query, instead of relying only on what it learned during training. Two systems feed it: a partnership with Microsoft that surfaces results from the Bing index, and OpenAI's own crawler, which independently fetches and evaluates pages for inclusion in search answers. Because ChatGPT search leans on Bing's index for a meaningful share of its results, being indexed by Bing carries over — see how AI assistants index your site and setting up Bing and ChatGPT indexing for the practical setup.
Which bots does OpenAI use, and what does each one do?#
OpenAI documents three separate crawlers, each with a distinct job. Treating them as interchangeable is the single most common configuration mistake:
| Bot | Job | Effect of blocking it |
|---|---|---|
OAI-SearchBot | Crawls and indexes content specifically for ChatGPT search results | Removes the page from ChatGPT search answers; it can still appear as a bare navigational link, per OpenAI |
ChatGPT-User | Fetches a specific page in real time when a user or agent asks ChatGPT to look at it during a conversation | No effect on search inclusion — OpenAI states directly it "is not used to determine whether content may appear in Search" |
GPTBot | Crawls content for OpenAI's model training data | No effect on ChatGPT search visibility — a separate purpose entirely |
OpenAI also documents that robots.txt changes take roughly 24 hours to take effect, and publishes OAI-SearchBot's IP ranges at openai.com/searchbot.json so you can verify traffic claiming to be it. The costly mistake worth naming explicitly: blocking OAI-SearchBot while intending only to opt out of training data removes the page from ChatGPT search entirely — the bot to block for that goal is GPTBot.
How does content negotiation affect what ChatGPT search sees?#
Content negotiation is serving a different format of the same page depending on what the requesting client asks for, via the HTTP Accept header — text/html gets one response, text/markdown gets another, from the same URL. It's a real, standards-based mechanic, not a myth: a server can inspect the Accept header on an incoming request and respond accordingly.
CrawlReady's own crawl pipeline uses an HTML-first version of this: optimized, noise-stripped HTML — with ARIA landmarks and structured data — is served by default to every eligible crawler, and a markdown alternate is served only when a request explicitly carries Accept: text/markdown. That default exists because the crawlers that matter most for AI Overview-style citation (Google's rendering pipeline in particular) get more value from enriched HTML than from markdown, and markdown-only serving would leave that channel unaddressed. This is CrawlReady's own architecture decision, not a claim about how every AI product behaves — other products may default differently.
Do AI bots prefer markdown over HTML?#
Less than the current wave of "serve markdown to AI crawlers" advice implies. Practitioner crawl telemetry — not an official OpenAI or Google statement, so treat it as corroborating rather than definitive — offers a useful correction here. TrustMRR's crawl analysis, covering more than 1 million AI bot requests over 30 days and surfaced in August 2026, found that when a site offers a markdown alternate via content negotiation, AI bots — including OpenAI's — still fetch the HTML version roughly half the time. That figure is self-reported from a single product's traffic, not an independently audited industry study, and should be cited with that caveat every time.
Two things follow from this. First, markdown negotiation is a real, additive distribution surface for the requests that do ask for it — not a myth to dismiss. Second, it isn't a substitute for making the default HTML response itself crawlable and content-complete, since a large share of AI bot traffic never requests the markdown version at all. This is the same conclusion CrawlReady's product architecture already reflects: render-visible HTML first, markdown negotiation as an addition on top of it, not instead of it.
How does ChatGPT decide what to cite?#
OpenAI hasn't published a full ranking algorithm for ChatGPT search citations the way Google has for AI Overviews. What is documented: a page has to be crawlable by OAI-SearchBot to be a candidate at all, and Bing indexing status feeds into results sourced from that partnership. Beyond that, the same fundamentals that make a page usable for any retrieval-based AI answer apply — a clear, accurate, self-contained answer near the top of the relevant section, and a page that doesn't depend on JavaScript execution to reveal its content.
Run a free scan to check whether your robots.txt correctly allows OAI-SearchBot and whether your content reaches a non-JavaScript crawler intact.
