Google's AI Overviews and OpenAI's ChatGPT search are the two most-searched AI answer engines, and both publish real documentation about how they work — covered on Google AI Overviews and how ChatGPT search works. The rest of the AI search landscape — Perplexity, Microsoft Copilot, Gemini's search grounding, Claude's web search — matters too, but each sources citations differently, and none of them publish a ranking methodology as detailed as Google's. This page covers what's documented for each, and is explicit about what isn't.
Which AI search engines have their own crawlers, and which read a shared index?#
| Engine | How it finds pages | Direct submission available? |
|---|---|---|
| Google AI Overviews / AI Mode | Google's own Search index, via retrieval-augmented generation and query fan-out | Yes — standard Google indexing (see how AI assistants index your site) |
| ChatGPT search | Bing index, plus OpenAI's own crawler, OAI-SearchBot | Yes, via Bing |
| Microsoft Copilot | The Bing index | Yes, via Bing — the same setup that covers ChatGPT search |
| Gemini (when it searches) | Google Search results — same prerequisite as AI Overviews | Yes — standard Google indexing |
| Perplexity | Its own crawler (PerplexityBot), blended with third-party indexes | No direct push; crawlability and being indexed elsewhere both help |
| Claude (when it searches) | Third-party search results | No direct push |
The practical split: two setups — pushing to Google and pushing to Bing — cover Google AI Overviews, Gemini, ChatGPT search, and Copilot between them, because those four all draw on one of those two indexes. Perplexity and Claude don't have an equivalent direct-submission path; for those, being crawlable and being present in the indexes they do draw from is the whole lever available.
How does Perplexity find and cite pages?#
Perplexity crawls the web with its own bot, publicly identified as PerplexityBot, and blends what it finds with results from third-party indexes rather than relying on one canonical index the way Copilot relies on Bing. Neither Perplexity nor any independent party has published a detailed public ranking methodology for how it selects and weights citations, so treat any specific claim about Perplexity's ranking factors with skepticism unless it cites a primary source. What is directly checkable: whether PerplexityBot is allowed in your robots.txt, and whether your content is visible to a crawler that doesn't execute JavaScript — the same crawlability baseline every AI crawler needs, regardless of ranking specifics.
How does Microsoft Copilot decide what to cite?#
Copilot's web-grounded answers read from the Bing index — the same index ChatGPT search partly relies on through its OpenAI/Microsoft partnership. That means the setup work overlaps directly: getting a page indexed by Bing (see setting up Bing and ChatGPT indexing) benefits both. Beyond indexing status, Microsoft hasn't published Copilot-specific citation criteria distinct from ordinary Bing ranking factors.
How does Gemini use search results when it searches the web?#
When Gemini performs a web search rather than answering from its trained knowledge, it draws on Google Search results — meaning the same prerequisites documented for Google AI Overviews apply: the page has to be crawlable, indexed, and eligible to rank in Google Search. There's no separate Gemini index or ranking system layered on top. See how to appear in Google AI Overviews for the checklist, since it applies here too.
How does Claude cite web sources when it searches?#
Claude's web search draws on third-party search results rather than a proprietary crawler and index the way Google does. Anthropic hasn't published a detailed public methodology for how those results are selected or weighted for citation. As with Perplexity, the checkable fundamentals — crawlability, indexing status in the underlying search sources it draws from, and content that answers the query clearly — are the practical levers, even without a published ranking algorithm to optimize against directly.
What should you check?#
- Robots.txt coverage. Confirm your robots.txt explicitly allows the crawlers relevant to your audience —
PerplexityBot,Google-Extended,OAI-SearchBot,ClaudeBot— rather than relying on a default that predates these bots existing. - Bing and Google indexing status, since four of the six engines above route through one or the other.
- Content visibility without JavaScript execution, the baseline every crawler in this list needs regardless of which index or ranking system sits behind it.
Run a free scan to check which of these crawlers your site currently allows and how much of your content is visible to them.
