Terminology#
- SEO
- Making a website easy for a search engine to crawl, index, and rank favorably for relevant queries. See What is SEO, and what changed with AI search.
- AEO
- Structuring content so a system can extract one direct, correct answer from it. See What is Answer Engine Optimization?
- GEO
- Making content usable as source material for an AI system that generates a synthesized answer. See What is Generative Engine Optimization?
- AI Overview
- A Google Search feature that generates a written summary above the normal results, grounded in Google's core Search index. See What is an AI Overview on Google?
- AI Mode
- A Google Search experience built around conversational, AI-generated answers, using the same underlying index and generative pipeline as AI Overviews.
- Retrieval-Augmented Generation (RAG)
- Retrieving relevant documents from an index and generating an answer grounded in what that retrieval returns, rather than answering from trained knowledge alone. The mechanism behind Google AI Overviews.
- Query fan-out
- Google's technique of issuing several related searches based on a user's original query to gather a wider set of candidate sources before generating an AI Overview.
- Scaled content abuse
- Google's spam policy classification for publishing many thin, near-duplicate pages — such as one page per query fan-out variation — instead of one thorough page. A named violation, not a gray-area tactic.
AI crawlers and bots#
- OAI-SearchBot
- OpenAI's crawler that determines whether a page can appear in ChatGPT search results. Blocking it removes a page from those answers.
- ChatGPT-User
- OpenAI's agent that fetches a specific page in real time when a user or agent asks ChatGPT to look at it mid-conversation. Has no effect on search inclusion, per OpenAI.
- GPTBot
- OpenAI's crawler for gathering AI model training data — separate from OAI-SearchBot, and unrelated to ChatGPT search visibility.
- Google-Extended
- The crawler control that lets a site opt its content in or out of use for training Google's generative AI models, separate from Googlebot's ordinary indexing crawl.
- PerplexityBot
- Perplexity's crawler for finding and citing web content in its AI-generated search answers.
- ClaudeBot
- Anthropic's crawler used to fetch web content for Claude, including when Claude performs a web search.
Mechanics and infrastructure#
- llms.txt
- A proposed convention file pointing AI systems to a curated, machine-readable summary of a site's content. Not required for Google AI Overview eligibility; practitioner crawl telemetry indicates OpenAI and Anthropic bots do fetch it where it exists. See what is llms.txt?.
- Content negotiation
- A server serving a different format of the same page depending on the request's
Acceptheader — HTML by default, with a markdown alternate served only when explicitly requested. See how ChatGPT search works. - IndexNow
- A protocol that lets a site push a URL directly to a search engine's indexing queue the moment it changes, instead of waiting for the engine's next scheduled crawl. See how AI assistants index your site.
- Schema.org / JSON-LD
- A shared vocabulary for describing page content in structured form, most often embedded as JSON-LD, that any system parsing a page can read unambiguously.
- ARIA
- A set of HTML attributes describing a page's interactive elements and structure for assistive technologies and, increasingly, for AI agents navigating a page via its accessibility tree.
CrawlReady scoring terms#
- Crawlability
- How much of a page's content is visible and structured correctly to a crawler that does not execute JavaScript. See how the AI Readiness Score is calculated.
- Agent Readiness
- Whether a page's structured data and standards-based markup let an AI agent act on its content correctly.
- Agent Interaction
- Whether a page's semantic HTML and navigation structure let a visual AI agent operate the page correctly.
- AI Readiness Score
- CrawlReady's 0-100 metric combining Crawlability, Agent Readiness, and Agent Interaction into a single measure of how well a site works with AI crawlers and agents.
Run a free scan to see how your own site measures against the terms defined here.
