llms.txt, schema and Open Graph: the technical side of AI search visibility

The technical side of AI search visibility comes down to three things: making sure AI crawlers can reach your pages, describing your business and content clearly with schema markup, and controlling how your pages look when shared with Open Graph tags. llms.txt, a proposed file that summarizes your site for language models, is fine to add but is not a major factor today; Google has said it does not use it for Search. Get crawl access, structured data and clean HTML right first.

This is the technical companion to our broader guide on answer engine optimization, which covers the content side.

How AI tools find and use your pages

AI answer tools get information about your business in a few ways:

• Search indexes. Google's AI Overviews and AI Mode draw on Google's index. ChatGPT search, Copilot and others draw on their own or partner search indexes, including Bing's. If you aren't indexed, you're unlikely to be cited.
• Live retrieval. When a user asks a question, some assistants fetch pages in real time to ground their answer.
• Training data. Models learn from large crawls of the public web, though that is slow to change and hard to influence directly.

So the technical work is mostly familiar SEO work, done carefully: crawlable, fast, well-structured pages with clear signals about what they are.

The signals at a glance

Crawl access comes first

The most common technical reason a site is invisible in AI answers is that crawlers can't get to it.

robots.txt and AI crawlers

AI companies run several crawlers, often with different jobs. As of this writing, the main ones include:

• OpenAI: GPTBot (training), OAI-SearchBot (search results in ChatGPT) and ChatGPT-User (fetches made on behalf of a user).
• Anthropic: ClaudeBot and related agents for search and user-initiated fetches.
• Perplexity: PerplexityBot.
• Google: Googlebot for Search (which feeds AI Overviews), plus the Google-Extended token, which controls whether your content is used for Gemini models without affecting Search.

Each company documents its user agents, and the lists change, so check their current documentation before you write rules. The key decision: blocking a search or retrieval crawler can keep you out of that tool's answers. Many businesses allow search and retrieval agents and decide separately about training-only crawlers.

Other things that block crawlers

• Firewall and bot protection. Aggressive bot-blocking at your CDN or host can stop legitimate crawlers. Check your settings and logs.
• JavaScript-only content. If your content only appears after client-side JavaScript runs, some crawlers will see an empty page. Server-side rendering or static generation fixes this, which is one reason we like Next.js for business websites.
• Leftover noindex tags from staging, a classic redesign mistake we covered in how to redesign without losing SEO.
• Login walls and gated content. If it's behind a form, it won't be cited.

Register your site in Google Search Console and Bing Webmaster Tools. Bing matters more than many people assume, because several AI tools draw on it.

Schema markup: describing your business to machines

Schema markup is structured data, usually added as a JSON-LD block in the page head, that tells search engines exactly what a page describes: this is a local business with these hours, this is an article by this author, this is a product with this price.

The types most businesses need

A note on FAQ markup: Google now shows FAQ rich results only for a narrow set of authoritative sites, so most businesses won't see the expandable FAQ in results. The markup is still valid, and clear question-and-answer content is useful for answer engines regardless.

Rules we follow

• Mark up what is visible. Schema must match the content on the page. Marking up things users can't see risks penalties and erodes trust.
• Be consistent. Your business name, address and phone should match across schema, your site footer, Google Business Profile and directories.
• Connect entities. Use sameAs links to your official social profiles and listings, and reference the same Organization from your articles and author pages.
• Keep dates honest. Update dateModified only when the content meaningfully changes.
• Validate. Test with Google's Rich Results Test and the Schema.org validator after every template change.
• Build it into templates. On WordPress, an established SEO plugin handles most of it. In Webflow, use CMS fields to populate JSON-LD. In a headless build, generate it from the content model, which is easy if your content is well structured, as we described in Contentful content models and reusable blocks.

Open Graph: how your pages look when shared

Open Graph tags are meta tags that tell platforms what title, description and image to show when someone shares your link. They are used by social networks, messaging apps, collaboration tools and many AI chat interfaces that show link previews.

The essential tags:

• og:title: the page title as you want it to appear when shared.
• og:description: one or two sentences.
• og:image: an absolute URL to a preview image (a wide landscape image, commonly 1200 by 630 pixels).
• og:url: the canonical URL.
• og:type: usually website or article.

Add Twitter/X card tags (twitter:card set to summary_large_image) for consistent previews there. Then check a few key pages with the platforms' sharing debuggers, because they cache previews.

Open Graph doesn't directly change rankings, but a clear title, description and branded image make shared links look trustworthy and get clicked, and that includes links surfaced inside AI tools.

llms.txt: what it is and what it isn't

llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI. The idea: place a Markdown file at /llms.txt that gives language models a short summary of your site and a curated list of links to the most useful pages, ideally with clean Markdown versions. Think of it as a reading list for an AI, rather than a set of rules like robots.txt.

Where it stands today

• It is a community proposal, not a standard from a standards body.
• Google has said it does not use llms.txt for Search, and its guidance says you don't need special AI text files or markup to appear in Google's AI features.
• Support from other AI companies is limited and not consistently documented.
• It has found real use in developer documentation, where coding assistants and tools can pull the file to get a clean map of the docs. Several documentation platforms generate one automatically.

Should you add one?

For most business sites, it's optional. It takes little effort, does no harm, and may help some tools. Just don't expect it to move AI citations on its own, and don't spend time on it before crawl access, content and schema are solid.

If you add one, keep it simple:

1. An H1 with your business name.
2. A short blockquote summary of what you do and who you serve.
3. Sections of links to your key pages (services, locations, pricing, key guides), each with a one-line description.
4. Keep it current when pages change, or it becomes misleading.

A practical checklist

1. Confirm robots.txt allows the search and AI crawlers you want, and check your firewall isn't blocking them.
2. Make sure key content is in the server-rendered HTML.
3. Register with Google Search Console and Bing Webmaster Tools and submit your sitemap.
4. Add Organization or LocalBusiness schema, plus Article, Service, Product and BreadcrumbList where they apply.
5. Validate structured data after every template change.
6. Add Open Graph and Twitter card tags to every template, with a good default image.
7. Lead each page with a direct answer to the question it targets.
8. Optionally, publish a concise llms.txt.
9. Monitor referral traffic from AI tools in your analytics.

Fast, accessible pages help too. See Core Web Vitals for small business and website accessibility for business owners. And if you're still choosing a platform, our complete guide to choosing a website platform covers how each one handles these technical foundations.

The short version

Let the right crawlers in, render content in HTML, describe your business accurately with schema, and control shared previews with Open Graph. Add llms.txt if you like, but treat it as a nice extra, not a strategy. The fundamentals are what get you found and cited.

Want a technical AI visibility audit? See our SEO and AEO services or contact us.