llms.txt and Markdown Mirrors: What They Are and How to Implement Them Safely

What the proposal says

The community proposal places a Markdown file at /llms.txt with a short site description and curated links to detailed Markdown documents. For directory-style page URLs, the proposal describes mirrors using index.html.md.

What Google says

Google says websites do not need llms.txt, special AI text files, or Markdown copies to appear in Search or its generative AI experiences. Google may crawl many file types, but the presence of a file is not a ranking or inclusion signal.

A safe implementation pattern

1. Keep the human-readable HTML page canonical and fully crawlable.
2. Publish only public, useful URLs in llms.txt.
3. Keep Markdown facts synchronized with the canonical page.
4. Use an HTTP X-Robots-Tag: noindex, follow for mirrors when duplicate search results are not desired.
5. Exclude mirrors from the XML sitemap and link them from the canonical page with rel=alternate.
6. Do not serve materially different claims to bots and people.

What belongs in llms.txt?

Prioritize product facts, documentation, methods, terminology, policies, current release information, and a small set of high-value guides. Avoid dumping every URL or building a keyword list.

Maintenance risks

The main risk is stale machine-readable content. Version changes, pricing, permissions, feature claims, and policy pages must update in the HTML, structured data, llms.txt, and Markdown mirrors together.

How GSC Pulse implements it

GSC Pulse publishes a curated llms.txt, a larger full context file, and a noindex index.html.md mirror for every canonical page. The HTML remains the source of truth.

Primary sources

• Answer.AI llms.txt proposalSource
• Google generative AI optimization guideSource
• Google robots meta and X-Robots-Tag documentationSource