llms.txt

The llms.txt idea was proposed by Jeremy Howard in 2024 as a way to address a real problem: large language models have limited context windows and web pages are often cluttered with navigation, ads, and JavaScript that make it hard for an AI tool to extract the actual content efficiently. The convention specifies a Markdown file at the site root (/llms.txt) with a required H1 title, an optional blockquote summary, and then sections of links, each ideally pointing to clean Markdown versions of pages rather than full HTML. Some documentation platforms and site owners also publish an llms-full.txt file that concatenates complete pages. This is a community extension, not part of the current llmstxt.org v2 specification. It is a proposal rather than a protocol requirement. Some documentation platforms generate these files and several AI labs publish their own documentation indexes, but publication does not mean every search crawler or AI product fetches, follows, or ranks by them. Treat the file as an optional author-curated index and verify a target tool’s behavior before relying on it. A key misconception is treating llms.txt as equivalent to robots.txt in enforcement — robots.txt is a widely respected directive about crawler access, while llms.txt is only a curated pointer file that a given AI tool may or may not choose to fetch and use. A site owner can experiment after checking for stale or sensitive links, and can measure whether known tools fetch the file. Do not promise an SEO or citation benefit, and do not use it to replace content quality, accessible navigation, or crawler controls.