The llms.txt file is a plain text document, placed at the root of a website, that gives language models and AI agents a structured summary of the available content. The idea is relatively recent and emerged from the practical need to make websites easier to parse for systems that don't browse visually but instead read and interpret raw text. In this article we explain what a well written llms.txt contains, how it differs from robots.txt and sitemap.xml, and how widespread this practice actually is.
A community convention, not an official standard
It's important to clarify one aspect that is often overlooked: llms.txt is not a standard issued by the World Wide Web Consortium or any other regulatory body. There is no official protocol requiring language models to read it, the way search engines comply, to varying degrees, with robots.txt.
It's a convention proposed by the technical community, voluntarily adopted by site administrators and partially supported by a handful of development tools and platforms. In practice, it works as a best practice recommendation, not as a binding rule.
Why it's still worth adopting
Even without official status, llms.txt brings a concrete benefit: it reduces ambiguity for any automated system trying to understand a site's structure without manually parsing hundreds of HTML pages. A clean summary, written to be read by a machine, is more efficient than extracting information from a complex navigation menu.
On top of that, the process of writing an llms.txt forces you to think clearly about what is truly essential on your site. This clarifying exercise has value regardless of whether a particular model reads it right now or not.
The recommended structure of an llms.txt
The convention proposes a simple format, written in Markdown, easy to read by both humans and automated systems. The basic structure includes a title, a short summary, and one or more sections with annotated links to relevant pages.
There's no need for a complicated structure or additional metadata. Simplicity is actually an advantage, since it avoids misinterpretation by systems parsing the file.
The title and summary
The file usually starts with a level one heading naming the site or organization, followed immediately by a short paragraph explaining, in a few sentences, what the site does and what kind of information a reader or an AI agent can find there. This summary acts as an introduction for someone who has never visited the site before.
We recommend clear, direct wording, without heavy marketing language. A language model doesn't need superlatives, it needs precise information about what you actually offer.
Sections with annotated links
After the summary, sections organized by category usually follow, each containing a list of links accompanied by a short description. These annotations are essential, because they tell the automated reader what it will find at each address, without having to access the page to find out.
A typical structure might look like this:
- a section with core documentation, with links to the main product or service pages;
- a section with optional resources, such as blog posts or case studies;
- a section with contact or company information.
You don't have to follow these exact categories, but organizing content into thematic sections helps readability.
The difference from robots.txt
The robots.txt file has had a narrow, well defined purpose for over twenty years: it tells search engines and other crawlers which areas of a site can be accessed and which should be avoided. It is, essentially, a set of allow or disallow rules.
Llms.txt has a completely different role. It doesn't restrict access or impose rules, it provides context and guidance. You can have both files at the same time, since they don't overlap functionally and don't conflict with each other.
The difference from sitemap.xml
Sitemap.xml is a technical, rigorously formatted file that exhaustively lists all the pages of a site to help search engines with indexing. It's designed for machines, not for human reading, which is why it contains data such as the last update date or the relative priority of pages.
Llms.txt, by contrast, is a selective and readable document, which doesn't aim to list every single page but to highlight what's actually relevant. A sitemap can have thousands of entries, while an effective llms.txt usually stays short and focused.
What to include in llms.txt
The file should include the pages that describe the essence of your offering: the main services page, product documentation, important guides, and contact or about pages. The idea is to provide a selective map, not a complete index.
For example, if you run a site with technical services, it makes sense to include links to the services page or the about us page, each accompanied by a clear description.
What to leave out of llms.txt
Avoid including administrative pages, internal forms, dynamically generated addresses, or content with no independent informational value. It's also not useful to add dozens of links without annotations, since the file loses its purpose as a synthesis.
Don't include sensitive information or data that shouldn't be exposed publicly, even if that address isn't listed in the main menu. The fact that a file is intended for machines doesn't make it any less public.
The llms-full.txt variant
Alongside llms.txt, some sites also publish an llms-full.txt file, which usually contains the full content of the essential pages, not just links to them. This is useful for situations where a language model prefers to receive the complete text in a single document, without making additional requests for each page.
Not every site needs such an extended version. For a small or medium site, a well structured llms.txt with annotated links is usually enough to serve the intended purpose.
Where to place the file and how to maintain it
The convention calls for placing the file at the root of the domain, at an address like example.com/llms.txt, exactly as is done with robots.txt or sitemap.xml. The simple text format allows for quick editing without any special tools.
An important aspect that's often overlooked is keeping it updated. If you add important new pages or restructure the site, the file needs to be revised, otherwise it quickly becomes an outdated document that misleads rather than helps.
The current level of adoption
We need to be honest: adoption of llms.txt is still limited and uneven. Some technical documentation platforms and AI focused companies have started publishing such files, but the vast majority of websites, including many large and well known ones, still don't have one.
There's also no clear and consistent confirmation from major language model providers that they actually read and use this file in their response process. The practice is promising and makes logical sense, but it remains, for now, a field still taking shape, and concrete results can vary from one system to another.
How to check if your file is correct
Once you've written an llms.txt, it's worth checking whether it follows the recommended format and whether the links in it work correctly. You can use the WebMCP validator to check the file's structure and identify potential issues before publishing it.
Periodic checks are especially useful after major site changes, when the risk of broken links or outdated sections increases considerably. A malfunctioning file can be less useful than having no file at all.
The connection to AI agent readiness
Llms.txt is part of a broader context, in which websites are increasingly interfacing not only with human users but also with automated agents that make decisions or perform actions on their behalf. If you want a full assessment of how ready your site is for this type of interaction, an AI agent readiness audit can identify existing gaps.
For sites that want to go further, with interactive support for agents rather than just static documents, the WebMCP implementation service offers a broader technical framework than a simple text file.
Conclusion
Llms.txt remains a young convention, without the backing of an official body, but with an easy to understand practical logic: it provides clarity where there would otherwise only be ambiguity. It's worth writing carefully, updating periodically, and treating as a complementary tool, not as a universal solution for visibility in front of AI agents.
