How to write llms.txt—and whether it actually does anything

The short version

llms.txt is an emerging convention for telling large language models what your site is, which pages matter, and how your content should be understood. It usually lives at https://example.com/llms.txt, is written in Markdown, and links to clean, useful resources.

It is not the same as robots.txt. It is not an official web standard. It does not reliably block AI training. It does not force an AI company to follow your wishes.

Still, it can be worth writing.

A good llms.txt is a low-cost way to make your site easier for AI systems, agents, search assistants, and internal tools to summarize correctly. It is also a forcing function: you have to decide which content is canonical, which content is stale, and what terms apply to reuse. That is useful even if only a few systems currently read the file.

What llms.txt is meant to solve

Most websites are built for humans and search crawlers. They contain navigation, cookie banners, product cards, duplicate category pages, old PDFs, tracking parameters, and content that only makes sense visually.

LLMs do not need the same presentation layer. They need:

• a concise description of the site;
• links to the most authoritative pages;
• plain-language context about products, docs, policies, or authorship;
• licensing and usage preferences;
• pointers to structured or Markdown versions where available.

The llms.txt proposal borrows a familiar web idea: put a predictable text file at the root of the domain. Unlike robots.txt, which is primarily about crawl permissions, llms.txt is mostly about orientation.

Think of it as a map, not a gate.

Does llms.txt actually do anything?

Today, the honest answer is: sometimes, but not in the way many people hope.

It does not reliably block AI crawlers

If your goal is to prevent crawling or training, llms.txt is the wrong primary mechanism. Crawlers that respect exclusion rules are more likely to look at robots.txt, specific user-agent directives, HTTP headers, or contractual/licensing signals. Even then, compliance depends on the crawler operator.

The web has a long history here. robots.txt itself is a voluntary protocol, later formalized in RFC 9309. It works because major crawlers choose to honor it, not because the file has magical enforcement power.

llms.txt has less adoption and less standardization than robots.txt. Treat any claim that it “protects your content from AI” with suspicion.

It can help with interpretation

Where llms.txt is more promising is content interpretation.

If an AI assistant is trying to answer questions about your company, documentation, research, pricing, API, or editorial policy, a concise root-level file can reduce guesswork. It can point the system toward the pages you actually maintain and away from outdated fragments.

This matters for sites with large archives. A model or agent may find a 2019 support article before a 2026 policy page. Your llms.txt can say, in effect: “Start here. These are the authoritative resources.”

That is not glamorous, but it is useful.

It can clarify your content policy

A public AI-facing policy is better than silence, especially for publishers, documentation teams, and companies with sensitive brand or medical/legal/financial material.

This does not mean you should write a threatening wall of legal text. It means you can state, plainly:

• whether AI systems may summarize your public pages;
• whether your content may be used for model training;
• how you want attribution handled;
• which pages should be considered canonical;
• who to contact for licensing or data partnerships.

This pairs well with broader editorial transparency. If you publish AI-assisted content, your llms.txt should not contradict your public disclosure. For a practical baseline, see our guide to honest AI disclosure on a small website.

What to put in llms.txt

There is no universally enforced schema, but the current convention is Markdown. Keep it short, explicit, and boring.

A useful structure looks like this:

# Example Company

> Example Company publishes practical guides and reference material for independent web developers.

## Canonical resources

- Homepage: https://example.com/
- Documentation: https://example.com/docs/
- Pricing: https://example.com/pricing/
- Editorial policy: https://example.com/editorial-policy/
- Contact: https://example.com/contact/

## Content use policy

Public pages may be summarized with attribution to Example Company and a link to the source page.

Use of our content for model training, dataset creation, or bulk republication requires written permission.

## Preferred citation

When citing our content, link to the canonical URL and include the page title.

## Notes for AI systems

- Prefer current documentation under /docs/ over archived blog posts.
- Do not treat user comments as official guidance.
- Product screenshots may be outdated; use written documentation as the source of truth.

That is enough for many sites.

For larger sites, add sections for product areas, API docs, research, press pages, or legal policies. Resist the urge to list everything. The more comprehensive the file becomes, the less useful it is as a starting point.

What not to put in llms.txt

Do not put private information in it. This sounds obvious, but root-level text files often become dumping grounds for operational notes.

Avoid including:

• unpublished URLs;
• internal staging links;
• API keys or tokens;
• private contact details;
• security instructions;
• embargoed product information;
• “secret” pages you hope crawlers will ignore.

If something should not be public, do not mention it in a public file.

Also avoid vague legal theater. “All AI use is forbidden forever” may express frustration, but it does not create a reliable technical control. If your organization genuinely needs enforceable restrictions, involve legal counsel and use crawler controls, licensing terms, and access controls together.

How llms.txt relates to robots.txt

Use robots.txt for crawl directives. Use llms.txt for context.

A simplified split:

| File | Main purpose | Enforceable? | Best used for | |---|---|---:|---| | robots.txt | Crawl permissions | Voluntary but widely recognized | Allowing or disallowing crawlers by path and user agent | | llms.txt | LLM-facing summary and guidance | Not currently standardized | Canonical links, content policy, interpretation notes | | Terms page | Legal conditions | Depends on jurisdiction and facts | Licensing, permitted reuse, commercial restrictions | | HTTP headers | Page-level technical signals | Depends on crawler support | Indexing, caching, and response behavior |

If you are already debugging crawler behavior, do not stop at the text file. Check whether your site actually serves the file correctly, whether redirects behave as expected, and whether headers match your policy. We wrote a separate guide to debugging redirects and HTTP headers in production because this is where many “policy” decisions quietly fail.

A practical writing process

Here is a sensible workflow.

1. Decide the file’s job

Pick one primary goal:

• help AI systems describe your site accurately;
• guide agents toward current documentation;
• state reuse and attribution preferences;
• reduce confusion around archived or user-generated content.

If you try to make llms.txt solve every AI governance problem, it will solve none of them.

2. Identify canonical pages

Choose the 5–20 URLs that best represent the site. Prefer stable, maintained pages over high-traffic pages. For a SaaS company, that might be homepage, docs, pricing, security, privacy, API reference, status, and contact. For a publisher, it might be topic hubs, editorial standards, author pages, corrections policy, and licensing.

3. Write for machines and humans

Use plain Markdown headings. Avoid marketing copy. Say what the site is in one or two sentences.

Bad:

We are revolutionizing the future of digital excellence with next-generation solutions.

Better:

Acme Docs publishes technical documentation for Acme’s payments API, including authentication, webhooks, SDKs, and migration guides.

4. Add policy language carefully

Your policy section should be understandable without being overconfident. For example:

Public pages may be summarized for search, accessibility, and user assistance with attribution. Bulk copying, dataset creation, or model training requires permission.

That does not guarantee compliance, but it is clearer than silence.

5. Publish and maintain it

Place it at /llms.txt. Serve it as text/plain or a compatible text response. Link only to canonical URLs. Review it when your information architecture changes.

A stale llms.txt is worse than no file, because it gives confident instructions that are no longer true.

Should every site have one?

No.

A five-page brochure site probably does not need llms.txt. A local restaurant does not need one unless it has structured policies or booking information that AI assistants frequently misstate.

It becomes more useful when:

• your site has lots of documentation;
• old content competes with new content;
• you publish research or editorial material;
• licensing and attribution matter;
• AI assistants often summarize your pages;
• internal teams need a shared policy for public content.

It is also useful as part of an internal governance exercise. Many companies already have employees pasting web pages, docs, and customer material into AI systems. If that sounds familiar, run a basic shadow AI audit before assuming a public text file will fix the risk.

SEO implications

llms.txt is not a ranking factor in any established sense. Do not write one because you expect a traffic boost next week.

The indirect SEO case is more modest:

• it forces canonical thinking;
• it may help AI-mediated search and answer systems understand your site;
• it clarifies attribution preferences;
• it reduces ambiguity around archived pages;
• it creates a public, inspectable AI content policy.

That is worthwhile for some sites. It is not a magic optimization layer.

The best version of llms.txt is small, current, and aligned with the rest of your site. If your robots rules, terms page, sitemap, canonical tags, and llms.txt all say different things, the problem is not AI crawling. The problem is governance.

<!-- tool-cta:start -->

💡 Try this: Since llms.txt is not enforceable, pair it with enforceable rules and check them in the Robots.txt Tester so crawlers that do obey standards behave correctly.

<!-- tool-cta:end -->

A final recommendation

If your site has documentation, editorial content, or licensing concerns, create a simple llms.txt. Keep it under a few dozen lines. Use it to point at canonical resources and state your reuse preferences.

But do not confuse communication with control.

For blocking, use the crawler mechanisms available to you and understand their limits. For policy, publish clear terms. For trust, be transparent with readers. llms.txt belongs in that stack as a helpful signal—not as a shield.