What it is
Proposed in 2024, llms.txt sits at your site root and gives a short, curated index of important pages. It is not an official standard. Many companies, including developer-tool and infrastructure providers, publish one, mainly to help AI coding assistants read documentation.
What Google and the data say
- Google’s guidance on AI features says you do not need new machine-readable files or Markdown to appear in Google Search, including its generative features.
- Google representatives have publicly compared llms.txt to the old keywords meta tag.
- Analyses of large numbers of domains report most llms.txt files are rarely or never fetched and show no measurable citation uplift.
See summaries at DEV Community and llmtxt.info. Standards and behaviour can change, so re-check periodically.
Should you still add one?
Yes if it takes ten minutes and you maintain it, especially if you publish developer docs. No if it would distract from real work. We publish one at awidigital.com/llms.txt as a convenience, not as a ranking tactic.
What to do instead
- Make sure search and AI crawlers can fetch your pages.
- Server-render important content.
- Write answer-first pages with specific facts and sources.
- Use accurate schema.
- Earn independent mentions.
- Measure citations monthly.
Full checklist: How to make your website LLM-friendly.
A minimal llms.txt you can copy
If you decide to add one, keep it short and accurate:
> One-sentence description of what you do, where, and for whom.
## Key pages
- [Services](https://example.com/services): what we offer and prices
- [Pricing](https://example.com/pricing): current price ranges
- [Guides](https://example.com/guides/): in-depth articles
- [Contact](https://example.com/contact): how to reach us
Update it when your key pages change, and never include pages you would not want quoted.
How to test whether AI crawlers can actually see your site
- Check server logs for requests from OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Googlebot.
- Fetch a key page with JavaScript disabled to see what a non-rendering crawler sees.
- Check your CDN or firewall bot settings; some block unknown bots by default.
- Test robots.txt in Search Console and review each AI vendor’s current crawler documentation.
Skip the myths Free AI-visibility review
We focus on what moves citations.
Frequently asked questions
Will llms.txt hurt SEO?
No, but it will not help Google rankings.
Is robots.txt more important?
Yes. Crawl access in robots.txt and your CDN decides whether bots can read your pages at all.
Get a free quote in 24 hours
Tell us what you need. We reply with scope, timeline and a fixed price.