Quick answer: Lighthouse fails "llms.txt does not follow recommendations" when /llms.txt cannot be fetched, returns a 5xx or 429, or is not Markdown with an H1 title and at least one link. A missing file that returns 404 is not a failure. Fix it by serving a short Markdown file: "# Site name", a one-line summary, and links to your key pages.
This audit is part of Agentic Browsing, the newest Lighthouse category, which checks how well AI agents can read and use a page. Google marks the category as under development, so its audits may change. The llms-txt audit fetches /llms.txt from the root of your site and checks that it looks like the format proposed at llmstxt.org: a Markdown file with a title and links to your important pages. It fails with the title "llms.txt does not follow recommendations". On lighthouse-md.com, 75 of 698 analyses in the last 30 days failed it. This page covers exactly what the audit checks, why most failures come from sites that never wrote an llms.txt at all, and a minimal file that passes.
TL;DR
- What: Lighthouse requests
https://your-site/llms.txt. It fails if the request errors or times out, if the server returns a 5xx or 429, or if the file has no Markdown H1, no Markdown link, or under 50 characters. - The surprise: a missing file that returns 404 passes. The audit is marked not applicable. Most failures come from servers that answer
/llms.txtwith an HTML page and a 200 status, or that time out. - Fix: Either return a clean 404 for the path, or serve a short Markdown file:
# Site name, a one-line summary, and links to your key pages.
What does "llms.txt does not follow recommendations" mean?
Lighthouse runs one request for /llms.txt on the origin of the page you audited, and then applies these checks in order:
| Result of the request | Audit result | What the report shows |
|---|---|---|
| Network error or timeout | Fail | Fetch of llms.txt failed: Timed out fetching resource (or another error) |
| No status at all | Fail | Fetch of llms.txt failed |
| HTTP 500 to 599, or 429 | Fail | Failed with HTTP status 503 |
| HTTP 400 to 499 (other than 429), including 404 | Not applicable | Nothing counts against you |
| HTTP 200, file checked | Pass or fail on content | One row per problem, below |
When the file comes back, Lighthouse checks the content for three things:
| Check | Rule in the audit | Error row when it fails |
|---|---|---|
| Title | A line that starts with #, a space, then text | File is missing a required H1 header (e.g., "# Title"). |
| Links | At least one Markdown link, <a href="url">text</a> | File does not appear to contain any links. |
| Length | 50 characters or more | File is suspiciously short. |
Any one row fails the audit. The checks are format checks only. Lighthouse does not check whether the links work, whether the content is accurate, or which content type the server sends.
Why does Lighthouse say my llms.txt is missing an H1 when I never made one?
Because your server answered /llms.txt with something else and a 200 status. This is the most common failure, and the fix is not to write an llms.txt. Typical causes:
- Single-page apps and catch-all routes. React, Vue and Angular apps on many hosts, and frameworks with a catch-all route, return
index.htmlfor any unknown path. HTML has no# Titleline and no<a href="url">text</a>link, so the audit reports both the H1 and the links error. - Custom "not found" pages served with 200. Some CMS themes and site builders show a friendly not-found page without sending a 404 status. Search engines call this a soft 404, and it fails this audit the same way.
- A redirect to the homepage. Some sites send every unknown URL to
/, and the homepage HTML fails the content checks.
Check with:
curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/llms.txt
200 text/html on a site that has no llms.txt is this problem. Make the path return a real 404, or add a real file. Either one clears the failure.
What does "Fetch of llms.txt failed: Timed out fetching resource" mean?
Lighthouse asked for /llms.txt and did not get a complete answer in time. The audit does not report a status in this case, only the error text. The usual causes:
- Bot protection. A firewall or bot-management rule holds the request for a challenge that an automated client never completes. A challenge returned as a 403 is a 4xx and reads not applicable; one that hangs or returns a 503 fails.
- A slow origin with no cache. The path falls through to an application route that takes too long to answer.
- A route that never responds. A misconfigured rewrite or proxy rule that loops or waits.
Request the URL with curl -i from outside your network and time it. Serve the file as a static asset (or return a fast 404), exclude the path from bot challenges, and cache it at the CDN.
How do I write an llms.txt that passes?
A minimal valid file:
# Example Store
> Handmade ceramics shipped from Lisbon. Product pages, care guides and shipping policy.
## Key pages
- [Shop all products](https://example.com/collections/all): the full catalog
- [Care guide](https://example.com/pages/care): washing and storing ceramics
- [Shipping and returns](https://example.com/policies/shipping-policy): rates, times and returns
That file passes all three checks: an H1, Markdown links, and well over 50 characters. The llmstxt.org proposal adds more structure:
- One H1 with the site or project name. It is the only required part of the proposal.
- A blockquote with a short summary of what the site is.
- Optional paragraphs with details a reader needs to use the links.
- H2 sections, each a list of Markdown links with a short note after each.
- An optional
## Optionalsection for links an agent can skip when its context is short.
Serve it at the root, https://your-site/llms.txt, as a static file. text/plain or text/markdown both work; the audit does not check the content type. For a live example, see lighthouse-md.com/llms.txt.
Should I add an llms.txt file at all?
It is optional. llms.txt is a proposed convention, not a standard, and Google Search representatives have said publicly that Search does not use it. Some AI tools and coding agents do read it, and it is cheap to maintain for documentation sites, developer tools and stores with many pages. If you do not want one, a clean 404 passes this audit. What you should not leave in place is a path that returns your HTML page with a 200, or one that times out.
How do I fix it on Shopify, WordPress, or Next.js?
- Shopify: check what
/llms.txtreturns on your primary domain before changing anything. If it is a 404, the audit is not applicable and nothing needs fixing. If an app serves the path, its content must meet the three checks above. Bot-protection apps can cause the timeout error. - WordPress: an SEO plugin may already generate llms.txt; check its settings before adding a second one. Otherwise upload a static
llms.txtto the web root. A theme that serves a not-found template with status 200 causes the H1 and links errors. - Next.js and other SPA hosts: put
llms.txtinpublic/so it is served as a static file. Without it, a catch-all route or a rewrite toindex.htmlreturns HTML with a 200, which fails. Make sure the host's fallback rewrite does not catch/llms.txt.
How do I verify the fix?
- Run
curl -i https://your-site/llms.txt. You want either a404, or a200with your Markdown in the body. - Re-run Lighthouse with the Agentic Browsing category. On lighthouse-md.com it runs on every analysis; with the PageSpeed Insights API, add
category=agentic-browsing. The audit should read "llms.txt follows recommendations" or show as not applicable.
What mistakes should I avoid?
- Writing a file only to silence the audit. If the failure was a 200 HTML response, a 404 is the honest fix.
- Serving HTML at the path. A generated page that renders the Markdown as HTML fails the content checks. Serve the raw file.
- Putting secrets or private URLs in it. The file is public, and its purpose is to be read by machines.
- Expecting a ranking change. The audit checks format. Passing it does not change how Google ranks the site.
Related audits
- Links are not crawlable: links that agents and crawlers can actually follow
- Document does not have a main landmark: agents read the same accessibility tree as screen readers
- Avoid multiple page redirects: a redirect from /llms.txt to your homepage is one of the causes above
Audit your URL at https://lighthouse-md.com.
Audit your page now
Paste your URL, get scores plus a CLAUDE.md plan for Claude Code.