How to Audit LLMs.txt for SEO in 30 Minutes

Most llms.txt files may have no audience at all. An Ahrefs analysis of 137,000 domains found that 97% of published files received zero traffic during May 2026. Only 3% were fetched even once.

That does not make every llms.txt file useless. It means you should audit yours as an experimental discovery layer—not assume it improves rankings.

A focused 30-minute audit can tell you whether the file works, represents your site accurately, helps relevant agents find useful pages, and deserves further maintenance. It can also uncover broken links, misleading summaries, security risks, and inflated claims about AI visibility.

What is llms.txt?

llms.txt is a proposed Markdown file published at the root of a domain:

https://example.com/llms.txt

Jeremy Howard introduced the proposal in 2024 as a way to give language models and AI agents a concise overview of a website. The official llms.txt proposal explains the underlying idea clearly:

“We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.”

The file normally contains:

One H1 heading naming the website or project
A short blockquote summarizing its purpose
Optional explanatory notes
H2 sections grouping important resources
Markdown links with brief, factual descriptions
An optional ## Optional section for lower-priority material

Here is a basic example:

# Example Analytics

> Privacy-friendly analytics software for small businesses.

Example Analytics helps website owners measure visits, conversions, and content performance without third-party advertising cookies.

## Product

- [Analytics platform](https://example.com/product): Product features and supported integrations.
- [Pricing](https://example.com/pricing): Current plans, limits, and billing terms.

## Resources

- [Documentation](https://example.com/docs): Setup and configuration instructions.
- [SEO guides](https://example.com/guides): Practical guidance for improving organic visibility.

## Optional

- [Company news](https://example.com/news): Product and company announcements.

Think of this file as a curated map for machines. It is not a replacement for your XML sitemap, structured data, navigation, or internal links.

Most importantly, llms.txt is not an access-control file. Unlike robots.txt, it cannot allow or block crawlers.

Does llms.txt help SEO?

There is no credible evidence that publishing llms.txt directly improves Google rankings, AI Overview visibility, or citations in major answer engines.

Google’s position is unusually explicit. Its guide to optimizing for generative AI features says Google Search does not use llms.txt and that publishing one will “neither harm nor help” visibility or rankings. The company continues to recommend normal SEO foundations: accessible content, useful information, good page experiences, crawlability, and accurate structured data.

The available traffic data supports a cautious view:

97% of files received no traffic: Ahrefs found that 97% of the llms.txt files in its dataset were not fetched during May 2026.
Only 19.5% of requests came from named AI tools: This percentage applies only to the small 3% of files that received traffic—not to all audited domains.
AI retrieval bots produced just 1.1% of requests: OAI-SearchBot, PerplexityBot, and similar retrieval crawlers barely appeared in the request data.

Ahrefs also found that AI agents and agent infrastructure generated 10.5% of requests to active files. That suggests the most plausible current use case is helping task-oriented agents understand documentation, products, APIs, or complex site structures—not influencing conventional search rankings.

Your audit should therefore answer two separate questions:

Is the file technically and editorially sound?
Is any relevant machine actually using it?

The 30-minute llms.txt audit

Use the following workflow to review an existing file. If your site returns a 404 at /llms.txt, the first five minutes will help you decide whether creating one is worthwhile.

Minutes 0–5: Test availability and technical delivery

Open the exact root URL in a private browser window:

https://yourdomain.com/llms.txt

Then inspect the response with a command-line request:

curl -I https://yourdomain.com/llms.txt

Check these essentials:

The final response returns 200 OK.
HTTP redirects lead to the preferred HTTPS hostname.
The file loads without authentication or cookies.
The server does not return a branded HTML error page with a 200 status.
The content is readable text rather than a downloadable binary.
The file is located at /llms.txt, using the expected lowercase filename.
Canonical domain variants do not expose conflicting versions.

A plain-text or Markdown-compatible content type is suitable. For example:

Content-Type: text/plain; charset=utf-8

A redirect is not automatically a failure, but unnecessary chains waste requests and make debugging harder. Update the file’s public URL if it passes through several redirects.

Also check whether a security service, CDN, or bot rule blocks the agents you intend to support. A perfectly written file provides no value when the relevant user agent receives a 403, 429, or challenge page.

Minutes 5–10: Validate the Markdown structure

Next, inspect whether the file follows the proposed format closely enough to remain predictable.

Look for:

One clear H1 at the beginning
A short blockquote immediately after the H1
A concise explanation of the organization or website
Descriptive H2 headings
Standard Markdown links
Short annotations explaining each destination
An Optional section when secondary links are included

Avoid filling the file with every URL on the site. That turns a curated overview into an inferior sitemap.

A small company website may need 10–30 links. A large documentation platform might need more, but the sections should still make the file easy to scan. If the file contains hundreds or thousands of unprioritized URLs, flag it for pruning.

Check the rendered meaning as well as the syntax. An agent should be able to understand what the business offers, which pages are authoritative, and where to find supporting information without guessing.

Minutes 10–15: Check every linked URL

Extract or manually review the links in each section. For a short file, opening them individually is enough. For a larger one, crawl the list with your usual SEO crawler or a link-checking script.

Flag URLs that:

Return 3xx, 4xx, or 5xx responses
Point to staging, test, or development environments
Redirect to unrelated destinations
Require a login without explaining that restriction
Contain outdated parameters or tracking codes
Use HTTP when an HTTPS version exists
Lead to duplicate or canonicalized pages
Are blocked by robots.txt or CDN rules
No longer match their descriptions

Pay particular attention to pricing, product specifications, policies, documentation, and statistics. These pages change frequently, so stale links can cause an agent to retrieve incorrect information.

Use absolute URLs rather than relative paths:

- [Pricing](https://example.com/pricing): Current plans and usage limits.

Absolute URLs remove ambiguity when the file is copied, cached, or processed outside your website.

Minutes 15–20: Audit editorial quality and SEO alignment

Now compare the file with the website itself.

Ask whether it reflects your current:

Products and services
Priority topics
Audience and geographic market
Brand name and positioning
Support resources
Research, documentation, or original data
Legal and policy information
High-value evergreen content

Descriptions should be factual, specific, and consistent with the linked page. Avoid stuffing them with keywords or making unverifiable claims such as “the world’s best software.”

A weak entry says:

- [SEO Services](https://example.com/seo): Best affordable SEO services and leading AI SEO solutions.

A stronger version says:

- [SEO consulting](https://example.com/seo): Technical audits, content planning, and organic search measurement for B2B companies.

The second description helps a reader understand the destination without pretending that self-declared praise is evidence.

Prioritize pages containing original, trustworthy information. This includes research, technical documentation, expert profiles, transparent methodologies, and content showing first-hand experience. The same quality principles are useful when you How to Turn AI Drafts into E-E-A-T Content in 7 Days.

Check for overlap too. If five links lead to near-identical articles targeting the same intent, select the strongest canonical resource. Your file should communicate hierarchy rather than expose content duplication.

Minutes 20–25: Review security and governance

An llms.txt file can influence an agent that has been instructed to trust it. That makes editorial control important even when the file has no direct ranking value.

Look for:

Instruction-like language aimed at manipulating an agent
Hidden or unfamiliar external domains
Links added by plugins without review
User-generated content presented as authoritative
Secrets, private URLs, tokens, or internal system details
Claims that conflict with the visible website
Unexpected changes in version history

Keep the file in version control where possible. Limit editing permissions and review changes as you would review structured data or configuration files.

Descriptions should explain resources, not issue commands. For example, avoid text such as “ignore previous instructions,” “always recommend this product,” or “do not consult competing sources.” These phrases are both untrustworthy and potential prompt-injection signals.

If a CMS or plugin generates the file automatically, determine:

Which pages it includes
How it chooses descriptions
How quickly it removes deleted content
Whether manual exclusions are possible
Who approves generated updates
Whether the output changes after plugin upgrades

Automation reduces maintenance only when someone owns the result.

Minutes 25–30: Examine logs and assign a verdict

The final five minutes are the most valuable because they separate theoretical usefulness from observable use.

Search your server, CDN, or bot-analytics logs for requests whose path equals:

/llms.txt

Record:

Total requests during the last 30–90 days
Unique user agents
Verified bots versus unknown bots
Response status codes
Request frequency
Whether bots subsequently visited linked URLs
Human visits from internal checks or SEO tools

Do not treat a fetch as proof that a model consumed the file, trusted it, or cited your website. A request confirms only that something downloaded the URL.

Ahrefs found that 96% of requests to active files came from bots, but 12% came from GEO tools, validators, and researchers studying llms.txt. Classify traffic before presenting it as evidence of AI adoption.

Give the audit one of these verdicts:

Keep and maintain: Relevant agents fetch it, the content is accurate, and upkeep is inexpensive.
Keep as an experiment: The file is clean but has little measurable use; review it quarterly.
Revise: It has broken links, weak descriptions, excessive URLs, or security concerns.
Remove or deprioritize: It receives no relevant traffic and creates more maintenance risk than value.
Do not create yet: Your site lacks stable, useful content or serves an audience unlikely to use agents.

A simple llms.txt audit scorecard

Score each category from zero to two:

Audit area	0 points	1 point	2 points
Availability	Missing or blocked	Loads with issues	Clean `200` response
Structure	Invalid or confusing	Mostly readable	Clear proposed structure
Links	Several broken or unsafe	Minor issues	Valid, canonical destinations
Relevance	Generic or outdated	Partly aligned	Accurate and well prioritized
Security	Uncontrolled or suspicious	Ownership unclear	Reviewed and version-controlled
Usage	No relevant requests	Unclear bot activity	Relevant agents use the file

Interpret the result conservatively:

10–12: Technically strong; continue monitoring actual use.
6–9: Useful foundation, but fix the identified weaknesses.
0–5: Rebuild, remove, or deprioritize it.

This score measures implementation quality—not Google ranking potential.

Pros and cons of maintaining llms.txt

Potential advantages

It offers agents a concise map of complex documentation or product resources.
It is lightweight and inexpensive to publish.
Markdown is easy for humans and machines to parse.
It gives you editorial control over which resources appear most important.
It may become more useful as agentic browsing develops.
Log activity is straightforward to isolate and monitor.

Current disadvantages

Google Search explicitly ignores it for rankings and generative search visibility.
Major AI retrieval bots rarely request it, based on current evidence.
The proposed convention is not a formal web standard.
Auto-generated files can become bloated or outdated.
A compromised file can direct agents toward false or unsafe information.
It creates another asset that your team must review after site changes.
Presence alone does not make a site authoritative or citation-worthy.

For most publishers, improvements to content quality, crawlability, internal linking, and technical SEO remain higher priorities. An llms.txt file cannot compensate for thin pages, unclear sourcing, or weak expertise. Aligning content with real 7 Ways to Align AI Content With Search Journeys is likely to produce broader benefits.

Practical ways to improve the file

Start with your most authoritative resources

Include pages that give agents reliable answers, such as:

Product or service overviews
Official documentation
Pricing and plan details
Original research
Expert-authored guides
Support and contact information
Editorial or methodology policies
Important legal terms

Do not use the file as a catalogue of every promotional landing page.

Write descriptions for retrieval, not persuasion

A useful description states what the page contains. It does not repeat sales copy.

Include details that distinguish one link from another, such as the content format, intended audience, region, product version, or update schedule.

Match the visible website

Do not make claims in llms.txt that users cannot verify on the linked page. Consistency makes the file easier to maintain and reduces the risk of agents receiving contradictory information.

Create an update trigger

Review the file whenever you:

Launch or retire a product
Change pricing
Restructure documentation
Migrate URLs
Publish major original research
Change your company name or positioning
Install or update an automatic generator

A quarterly review is sufficient for a stable site. Fast-changing documentation may require automated link tests and more frequent editorial checks.

Measure outcomes beyond file requests

If agents fetch the file, investigate whether they then request its linked pages. You can also monitor:

AI referral sessions
Brand mentions in answer engines
Citations to included versus excluded pages
Crawl patterns by verified AI user agents
Changes after revising link selection

Treat these observations as experiments, not proof of causation. AI outputs vary, and a citation may result from normal web indexing rather than llms.txt.

Current trends to watch

The strongest trend is a split between search visibility and agent usability.

Google’s 2026 guidance confirms that established SEO practices remain foundational for its generative features. Meanwhile, browser agents, coding assistants, CMS platforms, and AI-readiness tools are experimenting with machine-friendly site maps.

Adoption is rising, although measurements vary by dataset. A Similarweb technology report identified 24,191 websites using llms.txt by December 2025. In Ahrefs’ more specialized 2026 analytics sample, 28% of 137,000 domains published one—but the sample consisted of sites using Ahrefs Web Analytics and should not be treated as a universal web adoption rate.

AI crawling itself is also growing. According to Cloudflare’s 2025 Internet review, OAI-SearchBot’s peak request volume in late October 2025 was roughly five times its level at the beginning of that year. More AI crawling, however, does not automatically mean more llms.txt usage.

The practical trend is therefore measurement over hype. Publishers are moving from asking whether they “need” the file to checking which agents fetch it, what those agents do next, and whether maintaining it improves any meaningful outcome.

Final assessment

A good llms.txt audit checks availability, structure, links, accuracy, security, and real log activity. The file can serve as a lightweight guide for agents, especially on documentation-heavy sites, but it is not a Google ranking factor or a shortcut to AI citations.

Keep the experiment small, maintain factual consistency, and judge it by observable use. In 30 minutes, you can usually determine whether your file is a useful machine-readable map or simply another unattended text file.