llms.txt Guide:Deployment, Best Practices, and AI Crawler Optimization
In a web landscape dominated by conversational assistants and generative engines (AEO & GEO), the way your site's data is consumed has shifted. llms.txt is a proposal for a Markdown summary of your site, usually placed at the root, for AI tools to read. It is not a requirement: Google says you don't need new machine-readable files, AI text files or Markdown to appear in its generative AI features in Search, and that Search ignores llms.txt.
So treat llms.txt for what it is: a cheap, optional file for the AI tools that choose to read it, not a condition for being found and not a standard.
This guide provides the technical specifications, deployment options for WordPress, Shopify, Webflow, and Next.js, and validation processes, so that if you publish the file, it is correct.
What is llms.txt and Its Technical Specification
The llms.txt file is a proposal by Jeremy Howard, first published on September 3, 2024 and now in its second version (v2): a plain text file written in Markdown, placed at the root of a site (https://domain.com/llms.txt) or in a subfolder, where it describes only the pages under it.
This file acts as a machine-readable sitemap for LLMs. It leaves out layouts, tracking pixels and HTML wrappers: a short summary and clean links, each with an optional note about the page.
Syntactic Formatting Rules
The format proposed at llmstxt.org is plain Markdown, with the sections in the order below; only the H1 with the site's name is required, the rest is optional:
- H1 (
# Title): the name of the project or site; it is the only required part. - Blockquote (
> …): a short summary, with the information needed to understand the rest of the file. - Paragraphs or lists without headings: details about the project and how to read the files it points to.
##sections: lists of links, each item a Markdown link[Page title](URL), optionally followed by a colon and notes about the page.## Optional: by convention, secondary links that an agent can skip when it needs a shorter context.
You will also see /llms-full.txt, a single file with the full, concatenated text of key pages, meant to let a tool read everything in one request. The current version (v2) of the proposal doesn't mention it.
What It Does and Does Not Do for Google AI Mode
Traditional search bots load HTML documents, parse the DOM tree, and execute JavaScript to find out what a page is about. Google AI Mode doesn't change that: Google Search, AI Mode included, ignores llms.txt. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Search with a snippet.
By exposing a clean /llms.txt file, you can at most:
- Offer a Map: tools that read the file see in one pass which URLs hold your key facts.
- Describe Your Pages in Your Own Words: you write the summaries yourself; tools may use them, and Google ignores them.
- Low-Carbon Crawling (Digital Sustainability): From an environmental engineering perspective, rendering JavaScript-heavy pages for thousands of URLs costs server work and energy. When a tool reads the raw Markdown file instead of your pages, that processing layer disappears: there is no DOM tree to build, no scripts to execute, no secondary resources to download — the server delivers a minimal payload.
To understand how this fits into a broader structured data architecture, see our sustainable AEO & GEO guide.
Don't Take Our Word for It — Measure Us Yourself
Focusing on code efficiency is not a marketing trend but an engineering decision you can verify directly, without relying on figures published by any agency (including us):
- The speed test: Run
verdantmindset.comthrough PageSpeed Insights (pagespeed.web.dev). Then run, with the exact same tool, the site of the agency that promises you speed. Compare the results side by side — the tool belongs to Google, not to us. - The llms.txt test: Open
https://verdantmindset.com/llms.txtdirectly in your browser. See for yourself what a semantic map served as raw text looks like: clean links, short descriptions, zero ballast. Then try the same URL on your own site or on your current provider's.
This is the standard you should hold any technical partner to: performance claims are verified with public tools, not accepted from slide decks.
Multi-Platform Deployment Guides
For llms.txt to be useful, it has to reflect the current state of your site: generated automatically or updated with every important change. Below are implementation examples for the most common stacks:
WordPress:Dynamic Output via PHP Hook
Add this code snippet to your child theme's functions.php file (or a custom site-specific plugin) to serve a dynamic text file:
<?php
// Register a custom rewrite rule for llms.txt
add_action('init', 'vm_register_llms_rewrite');
function vm_register_llms_rewrite() {
add_rewrite_rule('^llms\.txt$', 'index.php?vm_llms_trigger=1', 'top');
}
// Add query variable to WordPress query vars
add_filter('query_vars', 'vm_register_llms_query_vars');
function vm_register_llms_query_vars($vars) {
$vars[] = 'vm_llms_trigger';
return $vars;
}
// Intercept request and output dynamic Markdown content
add_action('template_redirect', 'vm_serve_llms_txt');
function vm_serve_llms_txt() {
if (get_query_var('vm_llms_trigger')) {
header('Content-Type: text/plain; charset=utf-8');
header('Cache-Control: public, max-age=3600');
echo "# " . get_bloginfo('name') . "\n\n";
echo get_bloginfo('description') . "\n\n";
echo "## Sitemaps\n";
echo "- [Main Sitemap](" . home_url('/sitemap_index.xml') . ")\n\n";
// Fetch specific pages to list
$pages = get_pages(array('meta_key' => '_vm_include_llms', 'meta_value' => 'yes'));
if (!empty($pages)) {
echo "## Key Pages\n";
foreach ($pages as $page) {
$desc = get_post_meta($page->ID, '_yoast_wpseo_metadesc', true);
if (empty($desc)) {
$desc = wp_strip_all_tags(wp_trim_words($page->post_content, 20));
}
echo "- [" . $page->post_title . "](" . get_permalink($page->ID) . "): " . esc_html($desc) . "\n";
}
echo "\n";
}
// Fetch posts
$posts = get_posts(array('numberposts' => 100, 'post_status' => 'publish'));
if (!empty($posts)) {
echo "## Articles & Insights\n";
foreach ($posts as $post) {
$desc = get_post_meta($post->ID, '_yoast_wpseo_metadesc', true);
if (empty($desc)) {
$desc = wp_strip_all_tags(wp_trim_words($post->post_content, 20));
}
echo "- [" . $post->post_title . "](" . get_permalink($post->ID) . "): " . esc_html($desc) . "\n";
}
}
exit;
}
}
Note: Re-save your permalinks under Settings > Permalinks in the WordPress admin panel to flush the rewrite rules after adding this snippet.
Shopify:The Platform-Managed File
Shopify already serves /llms.txt by default: unless a store overrides it, the URL mirrors the /agents.md file the platform generates, as does /llms-full.txt. For most stores, Shopify says, the managed file is all you need, so you don't need the old method, a page template plus a redirect from /llms.txt.
If you have requirements the managed file doesn't cover:
- Add an
llms.txt.liquidtemplate to your theme'stemplatesfolder, used for/llms.txtonly, or anagents.md.liquid, used for all three URLs; the latter can't be a JSON template. - Write Markdown in it.
agents.md.liquidrenders with a restricted Liquid context, with only therequestandagentsobjects; standard globals such asshop,collectionsorpagesrender blank, so you take store data fromagentsor hardcode it. - Once you add the template, keeping its content current is your job.
Next.js:Route Handler (app/llms.txt/route.ts)
In modern Next.js (App Router), you can deploy a dynamic endpoint returning clean plain text headers with minimal overhead:
import { NextResponse } from 'next/server';
import { getActiveBlogPosts, getCoreServices } from '@/lib/api';
export const revalidate = 3600; // Cache for one hour
export async function GET() {
try {
const posts = await getActiveBlogPosts();
const services = await getCoreServices();
let markdown = `# Verdant Mindset\n\n`;
markdown += `Digital engineering studio, technical SEO, and sustainable web design.\n\n`;
markdown += `## Sitemaps\n`;
markdown += `- [Main Sitemap](https://verdantmindset.com/sitemap.xml)\n\n`;
markdown += `## Core Services\n`;
services.forEach((service) => {
markdown += `- [${service.title}](https://verdantmindset.com/en/services/${service.slug}): ${service.summary}\n`;
});
markdown += `\n## Articles & Guides\n`;
posts.forEach((post) => {
markdown += `- [${post.title}](https://verdantmindset.com/en/resources/${post.slug}): ${post.description}\n`;
});
return new NextResponse(markdown, {
status: 200,
headers: {
'Content-Type': 'text/plain; charset=utf-8',
'Cache-Control': 'public, max-age=3600, s-maxage=3600',
},
});
} catch (error) {
return new NextResponse('Error generating llms.txt data', { status: 500 });
}
}
Webflow:The LLMs.txt Setting or a Cloudflare Worker
Webflow now has its own setting: in Site settings > SEO > LLMs.txt you upload the file (UTF-8 encoded, under 100 KB), and once you publish, it is served at https://domain.com/llms.txt on your custom domain, not on the Webflow staging domain. The file served is the one you uploaded. The older route, a Cloudflare Workers proxy in front of the site, only makes sense if you want to serve the file from code:
addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request))
})
async function handleRequest(request) {
const url = new URL(request.url)
if (url.pathname === '/llms.txt') {
const llmsContent = `# Webflow Project Name\n\n` +
`Technical summary of your brand's digital presence.\n\n` +
`## Sitemaps\n` +
`- [Sitemap](https://domain.com/sitemap.xml)\n\n` +
`## Core Services\n` +
`- [Service 1](https://domain.com/service-1): Core description.\n` +
`- [Service 2](https://domain.com/service-2): Core description.\n`;
return new Response(llmsContent, {
headers: {
'content-type': 'text/plain; charset=utf-8',
'cache-control': 'public, max-age=86400'
},
})
}
return fetch(request)
}
Validation and Testing Checklist
After launching, run these checks to ensure AI crawlers process your file correctly:
- Test HTTP Headers: Run
curl -I https://domain.com/llms.txtin your terminal. Ensure the response includesContent-Type: text/plain; charset=utf-8. If it returnstext/html, fix it: the file is plain text and should be served as such. - Verify Markdown Syntax: Verify that all link syntax is formatted correctly (
[Title](URL)) and that no unescaped HTML tag fragments reside in descriptions. - Check Access Settings: Ensure the file isn't blocked by your
robots.txtconfiguration or a DDoS mitigation screen (like a Cloudflare JavaScript challenge). AI bots must be able to pull this file via a direct, anonymous HTTP GET request.
FAQ.PROTOCOL
Frequently Asked Questions
Corrections and updates
- October 31, 2026. Checked against Google's documentation, the llms.txt proposal (v2), the OpenAI and Perplexity crawler pages, Shopify's developer documentation, Webflow's help center and the Ahrefs study. The guide no longer presents llms.txt as a standard, as something AI crawlers require, or as a way to control how LLMs read your site: it is a proposal, and Google Search ignores it. Corrected: the file format, after the second version of the proposal, in which only the H1 is required and the note after a link follows a colon, also in the code examples;
/llms-full.txt, which the current version doesn't mention; Shopify, with the file the platform generates and thellms.txt.liquidandagents.md.liquidtemplates, in place of the page with a redirect; Webflow, with its LLMs.txt setting; the line that a site without the file becomes invisible to AI search, and the prediction that it is becoming the standard; the code, now presented as examples, not as tested setups; what to do when the file is served astext/html; the FAQ answers on blocking crawlers, with robots.txt as a request, not a barrier, and on keeping the file current, which point to the WordPress and Next.js examples. The AI Mode section now spells out that the file changes nothing there; the sustainability answer no longer promises stable Core Web Vitals or calls TTFB one of them; the sitemap answer says what Google does with<lastmod>,<priority>and<changefreq>.
