Skip to main content
ai search aeopublished 17 Jun 2026updated 31 Oct 2026·12 min read

llms.txt Guide: Robots.txt for AI Search and Intelligent Web

Dragoș-Adrian BuhoiuFounder · Web & SEO Engineer
Verdant Mindset cover: llms.txt Guide: Robots.txt for AI Search and Intelligent Web
FEATURED.IMG
Verdant Mindset cover: llms.txt Guide: Robots.txt for AI Search and Intelligent Web

Learn the llms.txt guide at Verdant Mindset: the spec, what it doesn't do for Google AI Mode, measure us yourself and deployment guides.

llms.txt Guide:Deployment, Best Practices, and AI Crawler Optimization

In a web landscape dominated by conversational assistants and generative engines (AEO & GEO), the way your site's data is consumed has shifted. llms.txt is a proposal for a Markdown summary of your site, usually placed at the root, for AI tools to read. It is not a requirement: Google says you don't need new machine-readable files, AI text files or Markdown to appear in its generative AI features in Search, and that Search ignores llms.txt.

So treat llms.txt for what it is: a cheap, optional file for the AI tools that choose to read it, not a condition for being found and not a standard.

This guide provides the technical specifications, deployment options for WordPress, Shopify, Webflow, and Next.js, and validation processes, so that if you publish the file, it is correct.

What is llms.txt and Its Technical Specification

The llms.txt file is a proposal by Jeremy Howard, first published on September 3, 2024 and now in its second version (v2): a plain text file written in Markdown, placed at the root of a site (https://domain.com/llms.txt) or in a subfolder, where it describes only the pages under it.

This file acts as a machine-readable sitemap for LLMs. It leaves out layouts, tracking pixels and HTML wrappers: a short summary and clean links, each with an optional note about the page.

Syntactic Formatting Rules

The format proposed at llmstxt.org is plain Markdown, with the sections in the order below; only the H1 with the site's name is required, the rest is optional:

  • H1 (# Title): the name of the project or site; it is the only required part.
  • Blockquote (> …): a short summary, with the information needed to understand the rest of the file.
  • Paragraphs or lists without headings: details about the project and how to read the files it points to.
  • ## sections: lists of links, each item a Markdown link [Page title](URL), optionally followed by a colon and notes about the page.
  • ## Optional: by convention, secondary links that an agent can skip when it needs a shorter context.

You will also see /llms-full.txt, a single file with the full, concatenated text of key pages, meant to let a tool read everything in one request. The current version (v2) of the proposal doesn't mention it.

What It Does and Does Not Do for Google AI Mode

Traditional search bots load HTML documents, parse the DOM tree, and execute JavaScript to find out what a page is about. Google AI Mode doesn't change that: Google Search, AI Mode included, ignores llms.txt. To be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Search with a snippet.

By exposing a clean /llms.txt file, you can at most:

  1. Offer a Map: tools that read the file see in one pass which URLs hold your key facts.
  2. Describe Your Pages in Your Own Words: you write the summaries yourself; tools may use them, and Google ignores them.
  3. Low-Carbon Crawling (Digital Sustainability): From an environmental engineering perspective, rendering JavaScript-heavy pages for thousands of URLs costs server work and energy. When a tool reads the raw Markdown file instead of your pages, that processing layer disappears: there is no DOM tree to build, no scripts to execute, no secondary resources to download — the server delivers a minimal payload.

To understand how this fits into a broader structured data architecture, see our sustainable AEO & GEO guide.

Don't Take Our Word for It — Measure Us Yourself

Focusing on code efficiency is not a marketing trend but an engineering decision you can verify directly, without relying on figures published by any agency (including us):

  • The speed test: Run verdantmindset.com through PageSpeed Insights (pagespeed.web.dev). Then run, with the exact same tool, the site of the agency that promises you speed. Compare the results side by side — the tool belongs to Google, not to us.
  • The llms.txt test: Open https://verdantmindset.com/llms.txt directly in your browser. See for yourself what a semantic map served as raw text looks like: clean links, short descriptions, zero ballast. Then try the same URL on your own site or on your current provider's.

This is the standard you should hold any technical partner to: performance claims are verified with public tools, not accepted from slide decks.

Multi-Platform Deployment Guides

For llms.txt to be useful, it has to reflect the current state of your site: generated automatically or updated with every important change. Below are implementation examples for the most common stacks:

WordPress:Dynamic Output via PHP Hook

Add this code snippet to your child theme's functions.php file (or a custom site-specific plugin) to serve a dynamic text file:

<?php
// Register a custom rewrite rule for llms.txt
add_action('init', 'vm_register_llms_rewrite');
function vm_register_llms_rewrite() {
    add_rewrite_rule('^llms\.txt$', 'index.php?vm_llms_trigger=1', 'top');
}

// Add query variable to WordPress query vars
add_filter('query_vars', 'vm_register_llms_query_vars');
function vm_register_llms_query_vars($vars) {
    $vars[] = 'vm_llms_trigger';
    return $vars;
}

// Intercept request and output dynamic Markdown content
add_action('template_redirect', 'vm_serve_llms_txt');
function vm_serve_llms_txt() {
    if (get_query_var('vm_llms_trigger')) {
        header('Content-Type: text/plain; charset=utf-8');
        header('Cache-Control: public, max-age=3600');
        
        echo "# " . get_bloginfo('name') . "\n\n";
        echo get_bloginfo('description') . "\n\n";
        
        echo "## Sitemaps\n";
        echo "- [Main Sitemap](" . home_url('/sitemap_index.xml') . ")\n\n";
        
        // Fetch specific pages to list
        $pages = get_pages(array('meta_key' => '_vm_include_llms', 'meta_value' => 'yes'));
        if (!empty($pages)) {
            echo "## Key Pages\n";
            foreach ($pages as $page) {
                $desc = get_post_meta($page->ID, '_yoast_wpseo_metadesc', true);
                if (empty($desc)) {
                    $desc = wp_strip_all_tags(wp_trim_words($page->post_content, 20));
                }
                echo "- [" . $page->post_title . "](" . get_permalink($page->ID) . "): " . esc_html($desc) . "\n";
            }
            echo "\n";
        }
        
        // Fetch posts
        $posts = get_posts(array('numberposts' => 100, 'post_status' => 'publish'));
        if (!empty($posts)) {
            echo "## Articles & Insights\n";
            foreach ($posts as $post) {
                $desc = get_post_meta($post->ID, '_yoast_wpseo_metadesc', true);
                if (empty($desc)) {
                    $desc = wp_strip_all_tags(wp_trim_words($post->post_content, 20));
                }
                echo "- [" . $post->post_title . "](" . get_permalink($post->ID) . "): " . esc_html($desc) . "\n";
            }
        }
        exit;
    }
}

Note: Re-save your permalinks under Settings > Permalinks in the WordPress admin panel to flush the rewrite rules after adding this snippet.

Shopify:The Platform-Managed File

Shopify already serves /llms.txt by default: unless a store overrides it, the URL mirrors the /agents.md file the platform generates, as does /llms-full.txt. For most stores, Shopify says, the managed file is all you need, so you don't need the old method, a page template plus a redirect from /llms.txt.

If you have requirements the managed file doesn't cover:

  1. Add an llms.txt.liquid template to your theme's templates folder, used for /llms.txt only, or an agents.md.liquid, used for all three URLs; the latter can't be a JSON template.
  2. Write Markdown in it. agents.md.liquid renders with a restricted Liquid context, with only the request and agents objects; standard globals such as shop, collections or pages render blank, so you take store data from agents or hardcode it.
  3. Once you add the template, keeping its content current is your job.

Next.js:Route Handler (app/llms.txt/route.ts)

In modern Next.js (App Router), you can deploy a dynamic endpoint returning clean plain text headers with minimal overhead:

import { NextResponse } from 'next/server';
import { getActiveBlogPosts, getCoreServices } from '@/lib/api';

export const revalidate = 3600; // Cache for one hour

export async function GET() {
  try {
    const posts = await getActiveBlogPosts();
    const services = await getCoreServices();
    
    let markdown = `# Verdant Mindset\n\n`;
    markdown += `Digital engineering studio, technical SEO, and sustainable web design.\n\n`;
    
    markdown += `## Sitemaps\n`;
    markdown += `- [Main Sitemap](https://verdantmindset.com/sitemap.xml)\n\n`;
    
    markdown += `## Core Services\n`;
    services.forEach((service) => {
      markdown += `- [${service.title}](https://verdantmindset.com/en/services/${service.slug}): ${service.summary}\n`;
    });
    
    markdown += `\n## Articles & Guides\n`;
    posts.forEach((post) => {
      markdown += `- [${post.title}](https://verdantmindset.com/en/resources/${post.slug}): ${post.description}\n`;
    });
    
    return new NextResponse(markdown, {
      status: 200,
      headers: {
        'Content-Type': 'text/plain; charset=utf-8',
        'Cache-Control': 'public, max-age=3600, s-maxage=3600',
      },
    });
  } catch (error) {
    return new NextResponse('Error generating llms.txt data', { status: 500 });
  }
}

Webflow:The LLMs.txt Setting or a Cloudflare Worker

Webflow now has its own setting: in Site settings > SEO > LLMs.txt you upload the file (UTF-8 encoded, under 100 KB), and once you publish, it is served at https://domain.com/llms.txt on your custom domain, not on the Webflow staging domain. The file served is the one you uploaded. The older route, a Cloudflare Workers proxy in front of the site, only makes sense if you want to serve the file from code:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url)
  
  if (url.pathname === '/llms.txt') {
    const llmsContent = `# Webflow Project Name\n\n` +
      `Technical summary of your brand's digital presence.\n\n` +
      `## Sitemaps\n` +
      `- [Sitemap](https://domain.com/sitemap.xml)\n\n` +
      `## Core Services\n` +
      `- [Service 1](https://domain.com/service-1): Core description.\n` +
      `- [Service 2](https://domain.com/service-2): Core description.\n`;

    return new Response(llmsContent, {
      headers: { 
        'content-type': 'text/plain; charset=utf-8',
        'cache-control': 'public, max-age=86400'
      },
    })
  }
  
  return fetch(request)
}

Validation and Testing Checklist

After launching, run these checks to ensure AI crawlers process your file correctly:

  1. Test HTTP Headers: Run curl -I https://domain.com/llms.txt in your terminal. Ensure the response includes Content-Type: text/plain; charset=utf-8. If it returns text/html, fix it: the file is plain text and should be served as such.
  2. Verify Markdown Syntax: Verify that all link syntax is formatted correctly ([Title](URL)) and that no unescaped HTML tag fragments reside in descriptions.
  3. Check Access Settings: Ensure the file isn't blocked by your robots.txt configuration or a DDoS mitigation screen (like a Cloudflare JavaScript challenge). AI bots must be able to pull this file via a direct, anonymous HTTP GET request.

FAQ.PROTOCOL

Frequently Asked Questions

No, it is a community proposal, and no major AI platform has committed to reading it. Some platforms, Shopify among them, now generate one automatically, but that doesn't make it a standard for AI search: Google Search ignores it, and OpenAI and Perplexity describe robots.txt as the way to manage their crawlers.
No. `llms.txt` only describes your site; it can't block anything. To ask crawlers not to access your pages, you use your `robots.txt` file, with `Disallow: /` under each crawler's `User-agent`; it is a request, not a barrier.
It can lower server load, but only when a tool reads the file instead of rendering your pages: a light Markdown file takes less work to serve and parse than a JavaScript-heavy page. Don't expect it to move your Core Web Vitals: they measure loading (LCP), responsiveness (INP) and visual stability (CLS) in real users' visits, and TTFB is not one of them.
For dynamic sites like e-commerce or daily blogs, the file must be generated dynamically (like our WordPress and Next.js examples). A static text file quickly gets outdated, and any tool that reads it gets wrong prices, stock statuses or broken links.
A sitemap XML is a raw list of URLs for search engines (Googlebot, Bingbot), with an optional last-modified date that Google uses only when it is consistently and verifiably accurate; Google ignores `<priority>` and `<changefreq>`. `llms.txt` provides semantic hierarchy, titles, and descriptive context written in Markdown, which LLMs can parse directly to generate conversational answers.

Corrections and updates

  • October 31, 2026. Checked against Google's documentation, the llms.txt proposal (v2), the OpenAI and Perplexity crawler pages, Shopify's developer documentation, Webflow's help center and the Ahrefs study. The guide no longer presents llms.txt as a standard, as something AI crawlers require, or as a way to control how LLMs read your site: it is a proposal, and Google Search ignores it. Corrected: the file format, after the second version of the proposal, in which only the H1 is required and the note after a link follows a colon, also in the code examples; /llms-full.txt, which the current version doesn't mention; Shopify, with the file the platform generates and the llms.txt.liquid and agents.md.liquid templates, in place of the page with a redirect; Webflow, with its LLMs.txt setting; the line that a site without the file becomes invisible to AI search, and the prediction that it is becoming the standard; the code, now presented as examples, not as tested setups; what to do when the file is served as text/html; the FAQ answers on blocking crawlers, with robots.txt as a request, not a barrier, and on keeping the file current, which point to the WordPress and Next.js examples. The AI Mode section now spells out that the file changes nothing there; the sustainability answer no longer promises stable Core Web Vitals or calls TTFB one of them; the sitemap answer says what Google does with <lastmod>, <priority> and <changefreq>.
WhatsApp direct