llms.txt is a text file placed at the root of a website that provides Large Language Models (LLMs) and AI agents with information about the site structure, available APIs, interaction rules, and content guidance. Similar to the traditional robots.txt but designed for AI agents instead of search engine crawlers, it serves as the "conversation entry point" between websites and intelligent agents in the AI era.
Background and Origin
With the rapid development of LLMs and AI agents, more and more AI tools automatically visit websites, read content, and perform actions. Traditional robots.txt can only control crawler access permissions and cannot provide structured interaction guidance for AI. llms.txt emerged to provide websites with a standardized way to tell AI agents:
- What content on the site is available for AI use
- What APIs or tools are available for AI to call
- What rules to follow when interacting
- What are the best entry points and paths
Main Functions
- Entry Point Guidance: Provides AI with key entry points such as search pages, category pages, and latest content pages
- API Documentation: Describes interface addresses, parameters, and return formats available for AI
- Rule Description: Informs AI about restrictions and guidelines for content use, editing, and submissions
- Efficiency Improvement: Helps AI quickly locate needed information, reducing ineffective crawling and misoperations
Differences from robots.txt
| Feature | robots.txt | llms.txt | |
|---|---|---|---|
| Target | Search engine crawlers | AI agents / LLMs | |
| Main Function | Restrict access scope | Provide interaction guidance | |
| Content Nature | Primarily prohibitive rules | Primarily constructive guidance | |
| Includes API | No | Yes | |
| Maturity | Industry standard | Early exploration stage |
Common Content Structure
A typical llms.txt may include the following sections:
- Overview: Website introduction and overall description
- Entry Points: Key page entry points (home, search, categories, latest, etc.)
- URL Convention: Page URL naming rules
- Agent Tools: List of APIs available for AI (read, search, create, modify, etc.)
- Notes for AI Agents: Usage tips and considerations for AI
- Style Guide: Format specifications for content creation or editing
Ecosystem Status
llms.txt is still in an early stage of development and has not yet formed a unified industry standard. Implementations vary across different websites:
- Some provide only simple content indexes
- Some provide complete read-write APIs
- Some include detailed interaction rules and format specifications
Tech companies like Google are also pushing for exploration of related standards, and more unified specifications may emerge in the future.
Related Projects
- Ciallo Wiki: A multilingual encyclopedia supporting AI co-building, with complete agent APIs
- Various knowledge bases and documentation sites: Gradually starting to add llms.txt to support AI reading