Mintlify Automatically Generates llms.txt and llms-full.txt
In recent years, LLMs have rapidly developed, and cases of using information on the web as training data have increased. Along with this trend, llms.txt has been proposed as a mechanism for content creators to explicitly indicate how they want their documents to be used by LLMs.
The documentation platform Mintlify supports a feature that automatically generates this llms.txt, allowing documentation creators to indicate their LLM access control policies without special configuration.
This article explains the overview of llms.txt and its automatic generation feature in Mintlify.
llms.txt is a text file placed in the root directory of a website that instructs LLM crawlers about which content within the site can be used for training data or response generation, or which content should not be used.
This is similar to robots.txt, which provides instructions to search engine crawlers, but differs in that it specifically targets LLMs.
For details, please refer to llmstxt.org, which proposes the standard specification.
Mintlify automatically generates and places llms.txt files on created documentation sites. This allows users to apply basic LLM policies to their sites without being conscious of the llms.txt specification.
In addition to llms.txt, Mintlify also generates a file called llms-full.txt. llms-full.txt compiles all text within the site into a single markdown file, making it easy to load context into AI tools by simply pasting a single URL.
Here are examples of llms.txt and llms-full.txt automatically generated by Mintlify:
Mintlify’s automatic llms.txt generation feature is a useful function for addressing document management challenges in the LLM era.
This allows documentation creators to more clearly control how their content is handled by LLMs. As LLMs continue to develop, the importance of mechanisms like llms.txt will likely increase further.
That’s all from the Gemba.