How cats.txt showed llms.txt evidence is GEO astrology
40 points by kaycebasques
40 points by kaycebasques
As Lobsters' token technical writer, I figured the community might be interested to know that among documentarians there's still a lot of debate and unknowns about how agents (and model training pipelines) consume technical documentation.
It was a wonderful read. For at least two reasons. One, it's a beautifully silly way, with the appropriate amount of seriousness and investment, to tell people "to an llm, a file is a file". And another, I've enjoyed reading the article itself, I don't know if an llm or a human generated the text, but it was a nice amusing read with my breakfast.
I'm confused about this whole concept of llms.txt as a SEO thing. It was not the purpose when the idea was created. The usage doesn't depend on the search in any way either. https://llmstxt.org/
As the document says, it's a "file to provide information to help agents use a website". It's not for search. It's for "look at https://docs.convex.dev/llms.txt and tell me how to deploy this application" without the need to burn the tokens on parsing html pages meant for people.
i think it’s mostly that the “standard” part of it seems meaningless: as you say, it is just a text file that the model can read. the model doesn’t understand or specially support the “standard,” it’s just more text. nothing was stopping people from adding README.txt to their website before, and the value of conforming to the llms.txt format is unclear.
edit: and intention aside, TFA states that they aren’t complaining about the llms.txt thing specifically, they are complaining about the GEO astrology that has come about as a post-hoc justification for using this thing
I had no idea people were trying to use llms.txt for GEO. llms.txt is designed only to make a website easy to view and navigate for agents:
We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files.
It was never about search.
FYI, GEO = "Generative Engine Optimisation", for those like me that did not recognise the acronym.