Where is it like to be a language model?

This sort of worldview was fairly well-analyzed previously, on Lobsters and previously, on Awful, where language models are idealized as bags of words, phrases, sentences, memes, tropes, narremes, etc.