LLMs and self-referentiality

While LLMs are often not very self-referential, agentic systems built on LLMs often are and that's also what makes them quite magical. Just ask pi or another coding agent that understands itself to make changes to it, or to explain how it works. It's a very neat experience.

It gets particularly meta if it encounters problems and starts debugging itself.

I've been recently (and very very slowly) reading David Bentley Hart's "All Things Are Full of Gods," which is a quite engaging little philosophical dialogue about minds and machines. I recommend it as a fun little attack on the purely mechanistic view of consciousness that is the current regime.

But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence?

Was this ever the argument? People have been convinced by chatbots for a long, long time. World-changing, though, is new.

Hofstadter has another book, called Surfaces and Essences that I need to get around to finishing, but I recall the thrust of the argument being something like metaphors being the fuel of intelligence. I suppose you could think of the manipulation of WordVec weights as a sort of model of metaphors, but I don't know enough to continue the idea...

The rejection of The Philosopher's four causes at the end bristled me a little bit... Don't throw it away just yet! Maybe Aristotle has more to teach us, now that we've reached the end of our mechanistic rope.

I think it is a complicated question whether LLMs are explicitly trained for a bit of capacity of self-referentiality.

Training on the desirable chain-of-«thought» style is absolutely a thing, and it includes the ability to evaluate the previous parts of the generated attempts and evaluating which of them is more promising to build upon. In case of experiments with local-scale models we sometimes know what exactly people tried with reasoning-like trace selection, and it is very visible if one follows the trace as it is generated.

It is even more visible if one is watching the beginning of LLM's rehashing of the request, and is interrupting/adjusting the prompt/restarting if there is an omission or ambiguity in the request (until the attempt looks worth no longer watching and letting to run its minutes to completion).

I do not belive that those frontier models where we do not know for sure are exceptions to this style of curation of «analysis» traces, e.g. for recognising dead-end attempts and switching the approach.