On the Limits of Self-Improving in Large Language Models: The Singularity Is Not Near Without Symbolic Model Synthesis
11 points by darccio
11 points by darccio
Systems with non-vanishing exogenous grounding fall outside this regime. However, mainstream Singularity, AGI, and ASI narratives typically posit systems that become increasingly autonomous and require little to no human or external intervention for self-improvement. In that autonomy regime, the vanishing-signal condition is satisfied
AI systems can interact with the world and get feedback in the form of "did the program compile" or "did the robot enter the kitchen" or whatever. And indeed incorporating real-world feedback during training (so-called "Reinforcement Learning with Verifiable Rewards") is pretty much universal now, which is why model collapse as discussed in this paper has basically ceased to be a relevant talking point.
The author is apparently bucketing these kinds of things under "external intervention" (section 4.6 talks about this some), which I guess is fine; it just means this paper only applies under extremely narrow conditions. But I do not buy that mainstream narratives are assuming AI will not be running compilers and inspecting the output, as the abstract implies.
I think these sorts of pure-theory papers very rarely end up contributing much, and they harm the discourse because people end up assuming that they apply outside of the extremely narrow assumptions they make.
The intuition here is pretty straightforward: if you sample from a distribution, interpolate a new distribution from that, and repeat the process, you eventually just get something that is nothing but interpolation.
Their suggested solution is to fit a symbolic model like small Turing machine or causal graph to the previous model, then train the next model to fit that symbolic model. This does prevent the models from converging solely to whatever is most regular according to whatever interpolation method you're using originally, but it seems like it's just changing your interpolation method while still measuring convergence to the stable state of the original one.
If we break this down to the simplest case, imagine I have a set of (x,y) points I have measured. I fit a series of orthogonal polynomials to this with a regularization that wants to keep the size of the coefficients similar. If I take that fitted polynomial and sample new (x,y) pointed from it, they're going to be closer to a series with uniform coefficients, and if I keep going I end up with uniform coefficients eventually. If I add fitting a parsimonious tree or something to the polynomial before I generate samples for the next round I change where I converge, but I'm still converging somewhere that's largely unrelated to my initial measurements or the real world.
I'm fairly convinced that neural nets that have a distinction between training phase and the state carried inference time (LSTM's short term memory or the context window in transformers) is a dead end for AI research. My guess is that we're going to be looking at reinforcement learning of a system that can learn to play language games and generate something like Piagetian schemas in the process.
One link that was shared here a while ago had an interesting modification to transformers, which somehow enabled them to directly execute wasm bytecode reliably (not through tools, but as a "way of thinking")
this could reduce the hallucination by a good margin, and also give models good capabilities for deterministic work.
paired with some way to have some kind of hierarchical knowledge graphs as a long-term memory system as well as more "sensory input" (and not just text), we can probably achive way more flexible and powerful AIs.
but i'm not a pro and should look into how transformers actually work by building one myself...