Compression is prediction
33 points by gmem
33 points by gmem
Of course, and there is this awesome project (not mine) that actually uses gzip as a language model.
Youtuber 3blue1brown has a good series on this (still in-progress at time of writing). The first video in the series: https://www.youtube.com/watch?v=l6DKRf-fAAM
I second this. If you haven't watched 3blu1brown series on this it's excellent. These ideas are old and looking at LLMs like this is also relatively old. I'm happy to see people having these 'Aha' moments discovering this on their own!
Here is Claude Shannon's (founder of information theory) paper about combining the idea of entropy and prediction. https://www.princeton.edu/~wbialek/rome/refs/shannon_51.pdf
Apparently he used his wife as a sort of LLM.
Here is a great paper called Language Modeling Is Compression (2023) https://arxiv.org/abs/2309.10668 by Grégoire Delétan
Repo from DeepMind https://github.com/google-deepmind/language_modeling_is_compression
The idea itself has been, of course, popularised well before the current wave of language models: https://en.wikipedia.org/wiki/Hutter_Prize for a version with money behind the promotion
To add, the Hutter prize was originally set in hopes that progress in compression will lead to improvements in AI. Ironically, it worked out the other way round -- the best modern text compression algorithms (measuring output size only, ignoring performance) are based on LLMs.
One could argue, however, whether some of the steps in ML progress were basically training models to compress/decompress text with not too much (prediction) error. Cross-feeding between the two tasks seems (to me) quite aligned with the idea of Hutter prize.
Of course. And "AI hallucinations" are artifacts from lossy compression, like JPEG artifacts confidently mis-modeling reality with statistical noise.
Not exactly JPEG artifact like, it's more like lossy vectorisation, or maybe similar glyph grouping, sometimes that cleans up the noise a bit and goes to the median in a desired (under some circumstances) way, sometimes it mixes up 6 and 8 like https://www.dkriesel.com/en/blog/2013/0802_xerox-workcentres_are_switching_written_numbers_when_scanning