How Is Compression Prediction?
14 points by lukefleed
14 points by lukefleed
The math here is frankly a bit over my head, but just yesterday I read this same idea in What Is Intelligence? by Blaise Agüera y Arcas.
In Chapter 5, "Other Minds", there's this footnote:
There's likely no sharp boundary between general skills and specific knowledge...because everything you learn is represented in terms of your existing conceptual vocabulary, and in turn extends that vocabulary. One can think of it almost like a compression algorithm, in which one's experiences so far make up the "dictionary" with which subsequent knowledge and experiences are compressed. This may at least partly explain why time seems to pass more quickly as we age: compressing our life experiences more efficiently—alas.
The book is about how intelligence is, at its heart, prediction. Seen this way, our brains (and many other biological predictors) are meat-computers that implement really good compression algorithms!
Idk, I think OP's point, as I understand it, is more that though on the surface the claim that compression and prediction are equivalent is true, it's also needlessly reductionist to the point of being almost tautological and really removes the interesting meat of what's going on.
The identity above assumes that the model QQ is already available. In practice, however, the source distribution is usually unknown and must be estimated from the same sequence we want to encode. This raises a question: does empirical entropy describe a genuine property of the sequence, or only the probabilistic model we have fitted to it?
OP seems to be saying that the more interesting part to look at it is how dictionaries are formed, how they are pre-shared, how efficiently you can do that work, and how much of a corpus you need to extract a good enough predictive model: once you have that model, the actual encoding/decoding part is trivial and not especially interesting.
I think the dictionary is one example of a slightly broader issue. Before we can talk about prediction, we have already decided what counts as an object, how it is serialized, what the decoder already knows, and what operations the representation must support. Once those choices and the model are fixed, turning probabilities into code lengths is almost the mechanical part. I wouldn’t call the actual coder trivial to implement, though. It can get pretty hairy in practice.
3blue1brown effectively covers this in his "Reinventing Entropy" series, too, and it's quite an enjoyable watch.