Can gzip be a language model?
- A while back I wrote about language modeling without neural networks, where I generated Shakespeare with an unbounded n-gram model: no weights, no training, just counting.
- Fortuitously, I came across the paper Language Modeling is Compression, which mentioned the compression–prediction equivalence:every prediction model is inherently a compressor, and all compression algorithms are prediction models.
- This led to the natural question: can gzip do language modeling?1 No neural network, no learned parameters, nothing.
Unverified
- A while back I wrote about language modeling without neural networks, where I generated Shakespeare with an unbounded n-gram model: no weights, no training, just counting.
- Fortuitously, I came across the paper Language Modeling is Compression, which mentioned the compression–prediction equivalence:every prediction model is inherently a compressor, and all compression algorithms are prediction models.
- This led to the natural question: can gzip do language modeling?1 No neural network, no learned parameters, nothing.
Sources: Nathan