The fuuuucking annoying part is that these weight trained models would be perfect for translation models, compression, etc. An llm is already kind of a really efficient lossy compressor but you could actually make it lossless and an actual compressor if used properly. But instead people are literally telling llms to translate instead of training models that are for translating. The technology isnt the problem itself, its the industry and capitalism.
There are not many places where 95% compression of UTF-8 plain text could outweigh needing a 30GB model in memory and a significant fraction of current LLM inference cost to decompress it − and good luck convincing librarians to adopt it.
The image versions are probably more useful though.
The important part for these codes is that it has pre-LLM functions for quality metrics to judge if the output is close enough to indistinguishable (preserves detail, doesn’t add any).
Although there is also variants deriving a neural net from the media to recreate it from the smaller model.
The fuuuucking annoying part is that these weight trained models would be perfect for translation models, compression, etc. An llm is already kind of a really efficient lossy compressor but you could actually make it lossless and an actual compressor if used properly. But instead people are literally telling llms to translate instead of training models that are for translating. The technology isnt the problem itself, its the industry and capitalism.
There are not many places where 95% compression of UTF-8 plain text could outweigh needing a 30GB model in memory and a significant fraction of current LLM inference cost to decompress it − and good luck convincing librarians to adopt it.
There’s a 300 MB library for audio compression using it. If you have large audio libraries it could eventually become worth the tradeoff.
https://huggingface.co/facebook/encodec_32khz
The image versions are probably more useful though.
The important part for these codes is that it has pre-LLM functions for quality metrics to judge if the output is close enough to indistinguishable (preserves detail, doesn’t add any).
Although there is also variants deriving a neural net from the media to recreate it from the smaller model.
lol no you cannot make it lossless.
Yeah you can? A neural network that only relies on wheights and doesnt use random numbers will always give the same output for the same input.
So does lossy compression… same output for input of an algorithm is deterministic, not lossless…
Enhance!
I think, they don’t mean lossless compression with LLM, but with a neural network in general. I think it might be possible, but I’m not sure