Science Journalism is like a blurry image of science. Image courtesy of Joshua Sortino
I really enjoyed the recent New Yorker article by Ted Chiang 1 that draws an analogy between the way that lossy photo formats like JPEG store representations of images and the way that large language models store knowledge. I found this analogy to very useful. Its a great way to describe the current state of these models to folks tangential to ML and NLP without dropping into transformer architecture and attention mechanisms. The post has also drawn some criticism2 from scientists working in deep learning for not being “in keeping with our scientific understanding of LMs or deep learning”. Whilst Chiang may miss the finer strokes, the picture he paints is broadly representative. In a very meta way, his own work is like a blurry JPEG of how LLMs work. You might even consider that scientific journalism in general is like a blurry JPEG of scientific writing. I believe that in this context, such broad metaphor is ok most of the time. Let me explain.
Read more...