Published on
by 
James Ravenscroft
Fair warning: Some of the quotes in this post are a bit sweary.
I've been reflecting on a few articles that I've read over the last couple of days about consumer trends in devices and parallels with generated content. I'm sure everyone's sick of "yet another hot take on enshittification" - but what about an optimistic one?
Published on
by James Ravenscroft
Typesetting blocks form alphabet spaghetti a bit like a language model might spit out. Photo by Raphael Schaller on Unsplash
There is sooo much hype around LLMs at the moment. As an NLP practitioner of 10 years (I built Partridge 1 in 2013), it’s exhausting and quite annoying and amongst the junior ranks, there’s a lot of despondency and dejection and a feeling of “what’s the point? ClosedOpenAI have solved NLP”.
Today I read via Simon Willison’s blog that [someone has managed to get LlaMA running on a raspberry pi]. That’s pretty incredible progress and it made me think of this excerpt from Hitchiker’s Guide To the Galaxy:
O Deep Thought computer," he said, “the task we have designed you to perform is this. We want you to tell us….” he paused, “The Answer.”
“The Answer?” said Deep Thought. “The Answer to what?”
Read more...
Published on
by James Ravenscroft
ChatGPT is a really impressive tech demo and it shows us the power of large language models but it’s important to remember that ChatGPT is a machine learning model and, like any AI, it’s only as good as the data it’s trained on. This means that it’s prone to making errors, and it’s important for humans to validate the answers it produces. I fully expect any executives wringing their hands with glee about “cutting resource” and making redundancies are going to have a real shock when they realise that they still need those people to supervise the model and verify its outputs. So maybe our relationship with coding changes and the quality and speed with which we can build systems increases. However, would you ask GPT “generate the control code for a pacemaker” and trust that device to help your own Grandma or would you prefer a team of medical systems engineer with 20+ years experience to review that code first?
Read more...
Published on
by James Ravenscroft
Token dropping aims to accelerate the pretraining of transformer models such as BERT without degrading its performance on downstream tasks.
A BERT model pretrained using this token dropping method is not different to a BERT model pretrained in the conventional way: a BERT checkpoint pretrained with token dropping can be viewed and used as a normal BERT checkpoint, for finetuning etc.
Published on
by James Ravenscroft
Published on
by James Ravenscroft
This is a superb visual summary of the new generation retrieval transformer models (e.g. DeepMind RETRO).
Published on
by James Ravenscroft
MLFlow is a powerful open source MLOps platform with built in framework for serving your trained ML models as REST APIs. The REST framework will load data provided in a JSON or CSV format compatible with pandas and pass this directly into your model. This can be handy when your model is expecting a tabular list of numerical and categorical features. However it is less clear how to serve with models and pipelines that are expecting unstructured text data as their primary input. In this post we will explore how to train and then serve an NLP model using MLFlow, scikit-learn and spacy.
Read more...In recent weeks and months the impending global climate catastrophe has been at the forefront of many peoples’ minds. Thanks to movements like Extinction Rebellion and high profile environmentalists like Greta Thunberg and David Attenborough as well as damning reports from the IPCC, it finally feels like momentum is building behind significant reduction of carbon emissions. That said, knowing how we can help on an individual level beyond driving and flying less still feels very overwhelming.
Read more...
Published on
by James Ravenscroft
If you’re a frequent user of spacy and virtualenv you might well be all too familiar with the following:
python -m spacy download en_core_web_lg
Collecting en_core_web_lg==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz#egg=en_core_web_lg==2.0.0
Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_lg-2.0.0/en_core_web_lg-2.0.0.tar.gz (852.3MB)
5% |█▉ | 49.8MB 11.5MB/s eta 0:01:10
If you’re lucky and you have a decent internet connection then great, if not it’s time to make a cup of tea.
Read more...