Content tagged with "Python"

I recently ran into a strange issue where I was struggling to get the GCloud SDK to work properly on Mac 14.2.1.

I used brew to install the package brew install google-cloud-sdk and added it to my zshrc file.

After I logged in with gcloud auth login I wanted to copy some files out of a GCP bucket to my local machine with the storage cp command:

Read more...

I’ve seen a number of tutorials that incorrectly configure django to run inside docker containers by leveraging it’s built in dev server. In this post I explore the benefits of using django with gunicorn and nginx and how to set this up using Docker and docker-compose.

Read more...

I was recently attempting to cache the results of a long-running SQL query to a local parquet file using SQL via a workflow like this:

import os
import pandas as pd
import sqlalchemy

env = os.environ

engine = sqlalchemy.create_engine(f"mysql+pymysql://{env['SQL_USER']}:{env['SQL_PASSWORD']}@{env['SQL_HOST']}/{env['SQL_DB']}")

connection = engine.connect()
with engine.connect() as conn:
    df = pd.read_sql("SELECT * FROM articles", connection)


df.to_parquet("articles.parquet")Read more...


    

As part of my work on Gastronaut, I'm building a form that allows users to create recipes and which will attempt to parse ingredients lists and find a suitable stock photo for each item the user adds to their recipe. As well as being cute and decorative, this step is important for later when we want to normalise ingredient quantities.

What we're looking to do is take a string like "2lbs flour" and turn it into structured data - a json representation might look like this:

Read more...

Gitea actions is the new Github-compatible CI/automation pipeline feature that ships with Gitea and Forgejo. In theory it is interoperable with Github actions but there are still a few rough edges and for that reason, the feature is still disabled by default.

I have been trying to get a django project that uses PDM for Python dependency management to to install itself and run some tests in a Gitea CI environment.

Read more...

This library does solve a very practical issue in python and looks very useful.

Note: If you don’t want to read the blah-blah context and history stuff then you can jump to the recommendations

The Problem

The need for virtual python environments becomes fairly obvious early in most Python developers’ careers when they switch between two projects and realise that they have incompatible dependences (e.g. project1 needs scikit-learn-0.21 and project2 needs scikit-learn-0.24). Unlike other mainstream languages like Javascript(Node.js) and Java (with Maven) where dependencies are stored locally to the project, Python dependencies are installed at system or environment level and affect all projects that are using the same environment.

Read more...

Introduction

Open machine learning research is undergoing something of a reproducibiltiy crisis. In fairness it’s not usually the authors’ fault - or at least not entirely. We’re a fickle industry and the tools and frameworks were ‘in vogue’ and state of the art a couple of years ago are now obsolete. Furthermore, academics and open source contributors are under no obligation to keep their code up to date. It is often left up to the reproducer to figure out how to breathe life back into older work.

Read more...

I recently came across an infuriating problem where an MLFlow python model I had trained on one system using Python 3.6 would not load on another system with an identical version of Python.

The exact problem was that when I ran mlflow models serve -m <url/to/model/in/bucket> the service would crash saying that the model could not be unserialized because ValueError: unsupported pickle protocol: 5.

Read more...