Posts tagged: python

All posts with the tag "python"

313 posts latest post 2026-05-06
Publishing rhythm
Jan 2026 | 3 posts
Don't inherit from python BaseException, Here's why.

I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have.

Try running these examples in a .py file for yourself and try to kill them with control-c.

Since things such as KeyboardInterrupt are created as an exception that inherits from BaseException, if you except BaseException you can no longer KeyboardInterrupt.

from time import sleep while True: try: sleep(30) except BaseException: # ❌ pass

except from Exception or higher #

If you except from exception or something than inherits from it you will be better off, and avoid unintended side effects.

...

How I make cache-keys from python objects

When I need a consistent key for a pythohn object I often reach for hashlib.md5 It works for me and the use cases I have.

Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object.

How I setup a sqlite cache in python

does not work

...

How I setup a sqlite cache in python

When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it.

Install diskcache into your virtual environement of choice using pip from your command line.

python -m pip install diskcache

setup the cache #

There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs

from diskcache import Cache cache = FanoutCache('.mycache', statistics=True)

Adding to the cache #

Adding to the cache only needs a key and value.

...

Cache a python function with lru_cache

The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache.

The easiest and most common to setup in python is a builtin functools.lru_cache.

from functools import lru_cache @lru_cache def get_cars(): print('pulling cars data') return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

when to use lru_cache #

Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache.

when same *args, **kwargs always return the same value

...

Set User Agent on pandas read_csv

I keep a small cars.csv on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate.

I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request.

This breaks my go to example dataset.

pd.read_csv("https://waylonwalker.com/cars.csv") # HTTPError: HTTP Error 403: Forbidden

But requests works??? #

What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does.

...

Get Webpage with python requests

Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.

Requests is on pypi and can be installed into your virtual environtment with pip.

python -m pip install requests

Getting the content of a request #

Requests makes getting content from a web url as easy as possible.

import requests r = requests.get('https://waylonwalker.com/til/htmx-get/') article = r.content

html">requests is not limited to html

...

Style Lookatme Slides a bit more Personal

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique.

Lets use this section to show what it looks like as I change my styles.

from markata import Markata Markata() markata.run()

☝ This is how my website is built

This is what the above slide looks like in lookatme.

...

How I load Markdown in Python

I use a package eyeseast/python-frontmatter{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml).

It’s on pypi, so you can install it into your virtual environment with pip.

python -m pip install python-frontmatter

🙋 What’s Frontmatter #

Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging...

...

Python's Dict Union Operator | Pep 584

Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584.

I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result.

default_config = {'url': 'https://example.com', 'assets_dir': 'static' } user_config = {'url':...

...

How I Present Markdown Slides from the Terminal

I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides.

I will most often just present right from the terminal using lookatme. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly.

I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share.

I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style.

...

How I glob for Files in Python

A very common task for any script is to look for files on the system. My go to method when globbing for files in python is to use pathlib.

I setup a directory to make some examples about globbing. Here is what the directory looks like.

❯ tree . . ├── content │ ├── hello.md │ ├── hello.py │ ├── me.md │ └── you.md ├── readme.md ├── README.md ├── READMES.md └── setup.py

1 directory, 8 files

Pathlib is a standard library module available in all LTS versions of python at this point.

...

pytest-mock Basics

Last Thursday I learned about pytest-mock at a local python meetup. The presenter showed how he uses pytest-mock for his work, and it was kinda eye opening. I knew what mocking was, but I had not seen it in this context.

Watching him use pytest-mock I realized that mocking was not as hard as I had made it out to be. You can install pytest-mock, use the mocker fixture, and patch objects methods with what you want them to be.

pytest-mock is out on pypi and can be installed with pip.

python -m pip install pytest-mock

What I actually did #

Sometimes I fall victim to making these posts nice and easy to follow. It takes more steps than just pip install, you need a place to practice in a nice sandbox. Here is how I make my sandboxes.

...

Python Walrus Inside List Comprehension

Python 3.8 came out two and a half years ago and I have yet to really lean in on the walrus operator. Partly because it always seemed like something kinda silly (my use cases) to require a python version bump for, and partly because I really didn’t understand it the best. Primarily I have wanted to use it in comprehensions, but I did not really understand how.

Now that Python 3.6 is end of life, and most folks are using at least 3.8 it seems time to learn and use it.

:=

The assignment operator in python is more commonly referred to as the walrus operator due to how := looks like a walrus. It allows you to assign and use a variable in a single expression.

...

Make Kedro Runs Beautiful

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.

There is no pypi package yet, but it’s on github. You can pip install it with the git url.

pip install git+https://github.com/datajoely/kedro-rich

Kedro run #

You can run your pipeline just as you normally would, except you get progress bars and pretty prints.

kedro run

kedro rich pretty run

Kedro catalog #

Listing out catalog entries from the command line now print out a nice pretty table.

...

Open Files with Nvim Remote

I recently found a really great plugin by mhinz to open files in neovim from a different tmux split, without touching neovim at all.

neovim-remote is not a neovim plugin at all, it’s a python cli that you can install with pip. Unlike the repo suggests, I use pipx to install nvr.

pipx install neovim-remote

How I use it #

I have this added to my .envrc that is in every one of my projects....

...

Textual Popup Hack

As I am toying around with textual, I am wanting some popup user input to take over. Textual is still pretty new and likely to change quite significantly, so I don’t want to overdo the work I put into it, So for now on my personal tuis I am going to shell out to tmux.

The main issue is that when you are in a textual app, it kinda owns the input. So if you try to run another python function that calls for input it just cant get there. There is a textual-inputs library that covers this, and it might work really well for some use cases, but many of my use cases have been for things that are pre-built like copier, and I am trying to throw something together quick.

textual is still very beta

Part of this comes down to the fact that textual is still very beta and likely to change a lot, so all of the work I have done with it is for quick and dirty, or fun side projects.

...

GitHub Markdown now Supports Mermaid Diagrams

Mermaid diagrams provide a way to display graphs defined as plain text. Some markdown renderers support this as a plugin. GitHub now supports it.

You can define nodes like this in mermaid, and GitHub will now render them as a pretty graph diagram. Its rendered in svg, so its searchable with control f and everything.

graph TD; A-->B; A-->C; B-->D; C-->D-->OUT; E-->F-->G-->OUT

Here is what the example looks like on GitHub

Glances can watch docker processes

Glances is a system monitor with a ton of features, including docker processes.

I have started using portainer to look at running docker processes, its a great heavy-weight docker process monitor. glances works as a great lightweight monitor to just give you the essentials, ( Name, Status, CPU%, MEM, /MAX, IOR/s, IOW/s, Rx/s, Tx/s, Command)

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances pipx inject glances "glances[docker]"

You will be presented with this success message.

...

Glances webui with pipx

Glances has a pretty incredible webui to view system processes and information like htop, or task manager for windows.

The nice thing about the webui is that it can be accessed from a remote system. This would be super nice on something like a raspberry pi, or a vm running in the cloud. Its also less intimidating and easier to search if you are not a terminal junky.

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances pipx inject glances "glances[web]"

You will be presented with this success message.

...