Today I Learned

Short TIL posts

174 posts latest post 2026-07-10 simple view
Publishing rhythm
Jul 2026 | 2 posts

Markata is a great python framework that allows you to go from markdown to a full website very quickly. You can get up and running with nothing more than Markdown. It is also built on a full plugin architecture, so if there is extra functionality that you want to add, you can create a plugin to make it behave like you want.

Full transparancy… I built markata.

The talk #

The talk is live on YouTube. Make sure you check out the other videos from the conference. There were quite a few quality talks that deserve a watch as well.

https://youtu.be/Wq9YBamSgs0

“An astronaut working in a lab, hacking on a computer terminal, htop is running, shallow depth of field beakers, test tubes, volumetric lighting, pink lighting, by victo ngai, killian eng vibrant colours, dynamic lighting, digital art” -s50 -W768 -H448 -C7.5 -Ak_lms -S3617210203

I spoke at python webconf in March 2022 about how I deploy this blog on a continuous basis.

Building this blog has brought me a lot of benefits. I have a set of custom curated notes to help describe a problem and how to solve it to me. At theis point it’s not uncommon to google an Issue I am having and finding my own blog with exactly the solution I need at the top.

I also bump into people from time to time that recognize me from the blog, its a nice conversation starter, and street cred.

The Talk #

The talk recently released on Youtube, you can watch it without having a ticket to the conference for free. There were a bunch of other talks that you should check out too!

https://youtu.be/JDZVjDCTKHE

I got all the pypi packages that I own behind 2 factor authentication. 💪

Recently this really made it’s rounds in the python news since pypi was requiring critical package maintainers to have 2FA on and even offering them hardware tokens to help them turn this on.

I feel like this caused a bit of confusion as turning on 2FA does not mean that you need to do anything different to deploy a package, and it DOES NOT require a hardware token. You can continue using your favorite 2FA app.

You might wonder what this means for my projects. It means that to edit any sensitive content such as pull a new api token, add/remove maintainers, or deleting a release I need to use a TOPT (time based one time password) application such as Google Authenticator, Microsoft Authenticator, Authy, or FreeOTP.

This has very little change to my overall workflow as my CI system still automatically deploys for me with the same api token as before.

This is one small thing that maintainers can do to prevent supply chain attacks on their projects that they put so much work into.

Login #

When I log in I now get this extra screen asking for an auth token.

pypi-2fa-code.webp

My packages #

Once I turned on 2FA for my account I could then turn on 2FA requirement for each project. I am not sure how much safety there is in pypi, it might require all maintainers to have it turned on before it allows packages to have it turned on.

my-pypi-packages-aug-2022.webp

Once turned on it requires anyone who maintains the project to have 2FA on to be able to edit any sensitive content.

I just love how some features of vim are so discoverable and memorable once you really start to grasp it. Sorting and uniqing your files or ranges is one of those examples for me.

" sort the file
:sort
" sort the file only keeping unique lines
:sort u


" sort a range
:'<,'> sort
" sort a range only keeping unique lines
:'<,'> sort u

I recently used this to dedupe my autogenerated links section for rich-syntax-range-style. More often I am using it to sort and uniqify objects like arrays and lists.

Here is what the markdown looks like.

* [py-tree-sitter](https://github.com/tree-sitter/py-tree-sitter)
* [rich](https://github.com/Textualize/rich)
* [@textualizeio](https://twitter.com/textualizeio)
* [rich](https://github.com/Textualize/rich)
* [another post](https://waylonwalker.com/designing-kedro-router)
* [print-register-pipelines](https://screenshots.waylonwalker.com/print-register-pipelines.webp)
* [rich](https://github.com/Textualize/rich)
* [console-print-register-pipelines](https://screenshots.waylonwalker.com/console-print-register-pipelines.webp)
* [rich](https://github.com/Textualize/rich)
* [syntax-print-register-pipelines](https://screenshots.waylonwalker.com/syntax-print-register-pipelines.webp)
* [rich](https://github.com/Textualize/rich)
* [syntax-print-register-pipelines-highlight-line](https://screenshots.waylonwalker.com/syntax-print-register-pipelines-highlight-line.webp)
* [py-tree-sitter](https://github.com/tree-sitter/py-tree-sitter)

Then typing vap:sort u yields a uniqly sorted list of links.

* [@textualizeio](https://twitter.com/textualizeio)
* [another post](https://waylonwalker.com/designing-kedro-router)
* [console-print-register-pipelines](https://screenshots.waylonwalker.com/console-print-register-pipelines.webp)
* [print-register-pipelines](https://screenshots.waylonwalker.com/print-register-pipelines.webp)
* [py-tree-sitter](https://github.com/tree-sitter/py-tree-sitter)
* [rich](https://github.com/Textualize/rich)
* [syntax-print-register-pipelines-highlight-line](https://screenshots.waylonwalker.com/syntax-print-register-pipelines-highlight-line.webp)
* [syntax-print-register-pipelines](https://screenshots.waylonwalker.com/syntax-print-register-pipelines.webp)

Today I’ve been playing with py-tree-sitter a bit and I wanted to highlight match ranges, but was unable to figure out how to do it with rich, so I reached out to @textualizeio for help.

https://twitter.com/_WaylonWalker/status/1562469770766589952

While waiting for that reply let’s show how we got this far.

imports #

Lets import all the classes that we need from rich and setup a console to print to.

from rich.console import Console
from rich.syntax import Syntax
from rich.style import Style

console = Console()

some code #

Now we need some code to highlight. I am going to rip my register_pipeline from another post.

code = '''
from find_kedro import find_kedro

def register_pipelines(self) -> Dict[str, Pipeline]:
    """Register the project's pipeline.
    Returns:
        A mapping from a pipeline name to a ``Pipeline`` object.
    """
    return find_kedro()
'''

print #

We could simply print out the code we have as a variable, but thats a bit hard to read.

print-register-pipelines.webp

console.print #

printing with rich’s console makes it a little better, but not much by default.

console-print-register-pipelines.webp

Syntax #

We can pull from rich’s syntax module to really pretty this up.

syntax = Syntax(code, 'python', line_numbers=True)
console.print(syntax)
syntax-print-register-pipelines.webp

Now we are getting some really impressive print outs right in the terminal!

note that I have ipython set to use rich, you will need to console.print() in scripts

highlight lines #

Now we can start highlighting lines right when we initialize our Syntax instance. It looks ok. It’s not super visible, but more importantly its not granular enough. I want to highlight specific ranges like the word register_pipelines.

syntax = Syntax(code, 'python', line_numbers=True, highlight_lines=[4])
console.print(syntax)
syntax-print-register-pipelines-highlight-line.webp

This hows the line, but still is not very accurate.

highlight text #

[@textualizeio] got back to me, let’s see if What we can do with stylize_range!

https://twitter.com/textualizeio/status/1562487302274043904

syntax = Syntax(code, 'python', line_numbers=True)
style = Style(bgcolor='deep_pink4')
syntax.stylize_range(style, (4, 4), (4, 22))
console.print(syntax)

This gives us the final result we are looking for, we can easily see what is being targeted here. In this case the function name register_pipelines.

syntax-highlight-range-register-pipelines.webp

This turns out to be exacly what I am looking for. Now I have an easy way to print out highlighted code wtih my py-tree-sitter query results.

How to vimgrep over hidden files.

I needed to delete all build pipeline steps that were named upload docs. I currently have about 60 projects running from the same template all running very similar builds. In the past I’ve scripted out migrations for large changes like this, they involved writing a python script that would load the yaml file into a dictionary, find the corresponding steps make the change and write it back out.

Today’s job was much simplar, just delete the step, were all steps are surrounded by newlines. My first thought was to just open all files in vim and run dap. I just needed to get these files:positions into my quickfix. My issue is that all the builds reside within hidden directories by convention.

The issue #

variability

After searching through all the projects it was clear that all the steps were in their own paragraph, though I was not 100% confident enough to completely automate it, and the word upload docs was in the paragraph.

some were a two liner

- name: upload docs
  script: aws s3 ...

Some had a variation in the name

- name: upload docs to s3
  script: aws s3 ...

some were more than 2 lines.

- name: upload docs
  script: |
    aws s3 ...

some used a different command.

- name: upload docs
  script: |
    python ...

Templates are great #

but they change

Templates are amazing, and tools like cookiecutter and copier are essential in my workflow, but those templates change over time. Some things are a constant, and others like this one are an ever evolving beast until they are tamed into something the team is happy with.

vimgrep over hidden files #

I know all the files that I care to search for are called build.yml, and they are in a hidden directory.

:args `fd -H build.yml`
:vimgrep /upload docs/ ##

Once opened as a buffer by using args, and a handy fd command I can vimgrep over all the open buffers using ##

Open buffers are represented by ##

Now I can just dap and :cnext my way through the list of changes that I have, and know that I hit every one of them when I am at the end of my list. And can double check this in about 10s by scrolling back through the quickfix list.

Vim points achieved #

You’re not a true vim enthusiast until you have spent 10 minutes writing a blog post about how vim saved you 5 minutes. Check out all the other times this has happened to me in the vim tag.

image from Dall-e

a sprinter edging out his opponent by Dall-e

It’s about time to release Markata 0.3.0. I’ve had 8 pre-releases since the last release, but more importantly it has about 3 months of updates. Many of which are just cleaning up bad practices that were showing up as hot spots on my pyinstrument reports

Markata started off partly as a python developer frustrated with using nodejs for everything, and a desire to learn how to make frameworks in pluggy. Little did I know how flexible pluggy would make it. It started out just as my blog generator, but has turned into quite a bit more.

Over time this side project has grown some warts and some of them were now becoming a big enough issue it was time to cut them out.

Let’s compare #

I like to use my tils articles for examples and tests like this as there are enough articles for a good test, but they are pretty short and quick to render.

mkdir ~/git/tils/tils
cp  ~/git/waylonwalker.com/pages/til/ ~/tils/tils -r
cd ~/git/tils/tils

running tils on 0.2.0 #

At the time of writing this is the current version of markata, so just make a new venv and run it.

python3 -m venv .venv --prompt $(basename $PWD)
pip install markata
markata clean
markata build

cold tils: 14.523 warm tils: 1.028

running tils on 0.3.0b8 #

python3 -m venv .venv --prompt $(basename $PWD)
# --pre installs pre-releases that include a b in their version name
pip install markata --pre
markata clean
markata build

cold tils: 11.551 (+20%) warm tils: 0.860 (+16%)

pyinstrument #

These measurements were taken with pyinstrument mostly out of convenience since there is already a pyinstrument hook built in, but also because I like pyinstrument.

pyinstrument-markata==0.3.0b8-tils-hot.webp

Here is the pyinstrument report from the last run.

My Machine #

This comparison was not very exhaustive. It was ran on my pretty new to me Ryzen 5 3600 machine.

neofetch-8-21-2022.webp

The changes #

Most of these changes revolve in how the lifecycle is ran. It was trying to be extra cautious and run previous steps for you if it thought it might be needes, in reality it was rerunning a few steps multiple times no matter what.

The other thing I turned off by default, but can be opted into, is beautifulasoup’s prettify. That was one of the slower steps ran on my site.

0.3.0 #

It should be out by the time you see this, I wanted to compare the changes I had made and make sure that it was still making forward progress and thought I would share the results.

Deliberative #

People exceptionally talented in the Deliberative theme are best described by the serious care they take in making decisions or choices. They anticipate obstacles.

I am risk-adverse. I want everything well thought out and calculated before I make any sort of change. I have never gambled in my life and just the thought of it makes me anxious.

Aim it #

I can use this as a strength to plan out potential issues and prevent them. I do this quite often with my role in infrastructure.

I need to make sure that I use deadlines to keep this as a strength and not hinderence.

Automation #

One of the biggest ways that I utilize this skill is automation. I am all about automating things, not just because I don’t want to do the manual work, but I am not sure when I am going to need to do something again.

A common meta thing that I need in python is to find the version of a package. Most of the time I reach for package_name.__version__, but that does not always work.

but not all projects have a __version__ #

In searching the internet for an answer nearly every one of them pointed me to __version__. This works for most projects, but is simply a convention, its not required. Not all projects implement a __version__, but most do. I’ve never seen it lie to me, but there is nothing stopping someone from shipping mismatched versions.

If you maintain a project ship a __version__ #

I appreciate it

While its not required its super handy and easy for anyone to remember off the top of their head. It makes it easy to start debugging differences between what you have vs what you see somewhere else. You can do this by dropping a __version__ variable inside your __init__.py file.

## __init__.py
__version__ = 1.0.0

SO #

stack overflow saves the day

Special thanks to this Stack Overflow post for answering this question for me.

So what do you do… #

importlib

Your next option is to reach into the package metadata of the package that you are interested in, and this has changed over time as highlighted in the stack overflow post.

for Python >= 3.8:

from importlib.metadata import version

version('markata')
# `0.3.0.b4`

I only really use python >= 3.8 these days, but if you need to implement it for an older version check out the stack overflow post.

Another option.. #

use the command line

Another common option uses pip at the command line.

❯ pip show markata
Name: markata
Version: 0.3.0b4
Summary: Static site generator plugins all the way down.
Home-page: https://markata.dev
Author: Waylon Walker
Author-email: [email protected]
License: MIT
Location: /home/waylon/git/waylonwalker.com/.venv/lib/python3.11/site-packages
Requires: anyconfig, beautifulsoup4, checksumdir, diskcache, feedgen, jinja2, more-itertools, pathspec, pillow, pluggy, pymdown-extensions, python-frontmatter, pytz, rich, textual, toml, typer
Required-by:

And if the package implements a command line its common to ship a version command such as --version or -V.

❯ markata --version
Markata CLI Version: 0.3.0.b4

Why did I need to do this? #

Well we have a cli tool that wraps around piptools and we wanted to include the version of piptools in the comments that it produces dynamically. This is why I wanted to dynamically grab the version inside python without shelling out to pip show. Now along with the version of our internal tool you will get the version of piptools even though piptools does not ship a __version__ variable.

Fin #

In the end, I am glad I learned that its so easy to use the more accurate package metadata, but still appreciate packages shipping __version__ for all of us n00b’s out here.

Astronaut doing a mic drop with explosion

Recently I added two new bash/zsh aliases to make my git experience just a tad better.

trackme #

Most of our work repos were recently migrated to new remote urls, we scriped out the update to all of the repos, but I was left with a tracking error for all of my open branches. To easily resolve this I just made an alias so that I can just run trackme anytime I see this error.

There is no tracking information for the current branch.
    Please specify which branch you want to merge with.
    See git-pull(1) for details

    git pull <remote> <branch>

If you wish to set tracking information for this branch you can do so with:

    git branch --set-upstream develop origin/<branch>

getting the branch #

The following command will always return the currently checked out branch name.

git symbolic-ref --short HEAD

Injecting this into the suggested git command as a subshell gives us this alias that when ran with trackme will automatically fix tracking for my branch.

alias trackme='git branch --set-upstream-to=origin/$(git symbolic-ref --short HEAD)'

rebasemain #

I sometimes get a bit lazy at checking main for changes before submitting any prs, so again I made a quick shell alias that will rebase main into my branch before I open a pr.

alias rebasemain='git pull origin main --rebase'

The Aliases #

Here are both of the alias’s, feel free to steal and modify them into your dotfiles. If you are uniniatiated a common starting place to put these is either in your ~/.bashrch or ~/.zshrc depending on your shell of choice.

alias trackme='git branch --set-upstream-to=origin/$(git symbolic-ref --short HEAD)'
alias rebasemain='git pull origin main --rebase'

So many terminal applications bind q to exit, even the python debugger, its muscle memory for me. But to exit ipython I have to type out exit<ENTER>. This is fine, but since q is muscle memory for me I get this error a few times per day.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-1-2b66fd261ee5>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NameError: name 'q' is not defined

After digging way too deep into how IPython implements its ExitAutoCall I realized there was a very simple solution here. IPython automatically imports all the scripts you put in your profile directory, all I needed was to create ~/.ipython/profile_default/startup/q.py with the following.

q = exit

It was that simple. This is not a game changer by any means, but I will now see one less error in my workflow. I just press q<Enter> and I am out, without error.

It’s no secret that I love automation, and lately my templating framework of choice has been copier. One hiccup I recently ran into was having spaces in my templated directory names. This makes it harder to run commands against as you need to escape them, and if they end up in a url you end up with ugly %20 all over.

Cookiecutter has the solution #

Yes the solution comes from a competing templating framework.

I install copier with pipx, so I need to inject cookiecutter in to my copier environment to use the slugify filter.

pipx inject copier cookiecutter

If you are using a normal virtual environment you can just pip install it.

pip install copier cookiecutter

add the extension to your template #

copier.yml

Now to enable the extension you need to declare it in your copier.yml file in your template.

_jinja_extensions:
    - cookiecutter.extensions.SlugifyExtension

Use it | slugify #

use-it

Now to use it, anywhere that you want to slugify a variable, you just pipe it into slugify.

❯ tree .
.
├── copier.yml
├── README.md
└── {{ site_name|slugify }}
    └── markata.toml.jinja

1 directory, 3 files

Here is a slimmed down version of what the copier.yml looks like.

site_name:
  type: str
  help: What is the name of your site, this shows in seo description and the site title.
  default: Din Djarin

_jinja_extensions:
    - cookiecutter.extensions.SlugifyExtension

Results #

Running the template looks a bit like this.

copier-cookiecutter-slugify.webp

straight from their docs #

The next section is straight from the cookiecutter docs

Slugify extension #

The cookiecutter.extensions.SlugifyExtension extension provides a slugify filter in templates that converts string into its dashed (“slugified”) version:

{% "It's a random version" | slugify %}

Would output:

it-s-a-random-version

It is different from a mere replace of spaces since it also treats some special characters differently such as ' in the example above. The function accepts all arguments that can be passed to the slugify function of python-slugify_. For example to change the output from it-s-a-random-version to it_s_a_random_version, the separator parameter would be passed: slugify(separator='_').

Textual has devtools in the upcoming css branch, and its pretty awesome!

It’s still early #

Textual is still very early and not really ready for prime time, but it’s quite amazing how easy some things such as creating keybindings is. The docs are coming, but missing right now so if you want to use textual be ready for reading source code and examples.

On to the devtools #

As @willmcgugan shows in this tweet it’s pretty easy to setup, it requires having two terminals open, or using tmux, and currently you have to use the css branch.

https://twitter.com/willmcgugan/status/1531294412696956930

Why does textual need its own devtools #

Textual is a tui application framework. Unlike when you are building cli applications, when the tui takes over the terminal in full screen there is no where to print statement debug, and breakpoints don’t work.

getting the css branch #

In the future it will likely be in main and not need this, but for now you need to get the css branch to get devtools.

git clone https://github.com/Textualize/textual
git fetch --alll
git checkout css

install in a virtual environment #

Now you can create a virtual environment, feel free to use whatever virtual environment tool you want, venv is built in to most python distributions though, and should just be there.

python3 -m venv .venv --prompt textual
source .venv/bin/activate
pip install .

Now that we have textual installed #

Once textual is installed you can open up the devtools by running textual console.

textual console
textual-console.webp

Using Different versions of python with pipx | pyenv

[1] I love using pipx for automatic virtual environment [2] management of my globally installed python cli applications, but sometimes the application is not compatible with your globally installed pipx Which version of python is pipx using?? # [3] This one took me a minute to figure out at first, please let me know if there is a better way. I am pretty certain that this is not the ideal way, but it works. My first technique was to make a package that printed out sys.version. # what version of python does the global pipx use? pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers # what version of python does the local pipx use? python -m pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers Let’s setup some other versions of python with pyenv # [4] If you don’t already have pyenv [5] installed, you can follow their install instructions [6] to get it. pyenv install 3.8.13 pyenv install 3.10.5 I usually require a virtual environment # [7] I set the PIP...
2 min read