Posts tagged: python Archive

1fa2e162-deeb-4bb5-b1c5-961632abd452.webm

Deprecation warnings are so easy to miss, ignore, become numb to. Creating tools and processes to catch and address these issues is important. I’m surprised such big projects let deprecations just hang around for years.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

One Year Of Shots

2025-12-09T12:44:39Z

I’ve been running my shot scraper api for a year now. It creates og images for my website and thumbnails for my [[ reader ]] using a headless chrome instance.

25870 shots
73 shots per day on average
12-09-2025 first shot taken

Histogram #

a histogram of shot counts by day

You can see in the histogram that I’ve had a few big spike days, This has been mostly for days that I’ve integrated into a new service or changed the endpoint. On February 13, 2025 I swapped over from using the post to using template specific to open graph images.

-content = "https://shots.waylonwalker.com/shot/?url={{ config.url }}{{ post.slug }}&height=600&width=1200&scaled_width=1200&scaled_height=600"
+content = "https://shots.waylonwalker.com/shot/?url={{ config.url }}{{ post.slug }}/og/&height=600&width=1200&scaled_width=1200&scaled_height=600"

Image Comparison

Original Post Image

originally I simply used an image of the post itself

New OG Image

In Feb 2025 I made OG specific templates to use for the OG images.

Swapping to og images

Collage #

For fun I made a collage of all the shots. It’s cool to see all of these together, I remember a lot of the thumbnails and posts. Many of them from my rss reader.

collage of all shots, click to see full size

I tried to make a video collage, but turns out it takes a long time to show all 25k shots in video form. It also turned out to be a bit of a strobe as I don’t yet have it figured out how to dark mode in headless chrome.

video collage of shots

video collage of 600 shots, I tried to include all, but even at 20fps its a 20 minute video.

Workspaces V1

2025-11-26T20:17:29Z

I’ve been using the same workflow since 2020, my post How I navigate tmux in 2021 is still very accurate to my workflow. It’s been great, but is a starting to feel like it needs some work. It feels like enough time has passed that I can make some significant improvements.

Issues #

COLUMNS env var

2025-11-26T13:24:38Z

setting COLUMNS env var to a number greater than 0 will make the terminal resize to that number of columns.

COLUMNS=80 uvx --from rich-cli rich myscript.py

Note

Not all programs respct the COLUMNS env var, but rich does, and a lot of stuff I’m building uses rich.

I discovered this when I was trying to make a low effort readme generated from the code, but did not depend on the size of terminal it was ran on.

# justfile
readme:
    echo "# Workspaces" > README.md
    echo "" >> README.md
    echo '``` bash' >> README.md
    COLUMNS=80 ./workspaces.py --help >> README.md
    echo '```' >> README.md

side effect of just stars

2025-11-11T21:33:33Z

I found an interesting side effect of manually running my script to generate [[ stars ]] posts is that you get notified when one gets renamed. Today I noticed that Ned Batchelder created a coveragepy org.

pydantic settings alias choices

2025-11-11T21:23:58Z

Today I learned how to use AliasChoices with pydantic settings to setup common aliases for the same field. I’m bad about remembering these things, and hate looking up the docs. I like things to be intuitive and just do the thing I want it to do. Especially when they get configured through something like yaml and do not have a direct lsp look up right from my editor. I figured out how to support what might be common aliases for a storage directory. These can be set up as environment variables and used by config.

from pathlib import Path

from pydantic import Field
from pydantic import AliasChoices
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    storage_dir: Path | None = Field(
        default=None,
        validation_alias=AliasChoices(
            "STORAGE_DIR", "STORAGE_DIRECTORY", "STORAGE_PATH", "STORAGE_PATHNAME",
            "DROPPER_STORAGE_DIR", "DROPPER_STORAGE_DIRECTORY", "DROPPER_STORAGE_PATH", "DROPPER_STORAGE_PATHNAME",
        ),
        description="Directory for stored files",
    )

Mcat Anything

2025-11-05T20:10:02Z

I’ve long looked for a way to cat anything in the terminal. I’m am terminally in the terminal. I manage all of my projects, code, website, notes, files, servers, infrastructure, almost everything from the terminal. I occasionally open a file manager, mostly at home, only so that I can browse images.

Compounding my issue, I’m a tmux user. It works great for me, and I barely have to think about it at this point. The keybindings are second nature to me. I can go between server, terminal, nvim, and between projects instantly, no loader, no lag, no animation, it just works for everything that really matters to me for really getting things done.

Mcat #

mcat is a new tool that seems like it can cat anything in the terminal, code, files, images, markdown, markdown with images, and even video, without leaving tmux!

mcat static/8bitcc.png
curl https://r.jina.ai/https://waylonwalker.com/store/ | mcat --theme dracula --md-image all
curl https://r.jina.ai/https://waylonwalker.com/shots/ | mcat --theme dracula
mcat ~/git/dropper/data/01b21044-0bf8-4b06-9db1-a002c0519df6.mp4
mcat ~/git/dropper/data/0e659c05-1c12-4524-aa54-ef52ba680865.webm

But it doesn’t always work #

The keen eyed of you will notice the blank screen at the start of the demo above, not sure what happened, but resizing the terminal fixed something. Sometimes video comes up as an image, sometimes it core dumps, once even it crashed my system…

The hope is there, I’ll continue to use it occasionally, but right now it feels like a novelty that is fun to look at when it works. For now I’m going to stick to my kitty/tmux combo that works even though they seem to hate each other and I’m going to be happy just getting shit done.

Missing Thoughts

2025-11-05T07:23:35Z

No one is perfect, this is why we have things like checkpoints or gates in the form of pull requests, linting, type checking, and tests. What happens when you work on small side projects by yourself that try to be content focused? What happens when you end up building a lot of the tech under that site and build it on the bleeding edge of all the tech you make? They are likely missing these things and occasionally there are some periods of regression. This is one reason I really like the term digital garden to describe one’s small corner of the internet where they share their thoughts.

There will be regressions

The Signs #

There were signs, signs I did not notice

Chat

is your rss feed broken?

I’m not seeing anything show up in my rss reader

Do I not put thoughts in my rss feed, I swore I did.

Chat

my fault, Turns out I must have already clicked it in my reader.

great, glat it’s working

…But it wasn’t

Later this week comes the next sign that I also choose to ignore… Google search has unindexed a number of pages due to a soft 404. I don’t look at all of the emails from search console, but I did happen to see this one, and thought “Huh, that’s odd” and went on with my day without another thought on the issue.

Huh, that’s odd

The Discovery #

I didn’t realize I even had an issue until later that week when I went to my [[ archive ]] myself and noticed the shape of it looks off. When I opened up my feed something felt off, it was only the big posts. Thoughts show up as a different kind of card

Then it hit me.

Something just felt off

Thoughts are Gone!

The Fix #

I popped open my config and immediately knew what happened. I turned off some plugins locally for faster build iteration and that change found its way into production…. because I have nothing to check myself on other than me. The plugin is now back and thoughts are flowing from thoughts.waylonwalker.com to waylonwalker.com, no problem.

❯ git log -p
commit adeb9812f7ecf4d9a68c6aa5e01e549e4dd91285 (HEAD -> main, origin/main, origin/HEAD)
Author: Waylon S. Walker <waylon@waylonwalker.com>
Date:   Mon Nov 3 19:56:40 2025 -0600

    thoughts were missing

diff --git a/markata.toml b/markata.toml
index 59539bab..a55a494d 100644
--- a/markata.toml
+++ b/markata.toml
@@ -31,7 +31,7 @@ hooks = [
     'plugins.chartjs',
     'plugins.md_video',
     'plugins.post_model',
-    # 'plugins.thoughts',
+    'plugins.thoughts',
     # "plugins.wikilink_hover",
     # "plugins.wa_wikilink_hover",
     "plugins.tippy_wikilink_hover",

Owning Everything has a cost #

You see here I am a one man show. I own the content, the build, the infra that does the build. I love it, but there is so much to do for one persons side project passion project. Did I bite off more than I can chew? Did I choose the wrong abstraction level for my needs? Should I have better tests in place? Or is this just a digital garden that has ebs and flows, growing weeds and pruning them every few months?

Rules

2025-10-30T08:38:48Z

There is no such thing as magic
Be ready to roll back live deployments
If CI was too fast be suspicious

uv s3 sync with dotenv

2025-10-26T10:16:41Z

I often want to run an s3 sync in an isolated environment, I don’t want to set any environment variables, I don’t want anything secret in my history, and I don’t want to change my dotenv into something that exports variables, I just want s3 sync to work. dotenv run is the tool that I’ve been using for this, and this uv one liner lets it run fully isolated from the project.

one liner #

uv tool run --from 'python-dotenv[cli]' dotenv run -- uv tool run --from awscli aws s3 sync s3://bucket data

multi-line #

same thing formatted for readability

uv tool run \
  --from 'python-dotenv[cli]' \
  dotenv run -- \
uv tool run \
  --from awscli \
  aws s3 sync s3://dropper data

There are probably 10 ways to skin this cat, but this is what I did, if you have a better way let me know, I’ll link you below.

First 3d Printed Threads

2025-10-24T14:19:49Z

Working on an upcoming project that requires some threaded screws. Trying to keep a low budget on this one with as much to come off of the printer as I can. It might become a slant3d portals product if it works out. I always like making test prints for stuff like this especially to see how the feel is off of the printer that is going to print the final product and take much longer. First try was a success.

b485b759-719a-4aa0-aa8d-f98e0a5e1ac3-1080p.mp4

What worked #

I started out looking up standard half inch thread pitch and size, but ran out of time to get the exact profile of a half inch bolt, so I will need to fix that later. Th

The print orientation is critical for strength here. This part is a full 1/2: so it should be strong either way, but to make sure we are printing the bolt horizontally to get nice long print layers. To do this we have to give it a bit of a flat spot on the top and bottom. This does not hurt performance, if anything it probably helps give some room for poor tolerances.

.5" od
13tpi pitch
non-standard profile… kids needed me and I called it good enough to run a test.
0.1mm offset on all surfaces
external threads printed horizontally
internal threads printed vertically
Chamfer all lead-in/lead-out

fastapi

2025-10-22T09:26:17Z

FastAPI is a modern and efficient web framework for Python, built on top of the Starlette web framework, and pydantic for data validation and serialization.

From the FastAPI documentation #

FastAPI is a modern, fast (high-performance), web framework for building APIs with Python based on standard Python type hints.

The key features are:

Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic). One of the fastest Python frameworks available.
Fast to code: Increase the speed to develop features by about 200% to 300%. *
Fewer bugs: Reduce about 40% of human (developer) induced errors. *
Intuitive: Great editor support. Completion everywhere. Less time debugging.
Easy: Designed to be easy to use and learn. Less time reading docs.
Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
Robust: Get production-ready code. With automatic interactive documentation.
Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.

Mentioned in 2025 Stack Overflow Survey #

The +5 point increase for FastAPI is one of the most significant shifts in the web framework space. This signals a strong trend towards using Python for building performant APIs and reflects the overall strength of the Python ecosystem.

starlette head request

2025-10-22T09:11:31Z

FastAPI.">Starlette has a head request that works right along side your get requests. This morning I fiddled around with custom routes for GET and HEAD, but had to manually set some things about the file, and was still missing e-tag in the end. Turns out as a developer you can just add a head route to your get routes and starlette will strip the content for you, while preserving all of those good headers that fastapi FileResponse created automatically for you.

from fastapi import APIRouter
from fastapi.response import FileResponse
from fastapi import Request
from pathlib import Path

router = APIRouter()

@router.get("/file/{filename}")
@router.head("/file/{filename}")
async def get_file(filename: str, request: Request,):
    headers = {
      "Cache-Control": "no-cache, no-store, must-revalidate",
    }
    from pathlib import Path
    filename = Path(f"data/{filename}")
    if not filename.exists():
        raise HTTPException(status_code=404, detail="File not found")
    return FileResponse(filename, headers=headers)

Here is an example of the response with curl.

❯ curl -I -L "http://localhost:8100/api/file/e5523925-1565-454c-bab3-c70c4deabc83.webp?width=250"
HTTP/1.1 200 OK
date: Wed, 22 Oct 2025 14:16:03 GMT
server: uvicorn
cache-control: no-cache, no-store, must-revalidate
content-type: image/webp
content-length: 17206
last-modified: Tue, 23 Sep 2025 14:03:20 GMT
etag: f891660c1543feb1af7564f08abdd511

❯ curl -I -L "http://localhost:8100/api/file/unknown-file.webp?width=250"
HTTP/1.1 404 Not Found
date: Wed, 22 Oct 2025 14:16:11 GMT
server: uvicorn
content-length: 27
content-type: application/json

python extras are for shipping

2025-10-03T20:18:23Z

Python has two ways of adding optional dependencies to your projects pyproject.toml file dependency-groups and optional-dependencies.

dependency-groups #

for development

Dependency grooups are used when working on the project, they do not ship with the project, users cannot select to install them with the project. These are for things like running tests, linting, or docs. You might want to run these in ci, or keep your dev machines tight. For the most part you can probably keep these in dev. Depending on your team, fluency, and tolerance for slower installs extra packages. Adding too many tight groups might make it hard for the team to remember all the groups and which one to use and end up with them using --all-groups anyways.

optional-dependencies #

for users

Optional dependencies are for shipping. These are for your users, not your development team. This is used for dependencies that are clearly not needed for all or main use cases. It is annoying to use projects that you need to add optionals to just to use at all so use them a bit sparingly.

fastapi">example fastapi #

fastapi provides a very simple example. fastapi itself provides almost everything you need with optional-dependencies for standard, standard-no-fastapi-cloud-cli, and all. These primarily add support for uvicorn websockets and multipart forms.

example pandas #

Pandas is a very good example here. As a data processing library there are a lot of different sources for data that you might want to use, but you probably won’t need most of them, and often don’t need them to just get a DataFrame going. They offer a long list of optional dependencies such as pyarrow, aws, gcp, postgresql and many more.

example kedro-datasets #

Very similar to pandas kedro-datasets uses a similar pattern to pandas but at a higher level. A dataset is an abstraction of a datasource that get defined in the catalog, and primarily provide the framework with load and save methods for datasets. There are many optionals for data providers like pandas, but also many for each dataframe abstraction like pandas, polars, dask, databricks.

kedro-datasets

let’s build a package #

Let’s make a package called learn-uv that implements these dependency features.

mkdir /tmp/learn-uv
cd /tmp/learn-uv
uv init --package
uv add httpx
# add development dependencies
uv add --dev pytest
uv add --group dev ipython
# add documentation dependencies to the docs group
uv add --group docs markata

# add optional dependencies
# these are dependencies that a user could install
# python -m pip install 'learn-uv[data-science]'
uv add --optional data-science pandas

# add extra dependencies
# this adds the jupyter extra from rich
# similar to `python -m pip install 'learn-uv[jupyter]'`
uv add --extra jupyter rich

This will result in the following pyproject.toml.

[project]
name = "learn-uv"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
authors = [
    { name = "Waylon S. Walker", email = "hello@waylonwalker.com" }
]
requires-python = ">=3.13"
dependencies = [
    # uv add httpx
    "httpx>=0.28.1",
    # uv add --extra jupyter rich
    "rich[jupyter]>=14.1.0",
]

[project.scripts]
learn-uv = "learn_uv:main"

[project.optional-dependencies]
data-science = [
    # uv add --optional data-science pandas
    "pandas>=2.3.3",
]

[build-system]
requires = ["uv_build>=0.8.22,<0.9.0"]
build-backend = "uv_build"

[dependency-groups]
dev = [
    # uv add --group dev ipython
    "ipython>=9.6.0",
    # uv add --dev pytest
    "pytest>=8.4.2",
]
docs = [
    # uv add --group docs markata
    "markata>=0.9.1",
]

Using the project #

Let’s start off with one common point of confusion. uv tool run or its alias uvx does not install any dependencies or anything from the project.

uv tool run ipython

✅ ipython
❌ learn_uv
❌ httpx
❌ pytest
❌ markata
❌ pandas
❌ rich
❌ ipywidgets (from rich[jupyter])

Using uv run is project aware, installs the dependencies from the pyproject.toml.

uv run ipython

✅ ipython
✅ learn_uv
✅ httpx
✅ pytest
❌ markata
❌ pandas
✅ rich
✅ ipywidgets (from rich[jupyter])

uv run --group docs markata

✅ ipython
✅ learn_uv
✅ httpx
✅ pytest
✅ markata
❌ pandas
✅ rich
✅ ipywidgets (from rich[jupyter])

Installing the optional dependency extra data-science gives me pandas in this project.

uv run --extra data-science ipython

✅ ipython
✅ learn_uv
✅ httpx
✅ pytest
✅ markata
✅ pandas
✅ rich
✅ ipywidgets (from rich[jupyter])

everything #

There are many combinations of include/exclude that you can use, check uv run --help to see all of the options. Here is the nuclear option to include everything.

uv run --all-groups --all-extras ipython

Potential `uv` bug #

At the time of writing once a dependency group is used it persists in the virtual environment for the rest of the project. This is not the case with extras it only includes the extras you ask for on each run. The state is clean and consistent with extras, but groups depend on what you have ran before in the project.

❯ uv --version
uv 0.8.22 (ade2bdbd2 2025-09-23)

example

# install docs group
uv sync --group docs
# docs dependencies are available
uv run markata
# sync resets the environment
uv sync
# docs dependencies are not available
uv run markata
error: Failed to spawn: `markata`
  Caused by: Permission denied (os error 13)

💭 python 3.14 highlights! - YouTube

2025-10-03T01:04:57Z

anthony has some of the best python highlight videos each year. This might be a good sign, but each year there seems to be less and less that I am chomping at the bit to get to. I thought the remote debugger looked every interesting, his use case for babi seemed very interesting. I wonder what textual would look like built in a 3.14 world, would it still have built its own debugger/console?

uv tool run --python=3.14 babi

Without a process flag you need sudo permissions to attach a pdb debugger similar to gdb.

ps -ef | grep babi
uv tool run --python=3.14 python -m pdb -p8605

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Kraft-Coordinates

2025-10-02T16:52:02Z

Handy reference for coordinates in the kraft world.

Home #

Overworld:-208 71 -291 Nether:-26 9 -36

Ocean Monument #

Overworld: 209 62 -752 Nether:26 1 -94

💭 PEP 735 – Dependency Groups in pyproject.toml | peps.python.org

2025-10-01T16:25:57Z

PEP 735 – Dependency Groups in pyproject.toml | peps.python.org

This PEP specifies a mechanism for storing package requirements in pyproject.toml files such that they are not included in any built distribution of the project.

Python Enhancement Proposals (PEPs) · peps.python.org

PEP 735 describes dependency groups as sets of optional dependencies that are not shipped with the package but intended for development purposes.

The PEP includes an example for groups that include test, docs, typing, and a combo typing-test.

[dependency-groups]
test = ["pytest", "coverage"]
docs = ["sphinx", "sphinx-rtd-theme"]
typing = ["mypy", "types-requests"]
typing-test = [{include-group = "typing"}, {include-group = "test"}, "useful-types"]

This is implemented in uv and can be used by several of their commands.

uv sync --group test
uv run --group test
uv add --group test pytest
uv remove --group test pytest
uv export --group test
uv tree --group test

Dependency Groups are not Extras #

The docs describe extras as being intended to ship with the application and dependency groups intended for development. The spec allows both to exist with the same name, but care should be taken as tools may have different implementations.

Tools MAY choose to provide the same interfaces for installing Dependency Groups as they do for installing extras.

Note that this specification does not forbid having an extra whose name matches a Dependency Group.

Users are advised to avoid creating Dependency Groups whose names match extras. Tools MAY treat such matching as an error.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 TIL: Loading .env files with uv run

2025-09-29T18:42:01Z

TIL: Loading .env files with uv run

Replacing python-dotenv with uv

https://daniel.feldroy.com · daniel.feldroy.com

I smell a dependency to python-dotenv dying in my workflow. I originally read the title of the post and thought, “I know how to manage .env and almost skipped it”. I’m leaning more and more on uv run these days, so this should just go in my [[ just file ]] to make it easy to run.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Shots Get An Upgrade To the Main Feed

2025-09-20T20:19:57Z

Today I updated my jinja templates and feeds to include shots posts in the main feed. These posts will prominently show the image in the feed with a small post for each one.

Shots are instagram style imge posts inspired by https://justin.searls.co/shots/.

zmk two hand hold down

2025-09-17T20:04:36Z

I don’t know about you, but I don’t re-flash my keyboard enough to ever remember where I put the bootloader. Sometimes its the last thing I think about in a refactor and I end up cornering myself into a place where I cant get into that layer anymore. I’ve started putting hardware switches on my newer builds, but some older builds don’t have a hardware one, so it requires disassembly and jumping the microcontroller. Even when I have one though, I gotta flip my board over and its annoying sometimes, so I prefer to have a keystroke for it.

Two Hand hold down #

What I’ve landed on recently is the idea of a two hand hold down combo for the bootloader. These combos are ones that there is no way I can hit without picking my hands up from their normal homerow position and pressing four keys simultaneously with pointer and thumbs.

Here are some example layouts from keymap-editor

40% layout - similar to corne

Here it is on my Son’s 3x5 macropad

Here it is on my 3x3 macropad

Timing #

You can even give it a long prior timeout to really make sure that its not an accidental hit.

uv script management

2025-09-07T20:22:56Z

I’ve been leaning on lazy-self-installing-python-scripts more and more, but I did not realize how much tooling that uv gives you to help manage your scripts.

uv init --script up
uv add --script up typer rich
uv remove --script up rich
sed -i '1i #!/usr/bin/env -S uv run --script' up
chmod +x up
./up

The result is a script that looks like this, its executable as what looks like regular command in your shell.

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "typer",
# ]
# ///


def main() -> None:
    print("Hello from up!")


if __name__ == "__main__":
    main()

Keycap Cut Away

2025-08-30T16:03:01Z

I was curious how/if my custom keycap design was hitting my switches. So I set out to find out what the fitup inside of this assembly looks like, but not theoretically, a fully sliced view into their fit up in the flesh.

The set up #

To setup for this cut, I flooded the edge of a 2x4 with hot glue, and inserted the cap such that the step was tangent with the edge. This way I could use the edge as a guide to cut one side off and leave the stem in tact. I took a handsaw to it and filed it smooth.

Removal was applying some isopropyl alcohol and it popped right off.

Results #

Actually the results aren’t that interesting. I found that the pictures turned out really good, but nothing was touching and it sits in the stem exactly as I thought it did. I had a thought, what if the cap is not allowing the switch to reach it’s full travel, and had no way to know, but now I know.

Key Up

Key Down

qr codes in python

2025-08-05T08:32:12Z

I have a couple of use cases for simple qr codes in python coming up. One is for blog posts, the other is for auth into a new server application logged to a terminal. I tried the qrcode library and it does not look as nice to me and I found pyqrcode to be quite nice.

import pyqrcode
url = pyqrcode.create('https://waylonwalker.com/qr-codes-in-python')
url.svg('qr-codes-in-python.svg', scale=8)
print(url.terminal(quiet_zone=1))
url.svg('qr-codes-in-python.svg', scale=12)
url.svg('qr-codes-in-python.svg', omithw=True) # width is controlled by the container

url.svg('qr-codes-in-python.svg', omithw=True, module_color='#ffd119')
url.svg('qr-codes-in-python.svg', omithw=True, module_color='#ff69b4', background='#2b034c')

result #

Here is the final svg result.

Here is what it looks like in the terminal.

If you want fancier qrcodes check out https://mydigitalharbor.com/

💭 rushter/selectolax: Python binding to Modest and Lexbor engine...

2025-08-04T13:52:15Z

GitHub - rushter/selectolax: Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python. - rushter/selectolax

uv run for running tests on versions of Python

Selectolax you have my attention! I will be giving this a try for markata which often suffers from slow beautifulsoup. It appears to have everything I need for my simple use cases.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 uv run for running tests on versions of Python

2025-07-21T20:25:40Z

Using uv run with make to replace tox or nox for testing multiple versions of Python locally.

https://daniel.feldroy.com · daniel.feldroy.com

Such a fantastic use of uv, its so fast and flexible and does everything I need that next time I go to set up some more complex testing like this I’m going to lean towards it more than i would something like tox. In the post Daniel sets up matrix testing for testing out different versions of python with the same pytest test suite.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

pygments htmlformatter

2025-07-17T19:45:19Z

I’ve been a long user of pygments, it’s been the thing that injects <spans> with funny little class names like sc and si into the code blocks of my website. I’ve even gone as far as implementing a plugin for md-it, but I had no idea how to re-style it. I long ago got a theme that looked good enough from somewhere and just used it, maybe I pulled something from their docs site and forgot. Today I learned you can list all the themes easily from the library itself, and render out new css.

from pygments.styles import get_all_styles
list(get_all_styles())
# [
#     'abap',
#     'algol_nu',
#     'algol',
#     'arduino',
#     'autumn',
#     'borland',
#     'bw',
#     'colorful',
#     'default',
#     'dracula',
#     'emacs',
#     'friendly_grayscale',
#     'friendly',
#     'fruity',
#     'github-dark',
#     'gruvbox-dark',
#     'gruvbox-light',
#     'igor',
#     'inkpot',
#     'lightbulb',
#     'lilypond',
#     'lovelace',
#     'manni',
#     'material',
#     'monokai',
#     'murphy',
#     'native',
#     'nord-darker',
#     'nord',
#     'one-dark',
#     'paraiso-dark',
#     'paraiso-light',
#     'pastie',
#     'perldoc',
#     'rainbow_dash',
#     'rrt',
#     'sas',
#     'solarized-dark',
#     'solarized-light',
#     'staroffice',
#     'stata-dark',
#     'stata-light',
#     'stata',
#     'tango',
#     'trac',
#     'vim',
#     'vs',
#     'xcode',
#     'zenburn'
# ]

from pygments.formatters import HtmlFormatter
from pygments.styles import get_style_by_name
style = get_style_by_name("monokai")
formatter = HtmlFormatter(style=style)
print(formatter.get_style_defs('.highlight'))

And now you get styles that you can add to your css and be any theme from the list above.

pre { line-height: 125%; }
td.linenos .normal { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
span.linenos { color: inherit; background-color: transparent; padding-left: 5px; padding-right: 5px; }
td.linenos .special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
span.linenos.special { color: #000000; background-color: #ffffc0; padding-left: 5px; padding-right: 5px; }
.highlight .hll { background-color: #49483e }
.highlight { background: #272822; color: #f8f8f2 }
.highlight .c { color: #959077 } /* Comment */
.highlight .err { color: #ed007e; background-color: #1e0010 } /* Error */
.highlight .esc { color: #f8f8f2 } /* Escape */
.highlight .g { color: #f8f8f2 } /* Generic */
.highlight .k { color: #66d9ef } /* Keyword */
.highlight .l { color: #ae81ff } /* Literal */
.highlight .n { color: #f8f8f2 } /* Name */
.highlight .o { color: #ff4689 } /* Operator */
.highlight .x { color: #f8f8f2 } /* Other */
.highlight .p { color: #f8f8f2 } /* Punctuation */
.highlight .ch { color: #959077 } /* Comment.Hashbang */
.highlight .cm { color: #959077 } /* Comment.Multiline */
.highlight .cp { color: #959077 } /* Comment.Preproc */
.highlight .cpf { color: #959077 } /* Comment.PreprocFile */
.highlight .c1 { color: #959077 } /* Comment.Single */
.highlight .cs { color: #959077 } /* Comment.Special */
.highlight .gd { color: #ff4689 } /* Generic.Deleted */
.highlight .ge { color: #f8f8f2; font-style: italic } /* Generic.Emph */
.highlight .ges { color: #f8f8f2; font-weight: bold; font-style: italic } /* Generic.EmphStrong */
.highlight .gr { color: #f8f8f2 } /* Generic.Error */
.highlight .gh { color: #f8f8f2 } /* Generic.Heading */
.highlight .gi { color: #a6e22e } /* Generic.Inserted */
.highlight .go { color: #66d9ef } /* Generic.Output */
.highlight .gp { color: #ff4689; font-weight: bold } /* Generic.Prompt */
.highlight .gs { color: #f8f8f2; font-weight: bold } /* Generic.Strong */
.highlight .gu { color: #959077 } /* Generic.Subheading */
.highlight .gt { color: #f8f8f2 } /* Generic.Traceback */
.highlight .kc { color: #66d9ef } /* Keyword.Constant */
.highlight .kd { color: #66d9ef } /* Keyword.Declaration */
.highlight .kn { color: #ff4689 } /* Keyword.Namespace */
.highlight .kp { color: #66d9ef } /* Keyword.Pseudo */
.highlight .kr { color: #66d9ef } /* Keyword.Reserved */
.highlight .kt { color: #66d9ef } /* Keyword.Type */
.highlight .ld { color: #e6db74 } /* Literal.Date */
.highlight .m { color: #ae81ff } /* Literal.Number */
.highlight .s { color: #e6db74 } /* Literal.String */
.highlight .na { color: #a6e22e } /* Name.Attribute */
.highlight .nb { color: #f8f8f2 } /* Name.Builtin */
.highlight .nc { color: #a6e22e } /* Name.Class */
.highlight .no { color: #66d9ef } /* Name.Constant */
.highlight .nd { color: #a6e22e } /* Name.Decorator */
.highlight .ni { color: #f8f8f2 } /* Name.Entity */
.highlight .ne { color: #a6e22e } /* Name.Exception */
.highlight .nf { color: #a6e22e } /* Name.Function */
.highlight .nl { color: #f8f8f2 } /* Name.Label */
.highlight .nn { color: #f8f8f2 } /* Name.Namespace */
.highlight .nx { color: #a6e22e } /* Name.Other */
.highlight .py { color: #f8f8f2 } /* Name.Property */
.highlight .nt { color: #ff4689 } /* Name.Tag */
.highlight .nv { color: #f8f8f2 } /* Name.Variable */
.highlight .ow { color: #ff4689 } /* Operator.Word */
.highlight .pm { color: #f8f8f2 } /* Punctuation.Marker */
.highlight .w { color: #f8f8f2 } /* Text.Whitespace */
.highlight .mb { color: #ae81ff } /* Literal.Number.Bin */
.highlight .mf { color: #ae81ff } /* Literal.Number.Float */
.highlight .mh { color: #ae81ff } /* Literal.Number.Hex */
.highlight .mi { color: #ae81ff } /* Literal.Number.Integer */
.highlight .mo { color: #ae81ff } /* Literal.Number.Oct */
.highlight .sa { color: #e6db74 } /* Literal.String.Affix */
.highlight .sb { color: #e6db74 } /* Literal.String.Backtick */
.highlight .sc { color: #e6db74 } /* Literal.String.Char */
.highlight .dl { color: #e6db74 } /* Literal.String.Delimiter */
.highlight .sd { color: #e6db74 } /* Literal.String.Doc */
.highlight .s2 { color: #e6db74 } /* Literal.String.Double */
.highlight .se { color: #ae81ff } /* Literal.String.Escape */
.highlight .sh { color: #e6db74 } /* Literal.String.Heredoc */
.highlight .si { color: #e6db74 } /* Literal.String.Interpol */
.highlight .sx { color: #e6db74 } /* Literal.String.Other */
.highlight .sr { color: #e6db74 } /* Literal.String.Regex */
.highlight .s1 { color: #e6db74 } /* Literal.String.Single */
.highlight .ss { color: #e6db74 } /* Literal.String.Symbol */
.highlight .bp { color: #f8f8f2 } /* Name.Builtin.Pseudo */
.highlight .fm { color: #a6e22e } /* Name.Function.Magic */
.highlight .vc { color: #f8f8f2 } /* Name.Variable.Class */
.highlight .vg { color: #f8f8f2 } /* Name.Variable.Global */
.highlight .vi { color: #f8f8f2 } /* Name.Variable.Instance */
.highlight .vm { color: #f8f8f2 } /* Name.Variable.Magic */
.highlight .il { color: #ae81ff } /* Literal.Number.Integer.Long */

💭 uv cache prune

2025-07-09T19:41:13Z

uv cache prune

If you're running low on disk space and are a uv user, don't forget about uv cache prune: uv cache prune removes all unused cache entries. For example, the cache …

Notes – 09:32 Thu 19 Jun 2025

Good point to check on your uv cache if you are running low on disk space. I checked mine today, and it wasn’t too bad so I left it alone.

du -sh `uv cache dir`

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

markata parallel render

2025-07-01T08:05:58Z

  _._   __/__   __  __ _/_   Recorded: 07:53:56  Samples:  71681
 /_//_/// /_\ / //_// / //_'/ //     Duration: 92.741    CPU time: 91.748
/_/                      v4.5.1

Program: /home/waylon/git/waylonwalker.com/.venv/bin/markata build --pdb

92.740 Markata.run  markata/__init__.py:443
`- 92.714 HookCaller.__call__  pluggy/_hooks.py:479
      [2 frames hidden]  pluggy
         92.714 PluginManager._hookexec  pluggy/_manager.py:106
         |- 38.207 wrapper_register  markata/hookspec.py:265
         |  |- 26.105 render  plugins/link_collector.py:59
         |  |  |- 10.012 BeautifulSoup.__init__  bs4/__init__.py:122
         |  |  |     [14 frames hidden]  bs4, html
         |  |  |- 5.599 <listcomp>  plugins/link_collector.py:181
         |  |  |- 4.050 <listcomp>  plugins/link_collector.py:173
         |  |  |- 3.466 Markata.map  markata/__init__.py:565
         |  |  |`- 2.092   markata/__init__.py
         |  |  `- 0.942 BeautifulSoup.find_all  bs4/element.py:2008
         |  |        [2 frames hidden]  bs4
         |  |- 9.107 render  markata/plugins/render_markdown.py:260
         |  |`- 8.902 result_iterator  concurrent/futures/_base.py:612
         |  |        [5 frames hidden]  concurrent, threading, <built-in>
         |  `- 2.079 glob  markata/plugins/glob.py:112
         |`- 1.909 <listcomp>  markata/plugins/glob.py:161
         |        `- 1.908 Future.result  concurrent/futures/_base.py:428
         |              [3 frames hidden]  concurrent, threading, <built-in>
         |- 25.708 render  markata/plugins/post_template.py:639
         |`- 25.612 render_article  markata/plugins/post_template.py:404
         |     |- 22.223 <dictcomp>  markata/plugins/post_template.py:421
         |     |  `- 22.200 render_template  markata/plugins/post_template.py:429
         |     |`- 22.185 Template.render  jinja2/environment.py:1269
         |     |        `- 21.210 root  templates/post.html:4
         |     |`- 20.937 root  templates/base.html:4
         |     |              `- 20.455 block_content  templates/post.html:17
         |     |                 |- 11.212 root  templates/recent.html:4
         |     |                 |`- 10.853 root  templates/feed_sm_partial.html:4
         |     |                 |     `- 8.392 Environment.getattr  jinja2/environment.py:480
         |     |                 |`- 8.328 Feed.posts  markata/plugins/feeds.py:329
         |     |                 |           `- 8.300 Feed.map  markata/plugins/feeds.py:352
         |     |                 |`- 7.963 Markata.map  markata/__init__.py:565
         |     |                 |                 `- 7.786 <listcomp>  markata/__init__.py:628
         |     |                 |`- 6.465   markata/__init__.py
         |     |                 `- 8.486 call  jinja2/runtime.py:260
         |     |`- 8.468 Feed.map  markata/plugins/feeds.py:352
         |     |                       `- 8.394 Markata.map  markata/__init__.py:565
         |     |`- 8.326 <listcomp>  markata/__init__.py:628
         |     |                             `- 7.086   markata/__init__.py
         |`- 3.015 Cache.set  diskcache/core.py:749
         |           [3 frames hidden]  diskcache
         |- 7.788 save  markata/plugins/feeds.py:462
         |  `- 7.779 create_page  markata/plugins/feeds.py:493
         |     |- 4.530 Template.render  jinja2/environment.py:1269
         |     |`- 2.437 root  templates/feed.html:4
         |     |     `- 2.360 root  templates/base.html:4
         |     |`- 2.231 block_content  templates/feed.html:17
         |     |           `- 2.118 root  templates/feed_partial.html:4
         |     |`- 1.298 Environment.getattr  jinja2/environment.py:480
         |     |                 `- 1.290 Feed.posts  markata/plugins/feeds.py:329
         |     |`- 1.290 Feed.map  markata/plugins/feeds.py:352
         |     |                       `- 1.276 Markata.map  markata/__init__.py:565
         |     |- 1.325 Feed.map  markata/plugins/feeds.py:352
         |     |`- 1.317 Markata.map  markata/__init__.py:565
         |     `- 1.237 Cache.set  diskcache/core.py:749
         |- 7.029 pre_render  markata/plugins/auto_description.py:185
         |`- 6.840 set_description  markata/plugins/auto_description.py:137
         |     `- 5.791 get_description  markata/plugins/auto_description.py:107
         |        |- 4.485 MarkdownIt.parse  markdown_it/main.py:256
         |        |     [8 frames hidden]  markdown_it
         |`- 1.066 MarkdownIt.__init__  markdown_it/main.py:33
         |- 5.730 pre_render  markata/plugins/analytics.py:102
         |  |- 2.397 savefig  matplotlib/pyplot.py:1238
         |  |     [4 frames hidden]  matplotlib
         |  `- 2.004 heatmap  seaborn/matrix.py:355
         |        [10 frames hidden]  seaborn, matplotlib
         |- 1.704 post_render  plugins/permalink_aria.py:40
         |`- 1.588 process_html_content  plugins/permalink_aria.py:13
         |- 1.619 save  markata/plugins/publish_source.py:83
         |  `- 1.216 Post.dumps  markata/plugins/post_model.py:333
         |`- 1.203 Post.yaml  markata/plugins/post_model.py:254
         |        `- 1.044 dump  yaml/__init__.py:248
         |              [3 frames hidden]  yaml
         `- 1.183 post_render  plugins/wikilink_hover.py:104
            `- 1.094 do_hover_links  plugins/wikilink_hover.py:126

[07:55:38] save complete                                                                            __init__.py:465
           teardown running                                                                         __init__.py:462
           teardown complete                                                                        __init__.py:465
           lifetime cache hit rate 0.17%                                                            __init__.py:471
           lifetime cache hits/misses 36/21118                                                      __init__.py:476
           run cache hit rate 0.17%                                                                 __init__.py:482
           run cache hits/misses 36/21118                                                           __init__.py:487

Map Cache Statistics:
Total calls: 17197
Cache hits: 15317
Cache misses: 1880
Hit rate: 89.1%
Cache size: 1880

csv

2025-06-26T20:44:33Z

name,age,city
Alice,30,New York
Bob,25,San Francisco
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago

name,age,city
Alice,30,New York
Bob,25,San Francisco
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago

graph TD
  A-->B
  A-->C
  A-->D
  D-->E

💭 Notes – 09:32 Thu 19 Jun 2025 – David Bushell – Web Dev (UK)

2025-06-19T13:18:20Z

dbushell.com · dbushell.com

David’s design on his blog is fantastic likely from years of small improvements like this converting ugly quotes to pretty quotes and optimizing fonts.

It’s common for markdown libraries to convert the first to the second like my build script does.

This is new to me, I had no idea that markdown libraries did this, I’m now interested if markdown-it does it.

For subsetting I use the fontTools library but I’ve no idea how to setup Python environments. I got it working once and failed to document the process.

David, David, David, I’m sorry python has done you this dirty. I should do a post on making python environments in the age of Posts tagged: uv. You got options to run in docker/podman, a whole ass vm, uv venv, uvx, uv run, uv script, python -m venv, virtualenv, poetry, hatch, and too many more. The ones that matter are containers or uv.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

copier vcs quirks

2025-06-19T08:04:51Z

Copier has a few quirks with vcs that I just discovered by trying to test out some changes. I may have some config that I have long forgotten about somewhere deep in my dotfiles, I don’t think so, but id love to be wrong and corrected, please reach out.

What Doesn’t Work #

I tried throwing everything at this template to make it work. I tried a bunch of flags that did not work. I tried making commits to the local repo to get rid of the dirty warning. I really wanted to test new changes locally without committing and pushing untested and potentially broken changes.

uvx copier copy ../markata-blog-starter .
uvx copier copy gh:waylonwalker/markata-blog-starter@develop .
uvx copier copy ../markata-blog-starter . -wlg --trust

What Works - –vcs-ref #

Finally after trying everything to get the local copy to work, and my guess of @branch not working I found this to work. It does require me to go to the repo on my develop branch.

uvx copier copy gh:waylonwalker/markata-blog-starter --vcs-ref develop .

git">What Works - delete .git #

Really this might be my best option to make quick changes and test them locally without going through a version control system. It is not ideal, but makes it easy to quickly iterate on. I might be renaming .git, or copying to /tmp for quick iteration.

rm -rf .git
uvx copier copy ../markata-blog-starter .

Copier I love #

Copier is a great templating tool. I really love it. I use it every single day to create posts on this blog using Tmux hotkey for copier templates. This is the first time this quirk has got me and it had me puzzled for 45 minutes as I did not expect this behavior whatsoever.

💭 Taming file zoos: Data science with DuckDB database files - Al...

2025-06-09T13:31:30Z

The ability to query s3 buckets so seamless looks like such a pleasure to work with if you have a use case for that. Kedro catalog takes care of this most of the time for me, but I wonder if there are some cross project searching use cases I might find for this.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

rich theme

2025-06-05T20:37:10Z

The rich console is themeable, I’ve been a long time user of rich and had no Idea. You can define your own theme keywords and use them just like you use normal rich keywords in square brackets like'[bold red]'.

from rich.console import Console
from rich.theme import Theme
custom_theme = Theme({
    "info": "dim cyan",
    "warning": "magenta",
    "danger": "bold red"
})
console = Console(theme=custom_theme)
console.print("This is information", style="info")
console.print("[warning]The pod bay doors are locked[/warning]")
console.print("Something terrible happened!", style="danger")

💭 The adapter pattern in python | Rob Parsons

2025-05-28T21:19:27Z

The adapter pattern in python

The Adapter pattern is a design pattern that allows objects with incompatible interfaces to work together. It provides a way to convert the interface of an object into another interface that client...

Rob Parsons · robp.dev

This has me wondering if I need to really learn more patterns, data structures, and algorithms. This looks particularly useful when trying to combine several objects that you dont have full control over and make them behave similarly.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

web wakelock

2025-05-21T20:50:22Z

I’m trying to replace my usage of google inline search apps with real apps, today I used a stopwatch to time some things out at work by opening stopwatch. This was something I just wanted running in a tab on another screen, it was not timing running code or anything, I was using it as a reminder to check browser caches every 5 minutes or so for some testing.

So tonight I whipped up a stopwatch, clock and timer, all of which are using the wakelock API to keep the screen on while the app is running.

    // Wake Lock support
    let wakeLock = null;
    async function requestWakeLock() {
      try {
        if ('wakeLock' in navigator) {
          wakeLock = await navigator.wakeLock.request('screen');
          console.log("Wake lock acquired");
        }
      } catch (err) {
        console.error("Wake lock error:", err);
      }
    }

    document.addEventListener("visibilitychange", () => {
      if (wakeLock !== null && document.visibilityState === "visible") {
        requestWakeLock();
      }
    });

    requestWakeLock();

💭 Python: The Documentary [OFFICIAL TRAILER] - YouTube

2025-05-18T16:43:12Z

Excited to hear this story, they have so many key players in the trailer this will be fantastic.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

m9a devlog 1

2025-05-11T09:33:44Z

It’s sad to see textualize.io close the doors, but textual is still alive and maintained as a n open source project. I tried to use it very early, and struggled, this was before docs and tutorials really existed, before a lot of the widgets and components existed. Then as we all do I got busy and moved on to other things in life and did not have the capacity to build TUIs.

I like tuis #

I like tuis, I like staying in the terminal. I use lf daily to move files around when I want something more than mv and cp. I use k9s hourly to monitor and manage my kubernetes cluster.

Are they worth the effort?? #

As awesome as tui’s are, they are more effort to build, and less automatable. I feel like the first stage into automation of a project really needs to be a good cli, and this is often good enough for the project and I move on.

m9a (em - nine - ah) #

inspired by k9s

Like I said I really like k9s and use it all the time, It really makes running kubectl commands a breeze and much less verbose. I don’t know how useful this will be, but as a learning exercise I am working on a k9s experience for my blog generator markata.

m9a-1-trimmed.webm

Learning #

So far this is just for learning and not quite the most useful thing, I am not sure if there is a way to do it, but I am interested in the idea of some sort of framework (maybe just widgets) that can more easily turn pydantic objects into this kind of tui. I don’t quite know how it would work, or if it could work, for now just exploring the idea, and I think I hit a fairly crude clone of k9s so far.

fix feed descriptions

2025-05-05T10:32:04Z

Today I fixed a bug in markata that has been occurring for a few months where the description for posts come out as None if coming from cache, the issue was a pretty simple check and pull properly from cache. This fixes all the descriptions in feeds and metadata on the post.

Better description #

While in there we went ahead and improved our get_description to more accurately return plain text without escaped characters, remove cutoff words, and add an elipsis if the description cuts off the text.

More description #

While I was there I made longer form posts, til, blog-post use the super description of 500 characters instead of the regular 120 character description.

Before #

After #

ruff single line import

2025-05-04T14:23:41Z

I’ve been using ruff to lint my python code for quite awhile now, I was pretty early to jump on it after release. Some of my projects have had a nice force-single-line setting and some have not. I dug into the docs and it was not clear what I needed to make it work.

[tool.ruff]
select = ['I'] # you probably want others as well

[tool.ruff.isort]
force-single-line = true

Turns out I was missing Isort in the select list.

💭 Red-Knot: Ruff's new Type Checker - Talk Python Live Stream - ...

2025-05-02T01:06:05Z

Astral is doing great things in the python industry. They are disrupting entire categories of tools with extremely fast, easy to use, and feature rich alternatives that make it really hard to keep using the incumbent. So far I am seeing no signs of evil, sometimes with such a disrupter there is some sort of downside that make it hard to want to do the switch. In the interview they even mention things like leaning on lsp so that it works across all editors rather than building out vscode integrations that work for most developers. As a neovim user I greatly apreciate this.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Playground | Red Knot

2025-05-02T01:02:31Z

Playground | ty

An in-browser playground for ty, an extremely fast Python type-checker written in Rust.

types.ruff.rs

ty, has a playground running at types.ruff.rs. You can edit code in there and see what the type checker results would be in browser. This looks good, excited to see it running in my lsp.

Here is an example where a Optional may not be defined.

Checking for existance before using it resolves the issue.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Markata list posts by year

2025-05-01T22:35:47Z

I was looking back at my analytics page today and wondered what were my posts about back at the beginning. My blog is managed by markata so I looked at a few ways you could pull those posts up. Turns out it’s pretty simple to do, use the markata map with a filter.

from markata import Markata

m.map('title, slug, date', filter='date.year==2016', sort='date')

Note

the filter is python eval that should evaluate to a boolean, all of the

attributes of the post are available to filter on.

Result #

[
    ('⭐ jupyterlab jupyterlab', 'jupyterlab-jupyterlab', datetime.date(2016, 12, 13)),
    ('⭐ nickhould tidy-data-python', 'nickhould-tidy-data-python', datetime.date(2016, 12, 9)),
    (
        '⭐ mikeckennedy write-pythonic-code-demos',
        'mikeckennedy-write-pythonic-code-demos',
        datetime.date(2016, 11, 22)
    ),
    (
        '⭐ mikeckennedy write-pythonic-code-for-better-data-science-webcast',
        'mikeckennedy-write-pythonic-code-for-better-data-science-webcast',
        datetime.date(2016, 11, 22)
    ),
    ('⭐ rajshah4 dlgroup', 'rajshah4-dlgroup', datetime.date(2016, 11, 18)),
    ('⭐ pandas-dev pandas', 'pandas-dev-pandas', datetime.date(2016, 10, 5))
]

You could use the list command as well right within your shell and the same map and filters work.

⬢ [devtainer-0.1.3] ❯ markata list --map title --filter='date.year==2016'
[22:35:06] 2088/2145 posts skipped                                                                       skip.py:36
           57/2145 posts not skipped                                                                     skip.py:37

⭐ pandas-dev pandas
⭐ rajshah4 dlgroup
⭐ mikeckennedy write-pythonic-code-for-better-data-science-webcast
⭐ mikeckennedy write-pythonic-code-demos
⭐ nickhould tidy-data-python
⭐ jupyterlab jupyterlab

You could also do it with jin right inside of a markdown post using the jinja_md plugin.

{% raw %}
{% for title, slug, date in markata.map('title, slug, date', filter='date.year==2016', sort='date') %}
* [{{title}}]({{slug}}) - {{date}}
{% endfor %}
{% endraw %}

Note

You do have to `jinja: true` in the frontmatter of the post.

Result #

{% for title, slug, date in markata.map(’title, slug, date’, filter=‘date.year==2016’, sort=‘date’) %}

{{title}} - {{date}} {% endfor %}

💭 pypi package ty

2025-05-01T20:44:27Z

An extremely fast Python type checker, written in Rust.

66c5b984-5450-48ee-9978-00fec4815807.mp4

Astral is working on some great things around python, they have created a high standard for python tooling built on rust that works really well, runs fast and covers everything in the space it resides in. ty appears to be their linter coming soon.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

portal-platformer-devlog-1

2025-04-28T19:56:58Z

Here is the current state of my platformer yet to really be named, I want to make something in between hollow knight and portal.

Starting #

I made one once in make code arcace on a pybadge. It was quick and dirty, but fun to work on. It had the basic of blocks that I could move, blocks i could put a portal onto, and a goal for each level. Some levels you can just walk through and some levels required you to really think about where to place the portal.

History #

So this version of the game is a least 2 years in the making, I open it every few months give it a day or two and move on. Its mostly something that I work on with my son. He really likes to jump around on projects so its hard to make real progress on something, but we are hitting an age where he is able to come back to projects a little better.

All of this is built in python, and mostly before vibe coding was a thing, its mostly me trying to get out ideas as quick as my son is spitting out the the next idea.

Coyote #

It includes a few frames of coyote so it feels a bit more like most games.

!! Note If you are unfamiliar with the term coyote in platformers it allows you to jump for a few frames after falling off a platform, like wiley coyoyte in Luney Toons.

Wall slide/jump #

Wall slide and jump work, but so is wall climb as an unintended side effect. When you are touching a wall, your fall speed is halved.

Levels and loader #

There is a crude level loader that loads json levels with pydantic. No editor yet, just hand editing levels with json.

Checkpoints #

It makes checkpoints, when you die, you go back to your last checkpoint. Checkpoints can be invisible, and have a link that turns them into a door to another level.

no art #

There is no art yet, just the skin of a platformer, levels, checkpoints, and coyote. No portals, but there is a pointer with a janky box that covers my lighting.

💭 PyApp

2025-03-03T01:52:54Z

PyApp

ofek.dev

I think I’m getting really close to having a good workflow setup for using pyapp. Such an amazing project to allow developers to create applications in python without passing on the hassle of python and managing installs to the user.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

pre-commit exclude

2025-02-20T08:11:35Z

I run tailwind for my personal blog, whenever I update it, pre-commit goes in and fixes end of file. I’m sick of these things fighting each other, since it is a generated app it is going to et ignored from pre-commit from now on.

exclude: ^static/app.*\.css$
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.4.0
    hooks:
      - id: trailing-whitespace
      - id: end-of-file-fixer
      - id: check-yaml
      - id: check-added-large-files

💭 valkey · PyPI

2025-02-17T02:22:12Z

valkey

Python client for Valkey forked from redis-py

Replay markata-search-1.mp4

python bindings for valkey, forked from redis.

one notable difference I see from redis is that you can install with libvalkey to autmatically get faster parsing support.

For faster performance, install valkey with libvalkey support, this provides a compiled response parser, and for most cases requires zero code changes. By default, if libvalkey >= 2.3.2 is available, valkey-py will attempt to use it for response parsing.

pip install "valkey[libvalkey]"

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

markata search

2025-01-30T20:21:36Z

A side effect of Markata DidYouMean is that we are able to implement some rudimentary search based on slug, title, tags, and description.

I made a video of the early work on using this side effect to search for content on markata.dev.

This was first implemented to solve 404 pages giving users a way to try to find the page that they tried to access. But it turned out to be a decent way to search through the docs.

Markata DidYouMean

2025-01-29T20:28:36Z

Coming in Markata 0.9.1 is far better documentation. i.e. Documentation that actually exists for everything. As part of poking around I realized that I often go to look up the docs for a plugin and forget that the path is /markta/plugins/feeds, sometimes I might try /markata/feeds or /plugins/feeds.py or /feeds or I might even forget the plugin name exactly and try something like feed and get a 404. So I added a didyoumean plugin to markata that takes care of this.

I made a quick recording of this early feature, pay close attention to the url as it automatically updates to the correct page.

markata-didyoumean.mp4

Happy Path #

direct forward

If you have one post called /markata/plugins/feeds, and it is the only post called feeds, any combination of /markata/feeds or /plugins/feeds or /feeds will all automatically redirect with an html page (not a server 3xx) to the /markata/plugins/feeds post.

Here is the snippet that does the redirect.

<div class="container mx-auto px-4 py-8">
    <h1 class="text-2xl font-bold mb-4">Redirecting...</h1>
    <p class="mb-4">You will be redirected to <a href="/markata/plugins/feeds" class="text-blue-500 hover:underline">/markata/plugins/feeds</a></p>
    <script>window.location.href = "/markata/plugins/feeds";</script>
</div>

You won’t see it unless you are on a really bad network, or you have js disabled, but this is what it would look like if you ever saw it. Notice there is a backup link if you have js disabled.

Multiple Similar Posts #

list of options

If you have multiple posts called didyoumean from different directories, in the video I made a clashing post at /markata/cli/didyoumean with /markata/plugins/didyoumean. If you go to /didyoumean it will notice that there are multiple options and present you with a list of all of the potential pages that match.

Finally No Matches #

404.html

Last ditch effort is to implement a 404 page. This page will know all of the possible paths in your project and give you a list of all the similar pages more like a traditional didyoumean plugin.

💭 Models - Pydantic

2025-01-28T17:27:20Z

Models

Pydantic Docs · docs.pydantic.dev

I came accross from_attributes today it allows creation of pydantic models from objects such as a sqlalchemy Base Model or while nesting pydantic models. I believe in the past I have ran into some inconsistencies with nesting pydantic models and I’ll bet one had from_attributes set and another did not.

Arbitrary class instances¶ (Formerly known as “ORM Mode”/from_orm).

Pydantic models can also be created from arbitrary class instances by reading the instance > attributes corresponding to the model field names. One common application of this functionality is integration with object-relational mappings (ORMs).

To do this, set the from_attributes config value to True (see the documentation on Configuration for more details).

The example here uses SQLAlchemy, but the same approach should work for any ORM.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Top Python libraries of 2024 | Tryolabs

2025-01-23T00:31:18Z

Top Python libraries of 2024

Dive into our 10th annual Python Libraries roundup for 2024, now featuring separate curated lists for General Use and AI / ML / Data tools. Discover this year's most innovative additions to the eco...

Tryolabs · tryolabs.com

Really good listicle of new modern top python libraries from 2024. Very well done article with images, links, and an actually quality listicle with many things I’ve never even heard of.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Choosing color palettes — seaborn 0.13.2 documentation

2025-01-22T13:47:54Z

Good overview of seaborn color palettes. They have all sorts of different types, some designed to purposfully give each color the same weight for catecorization. Some designd to give linear differences in value, some have a parabolic feel with a diverging nature.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

I fixed lists in my blog

2025-01-21T16:08:01Z

A small improvement to my blog today. I was checking out my /now post and noticed that lists have left padding but no discs or numbers. I popped open my tailwind/app.css added these lines and rebuilt.

#post-body li {
  @apply list-disc;
}
#post-body ol {
  @apply list-decimal;
}

And rebuild, which I actually do with ⭐ casey just, but showing the command here for clarity.

npx tailwindcss --input tailwind/app.css --output static/app-39.css

WHY the applies #

Before #

After #

nvim-manager

2025-01-12T21:14:19Z

I recently built a cli application as a nearly-one-shot-app called nvim-manager. It manages your nvim dotfiles install.

Why #

How is nvim manager any better #

nvim-manager allows you to install pinned versions of your dotfiles, your friends dotfiles, and distros in ~/.config. This allows you to have stable versions that will not break installed while you change things.

I’m sure most of us have experienced the pain of installing one plugin, only to update all of your plugins and break something.

Or, you have small changes on every machine you use, because they are all just a bit different and now you have big merge conflicts to deal with.

All of this aside you can install a distro to get you by, or a known working version of your own dotfiles.

So all these versions in ~/.config #

ya, thats the magic of NVIM_APPNAME, I can boot up any of these intalled working versions in an instant with NVIM_APPNAME=nvim-waylonwalker-v0.0.1 nvim. I can still cowboy up and break my main install, but as long as I am diligent to keep these installs untouched I will always have a version to fall back to in that moment of need.

So what do I need to do? #

start tagging your dotfiles as you hit stable versions
Install curl https://i.jpillora.com/waylonwalker/nvim-manager | bash
Add some env variables to your shell startup (~/.bashrc or ~/.zshrc for example)
Install that pinned version of your dotfiles nvim-manager install v0.0.1
Install a distro nvim-manager install --distro astronvim

Here are those environment variables I was talking about, set them to use your dotfiles repo, name it what you like, and set your NVIM_APPNAME if you want to default to a stable nvim and force yourself to unset NVIM_APPNAME to live on the edge.

Install nvim-manager #

apt install curl git unzip
curl https://i.jpillora.com/MordechaiHadad/bob | bash
bob install nightly
bob use nightly
ln -s ~/.local/share/bob/nvim-bin/nvim ~/.local/bin
curl https://i.jpillora.com/waylonwalker/nvim-manager | bash

Install your own dotfiles #

Setup with the following environment variables.

# nvim-manager
export NVIM_MANAGER_REPO=https://github.com/WaylonWalker/devtainer
export NVIM_CONFIG_PATH=nvim/.config/nvim
export NVIM_MANAGER_INSTALL_DIR=$HOME/.config
export NVIM_MANAGER_PREFIX="nvim-waylonwalker"
export NVIM_APPNAME=${NVIM_MANAGER_PREFIX}-v0.0.1

nvim-manager install v0.0.1
nvim

Note I like installer by jpillora, I self host it for my own security, but feel free to download from GH if it makes you feel safer.

Ubuntu Container Speedrun #

Here is a speedrun to getting nvim up and running in an ubuntu container.

set -euxo
mkdir -p ~/.local/bin
export PATH=$PATH:~/.local/bin
apt update
apt install curl git unzip -y
curl -LsSf https://astral.sh/uv/install.sh | sh
curl https://i.jpillora.com/MordechaiHadad/bob | bash
mv bob ~/.local/bin
bob install nightly
bob use nightly
ln -s ~/.local/share/bob/nvim-bin/nvim ~/.local/bin
curl https://i.jpillora.com/waylonwalker/nvim-manager | bash
mv nvim-manager ~/.local/bin
export NVIM_MANAGER_REPO=https://github.com/WaylonWalker/devtainer
export NVIM_CONFIG_PATH=nvim/.config/nvim
export NVIM_MANAGER_INSTALL_DIR=$HOME/.config
export NVIM_MANAGER_PREFIX="nvim-waylonwalker"
export NVIM_APPNAME=${NVIM_MANAGER_PREFIX}-v0.0.1
nvim-manager install v0.0.1
nvim-manager install --distro lazyvim
nvim-manager install --distro astronvim
nvim-manager install --distro nvchad
nvim-manager install --distro kickstart
nvim-manager install --distro lunarvim

# plugins like treesiter need gcc and make
apt install gcc make -y
export TZ="America/Chicago"
export DEBIAN_FRONTEND=noninteractive
apt update
apt install tzdata -y
ln -fs /usr/share/zoneinfo/$TZ /etc/localtime
dpkg-reconfigure -f noninteractive tzdata
# Some of the mason installs need npm
apt install nodejs npm -y
# plugins like telescope require ripgrep
apt install fzf ripgrep -y

Give it a Star #

I’d appreciate a star if you find this app useful.

https://github.com/waylonwalker/nvim-manager

one-shot-s3-cleanup

2025-01-12T11:18:25Z

So I made a mistake in an app I am working on and ended up creating thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails… etc. I was able to delete them all, but I wanted to make a one-shot script to do it.

One Shot App #

I got the idea of the one shot app from Simon Willison and replicated his setup in a chatgpt project

Initial Prompt #

a mistake was made in my s3 bucket and I've created a bunch of extra files
write a script that deletes all files that contain _thumb_thumb

⬢ [devtainer] ❯ aws s3 ls s3://dropper
2024-12-29 14:32:32      16158 02271f4c-be18-4bea-b23e-d00f9fe42b9f.webp
2025-01-11 14:20:49       2878 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb.webp
2025-01-11 14:21:17       2858 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb_thumb.webp
2025-01-11 14:21:44       2856 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb_thumb_thumb.webp
2025-01-11 14:21:44       2856 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb_thumb_thumb_thumb.webp
2024-12-27 10:25:36       2812 06422c09-d0da-44ec-9339-786864ebccf2.webp
2025-01-11 14:20:49       2710 06422c09-d0da-44ec-9339-786864ebccf2_thumb.webp
2025-01-11 14:21:17       2652 06422c09-d0da-44ec-9339-786864ebccf2_thumb_thumb.webp
2025-01-11 14:21:45       2632 06422c09-d0da-44ec-9339-786864ebccf2_thumb_thumb_thumb.webp
2025-01-11 14:21:45       2632 06422c09-d0da-44ec-9339-786864ebccf2_thumb_thumb_thumb_thumb.webp
2024-12-29 20:44:14     146060 074edddb-8b46-4d94-9b55-d70a7bb74366.webp
2025-01-11 14:20:50      12476 074edddb-8b46-4d94-9b55-d70a7bb74366_thumb.webp
2025-01-11 14:21:17      12400 074edddb-8b46-4d94-9b55-d70a7bb74366_thumb_thumb.webp

create a typer application to do this job
include a --dry-run flag
make the search for _thumb_thumb editable

s3_cleanup #

Here is the result of the s3_cleanup script. It ended up being 5 or 6 passes to get everything I wanted with soft-delete enabled. Nice preview outputs in the dry run. This was 90% created by chatgpt 4o, with some slight hand edits by me along the way.

#!/usr/bin/env -S uv run --quiet --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "boto3",
#     "typer",
#     "rich"
# ]
# ///

import boto3
from datetime import datetime, timezone
from rich.console import Console
from rich.table import Table
import traceback
import typer

app = typer.Typer()
console = Console()

DEFAULT_SOFT_DELETE_DIR = "soft-delete/"


def human_readable_size(size: int) -> str:
    """Convert bytes to a human-readable format."""
    for unit in ["B", "KB", "MB", "GB", "TB"]:
        if size < 1024:
            return f"{size:.2f} {unit}"
        size /= 1024
    return f"{size:.2f} PB"


def calculate_file_age(last_modified: datetime) -> str:
    """Calculate the age of the file from the current time."""
    now = datetime.now(timezone.utc)
    age = now - last_modified
    days = age.days
    if days > 0:
        return f"{days} days"
    hours, remainder = divmod(age.seconds, 3600)
    minutes = remainder // 60
    if hours > 0:
        return f"{hours} hours"
    return f"{minutes} minutes"


@app.command()
def clean(
    bucket_name: str = typer.Argument(..., help="The name of the S3 bucket."),
    substring: str = typer.Argument(
        ..., help="The substring to search for in file names."
    ),
    dry_run: bool = typer.Option(
        False, help="If set, no files will be deleted; only listed."
    ),
    soft_delete: bool = typer.Option(
        False,
        help="If set, files will be moved to a soft delete directory instead of permanently deleted.",
    ),
    soft_delete_dir: str = typer.Option(
        DEFAULT_SOFT_DELETE_DIR, help="The directory to move soft-deleted files to."
    ),
    max_age: int = typer.Option(None, help="Maximum age of files to include, in days."),
    min_age: int = typer.Option(None, help="Minimum age of files to include, in days."),
):
    """Delete or soft-delete files in an S3 bucket that contain a specific substring in their names."""
    s3 = boto3.client("s3")

    try:
        # List all objects in the bucket
        response = s3.list_objects_v2(Bucket=bucket_name)

        if "Contents" not in response:
            console.print("[bold yellow]No files found in the bucket.[/bold yellow]")
            return

        now = datetime.now(timezone.utc)

        def file_within_age_range(obj):
            file_age = (now - obj["LastModified"]).days
            if max_age is not None and file_age > max_age:
                return False
            if min_age is not None and file_age < min_age:
                return False
            return True

        files_to_process = [
            obj
            for obj in response["Contents"]
            if substring in obj["Key"] and file_within_age_range(obj)
        ]

        if not files_to_process:
            console.print(
                "[bold yellow]No files matching criteria found in the bucket.[/bold yellow]"
            )
            return

        if dry_run:
            console.print(
                "[bold blue]Dry run mode: The following files would be processed:[/bold blue]"
            )
            table = Table(title="Files to be Processed")
            table.add_column("File Name", style="cyan", no_wrap=True)
            table.add_column("Size", style="magenta", justify="right")
            table.add_column("Age", style="green", justify="right")
            table.add_column("Action", style="yellow", no_wrap=True)

            total_size = 0
            for obj in files_to_process:
                human_size = human_readable_size(obj["Size"])
                file_age = calculate_file_age(obj["LastModified"])
                action = (
                    f"[bold yellow]-> {soft_delete_dir}{obj['Key']}[/bold yellow]"
                    if soft_delete
                    else "[bold red]DELETE[/bold red]"
                )
                table.add_row(obj["Key"], human_size, file_age, action)
                total_size += obj["Size"]

            console.print(table)
            console.print(
                f"\n[bold green]Summary:[/bold green] {len(files_to_process)} files, Total Size: {human_readable_size(total_size)}, Operation: {'Soft Delete' if soft_delete else 'Delete'}"
            )

        else:
            for obj in files_to_process:
                file_key = obj["Key"]
                if soft_delete:
                    copy_source = {"Bucket": bucket_name, "Key": file_key}
                    new_key = f"{soft_delete_dir}{file_key}"
                    s3.copy_object(
                        CopySource=copy_source, Bucket=bucket_name, Key=new_key
                    )
                    s3.delete_object(Bucket=bucket_name, Key=file_key)
                    console.print(
                        f"[bold yellow]Moved to soft-delete:[/bold yellow] {file_key} -> {new_key}"
                    )
                else:
                    s3.delete_object(Bucket=bucket_name, Key=file_key)
                    console.print(f"[bold red]Deleted:[/bold red] {file_key}")

            console.print(
                f"[bold green]Operation completed. {len(files_to_process)} files processed. Operation: {'Soft Delete' if soft_delete else 'Delete'}[/bold green]"
            )

    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] {e}", style="red")


@app.command()
def clear_soft_delete(
    bucket_name: str = typer.Argument(..., help="The name of the S3 bucket."),
    soft_delete_dir: str = typer.Option(
        DEFAULT_SOFT_DELETE_DIR, help="The soft delete directory to clear."
    ),
    dry_run: bool = typer.Option(
        False, help="If set, no files will be deleted; only listed."
    ),
):
    """Remove all files in the soft delete directory."""
    s3 = boto3.client("s3")

    try:
        response = s3.list_objects_v2(Bucket=bucket_name, Prefix=soft_delete_dir)

        if "Contents" not in response:
            console.print(
                f"[bold yellow]No files found in the soft delete directory '{soft_delete_dir}'.[/bold yellow]"
            )
            return

        files_to_delete = [obj for obj in response["Contents"]]

        if dry_run:
            console.print(
                "[bold blue]Dry run mode: The following files would be deleted from the soft delete directory:[/bold blue]"
            )
            table = Table(title="Files to be Deleted")
            table.add_column("File Name", style="cyan", no_wrap=True)
            table.add_column("Size", style="magenta", justify="right")
            total_size = 0

            for obj in files_to_delete:
                human_size = human_readable_size(obj["Size"])
                table.add_row(obj["Key"], human_size)
                total_size += obj["Size"]

            console.print(table)
            console.print(
                f"\n[bold green]Summary:[/bold green] {len(files_to_delete)} files, Total Size: {human_readable_size(total_size)}"
            )
        else:
            for file_key in files_to_delete:
                file_key = file_key["Key"]
                s3.delete_object(Bucket=bucket_name, Key=file_key)
                console.print(
                    f"[bold red]Deleted from soft-delete:[/bold red] {file_key}"
                )

            console.print(
                f"[bold green]Soft delete directory '{soft_delete_dir}' cleared. {len(files_to_delete)} files removed.[/bold green]"
            )

    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] {e}", style="red")
        console.print("[bold red]Operation failed.[/bold red]")

        console.print(traceback.format_exc())


@app.command()
def undo_soft_delete(
    bucket_name: str = typer.Argument(..., help="The name of the S3 bucket."),
    soft_delete_dir: str = typer.Option(
        DEFAULT_SOFT_DELETE_DIR, help="The soft delete directory to restore from."
    ),
    dry_run: bool = typer.Option(
        False, help="If set, no files will be restored; only listed."
    ),
):
    """Restore all files from the soft delete directory to their original locations."""
    s3 = boto3.client("s3")

    try:
        response = s3.list_objects_v2(Bucket=bucket_name, Prefix=soft_delete_dir)

        if "Contents" not in response:
            console.print(
                f"[bold yellow]No files found in the soft delete directory '{soft_delete_dir}'.[/bold yellow]"
            )
            return

        files_to_restore = [obj for obj in response["Contents"]]

        if dry_run:
            console.print(
                "[bold blue]Dry run mode: The following files would be restored:[/bold blue]"
            )
            table = Table(title="Files to be Restored")
            table.add_column("File Name", style="cyan", no_wrap=True)
            table.add_column("Size", style="magenta", justify="right")
            table.add_column("Original Location", style="green", no_wrap=True)
            total_size = 0

            for obj in files_to_restore:
                human_size = human_readable_size(obj["Size"])
                original_key = obj["Key"][len(soft_delete_dir) :]
                table.add_row(obj["Key"], human_size, original_key)
                total_size += obj["Size"]

            console.print(table)
            console.print(
                f"\n[bold green]Summary:[/bold green] {len(files_to_restore)} files, Total Size: {human_readable_size(total_size)}"
            )
        else:
            for obj in files_to_restore:
                file_key = obj["Key"]
                original_key = file_key[
                    len(soft_delete_dir) :
                ]  # Remove the soft delete prefix
                copy_source = {"Bucket": bucket_name, "Key": file_key}
                s3.copy_object(
                    CopySource=copy_source, Bucket=bucket_name, Key=original_key
                )
                s3.delete_object(Bucket=bucket_name, Key=file_key)
                console.print(
                    f"[bold green]Restored:[/bold green] {file_key} -> {original_key}"
                )

            console.print(
                f"[bold green]Restoration completed. {len(files_to_restore)} files restored.[/bold green]"
            )

    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] {e}", style="red")


if __name__ == "__main__":
    app()

full help text output #

Here is the help text for all of the commands in the script.

dropper on  main [!?] is 📦 v1.0.0 via  v22.13.0  v3.11.10 on  (us-east-1)  NO PYTHON VENV SET took 7s
⬢ [devtainer] ❯ ./scripts/s3_cleanup.py --help

 Usage: s3_cleanup.py [OPTIONS] COMMAND [ARGS]...

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                         │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.  │
│ --help                        Show this message and exit.                                                       │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ clean               Delete or soft-delete files in an S3 bucket that contain a specific substring in their      │
│                     names.                                                                                      │
│ clear-soft-delete   Remove all files in the soft delete directory.                                              │
│ undo-soft-delete    Restore all files from the soft delete directory to their original locations.               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

⬢ [devtainer] ❯ ./scripts/s3_cleanup.py clean --help

 Usage: s3_cleanup.py clean [OPTIONS] BUCKET_NAME SUBSTRING

 Delete or soft-delete files in an S3 bucket that contain a specific substring in their names.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    bucket_name      TEXT  The name of the S3 bucket. [default: None] [required]                               │
│ *    substring        TEXT  The substring to search for in file names. [default: None] [required]               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --dry-run            --no-dry-run                 If set, no files will be deleted; only listed.                │
│                                                   [default: no-dry-run]                                         │
│ --soft-delete        --no-soft-delete             If set, files will be moved to a soft delete directory        │
│                                                   instead of permanently deleted.                               │
│                                                   [default: no-soft-delete]                                     │
│ --soft-delete-dir                        TEXT     The directory to move soft-deleted files to.                  │
│                                                   [default: soft-delete/]                                       │
│ --max-age                                INTEGER  Maximum age of files to include, in days. [default: None]     │
│ --min-age                                INTEGER  Minimum age of files to include, in days. [default: None]     │
│ --help                                            Show this message and exit.                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

⬢ [devtainer] ❯ ./scripts/s3_cleanup.py clear-soft-delete --help

 Usage: s3_cleanup.py clear-soft-delete [OPTIONS] BUCKET_NAME

 Remove all files in the soft delete directory.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    bucket_name      TEXT  The name of the S3 bucket. [default: None] [required]                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --soft-delete-dir                    TEXT  The soft delete directory to clear. [default: soft-delete/]          │
│ --dry-run            --no-dry-run          If set, no files will be deleted; only listed. [default: no-dry-run] │
│ --help                                     Show this message and exit.                                          │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

⬢ [devtainer] ❯ ./scripts/s3_cleanup.py undo-soft-delete --help

 Usage: s3_cleanup.py undo-soft-delete [OPTIONS] BUCKET_NAME

 Restore all files from the soft delete directory to their original locations.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    bucket_name      TEXT  The name of the S3 bucket. [default: None] [required]                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --soft-delete-dir                    TEXT  The soft delete directory to restore from. [default: soft-delete/]   │
│ --dry-run            --no-dry-run          If set, no files will be restored; only listed.                      │
│                                            [default: no-dry-run]                                                │
│ --help                                     Show this message and exit.                                          │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

💭 cyclotruc/gitingest: Replace 'hub' with 'ingest' in any github...

2025-01-09T13:46:33Z

GitHub - coderamp-labs/gitingest: Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase

Replace 'hub' with 'ingest' in any GitHub URL to get a prompt-friendly extract of a codebase - coderamp-labs/gitingest

Gitingest has a python package on pypi that you can run with uvx, and it accepts the same arguments as the web version, right in your terminal

⬢ [devtainer] ❯ uvx gitingest --help
Usage: gitingest [OPTIONS] SOURCE

  Analyze a directory and create a text dump of its contents.

Options:
  -o, --output TEXT           Output file path (default: <repo_name>.txt in
                              current directory)
  -s, --max-size INTEGER      Maximum file size to process in bytes
  -e, --exclude-pattern TEXT  Patterns to exclude
  -i, --include-pattern TEXT  Patterns to include
  --help                      Show this message and exit.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 kitze 🚀 on X: "holy fk I thought package.json was bad but TIL ...

2024-12-27T02:51:27Z

X (formerly Twitter) · x.com

kitze hating on python packaging in new ways. Python packaging has a lot of quirks that can make it infuriating. Not once have I thought “you know what this needs, quotes and braces”

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 4 Tips for Building a Production-Ready FastAPI Backend - YouTube

2024-12-26T02:42:17Z

FastAPI Backends for Production" playlabel="Play: How I Actually Build FastAPI Backends for Production">

Great list of 4 tips for running fastapi applications.

Keep routes small #

Fat routers with all of the logic built in makes them hard to test, hard to refactor, causes lots of duplication, and makes it hard to reuse the business logic code later in something like a cli application.

Deploy Early #

I really like this advice! He reccommends deploying as early as you can get a healthcheck live in your application. I’ve found too many times developers build something that is really hard, or impossible to deploy, when if they had tried to deploy early they would have spotted some easy to fix issues. This is less important if you are building out of a template that your team commonly deploys from, but very important with new patterns.

https://youtu.be/XlnmN4BfCxw?si=ks1wvmgDyoQLgrv2&t=1093

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Building Python tools with a one-shot prompt using uv run and ...

2024-12-25T02:26:31Z

Building Python tools with a one-shot prompt using uv run and Claude Projects

I’ve written a lot about how I’ve been using Claude to build one-shot HTML+JavaScript applications via Claude Artifacts. I recently started using a similar pattern to create one-shot Python uti...

PEP 723 – Inline script metadata | peps.python.org

This is really interesting, the lazy uv scripts are really becoming quite appealing, especially for something like this to just pop out of an llm ready to run. The article features several examples of these one-shot prompt ideas that I suggest you give a try, and a prompt for creating them.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 PEP 723 – Inline script metadata | peps.python.org

2024-12-24T15:53:29Z

This PEP specifies a metadata format that can be embedded in single-file Python scripts to assist launchers, IDEs and other external tools which may need to interact with such scripts.

Python Enhancement Proposals (PEPs) · peps.python.org

PEP 723 is what is inspiring all of these lazy self installing python scripts, Authored by the author of hatch and pyapp. This is a really cool thing that uv has picked up and made python packaging just a bit easier.. maybe… dependency resolution still sucks.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Switching from virtualenvwrapper to direnv, Starship, and uv

2024-12-24T03:30:57Z

Switching from virtualenvwrapper to direnv, Starship, and uv

Earlier this week I considered whether I should finally switch away from virtualenvwrapper to using local .venv managed by direnv. I’ve never …

treyhunner.com

I’ve kinda fallen out of using direnv now that a lot of my projects use hatch, I generally just hatch shell into them. I just need to make sure I go through all of them and make my installer uv. Now I’ve been thinking about making uv my only needed dependency to run a python project and leaning more to something like uv run --with . uvicorn myapp --reload

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Lazy self-installing Python scripts with uv

2024-12-24T03:20:33Z

Lazy self-installing Python scripts with uv

I frequently find myself writing my own short command-line scripts in Python that help me with day-to-day tasks. It’s so easy to throw together …

treyhunner.com

I really like Trey’s steps to making an executable python script with uv

his old process seems to be the same with a new shebang

Add an appropriate shebang line above the first line in the file (e.g. #!/usr/bin/env python3)

Aet an executable bit on the file (chmod a+x my_script.py)

Place the script in a directory that’s in my shell’s PATH variable (e.g. cp my_script.py ~/bin/my_script)

And here is the new format the the shebang followed by the metadata comment block defined in PEP 723.

#!/usr/bin/env -S uv run --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "ffmpeg-normalize",
# ]
# ///

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

thoughts 0.0.4

2024-12-18T19:04:27Z

This is such an improvement to the backend of my website it warrants a blog post of celebration. For far too long I’ve been dealing with a tiny ass edit form on thoughts. I tend to not edit them, and try to get them right in one go. This is kinda the point of a thought, its a quick post meant to be the size of a tweet, but sometimes I’m leaving thoughts on a video or long post and want to make sure I have a good save point, but I just keep the thing in draft and hope I don’t loose if for far too long.

Results #

Let’s see this change in action!!

before #

This is the tiny ass form nested deeply in the flow of the feed. When I made it I naively just swapped out the post itself with the edit form, and swapped the post back in after edit.

thoughts is built with HTMX btw so all html is rendered in the backend and swapped by htmx client side.

after #

Now the edit is a full page modal with a nice blurry backdrop effect to the rest of the content. This feels pretty similar to making a new post on twitter.

How #

How did I do this with htmx? I had to break out of the mindset my brain was in with swapping in place and letting the edit form take over the entire screen.

First the empty #modal-container was added to the top of every page.

<div id="modal-container"></div>

Then each post that gets added to the page already had an edit button, but now the target is set to #modal-container, and the swap is set to innerHTML so that we keep the #modal-container in place.

<button class="h-8 w-8 p-1 text-center" hx-get="{{ config.root }}/edit-thought/{{ post.id }}"
        hx-target="#modal-container" hx-swap="innerHTML" title="Edit">
    {% include 'edit.svg' %}
</button>

Now the edit post that is returned from the server is turned into a full height and width modal with a nice backgrop blur over the content.


<div id="modal-container">
    <div class="fixed inset-0 z-50 overflow-y-auto" aria-labelledby="modal-title" role="dialog" aria-modal="true">
        <!-- Background backdrop -->
        <div class="fixed inset-0 bg-black/30 backdrop-blur-sm"></div>

        <!-- Modal panel -->
        <div class="flex min-h-screen items-center justify-center p-4">
            <div class="relative w-full max-w-4xl transform rounded-xl bg-zinc-900 p-6 shadow-2xl transition-all">
... similar to the original edit form in here
            </div>
        </div>
    </div>
</div>

One notable change to the original edit form is clearing the modal container on submit. It is done with an hx-on::after-request event and one line of js.

<form id="websiteForm"
      hx-patch="{{ config.root }}/post/html/"
      method="POST"
      name="newPost"
      hx-target="#post-{{ post.id }}"
      hx-swap="outerHTML"
      hx-on::after-request="document.getElementById('modal-container').innerHTML = ''">
    ```

Similarly on the Cancel button.

``` html
<button class="rounded-lg border border-black bg-zinc-950 px-6 py-3 text-lg hover:bg-zinc-900"
        type="button"
        onclick="document.getElementById('modal-container').innerHTML = ''">
    Cancel
</button>

Hotkeys #

Now I struggled to get this right with htmx, and it seemed like things were getting more complicated by trying to get the buttons to trigger using htmx triggered keyboard events, I ended up just using javascript. Its still in the same file, so locality of behavior is barely an issue on this one anyways, and the js was just working.

<script>
document.addEventListener('keydown', function(e) {
    if (e.key === 'Escape') {
        document.getElementById('modal-container').innerHTML = '';
    }
    if (e.key === 'Enter' && e.ctrlKey) {
        e.preventDefault();
        htmx.trigger('#websiteForm', 'submit');
    }
});
</script>

Other small change #

When the server is returns an empty list of posts the post container would change size and cause a layout shift. Now the empty posts element, while not great does not cause layout shift.

<ul id="posts" class="min-h-screen"><li>No posts</li></ul>

💭 Add a healthcheck to your FastAPI app | Pype.dev

2024-12-16T19:05:50Z

Add a healthcheck to your FastAPI app | Nic Payne

I'm building a few FastAPI apps to throw in docker and run on my homelab... I wanted to add healthchecks and here's a simple way to do it Make sure to

pype.dev

Nice example of adding a healthcheck to fastapi, and integrating it with docker. Don’t forget to include curl in the install, nice touch.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 watchfiles

2024-12-13T02:08:24Z

watchfiles

Simple, modern and high performance file watching and code reload in python.

watchfiles.helpmanual.io

Simple, modern and high performance file watching and code reload in python.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

price an stl print on slant3d

2024-11-18T15:29:30Z

I’ve been playing with 3d printing some items through the slant3d api. I’ve been pricing out different prints by running a slice request through their api.

make a project #

I’ve been using uv for project management. It’s been working well for quick projects like this while making it reproducible, I’m still all in on hatch for libraries.

mkdir slantproject
cd slantproject
uv init
uv venv
. ./.venv/bin/activate
uv add httpx rich python-dotenv

Get an api key #

You will need an api key from the slant api, which currently requires a google account and a credit card to create.

# .env
#  replace with your api key from https://api-fe-two.vercel.app/
SLANT_API_KEY=sl-**

slicing an stl with teh slant api #

Then you can run the python script to price out your print. I’m not exactly sure how this compares to an order, especially when you add in different materials.

from dotenv import load_dotenv
import httpx
import os

load_dotenv()

stl_url = ''
api_key = os.environ["SLANT_API_KEY"]

api = httpx.Client(base_url="https://www.slant3dapi.com/api/slicer")

res = httpx.post(
    "https://www.slant3dapi.com/api/slicer",
    json={"fileURL": stl_url},
    headers={"api-key": api_key, "Content-Type": "application/json"},
    timeout=60,
)


print(res.json())

💭 Production-ready Python Docker Containers with uv

2024-11-15T14:34:10Z

Production-ready Python Docker Containers with uv

Starting with 0.3.0, Astral’s uv brought many great features, including support for cross-platform lock files uv.lock. Together with subsequent fixes, it has become Python’s finest workflow too...

Hynek Schlawack · hynek.me

Hynek has a sick dockerfile setup for using uv in python projects.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Proper handling of None in WHERE condition · Issue #109 · fast...

2024-11-08T13:36:30Z

Proper handling of None in WHERE condition · Issue #109 · fastapi/sqlmodel

First Check I added a very descriptive title to this issue. I used the GitHub search to find a similar issue and didn't find it. I searched the SQLModel documentation, with the integrated search. I...

Document how to provide a negative number as an argument · fastapi typer · Discussion #798

SQLModel models ship with an is_, and is_not that you can use to compare to None without pesky linters complaining.

This comment summed it up quite well.

I believe this is concerned entirely with SQLAlchemy, not with SQLModel, and has to do with the required semantics to construct a BinaryExpression object. Hero.age == None evaluates to a BinaryExpression object which is eventually used to construct the SQL query that the SQLAlchemy engine issues to your DBMS. Hero.age is None evaluates to False immediately, and not a BinaryExpression, which short-circuits the query no matter the value of age in a row. From a cursory search, it does not seem that the is operator can be overridden in Python. This could help explain why the only possibility is by using the == operator, which can be overridden.

so rather than using Team.heros == None we can use Team.seros.is_(None) which checks for itentity not equality.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Document how to provide a negative number as an argument · fas...

2024-10-30T01:28:56Z

First Check I added a very descriptive title here. I used the GitHub search to find a similar question and didn't find it. I searched the Typer documentation, with the integrated search. I already ...

Today I learned that you cannot pass negative integers as values to typer. in this case context_settings={"ignore_unknown_options": True} is required so that the - does not look like a flag.

# script name: main.py

import typer

app = typer.Typer()


@app.command()
def failing(value: float):
    print(f"{value=}")
    

@app.command(
    context_settings={"ignore_unknown_options": True}
)
def working_good(value: float):
    print(f"{value=}")
    
    
if __name__ == "__main__":
    app()

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Will McGugan on X: "New Version of #Textual released! 😎🤓🦾 The ...

2024-10-27T02:51:07Z

X (formerly Twitter) · x.com

This new demo of textual is wildly awesome, so many widgets and features being added into the main library. The themes and animations are on point and far surpass my expectations of a tui.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Pin versions of dependencies · Issue #2200 · Kozea/WeasyPrint

2024-10-11T13:36:19Z

Pin versions of dependencies · Issue #2200 · Kozea/WeasyPrint

Stack trace: pdf = HTML(file_obj=html, base_url='/').write_pdf() File "/service/venv/lib/python3.9/site-packages/weasyprint/__init__.py", line 252, in write_pdf self.render(font_config, counter_sty...

Installing and managing Python | uv

weazyprint was throwing me some errors, turns out that it’s currently not compatible with the latest pydyf package.

my error

TypeError: __init__() takes 1 positional argument but 3 were give

I fixed it by locking in pydyf at 0.8.0

pydyf==0.8.0

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Installing Python - uv

2024-08-23T14:55:10Z

A guide to using uv to install Python, including requesting specific versions, automatic installation, viewing installed versions, and more.

docs.astral.sh

uv now can install python for you. uv is inspired by rust’s cargo, an all in one fast package and installation manager. uv is so fast and becoming feature complete, it’s really changing the python packaging ecosystem.

uv python install
uv python install 3.12
uv python list

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 python is simultaneously at the top and the bottom of the tier...

2024-07-11T13:41:04Z

Hard to argue this take, happy to see that its at the top. With it being such an old language its amazing that it still holds this position, and not surprising that it has warts, and thing that have turn users off from wanting anything to do with it.

timestamped in the link

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 markdown-it-pyrs · PyPI

2024-07-03T13:56:30Z

markdown-it-pyrs

A Python interface for markdown-it.rs, using Rust for blazingly fast Markdown parsing ⚡️

GitHub - dropbox/pyannotate: Auto-generate PEP-484 annotations

markdown it py running in rust claims to be 20x faster. I’ll definitely look into this if markdown it py is ever a bottleneck in my performance. At first glance it appears that plugins are written in rust not python, and there is no admonition plugin, so I’ll keep my eye on it for now, but I can’t use it.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DiskCache API Reference — DiskCache 5.6.1 documentation

2024-07-03T13:35:12Z

diskcache has a peekitem method that allows you to lookup the expire_time of a cached item without changing it. I recently used this to implement debounce for fastapi background tasks with multiple workers running. since all the workers I care about are on the same machine, but running in different processes diskcache was a great option. All workers have access to the same disk, but not the same variables in memory.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

diskcache as debounce

2024-07-03T11:14:11Z

I’ve been using fastapi more and more lately and one feature I just started using is background tasks [[ thoughts-333 ]].

Seealso

basic diskcache example <a href="/python-diskcache/" class="wikilink" data-title="How I setup a sqlite cache in python" data-description="When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is . It&#39;s built on sqlite with just enough..." data-date="2022-03-29">How I setup a sqlite cache in python</a>

One Background Task per db entry #

I am using it for longer running tasks and I don’t want to give users the ability to spam these long running tasks with many duplicates running at the same time. And each fastapi worker will be running in a different process so I cannot keep track of work in memory, I have to do it in a distributed fashion. Since they are all running on the same machine with access to the same disk, diskcache is a good choice

What I need #

check if a job is running
automatically expire jobs

Less infrastructure complexity #

My brain first went to thinking I needed another service like redis running alongside fastapi for this, then it hit me that I can use diskcache.

How I used diskcache #

Here is how I used diskcache to debounce taking screenshots for a unique shot every 60 seconds.

from diskcache import Cache

jobs_cache = Cache("jobs-cache")

@shots_router.get("/shot/{shot_id}", responses={200: {"content": {"image/webp": {}}}})
@shots_router.get("/shot/{shot_id}/", responses={200: {"content": {"image/webp": {}}}})
async def get_shot_by_id(
    background_tasks: BackgroundTasks,
    request: Request,
    shot_id: int,
):
    shot = Shot.get(shot_id)
    # check if the shot exists and return it or continue to create it.



    is_running = jobs_cache.get(shot_id)

    if is_running:
        expire_time = datetime.fromtimestamp(jobs_cache.peekitem(expire_time=True)[1]) - datetime.now()
        console.print("[red]Already running store_shot: ", shot_id)
        console.print(f"[red]Can retry in {expire_time.seconds}s")
    else:
        jobs_cache.set(shot_id, True, 60)
        background_tasks.add_task(
            store_shot,
            shot_id=shot_id,
        )

💭 PyAnnotate: Auto-generate PEP-484 annotations

2024-06-26T21:37:14Z

Auto-generate PEP-484 annotations. Contribute to dropbox/pyannotate development by creating an account on GitHub.

GitHub - tusharsadhwani/yen: The last Python environment manager you'll ever need.

pyannotate is a tool that uses runtime types to suggest type annotations to use. Test runners such as pytest can be used to generate the types.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 tusharsadhwani/yen: Create virtual environments for any Python...

2024-06-26T01:13:09Z

The last Python environment manager you'll ever need. - tusharsadhwani/yen

Best practices for Docker in production

Create virtual environments for any Python version, without needing Python installed.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Episode #323 Best practices for Docker in production - [Talk P...

2024-06-16T00:55:12Z

You

talkpython.fm

Great listen for anyone interested in productionizing python code with docker. Itamar brings up some

Don’t trust base images for security, upgrade your packages. Vulnerabilties become published and solved giving the bad guys istructions how to wreck your day and these fixes wont come to your docker application for up to two weeks due to image build tatency.

For job based containers pre-compile your pyc for faster startup.

Alpine linux is probably not what you want for python. Many packages such as postgres ship pre-copiled binaries that work for most linux distributions wich use glibc, but alpine uses musl so the binaries will be incompatable requiring you to need to install a bunch of build dependencies.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 xxHash - Extremely fast non-cryptographic hash algorithm

2024-06-03T13:34:30Z

xxHash - Extremely fast non-cryptographic hash algorithm

xxhash.com

xxHash is an extremely fast non-cryptographic hash algorithm, working at RAM speed limit. It is proposed in four flavors (XXH32, XXH64, XXH3_64bits and XXH3_128bits). The latest variant, XXH3, offers improved performance across the board, especially on small data.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 xxhash · PyPI

2024-06-03T13:34:05Z

xxhash

Python binding for xxHash

Mastodon.py — Mastodon.py 2.2.1 documentation

I hit an issue with markata where even though a bunch of articles were cached, the site build was still slow because I was hitting hashlib.sha256 so hard for cache keys. I was shocked when this popped up in my profiler as a significant portion of the time spent. I swapped out for xxhash and that issue completely went away.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Latest Page in Markata

2024-06-01T21:01:26Z

I just implemented a latest blog post link in Markata by asking for the first post slug from the blog feed. The implementation uses the jinja_md plugin to render jinja against the markdown and a tag to redirect.

My latest blog post is [[ {{ markata.feeds.blog.posts[0].slug }} ]].  Click the
link if you are not automatically redirected.

<meta http-equiv="Refresh" content="0; url='/{{ markata.feeds.blog.posts[0].slug }}'" />

Setting up the feed #

Feeds are setup in markata.toml configuration. They provide a handy way to create an html feed, rss feed, and quickly reference a filtered set of posts like this.

# you will need to enable the jinja_md plugin along with the defaults
[markata]
hooks = [
    "markata.plugins.jinja_md",
    "default",
    ]

# set up the blog feed
[[markata.feeds]]
slug = 'blog'
template = "feed.html"
filter = "date<=today and templateKey in ['blog-post'] and published"
sort = "date"
reverse = true

For more information on markata check out the full markata post.

sick wikilink hover

2024-05-30T21:48:49Z

Today I set up some sick wikilink hover effects using tailwind see A Case For Tailwindcss. When you hover over them they show an image preview of the link that you are going to. I cant find where I have seen this but it comes from some docs sites.

I’ll finish this article later, just excited to see it up.

markata

2024-05-30T13:45:54Z

This post is a work in progress.

Markata is the static site generator that I created to build my website about this site. I built it for me and I enjoy using it. I know everying it can do and I can extend it to do more easily. I have set it up for some friends to also use it and am proud that it helps them publish their content.

It’s a meme to create your own static site generator to make your website. Yes its funny, I don’t recommend it if your not ready for the level of work that comes with it, but at the end of the day it’s very rewarding and a great way to learn.

Static Sites were all the rage #

JAMStack was 🔥

Gatsby and Next.js hit the scene as the next generation of static site builders and were getting big around the time I started building my site in 2017. They were based on react. I dove into react and learned it enough to build my website, but I really lacked the depth of knowledge in the js ecosystem to really work on it effectively. For instance when it got slow, it was hard for me to profile and find out why. What I really wanted was my site written in python, which I knew the ecosystem for very well, but I did not find the existing site generators easy to extend to do the things I needed. Naively I thought it would be easier to just build my own than learn how to make one do what I wanted it to. Not invented here syndrome hitting hard.

In their Defense

I really lacked the depth of knowledge in the js ecosystem to really work on

it (gatsby) effectively.

Plugins all the way down #

I started building this as I dove deeper into the Kedro framework for Data Engineering pipelines. They use a plugin framework for allowing users to extend it called pluggy. I had a great experience extending kedro using pluggy and wanted to build something with based on pluggy when I had started markata.

A Minimal Pluggy Example

thinking about static sites in 2024

2024-05-30T12:57:24Z

actions build

k8s build

fastapi sqlite

💭 Hatch v1.10.0 - Hatch

2024-05-02T14:06:57Z

Hatch v1.10.0 - Hatch

hatch.pypa.io

Hatch be flyin.

This new release of hatch includes support for the new package installer uv which is just mind blowing fast compared to anything else we have in python right now.

[tool.hatch.envs.default]
installer = "uv"

The other features are cool too, check them out. I’ll probably be using the test runner, but I’ve been waiting for the uv support since uv launched.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 rich.panel — Rich 13.6.0 documentation

2024-04-30T18:10:37Z

This is my go to rich response container for clis written in python. It creates a nice box around the content on the screen and provides some nice separation in the output. It can be overdone, but comes in clutch when looking for that print statement in a long output.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

control minio token access

2024-04-24T17:03:34Z

To allow access only to the , you can pass add the Resource field to the User Policy when you create a new token.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/*"
      ]
    }
  ]
}

How to List Sqlite tables

2024-04-23T20:03:27Z

You can inspect sqlite tables with the sqlite shell.

note that you get into the shell with sqlite3 database.db

.tables

I also learned that .tables, .index and .schema are helper functions that query the sqlite_master table on the main database.

Here is an output from my redka database. The sqlite_master table contains all the sqlite objects type, name, tbl_name, rootpage, and sql to create them.

❯ sqlite3 database.db "SELECT * from sqlite_master;"
table|rkey|rkey|2|CREATE TABLE rkey (
    id       integer primary key,
    key      text not null,
    type     integer not null,
        version  integer not null,
    etime    integer,
        mtime    integer not null
)
index|rkey_key_idx|rkey|3|CREATE UNIQUE INDEX rkey_key_idx on rkey (key)
index|rkey_etime_idx|rkey|4|CREATE INDEX rkey_etime_idx on rkey (etime)
where etime is not null
trigger|rkey_on_type_update|rkey|0|CREATE TRIGGER rkey_on_type_update
before update of type on rkey
for each row
when old.type is not new.type
begin
    select raise(abort, 'key type mismatch');
end
table|rstring|rstring|5|CREATE TABLE rstring (
    key_id integer not null,
    value  blob not null,

    foreign key (key_id) references rkey (id)
          on delete cascade
)
index|rstring_pk_idx|rstring|6|CREATE UNIQUE INDEX rstring_pk_idx on rstring (key_id)
view|vstring|vstring|0|CREATE VIEW vstring as
  select
    rkey.id as key_id, rkey.key, rstring.value,
        datetime(etime/1000, 'unixepoch') as etime,
        datetime(mtime/1000, 'unixepoch') as mtime
  from rkey join rstring on rkey.id = rstring.key_id
  where rkey.type = 1
    and (rkey.etime is null or rkey.etime > unixepoch('subsec'))
table|rhash|rhash|7|CREATE TABLE rhash (
    key_id integer not null,
    field text not null,
    value blob not null,

    foreign key (key_id) references rkey (id)
      on delete cascade
)
index|rhash_pk_idx|rhash|8|CREATE UNIQUE INDEX rhash_pk_idx on rhash (key_id, field)
index|rhash_key_id_idx|rhash|9|CREATE INDEX rhash_key_id_idx on rhash (key_id)
view|vhash|vhash|0|CREATE VIEW vhash as
  select
    rkey.id as key_id, rkey.key, rhash.field, rhash.value,
        datetime(etime/1000, 'unixepoch') as etime,
        datetime(mtime/1000, 'unixepoch') as mtime
  from rkey join rhash on rkey.id = rhash.key_id
  where rkey.type = 4
    and (rkey.etime is null or rkey.etime > unixepoch('subsec'))

copier trust

2024-04-19T20:41:06Z

I recently had to update my copier-gallery command to trust my own templates because some of them have shell scripts that run afterwards. Be warned that this could be a dangerous feature to run on random templates you get off the internet, but these are all mine, so if I wreck it its my own fault.

copier copy --trust <template> <destination>

All the the copier copy api can be found with help.

❯ copier copy --help
copier copy 8.3.0

Copy from a template source to a destination.

Usage:
    copier copy [SWITCHES] template_src destination_path

Hidden-switches:
    -h, --help                         Prints this help message and quits
    --help-all                         Prints help messages of all sub-commands and quits
    -v, --version                      Prints the program's version and quits

Switches:
    -C, --no-cleanup                   On error, do not delete destination if it was
                                       created by Copier.
    --UNSAFE, --trust                  Allow templates with unsafe features (Jinja
                                       extensions, migrations, tasks)
    -a, --answers-file VALUE:str       Update using this path (relative to
                                       `destination_path`) to find the answers file
    -d, --data VARIABLE=VALUE:str      Make VARIABLE available as VALUE when rendering the
                                       template; may be given multiple times
    -f, --force                        Same as `--defaults --overwrite`.
    -g, --prereleases                  Use prereleases to compare template VCS tags.
    -l, --defaults                     Use default answers to questions, which might be
                                       null if not specified.
    -n, --pretend                      Run but do not make any changes
    -q, --quiet                        Suppress status output
    -r, --vcs-ref VALUE:str            Git reference to checkout in `template_src`. If you
                                       do not specify it, it will try to checkout the
                                       latest git tag, as sorted using the PEP 440
                                       algorithm. If you want to checkout always the
                                       latest version, use `--vcs-ref=HEAD`.
    -s, --skip VALUE:str               Skip specified files if they exist already; may be
                                       given multiple times
    -w, --overwrite                    Overwrite files that already exist, without asking.
    -x, --exclude VALUE:str            A name or shell-style pattern matching files or
                                       folders that must not be copied; may be given
                                       multiple times

ipython f2

2024-04-18T20:40:41Z

Today I accidentally ran f2 in ipython to discover that it opens your $EDITOR! I use this feature quite often in zsh, it is bound to <c-e> for me, and since I have my environment variable EDITOR set to nvim it opens nvim when I hit <c-e>. Today I discovered that Ipython has this bound to F2. If you know how to set it to <c-e> let me know I’ve tried, a lot.

export EDITOR=nvim
ipython
<F2>

better yet add export EDITOR=nvim to your .zshrc

# ~/.zshrc
export EDITOR=nvim

sqlmodel indexes

2024-04-17T21:55:25Z

I’ve really been enjoying using sqlmodel for my projects that need a database. One thing that I definitely lacked on for too long was indexing my database. I hit a point with one database where it was taking 7s for pretty simple paginated queries to return 10 records.

For every field that you will be querying on, you can create an index, by setting it equal to Field(index=True)

class Hero(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str = Field(index=True)
    secret_name: str
    age: int | None = Field(default=None, index=True)

example courtesy of the docs

Note

 primary keys are indexed by default.

The docs cover this pretty well, and in quite depth - Optimizing Queries

💭 inline-snapshot

2024-04-17T18:37:15Z

Redirecting

15r10nk.github.io

This is a cool snapshot testing tool that automatically creates, and updates test values for you.

Starting with some test code.

from inline_snapshot import snapshot


def something():
    return 1548 * 18489


def test_something():
    assert something() == snapshot()

now if I run pytest my tests will fail because my assert will fail, but if I run pytest --inline-snapshot=create it will fill out my snapshot values and the file will then look like this.

from inline_snapshot import snapshot


def something():
    return 1548 * 18489


def test_something():
    assert something() == snapshot(28620972)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

python inline snapshot

2024-04-17T13:32:53Z

inline-snapshot is a new tool that I am trying out for python testing. It takes snapshots of your outputs and places them inline with the test.

Here is the most basic starter.

import inline_snapshot

def test_one():
    assert 1 == snapshot()

Now when I run pytest my tests will fail because my assert has no value, but if I run pytest --inline-snapshot=create it will fill out my snapshot values and the file will then look like this.

import inline_snapshot

def test_one():
    assert 1 == snapshot(1)

It also works with pydantic models.

class MyModel(BaseModel):
    name: str
    age: int
    nickname: str | None = None


def test_my_model_instance():
    assert MyModel(name="Waylon", age=1) == snapshot(MyModel(name="Waylon", age=1))


def test_my_model_fields():
    me = MyModel(name="Waylon", age=1, nickname='Waylon')
    assert me.name == snapshot("Waylon")
    assert me.age == snapshot(1)
    assert me.nickname == snapshot("Waylon")

Composing Typer clis

2024-04-13T08:27:26Z

Typer makes it easy to compose your cli applications, like you might with a web router if you are more familiar with that. This allows you to build smaller applications that compose into a larger application.

You will see similar patterns in the wild, namely the aws cli which always has the aws <command> <subcommand> pattern.

Lets setup the cli app itself first. You can put it in project/cli/cli.py.

import typer

from project.cli.api import api_app
from project.cli.config import config_app
from project.cli.user import user_app
from project.cli.run import run_app

app = typer.Typer()

app.add_typer(api_app, name="api")
app.add_typer(config_app, name="config")
app.add_typer(user_app, name="user")
app.add_typer(run_app, name="run")

Creating an app that will become a command is the same as creating a regular app in Typer. We need to create a callback that will become our command, and a command that will become our subcommand in the parent app.

import typer
from rich.console import Console

from project.config import get_config

config_app = typer.Typer()

@config_app.callback()
def config():
    "model cli"


@config_app.command()
def show(
):
    project_config = get_config(env)
    Console().print(fokais_config)

Setting up the entrypoint in pyproject.toml.

[project.scripts] # <- this project is part of the config DO NOT change it
project = "project.cli.cli:app" # <- This project is the project name, DO change it

Now you can see each cli application as a sub command.

❯ project --help

 Usage: project [OPTIONS] COMMAND [ARGS]...

╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion  [bash|zsh|fish|powershell|pwsh]  Install completion for the specified shell.│
│                                                        [default: None]                            │
│ --show-completion     [bash|zsh|fish|powershell|pwsh]  Show completion for the specified shell,   │
│                                                        to copy it or customize the installation.  │
│                                                        [default: None]                            │
│ --help                                                 Show this message and exit.                │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ────────────────────────────────────────────────────────────────────────────────────────╮
│ api                        model cli                                                              │
│ config                     config cli                                                             │
│ user                       user cli                                                               │
│ run                        run cli                                                                │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯

In the example above we can run the command project config show to see the current configuration of our project.

One Day Build - Play Outside

2024-04-10T12:32:20Z

Inspired by Adam Savage and his One Day builds on youtube. I often build things, and want to make them generally useful for others and over configure out of the gate. This project is purely for me inspired by a need I have.

play-outside

!How-To #

This post will not directly show how to make a weather app, but document the process that I went through to make mine. It will show the tools that I used to make it, and the final result.

The Situation #

It often goes in our house ask dad while he is busy and he will probably just say yes without thinking much. This happens a lot when kids ask to go outside. I think sure, go for it, you will figure it out. Then my wife walks in and asks where they are, followed by, did you even check the weather, its -11 degrees outside right now.

I need a tool for this decision making process

Lungs #

You we have a family of not the most heathly lungs, we have my wife with lung cancer, one lung missing, and kids with asthma. We need to account for temperature, humidity, wind chill, and air quality before heading outside and seeing the repercussions of it later.

Final result #

So this is what I built, its a web app that checks the weather and air quality in your area and determines if its safe to go outside. It will even recommend limiting your time, or wearing a coat.

The Stack #

This is a one day build, I have both kids at home from school, so this is realistically only like 2-3 hours at most, so this has to be chosen based on familiarity.

Docker
k8s
Python
FastAPI
tailwind
httpx
OpenWeatherMap API
ipwho.is

This is the same stack (minue the apis) that I am using to build my startup fokais.com with. I am quite familiar with it and should be able to quickly make progress with it.

❯ tree
Permissions Size User   Date Modified Git Name
drwxr-xr-x     - waylon 16 Jan 12:21   -N  .
.rw-r--r--  3.8k waylon 16 Jan 12:24   -N ├──  deployment.yaml
.rw-r--r--   278 waylon 16 Jan 12:21   -N ├──  docker-compose.yml
.rw-r--r--   552 waylon 16 Jan 12:20   -N ├──  Dockerfile
.rw-r--r--   15k waylon 15 Jan 14:37   -N ├──  favicon.ico
.rw-r--r--  1.9k waylon 15 Jan 15:58   -N ├──  justfile
.rw-r--r--  1.1k waylon 15 Jan 11:39   -N ├──  LICENSE.txt
.rw-r--r--   51k waylon 15 Jan 14:38   -N ├──  package-lock.json
.rw-r--r--    69 waylon 15 Jan 14:38   -N ├──  package.json
drwxr-xr-x     - waylon 16 Jan 12:20   -N ├──  play_outside
.rw-r--r--   138 waylon 16 Jan 12:21   -N │  ├──  __about__.py
.rw-r--r--   115 waylon 15 Jan 11:39   -N │  ├──  __init__.py
.rw-r--r--  7.5k waylon 16 Jan 08:14   -N │  ├──  api.py
drwxr-xr-x     - waylon 15 Jan 21:30   -N │  ├──  cli
.rw-r--r--  3.5k waylon 15 Jan 21:30   -N │  │  └──  api.py
.rw-r--r--  2.8k waylon 16 Jan 12:20   -N │  ├──  config.py
.rw-r--r--  3.0k waylon 15 Jan 14:35   -N │  ├──  decorators.py
.rw-r--r--    51 waylon 15 Jan 14:50   -N │  └──  queries.py
.rw-r--r--  2.2k waylon 15 Jan 15:16   -N ├──  pyproject.toml
.rw-r--r--   506 waylon 15 Jan 11:39   -N ├──  README.md
drwxr-xr-x     - waylon 15 Jan 14:39   -N ├──  static
.rw-r--r--   21k waylon 16 Jan 08:10   -N │  ├──  app.css
.rw-r--r--   15k waylon 15 Jan 14:37   -N │  ├──  favicon.ico
.rw-r--r--   47k waylon 15 Jan 14:32   -N │  └──  htmx.org@1.9.8
drwxr-xr-x     - waylon 15 Jan 21:04   -N ├──  tailwind
.rw-r--r--  6.2k waylon 15 Jan 21:04   -N │  └──  app.css
.rw-r--r--   360 waylon 15 Jan 21:04   -N ├──  tailwind.config.js
drwxr-xr-x     - waylon 16 Jan 12:16   -N ├──  templates
.rw-r--r--  2.5k waylon 16 Jan 12:16   -N │  ├──  base.html
.rw-r--r--  2.2k waylon 16 Jan 12:11   -N │  ├──  card.html
.rw-r--r--   151 waylon 15 Jan 21:07   -N │  ├──  includestyles.html
.rw-r--r--   418 waylon 16 Jan 12:12   -N │  └──  index.html
drwxr-xr-x     - waylon 15 Jan 11:39   -N └──  tests
.rw-r--r--   115 waylon 15 Jan 11:39   -N    └──  __init__.py

htmx-btw">HTMX BTW #

I have been pairing up htmx with this stack quite a lot lately, and its fantastic, but honestly this idea just does not have a lot of endpoints, and I don’t think it needs it for a one day build, just toss everything into one page and call it good.

Getting Weather Data #

The first thing we need is the feels_like or Apartment Temperature. A quick google search lead me to https://openweathermap.org/ they have a very nice calculation for the feels like temerature already built in.

Now using openweathermap, we can get the feels like temperature, by latitude and longitude.

async def get_weather(lat_long):
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://api.openweathermap.org/data/2.5/weather?units=imperial&lat={lat_long['latitude']}&lon={lat_long['longitude']}&appid={config.open_weather_api_key}"
        )
        return response.json()

Where are you?? #

Since no one is going to know their current latitude and longitude we need a way to look this up for it to actually be useful. For this I leaned on https://ipwhois.io/

async def get_lat_long(ip_address):
    if ip_address is None:
        ip_address = "140.177.140.75"
    async with httpx.AsyncClient() as client:
        response = await client.get(f"https://ipwho.is/{ip_address}")
        return response.json()

Decision Tree #

For this I punted over to chatGPT to see what it came up with.

I am working on a web app to tell kids if its ok to play outside.  It is
targeted towards kids living in the  midwest united states.   I have the feels
like temperature already.  Set some values for the temperature

too cold,
no longer than 15 minutes,
coats required,
no longer than 15 minutes,
too hot

What it came up with wasn’t a terrible starting point, but its values essentially said kids cant play in the snow.

def determine_play_condition(feels_like_temperature):
    if feels_like_temperature < 20:
        return "It's too cold for extended play. Stay indoors and keep warm!"
    elif 20 <= feels_like_temperature <= 32:
        return "Perfect weather for snow play! Enjoy the winter wonderland!"
    elif 32 < feels_like_temperature <= 40:
        return "Coats and winter gear required for outdoor play. Stay cozy!"
    elif 40 < feels_like_temperature <= 50:
        return "You can play in the snow, but limit your time. It's getting warmer!"
    else:
        return "It's too warm for snow play. Find other fun activities indoors!"

# Example usage:
feels_like_temp = 25  # Replace with the actual feels like temperature
message = determine_play_condition(feels_like_temp)
print(message)

Some Massaging Later #

My wife sent me over this image from Tinkergarten, which is a chart made by the Iowa Department of Public Health. I used it as a guide to set some values, and added some conditions based on visibility and air quality index (aqi), which we have become all too familiar with over the past year with all the forest fires out Western US causing our lungs issues here in the Midwest.

(aqi) which we have become all too familiar with over the past year with all the forest fires out Western US

def determine_play_condition(weather, aqi=0):
    play_condition = PlayCondition()

    feels_like_temperature = weather["main"]["feels_like"]
    visibility = weather["visibility"]

    play_condition.message += hours_till_sunset(weather)

    if "after" in play_condition.message:
        play_condition.color = "bg-red-500"

    if visibility < 1000:
        play_condition.message += "It's too foggy. Find better activities inside!"
        play_condition.color = "bg-red-500"

    if aqi > 150:
        play_condition.message += "It's too polluted. Find better activities inside!"
        play_condition.color = "bg-red-500"
    elif aqi > 100:
        play_condition.message += "limit your time outside due to the poor air quality"
        play_condition.color = "bg-yellow-500"
    elif aqi > 50:
        play_condition.message += "Check the air quality outside at your discression."
        play_condition.color = "bg-yellow-500"
    else:
        play_condition.message += ""

    if feels_like_temperature < 10:
        play_condition.message += "It's too cold. Stay indoors and keep warm!"
        play_condition.color = "bg-red-500"
    elif feels_like_temperature < 30:
        play_condition.message += "You can play outside, but limit your time!"
        play_condition.color = "bg-yellow-500"
    elif feels_like_temperature < 40:
        play_condition.message += (
            "Coats and winter gear required for outdoor play. Stay cozy!"
        )
    elif feels_like_temperature < 50:
        play_condition.message += "Grab a warm jacket and enjoy your time outside!"
    elif feels_like_temperature < 60:
        play_condition.message += "Grab some long sleeves and enjoy your time outside!"
    elif feels_like_temperature > 90:
        play_condition.message += (
            "You can play outside, but limit your time in this heat!"
        )
        play_condition.color = "bg-yellow-500"
    elif feels_like_temperature > 109:
        play_condition.message += (
            "It's too hot for outdoor play. Find cooler activities indoors!"
        )
        play_condition.color = "bg-red-500"
    else:
        play_condition.message += "Enjoy your time outside!"
    return play_condition

Pulling the data together #

Since I will be needing all of the data together upon every request I put together one get_data function to return a dict of all of the data.

forecast

 I pulled the forecast endpoint from openweathermap as well, it looks like a

stripped down version of the regular weather endpoint, but every few hours over the course of the next 5 days.

async def get_data(request: Request):
    user_ip = request.headers.get("CF-Connecting-IP")
    lat_long = await get_lat_long(user_ip)
    weather = await get_weather(lat_long)
    forecast = await get_forecast(lat_long)
    air_quality = await get_air_quality(lat_long)
    weather["play_condition"] = determine_play_condition(
        weather,
        air_quality["list"][0]["main"]["aqi"],
    )

    forecast = [
        {"play_condition": determine_play_condition(x), **x}
        for x in forecast
        if datetime.fromtimestamp(x["dt"]).hour >= 6
        and datetime.fromtimestamp(x["dt"]).hour <= 21
    ]

    return {
        "request.client": request.client,
        "request.client.host": request.client.host,
        "user_ip": user_ip,
        "lat_long": lat_long,
        "weather": weather,
        "forecast": forecast,
        "air_quality": air_quality,
        "sunset": weather["sys"]["sunset"],
    }

FastAPI #

Fastapi here is a great framework, it uses pydantic to validate the data returned from the api, has a great dependency management system.

I am going to use none of that, all I need is one TemplateResponse using jinja. For good measure, I’ll toss in a /metadata route that returns the data.

from fastapi import FastAPI
app = FastAPI()


@app.get("/")
@no_cache
async def get_home(request: Request):
    data = await get_data(request)
    return config.templates.TemplateResponse("index.html", {"request": request, **data})

@app.get("/metadata")
async def root(
    request: Request,
):
    return await get_data(request)

No Cache Header #

I had some issues with cloudflare caching me and not letting me hit the api everytime. I’ve ran into this several times in the past, so I went to the cloudflare dashboard, manually busted the cache for the home route and popped a no_cache decorator on the get_home route.

def no_cache(func):
    not_cached_routes.append(f"{func.__module__}.{func.__name__}")

    @wraps(func)
    async def wrapper(*args, request: Request, **kwargs):
        # my_header will be now available in decorator
        if "request" in signature(func).parameters:
            kwargs["request"] = request

        if inspect.iscoroutinefunction(func):
            response = await func(*args, **kwargs)
        else:
            response = func(*args, **kwargs)

        response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
        response.headers["Pragma"] = "no-cache"
        response.headers["Expires"] = "0"
        return response

    return wrapper

@app.get("/")
@no_cache
async def get_home(request: Request):
    data = await get_data(request)
    return config.templates.TemplateResponse("index.html", {"request": request, **data})

This solved my caching issues.

Templates #

I used jinja for templating its built right into FastAPI.

from fastapi.templating import Jinja2Templates

templates = Jinja2Templates(directory="templates")

This will get you by for quite awhile, and I probably could deal with templates working just like this for a one day build, but I have some nice feautures that I like from other projects, and at least one specific to just this project. Once they are in my config object, I use them like so.

Note

 The `request` parameter is a requirement for all templates.

@app.get("/")
@no_cache
async def get_home(request: Request):
    data = await get_data(request)
    return config.templates.TemplateResponse("index.html", {"request": request, **data})

Globals #

I need a nice way to convert the openweathermap timestamps to human readable values.

from datetime import datetime

templates.env.globals["datetime"] = datetime

Then in jinja I can format the weather.dt variable like so.

{{ datetime.fromtimestamp(weather.dt).strftime(format = '%A') }}

Putting it together #

I put a get_templates function in my config, and the config object is passed the results as config. Again I copy pasted a function https_url_for from my other project so that I can use url_for in my templates and it work on both localhost and production.

import os
from datetime import datetime
from datetime import timezone

from fastapi.templating import Jinja2Templates
from rich.console import Console

console = Console()


@pass_context
def https_url_for(context: dict, name: str, **path_params: Any) -> str:
    request = context["request"]
    http_url = request.url_for(name, **path_params)
    return str(http_url).replace("http", "https", 1)

def get_templates(config: BaseSettings) -> Jinja2Templates:
    templates = Jinja2Templates(directory="templates")
    templates.env.filters["quote_plus"] = lambda u: quote_plus(str(u))
    templates.env.filters["timestamp"] = lambda u: datetime.fromtimestamp(
        u, tz=timezone.utc
    ).strftime("%B %d, %Y")
    templates.env.globals["https_url_for"] = https_url_for
    templates.env.globals["config"] = config
    templates.env.globals["datetime"] = datetime
    templates.env.globals["len"] = len
    templates.env.globals["int"] = int
    console.print(f'Using environment: {os.environ.get("ENV")}')

    if os.environ.get("ENV") in ["dev", "qa", "prod"]:
        templates.env.globals["url_for"] = https_url_for
        console.print("Using HTTPS")
    else:
        console.print("Using HTTP")

    return templates

Styles #

I’ve been all in on tailwind lately. I was a long holdout. I used Sass back in 2016/2017 when I was first getting into webdev, then went many years just straight vanilla. Tailwind just has some really well thought out color and rythm that makes it easy. I have also really been appreciating the locality of behavior part of it. I can make components in jinja with everything they need.

I set up my tailwind.config.js like so. It includes the typography plugin and some extra box shadows that are centered.

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: ["templates/**/*.html"],
  plugins: [require("@tailwindcss/typography")],
  theme: {
    extend: {
      boxShadow: {
        xlc: "0 0 60px 15px rgba(0, 0, 0, 0.3)",
        lgc: "0 0 20px 0px rgba(0, 0, 0, 0.3)",
      },

    },
  },
};

For my tailwind/app.css I set the background color of the page dark, text white, and make the scrollbar not default.

@tailwind base;
@tailwind components;
@tailwind utilities;

html {
  scroll-behavior: smooth;
  @apply bg-zinc-800 text-white autofill:bg-yellow-500;
}

::-webkit-scrollbar {
  @apply h-4 w-4;
}

::-webkit-scrollbar-track {
  @apply rounded-full bg-zinc-900;
}

body::-webkit-scrollbar-track {
  @apply rounded-full bg-pink-600;
}

::-webkit-scrollbar-thumb {
  @apply rounded-full bg-zinc-600 hover:bg-zinc-500;
}

body::-webkit-scrollbar-thumb {
  @apply rounded-full bg-cyan-500 hover:bg-cyan-400;
}

Compiling tailwind #

I run tailwind using npx after first installing the typography plugin. This will use my tailwind.config.js file, and the tailwind/app.css file as input, and output to static/app.css.

Note

 --watch will watch for changes, and automatically recompile as you make changes to any templates.

# install the typography plugin
npm i @tailwindcss/typography
# compile the css
npx tailwindcss --input tailwind/app.css --output static/app.css --watch

Deployment #

Docker #

CI #

K8s #

TEXT BASED #

play-outside

jinja macros

2024-04-07T14:54:08Z

I am working on a page for htmx-patterns and I ran into a situation with lots of duplication. Especially when i am using tailwind I run into situations where the duplication can get tedious to maintiain. The solution I found is macros.

Now I can use the same code for all of my links, and call the macro to use it.

{% macro link(id, text, boosted=false) -%}
<a
    class="
    {% if id is none %}
      pointer-events-none bg-terminal-950 text-terminal-900 ring-terminal-900
    {% else %}
      bg-terminal-950 hover:bg-terminal-900 hover:text-terminal-400 text-terminal-500 shadow-lg shadow-terminal-300/20 hover:shadow-terminal-300/30 ring-terminal-300
    {% endif %}
      cursor-pointer block text-center font-bold py-2 px-4 rounded w-full ring-1
    "
    {% if id is not none %}
    href="{{ url_for('boosted', id=id) }}"
    {% endif %}
    {% if boosted %}
    hx-boost="true"
    {% endif %}>
    {{ text }}
</a>
{%- endmacro %}

<h2 class='text-3xl font-light mt-0 max-w-xl text-center prose-xl mt-8 text-terminal-500'>
    Boosted Links
</h2>

<div class='flex flex-row gap-4'>
    {{ link(prev_id, 'Previous', boosted=True) }}
    {{ link(next_id, 'Next', boosted=True) }}
</div>

<h2 class='text-3xl font-light mt-0 max-w-xl text-center prose-xl mt-8 text-terminal-500'>
    Normal Links
</h2>

<div class='flex flex-row gap-4'>
    {{ link(prev_id, 'Previous', boosted=False) }}
    {{ link(next_id, 'Next', boosted=False) }}
</div>

jinja loop variable and htmx

2024-04-06T20:47:13Z

jinja has a loop variable that is very handy to use with htmx. Whether you want to implement a click to load more or an infinite scroll this loop variable is very handy.

{% for person in persons %}
<li
{% if loop.last %}
    hx-get="{{ url_for('infinite', page=next_page) }}"
    hx-trigger="intersect once"
    hx-target="#persons"
    hx-swap='beforeend'
    hx-indicator="#persons-loading"
{% endif %}
    {{ person.name.upper() }} -
    {{ person.phone_number }}
</li>
{% endfor %}

Now for every chunk of contacts that we load we will trigger the infinite scroll by loading more once the last one has intersected the screen.

fastapi jinja url_for with query params

2024-04-05T20:43:32Z

Out of the box FastAPI.">Starlette does not support url_for with query params. When trying to use url_for with query params it throws the following error.

starlette.routing.NoMatchFound: No route exists for name "infinite" and params "page"

In my searching for this I found starlette issue #560 quite helpful, but not complete, as it did not work for me.

import jinja2

if hasattr(jinja2, "pass_context"):
    pass_context = jinja2.pass_context
else:
    pass_context = jinja2.contextfunction

@pass_context
def url_for_query(context: dict, name: str, **params: dict) -> str:
    request = context["request"]
    url = str(request.url_for(name))
    if params == {}:
        return url
    from urllib.parse import parse_qs, urlencode, urlparse, urlunparse

    # Parse the URL
    parsed_url = urlparse(url)

    # Parse the query parameters
    query_params = parse_qs(parsed_url.query)

    # Update the query parameters with the new ones
    query_params.update(params)

    # Rebuild the query string
    updated_query_string = urlencode(query_params, doseq=True)

    # Rebuild the URL with the updated query string
    updated_url = urlunparse(
        (
            parsed_url.scheme,
            parsed_url.netloc,
            parsed_url.path,
            parsed_url.params,
            updated_query_string,
            parsed_url.fragment,
        )
    )

    if os.environ.get("ENV") in ["dev", "qa", "prod"]:
        updated_url = updated_url.replace("http", "https", 1)

    return updated_url

def get_templates():
    templates = Jinja2Templates(directory="templates")
    templates.env.globals["url_for"] = url_for_query
    return templates

https

 If you want url_for to work in production you need some way to convert http

to https. Here is how I make it work, for local development I export ENV=local then for each environment that I am running on a server I include it in the list and update ENV appropriately.

    if os.environ.get("ENV") in ["dev", "qa", "prod"]:
        updated_url = updated_url.replace("http", "https", 1)

The route might look something like this.

@infinite_router.get("/")
async def home(request: Request, page: int = 1, n: int = 10):
  ...

To access the home route using url_for in a jinja template you can use the following, once you have applied the url_for_query function as your default url_for

<a href="{{ url_for('home', page=1) }}">Home</a>

setting up a kind cluster with argocd installed

2024-04-04T18:42:18Z

Kind (Kubernetes in Docker) is a tool that makes it easy to create and tear down local clusters quickly. I like to use it to test out new workflows.

Argocd is a continuous delivery tool that makes it easy to setup gitops workflows in kubernetes.

Here is how you can setup a new kind cluster and install argocd into it using helm, the kubernetes package manager.

kind create cluster --name argocd

# your first time through you need to add the argocd repo
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# install argocd into the cluster
helm install argo argo/argo-cd --namespace argocd --create-namespace

# deploy the app of apps
kubectl apply -f apps/apps.yaml

If you want to add repos and apps to your cluster you can use the argo cli to do that, but first you will need forward the argocd port and login.

# Wait until Argo CD API server is available
echo "Waiting for Argo CD API server to be available..."
while ! kubectl wait --for=condition=available --timeout=60s deployment/argo-argocd-server -n argocd; do
  echo "Waiting for Argo CD API server to be ready..."
  sleep 10
done


kubectl port-forward svc/argo-argocd-server -n argocd 8080:443 &
argocd_admin_password=$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d)
argocd login localhost:8080 --insecure --username admin --password $argocd_admin_password
argocd repo add https://github.com/fokais-com/app.fokais.git --username waylonwalker --password ${GH_ARGO_PAT}
argocd app create app-fokais-local --repo https://github.com/fokais-com/app.fokais.git --path k8s/overlays/local --dest-server https://kubernetes.default.svc --sync-policy automated --sync-option Prune=true

💭 My New Python Lsp Setup

2024-03-09T02:58:03Z

![[None]]

I figured out the killer combination for python lsp servers, ruff and jedi! ruff does all of the diagnostics and formatting, then jedi handles all the code objects like go to definition and go to reference.

	local servers = {
		ruff_lsp = {},
		jedi_language_server = {},
}

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 flake8-to-ruff · PyPI

2024-03-09T02:29:51Z

Client Challenge

pypi.org

Underrated python library to on board ruff, or just use it on a project where its not the norm. ruff claims that its 99.9% compatible with black and when you read through the known differences they are clearly edge case bugs in black.

See this page for more about the comparison to black https://docs.astral.sh/ruff/faq/#how-does-ruffs-formatter-compare-to-black

oh and I just noticed that it is maintianed by Charlie, and comes straight out of astral.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 We Finally Agree On Unit Tests - YouTube

2023-12-22T14:52:25Z

Theo’s response puts a lot of my feelings about unit testing into words. It’s crazy how cargo culty it becomes that the echo chamber of twitter can bring in beliefs that we think we believe, but have not experienced enough or put enough thought in to form our own opinion.

This video made me think so much that it turned into it’s own blog post

Thoughts on Unit Testing

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

thoughts on unit tests

2023-12-22T08:49:07Z

Your browser does not support the audio element.

Theo’s response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements “Everything must be unit tested, tests make code more maintainable.” In reality when we are not writing complex low level code unit tests are probably the wrong approach.

Most of us are assemblers #

So much of software engineering is assembling existing well tested code. Crud applications, UI, Data Pipelines, building on top of battle tested code.

Manufacturing Analogy - Unit Testing #

This kind of reminds me of Manufacturing. Individual components are QA tested with tests that look more like unit test. Parts like bearings, pistons, shafts, valves, they are all tested against sophisticated statistics of sample measurements. This is quite similar to unit testing.

You see measuring the individual sizes of these components does not guarantee the actual function of the component. Before the Henry Ford assembly line parts like this were all different sizes, and they were all custom matched to each other. The one industry that I can thing of that still exists today like this is gunsmithing. parts are made very close to size, but they need some lapping and final finishing for high quality function.

Integration Testing #

So how are pumps, engines, valve assemblies, and so on tested? They are integration testsed based on their function. They are not generally unit tested to measurements of thier size. A pump is tested on a test bench and will be required to output a spec flow, a valve will have air or fluid applied and tested for leaks. An engine is a more complicated case where it will be ran on a suite of cycles and should output a spec level of power, while emissions, temperature, fuel consumption and hundreds of other measurements stay within their specs.

As assemblers we should be thinking of integration testing #

There is 100% a place for unit testing in the software engineering industry, just like manufacturing, but I don’t think it makes sense everywhere. Library development and lower level algorithms look a lot more like parts that can be physically measured. But that’s probably not the code most of us are writing most of the time.

We are likely assembling a lot of code to build a product such as a data pipeline, or a web application for instance. For these cases we should be thinking about the final function of the product. Like Prime says in the video, think of the product as a black box, as if you have no idea how anything works in the box, but you know how it should function.

I’m starting to lean more into simple tools for this qa work on we applications that I build. They are easy to build and maintain, and do not get in the way when the direction of the project changes. If I rip out half my code it would be likely to see 3/4’s of my unit tests failing. Not a fun thing to maintain.

Uptime #

This is the simplest form of integration testing that I am pulling into fokais, and it’s nearly fully automatable without much thought on my behalf. I am building web applications in fastapi, these api’s have endpoints that should be reachable. Tools like Uptime Kuma make it trivial to pull in all of my endpoints and simply ask if they are reachable once per minute.

With uptime tests I can more confidently move through dev to prod. If dev has been running for a number of minutes and alarms are not going off, it’s probably functioning.

Functionality Testing #

I’ve used tools built upon selenium in the past, such as testproject.io. It is a pretty great experience, and makes it pretty easy to define, when a user logs into the dev app, and clicks the dashboard, can they see their name and account information.

These tests are likely to remain true for the life of the project. No matter what changes inside the black box, when a user goes to their account they should see their account information. It might be server rendered with jinja, fetch with htmx, or client side rendered with reactjs, it does not matter.

Load Testing #

The last thing I want to really put as a tool in my toolbelt is load testing. It’s pretty easy to degrade the experience of a webpage with a bad database query or api call. Load testing can ensure that your application can handle the load that you want it to in dev before you release to prod. I am looking into bringing in Locust.io for load testing fokais.com.

Logging and Monitoring #

Inevitably users will use your product in ways that were never intended. It will be nearly impossible to cover everything with any sort of test. This is where logging and monitoring come in. If you can set up error monitoring and alerts, then you can check production logs to see what the heck happend and potentially resolve the issue. Without monitoring, you will have wild bugs that you never thought could exist.

Unit Tests are not a Golden fix all #

No tests will never be perfect, they will not catch everything. The point that I was inspired to make after watching Theo’s video is that unit tests may not the right thing for your product, and then requiring coverage metrics on PR may be more of a detriment to the product than help.

💭 Mastodon.py — Mastodon.py 1.8.1 documentation

2023-12-21T01:59:43Z

mastodonpy.readthedocs.io

Mastadon.py is a python api client for mastadon that makes it easy to cross post to mastadon.

from mastodon import Mastodon

Mastodon.create_app(
    'pytooterapp',
    api_base_url = 'https://mastodon.social',
    to_file = 'pytooter_clientcred.secret'
)

from mastodon import Mastodon

mastodon = Mastodon(client_id = 'pytooter_clientcred.secret',)
mastodon.log_in(
    'my_login_email@example.com',
    'incrediblygoodpassword',
    to_file = 'pytooter_usercred.secret'
)

mastodon.toot('Tooting from Python using #mastodonpy !')

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 logs with FastAPI and Uvicorn · Issue #1508 · tiangolo/fastapi

2023-12-15T22:04:42Z

logs with FastAPI and Uvicorn · Issue #1508 · fastapi/fastapi

Hello, Thanks for FastAPI, easy to use in my Python projects ! However, I have an issue with logs. In my Python project, I use : app = FastAPI() uvicorn.run(app, host="0.0.0.0", port=8000) And when...

Setting an additional log handler to the uvicorn logger for access logs in fastapi was not straightforward, but This post was very helpful.

@app.on_event("startup")
async def startup_event():
    logger = logging.getLogger("uvicorn.access")
    handler = logging.StreamHandler()
    handler.setFormatter(logging.Formatter("%(asctime)s - %(levelname)s - %(message)s"))
    logger.addHandler(handler)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 How to group FastAPI endpoints in Swagger UI?

2023-12-15T15:30:37Z

Show some equivalent list comprehensions in filter examples · Issue #1068 · pallets/jinja

    Setting tags in your fastapi endpoints will group them in the docs.  You can also set some metadata around the tags to get nice descriptions.

Here is a full example from the post.

from fastapi import FastAPI

tags_metadata = [
    {"name": "Get Methods", "description": "One other way around"},
    {"name": "Post Methods", "description": "Keep doing this"},
    {"name": "Delete Methods", "description": "KILL 'EM ALL"},
    {"name": "Put Methods", "description": "Boring"},
]

app = FastAPI(openapi_tags=tags_metadata)


@app.delete("/items", tags=["Delete Methods"])
@app.put("/items", tags=["Put Methods"])
@app.post("/items", tags=["Post Methods"])
@app.get("/items", tags=["Get Methods"])
async def handle_items():
    return

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Show some equivalent list comprehensions in filter examples · ...

2023-12-14T20:28:34Z

I'm willing to write a pull-request for this, but I just want to see what people think before I write it. So the issue is this. I'm very familiar with python. I'm new to Jinja2. Often I find myself...

GitHub - DataDog/ddqa: Datadog's QA manager for releases of GitHub repositories

I often want to reach for non existing list comprehensions in jinja 2, Here are a few nice equivalents.

a: {{ data | selectattr('x', 'gt', 5) | list }}
b: {{ data | map(attribute='c') | list }} 
c: {{ data | selectattr('x', 'gt', 5) | map(attribute='c') | list }}

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 DataDog/ddqa: Datadog's QA manager for releases of GitHub repo...

2023-12-13T22:18:06Z

Datadog's QA manager for releases of GitHub repositories - DataDog/ddqa

Litestar: Effortlessly Build Performant APIs

DataDog ddqa is building out a textual app and deploying it with pyapp. They have CI setup to fully build and cross compile their textual tui into github releases that you can just download from their releases page. This is something I am looking at for markata. This would be pretty sweet to be able to make it just work on places like windows. It would also be interesting to try to build a full desktop app with pyapp.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Stripe Cancellations in FastAPI and SQLModel

2023-12-09T22:29:15Z

Today I am working on fokais.com, trying to get to a point where I can launch by workig through stripe integrations. This is my first time using stripe, so there has been quite a bit to learn, and I am probably building in more than I need to before launching, but I am learning, and not in a rush to launch.

I am building the fokais backent in python primarilyt with fastapi and sqlmodel on sqlite. My billing integration is going to be all Stripe.

Stripe Subscription Cancellations Docs #

Here is a link to the stripe docs for your refrence, especially if you want to see how to cancel subscriptions in other languages. They include code samples for many popular languages.

User Model #

This is the part of the user model that includes the cancel and reactivate methods. It pretty much follows the stripe guide.

class UserBase(SQLModel, table=False):  # type: ignore[call-arg]
    username: str = Field(unique=True)
    full_name: str
    email: str
    email_verified: bool = False
    disabled: bool = False
    signup_date: Optional[datetime] = Field(default_factory=datetime.utcnow)
    stripe_customer_id: Optional[str]

    def cancel_subscription(self):
        for subscription in self.active_subscriptions:
            stripe.Subscription.modify(
                subscription.id,
                cancel_at_period_end=True,
            )
        self.refresh()

    def reactivate_subscription(self):
        for subscription in self.active_subscriptions:
            stripe.Subscription.modify(
                subscription.id,
                cancel_at_period_end=False,
            )
        self.refresh()

Cancellations api #

Here is the cancellations api. I created an are you sure form that I can link to from the accounts page with a normal anchor tag. Note that I am doing a POST request to do the cancellation from a form. I want this to work for any user whether there is js or not. This is an operation that will change the users data, and I want to make sure that it avoids all browser and cdn caching. As a scrappy startup we are running light on infrastructure and are caching hard at the CDN to avoid excessive server hits.

Note

 I am doing a `POST` request to do the cancellation from a form.

@pricing_router.get("/cancel")
@pricing_router.get("/cancel/")
def get_cancel(
    request: Request,
    current_user: Annotated[User, Depends(get_current_user_if_logged_in)],
):
    return config.templates.TemplateResponse(
        "cancel.html",
        {
            "request": request,
            "prices": products.prices,
            "products": products.products,
            "current_user": current_user,
        },
    )


@pricing_router.post("/cancel")
@pricing_router.post("/cancel/")
def post_cancel(
    request: Request,
    current_user: Annotated[User, Depends(get_current_user_if_logged_in)],
):
    current_user.cancel_subscription()
    return HTMLResponse('<p id="cancel" hx-swap-oob="outerHTML">Your Subscription has been Cancelled</p>')

Reactivations #

Reactivating accounts looks just about the same as cancelling, only flippng True to False.


@pricing_router.get("/reactivate")
@pricing_router.get("/reactivate/")
def get_reactivate(
    request: Request,
    current_user: Annotated[User, Depends(get_current_user_if_logged_in)],
):
    return config.templates.TemplateResponse(
        "reactivate.html",
        {
            "request": request,
            "prices": products.prices,
            "products": products.products,
            "current_user": current_user,
        },
    )


@pricing_router.post("/reactivate")
@pricing_router.post("/reactivate/")
def post_reactivate(
    request: Request,
    current_user: Annotated[User, Depends(get_current_user_if_logged_in)],
):
    current_user.reactivate_subscription()
    return HTMLResponse('<p id="reactivate" hx-swap-oob="outerHTML">Your Subscription has been reactivated</p>')

Full User Model #

This is the full user model, completely subject to change in the future, but it includes the cancel and reactivate methods.

class UserBase(SQLModel, table=False):  # type: ignore[call-arg]
    username: str = Field(unique=True)
    full_name: str
    email: str
    email_verified: bool = False
    disabled: bool = False
    signup_date: Optional[datetime] = Field(default_factory=datetime.utcnow)
    stripe_customer_id: Optional[str]

    @property
    def session(self):
        return next(get_session())

    @classmethod
    def get_by_id(cls, id):
        return next(get_session()).get(cls, id)

    def refresh(self):
        cache.set(f"active_subscriptions_{self.id}", None, 3600)
        cache.set(f"active_products_{self.id}", None, 3600)

    def get_checkout_sessions(self):
        return [
            stripe.checkout.Session.retrieve(s.stripe_checkout_session_id)
            for s in self.session.exec(select(CheckoutSession).where(CheckoutSession.user_id == self.id)).all()
            if s.stripe_checkout_session_id is not None
        ]

    def get_active_subscriptions(self):
        subscriptions = [
            s.subscription
            for s in [
                stripe.checkout.Session.retrieve(s.stripe_checkout_session_id)
                for s in self.session.exec(select(CheckoutSession).where(CheckoutSession.user_id == self.id)).all()
                if s.stripe_checkout_session_id is not None
            ]
            if s.status == "complete"
        ]
        active_subscriptions = [stripe.Subscription.retrieve(subscription) for subscription in subscriptions]
        return active_subscriptions

    def has_active_subscription(self):
        return len(self.active_subscriptions) > 0

    @property
    def active_subscriptions(self):
        active_subscriptions = cache.get(f"active_subscriptions_{self.id}")
        if active_subscriptions is not None:
            return active_subscriptions
        active_subscriptions = self.get_active_subscriptions()
        cache.set(f"active_subscriptions_{self.id}", active_subscriptions, 3600)

        return active_subscriptions

    @property
    def active_plans(self):
        subscriptions = self.active_subscriptions
        plans = [subscription.plan for subscription in subscriptions]
        return plans

    @property
    def subscription_to_plan(self):
        subscriptions = self.active_subscriptions
        plans = {subscription.id: subscription.plan.id for subscription in subscriptions}
        return plans

    @property
    def plan_to_subscription(self):
        plans = {v: k for k, v in self.subscription_to_plan.items()}

        return plans

    def get_active_products(self):
        plans = self.active_plans
        products = [stripe.Product.retrieve(plan.product) for plan in plans]
        return products

    @property
    def plan_to_product(self):
        plans = self.active_plans
        products = {plan.id: stripe.Product.retrieve(plan.product).id for plan in plans}
        return products

    @property
    def prodct_to_plan(self):
        plans = self.active_plans
        products = {stripe.Product.retrieve(plan.product).id: plan.id for plan in plans}
        return products

    @property
    def active_products(self):
        products = cache.get(f"active_products_{self.id}")
        if products is not None:
            return products
        products = self.get_active_products()
        cache.set(f"active_products_{self.id}", products, 3600)

        return products

    @property
    def best_active_subscription(self):
        subscriptions = self.active_subscriptions
        return subscriptions[0]

    @property
    def best_active_product(self):
        products = self.active_products
        products.sort(key=lambda p: p.metadata.get('level', 0))
        return products[0]

    @property
    def best_active_subscription(self):
        subscription_id = self.plan_to_subscription[self.prodct_to_plan[self.best_active_product.id]]
        return stripe.Subscription.retrieve(subscription_id)

    @property
    def config(self):
        product = self.best_active_product
        return product.metadata

    def subscription_status(self):
        subscriptions = self.active_subscriptions()

    def cancel_subscription(self):
        for subscription in self.active_subscriptions:
            stripe.Subscription.modify(
                subscription.id,
                cancel_at_period_end=True,
            )
        self.refresh()

    def reactivate_subscription(self):
        for subscription in self.active_subscriptions:
            stripe.Subscription.modify(
                subscription.id,
                cancel_at_period_end=False,
            )
        self.refresh()

Looking for a Heroku replacement, What I found was shocking!

2023-12-04T20:29:46Z

Your browser does not support the audio element.

I’ve long hosted my personal blog as a static site on waylonwalker.com. It’s all markdown, converted to html, and shipped as is. It’s been great, I’ve moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it’s just pointing ci to a different host after the site has built.

What about server side #

Now the part that I have struggled with is how to cheaply host a server rendered application that can just live on forever without me paying for it. This is a harder problem as it costs more to keep servers spinning, memory, and disk all ready for you to use at a moments notice.

Honestly #

I never really deployed anything that useful on heroku, but it seems like the klenex of the bunch that’s why they are in the title. I’ve moved between digital ocean and fly.io, and have had some great experiences with both. I just don’t want to build something that is going perpetually cost money, I’m cheap and dont want to feel the burden of paying for something that I might not be using all the time.

fly.io #

fly is absolutely amazing to get off the ground. Their cli is top notch. They have servers all over the place and have a great interface to get your application deployed close to your users. You just can’t run all that much on it before you end up off the free tier.

serverless #

These platforms so far have just not felt right for me, again, I have to pay for them, many of them I have to worry about how hard they would get hit, and often they are focused on javascript.

What I am looking for #

easy to deploy
cheap to host
free to use
generous free tier
no chance of big bills
reliable, not 9 9’s, but mostly up.
allows me to run a bunch of stuff that I lightly use
that buttery smooth cli that I get out of fly.io
runs containers

The last one is probably the most important. I don’t get tons of time to work on side projects, and they are mostly ideas I just want to get out there and play with, the idea may flop quickly, but I want that DX to be easy to go from nothing to publically deployed.

What about going back to self hosting?? #

I have a Gateway FX6860 built in 2011, and was gifted to me around 2016. She doesn’t have much power to her, but she does have more than free tier power. I have 8 dedicated cpu, 16GB RAM, 500GB boot disk, and a 4TB hard disk.

Cloudflare tunnels #

Cloudflare Tunnels allow me to route traffic to my own server sitting right next to me and have it publically accessible. They take care of ssl certs, they support wildcard routes, and configuration is pretty easy. I can map addresses to a port on the machine, wildcard to a reverse_proxy, and end with a 404.

I’m missing that buttery smooth cli #

At this point I’m hosting a few things that I use frequently, but I don’t have that buttery smooth deploy experience that I get from fly.io. I have to log into the machine, edit some files, and docker compose up every time I want to deploy.

I was certain there was a better way #

I knew there was something better out there that was not a complex pain in the ass to setup. I know small companies run their own infra with a small team. There has to be tools that don’t take an enterprise to manage.

What about kubernetes #

In my search I keep seeing kubernetes as the solution, just run k8s, k3s, k0s, minikube, or kind. But EVERYTHING I have heard about kubernetes is that its a pain in the ass to deploy, takes a team to manage, and if you don’t have a $1M problem to solve don’t bother cause k8s will create a $1M problem for you. For this reason alone I turned my nose up to k8’s and nearly didn’t even give it a chance. Everyone jokes about blogs and simple sites running on k8s, and how rediculous that must be.

If you don’t have a $1M problem to solve don’t bother cause k8s will create a $1M problem for you.

Let’s jump on k3s #

I’ve never ran kubernetes myself, but after seeing it so many times in my searches for a fly.io replacement, I decided to give it a shot. I chose k3s as it seems like a nice balance of easy to setup, maintain, and feature complete kubernetes service.

# install and start k3s
curl -sfL https://get.k3s.io | sh -
# check to see if your nodes are started
sudo kubectl get nodes

My main hiccup so far was the machine I am running on runs a fairly new ubuntu install with zfs on root, and it would not start the master node. Rather than figuring out how to make zfs play nice I just pointed k3s to a drive that is not zfs.

# manuallly
sudo k3s server -d /mnt/vault/.rancher/k3s
# without editing systemd service
sudo ln -s /mnt/vault/.rancher/k3s /var/lib/rancher/k3s

Next I needed to be able to completely manage my k3s cluster form my main machine while this one sits far away in a closet.

# from the server
sudo cp /etc/rancher/k3s/k3s.yaml ~/.config/kube

# from my local machine
scp falcon@falcon:~/.config/kube/k3s.yaml ~/.config/kube/falcon-k3s.yaml
sudo cp /etc/rancher/k3s/k3s.yaml /home/waylon/.config/kube
sudo chown -R waylon:waylon ~/.config/kube
export KUBECONFIG=~/.config/kube/k3s.yaml

kompose #

Since everything I was running prior was in docker compose, I found kompose.io to be a fantastic tool to help me start converting my docker deployments into kubernetes.

A Month Later #

I started this post a month ago, and I am still enjoying k3s. For clarity, I did have a bit of k8’s experience going in, but zero experience running it by myself. k3’s seems to have made it pretty straightforward so far. My worst issues have been with ingress. Docker registries were a bit of a pain due to their large blob sizes, and a service I wanted to try to self host (sshx) required grpc, which is not supported by cloudflare tunnels.

TLDR #

Don’t believe everyone’s pre-conceived notions about tech you have never tried. Most of these things come from the echo chamber that is twitter anyways. Create your own opinions by trying new things, learning for yourself, and forming your own opnions.

Hosted Platform Brainstorm

2023-11-15T08:52:44Z

shots
- shots templates
- gradient backgrounds
- premium feature upload your own background
ogs
- scrape og info, generate og image
- og’s build on shots, they are just hosted template endpoints that get a screenshots
- potentially auto approve og’s from rss
drops
- drop a file to share
- limited size, limited upload MB’s / month
archives
- deep permalink a website
- https://pypi.org/project/pywebcopy/
og-links
ai-covers
- give me an url, and a selector, I will read your article and give you an ai generated cover image
ai-summaries
- give me an url, and a selector, I will read your article and give you an ai generated summary
ai-voice - voices
- give me an url, and a selector, I will read your article and give you an ai generated voice
image-optimization
- use the image optimization pipelie for shots to give you an optimized image
blur ups

Community #

host it yourself

reader
markata

Limits #

cloudflare file size 100mb https://developers.cloudflare.com/cache/concepts/default-cache-behavior/

Free Tier #

100 shots
- 100kb
- 1080p
- watermark
100 drops
- 100kb
10 voice
- 4096 characters
- preroll add
10 summaries
- 4096 characters
10 covers
- 4096 characters

Starter Tier $10 #

500 shots
- 500kb
- 1080p
500 drops
- 500kb

Hobby Tier $25 #

500 shots
- 500kb
- 1080p
500 drops
- 500kb
100 voice
- 4096x10 characters
100 summaries
- 4096x10 characters
100 covers
- 4096x10 characters

Pro Tier $50 #

1000 shots
- 1000kb
- 1080p
1000 drops
- 1000kb
500 voice
- 4096x50 characters
500 summaries
- 4096x50 characters
500 covers
- 4096x50 characters

existing solutions #

og-images, similar to shots https://ogimage.org/templates
- $49/mo $99 setupfee

Roadmap to prod #

Shots #

Watermarks

💭 Episode #433 Litestar: Effortlessly Build Performant APIs - [T...

2023-10-17T18:45:08Z

We all know about Flask and Django. And of course FastAPI made a huge splash when it came on the scene a few years ago. But new web frameworks are being created all the time. And they have these ea...

talkpython.fm

Litestar is an interesting api framework similar to fastpi, that I am interested to check out to see if it fits into some project scope. It sounds like it comes with a lot more batteries included for things like auth, but does not have hard opinions like django. At this point I’m not jumping off of fastapi, but its something I want to try.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

how to host static content with fastapi

2023-10-04T08:32:58Z

I wanted to host some static files through fastapi. Typical use cases for this might be some static web content like html/css/js. It could also be images or some data that doesn’t need dynamically rendered.

From the Docs #

The docs cover how to host static files, and give this solution that is built into fastapi.

https://fastapi.tiangolo.com/tutorial/static-files/

from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

app = FastAPI()

app.mount("/static", StaticFiles(directory="static"), name="static")

Authenticated Static Files #

Thanks to #858.

OscartGiles posted this solution to add authentication to static files. I tried this out on my thoughts and it worked flawlessly.

import typing
from pathlib import Path
import secrets

from fastapi import FastAPI, Request, HTTPException, status
from fastapi.staticfiles import StaticFiles
from fastapi.security import HTTPBasic, HTTPBasicCredentials


PathLike = typing.Union[str, "os.PathLike[str]"]
app = FastAPI()
security = HTTPBasic()


async def verify_username(request: Request) -> HTTPBasicCredentials:

    credentials = await security(request)

    correct_username = secrets.compare_digest(credentials.username, "user")
    correct_password = secrets.compare_digest(credentials.password, "password")
    if not (correct_username and correct_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect email or password",
            headers={"WWW-Authenticate": "Basic"},
        )
    return credentials.username


class AuthStaticFiles(StaticFiles):
    def __init__(self, *args, **kwargs) -> None:

        super().__init__(*args, **kwargs)

    async def __call__(self, scope, receive, send) -> None:

        assert scope["type"] == "http"

        request = Request(scope, receive)
        await verify_username(request)
        await super().__call__(scope, receive, send)


app.mount(
    "/static",
    AuthStaticFiles(directory=Path(__file__).parent / "static"),
    name="static",
)

If you want both then, all you have to do is mount AuthStaticFiles to a different route. Now you can have private, or paid content behind /restricted.

app.mount("/static", StaticFiles(directory="static"), name="static")
app.mount(
    "/restricted",
    AuthStaticFiles(directory=Path(__file__).parent / "restricted"),
    name="restricted"
)

Setup Tailwind for Jinja

2023-09-17T17:10:32Z

I’ve recently given tailwindcss a second chance and am really liking it. Here is how I set it up for my python based projects.

https://waylonwalker.com/a-case-for-tailwindcss

Installation #

npm is used to install the cli that you will need to configure and compile tailwindcss.

npm install -g tailwindcss-cli

Setup #

You will need to create a tailwind.config.js file, to get this you can use the cli.

npx tailwindcss init

Using tailwind with jinja templates #

To set up tailwind to work with jinja templates you will need to point the tailwind config content to your jinja templates directory.

module.exports = {
  content: ["templates/**/*.html"],
};

Setting up the base styles #

I like to use the @tailwind base;, to do this I set up an input.css file.

@tailwind base;
@tailwind components;
@tailwind utilities;

Compiling #

Now that it’s all setup you can run the tailwindcss command. You will get an output.css with base tailwind plus any of the classes that you used.

tailwindcss -i ./input.css -o ./output.css --watch

💭 Bigger Applications - Multiple Files - FastAPI

2023-08-24T14:51:23Z

Bigger Applications - Multiple Files - FastAPI

FastAPI framework, high performance, easy to learn, fast to code, ready for production

sqlite_utils Python library - sqlite-utils

Fastapi lets you tag your APIRouter’s so that the swagger docs are grouped according to the router.

router = APIRouter(tags=['router'])

Now all routes in router will appear in the router group in the swagger docs.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite_utils Python library - sqlite-utils

2023-08-20T14:32:39Z

sqlite-utils.datasette.io

sqlite-utils is primarily a cli tool for sqlite operations such as enabling full text search, and executing searches, but it also has a nice python api that is exposed and pretty straightforward to use.

from sqlite_utils import Database
db = Database("database.db")
db["post"].enable_fts(["title", "message", "tags])
db["post"].search("water")

This returns a generator object that you can iterate over the row objects with.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 simonw/datasette-render-markdown: Datasette plugin for renderi...

2023-08-16T00:48:07Z

GitHub - simonw/datasette-render-markdown: Datasette plugin for rendering Markdown

Datasette plugin for rendering Markdown. Contribute to simonw/datasette-render-markdown development by creating an account on GitHub.

GitHub - simonw/shot-scraper: A command-line utility for taking automated screenshots of websites

datasette really does everything doesn’t it!

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 simonw/shot-scraper: A command-line utility for taking automat...

2023-08-16T00:05:33Z

A command-line utility for taking automated screenshots of websites - simonw/shot-scraper

shot-scraper: automated screenshots for documentation, built on Playwright

    > A command-line utility for taking automated screenshots of websites

Daaaang, this is such an elegantly simple way to get web screenshots with a cli. I was literally up and running with two commands on my arch linux machine (which it warned was unsupported by playwright).

pip install shot-scraper
# Now install the browser it needs:
shot-scraper install
shot-scraper waylonwalker.com
shot-scraper https://datasette.io/ 
shot-scraper https://datasette.io/ -h 1280 -w 1920
shot-scraper https://datasette.io/ -h 480 -w 720
shot-scraper shot --selector '#posts' https://thoughts.waylonwalker.com/post/89

Note shot-scraper https://datasette.io/ is a full length screenshot of the entire page.

Oh and its pretty dang fast, let alone the setup time, this crushes on startup time in my attempts to use a headless browser in the past.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 shot-scraper: automated screenshots for documentation, built o...

2023-08-15T23:55:17Z

shot-scraper is a new tool that I’ve built to help automate the process of keeping screenshots up-to-date in my documentation. It also doubles as a scraping tool—hence the name—which I …

Create Models with a Many-to-Many Link - SQLModel

An interesting way to build automatically annotaatd docs with arrows pointing to elements on a webpage.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Create Models with a Many-to-Many Link - SQLModel

2023-08-09T13:37:15Z

SQLModel, SQL databases in Python, designed for simplicity, compatibility, and robustness.

sqlmodel.tiangolo.com

Creating many to many relationships with sqlmodel requires a LinkTable Model. The link model will keep track of the linked id’s between each of the models.

from typing import List, Optional

from sqlmodel import Field, Relationship, Session, SQLModel, create_engine


class HeroTeamLink(SQLModel, table=True):
    team_id: Optional[int] = Field(
        default=None, foreign_key="team.id", primary_key=True
    )
    hero_id: Optional[int] = Field(
        default=None, foreign_key="hero.id", primary_key=True
    )


class Team(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str = Field(index=True)
    headquarters: str

    heroes: List["Hero"] = Relationship(back_populates="teams", link_model=HeroTeamLink)


class Hero(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str = Field(index=True)
    secret_name: str
    age: Optional[int] = Field(default=None, index=True)

    teams: List[Team] = Relationship(back_populates="heroes", link_model=HeroTeamLink)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 python - How to use a Pydantic model with Form data in FastAPI...

2023-08-08T13:46:19Z

`ValueError: Constraint must have a name` in alembic 1.10.0 · Issue #1195 · sqlalchemy/alembic

I went down the route of leveraging the json-enc extention in htmx, but later realized that this completely breaks browsers/users who do not wish to use javascript. While most of the web would feel quite broken with javascript disabled, I don’t want to contribute to that without good reason.

Taking a second look into this issue, rather than using json-enc, and using as_form to get form data into a model keeps the nice DX fo everything being a pydantic model, but the site still works without js. with js htmx kicks in, you get a spa like experience by loading partials onto the page, and without, you just get a full page reload.

the implementation #

copied from https://stackoverflow.com/questions/60127234/how-to-use-a-pydantic-model-with-form-data-in-fastapi

import inspect
from typing import Type

from fastapi import Form
from pydantic import BaseModel
from pydantic.fields import ModelField

def as_form(cls: Type[BaseModel]):
    new_parameters = []

    for field_name, model_field in cls.__fields__.items():
        model_field: ModelField  # type: ignore

        new_parameters.append(
             inspect.Parameter(
                 model_field.alias,
                 inspect.Parameter.POSITIONAL_ONLY,
                 default=Form(...) if model_field.required else Form(model_field.default),
                 annotation=model_field.outer_type_,
             )
         )

    async def as_form_func(**data):
        return cls(**data)

    sig = inspect.signature(as_form_func)
    sig = sig.replace(parameters=new_parameters)
    as_form_func.__signature__ = sig  # type: ignore
    setattr(cls, 'as_form', as_form_func)
    return cls

And the usage looks like

@as_form
class Test(BaseModel):
    param: str
    a: int = 1
    b: str = '2342'
    c: bool = False
    d: Optional[float] = None


@router.post('/me', response_model=Test)
async def me(request: Request, form: Test = Depends(Test.as_form)):
    return form

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Debugging | pywebview

2023-08-05T18:04:04Z

How to enable debug mode in pywebview.

import webview

webview.create_window('Woah dude!', 'https://pywebview.flowrl.com/hello')
webview.start(debug=True)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 `ValueError: Constraint must have a name` in alembic 1.10.0 · ...

2023-08-05T01:25:07Z

Describe the bug ValueError: Constraint must have a name in alembic 1.10.0. Expected behavior Migration succeeds. To Reproduce Please try to provide a Minimal, Complete, and Verifiable example, wit...

Read a Range of Data - LIMIT and OFFSET - SQLModel

After a nasty time with alembic upgrades, thoughts is about to get a new users table. This may have came from incorrectly setting up alembic for sqlite from the start, but I was able to fix the issue with this GitHub issue.

alembic sqlite ValueError: Constraint must have a name

The change I needed to make to get my migration to run.


        + batch_op.create_foreign_key('fk_post_author_id_user', 'user', ['author_id'], ['id'])

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Use Alembic Check to check for possible upgrades

2023-08-05T01:22:06Z

![[None]]

Since using alembic I have been just running out a new revision checking its content and deleting it if its empty, today I learned there is an alembic check command to check for operations that need to be created.

❯ alembic check
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
No new upgrade operations detected.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 API — Jinja Documentation

2023-08-04T23:54:05Z

#jinja2.Environment.compile_templates" class="embed-card-link" target="_blank" rel="noopener noreferrer">

API — Jinja Documentation (3.1.x)

jinja.palletsprojects.com

🤯 jinja comes with a loader to pre-compile templates! Defihnitely need to look at this for markata, as jinja is till one of the biggest hot spots.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 API — Jinja Documentation

2023-08-04T23:35:09Z

![https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.FileSystemLoader](/static/https://jinja.palletsprojects.com/en/3.0.x/api/#jinja2.FileSystemLoader)

I’ve definitely been missing out on setting up a proper jinja loader on a few projects, I need to lean on this a bit more.

class jinja2.FileSystemLoader(searchpath, encoding='utf-8', followlinks=False):
    '''
    Load templates from a directory in the file system.
    '''

The path can be relative or absolute. Relative paths are relative to the current working directory.

loader = FileSystemLoader("templates")
# A list of paths can be given. The directories will be searched in order, stopping at the first matching template.
loader = FileSystemLoader(["/override/templates", "/default/templates"])

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Read a Range of Data - LIMIT and OFFSET - SQLModel

2023-08-01T00:10:15Z

SQLModel, SQL databases in Python, designed for simplicity, compatibility, and robustness.

sqlmodel.tiangolo.com

Implement paging in sqlmodel with where, limit, and offset.

def select_heroes():
    with Session(engine) as session:
        statement = select(Hero).where(Hero.age > 32).limit(3)
        results = session.exec(statement)
        heroes = results.all()
        print(heroes)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Center things - Textual

2023-07-30T16:31:55Z

Textual - Center things

Textual is a TUI framework for Python, inspired by modern web development.

Textual Documentation · textual.textualize.io

How to center things in textual. Textual has a very unique way of styling text user interfaces for the terminal using css. If you know css it feels natural.

@willmcgugan, has put together a great article on how to center things in textual

here the final result

from textual.app import App, ComposeResult
from textual.widgets import Static

QUOTE = "Could not find you in Seattle and no terminal is in operation at your classified address."


class CenterApp(App):
    """How to center things."""

    CSS = """
    Screen {
        align: center middle;
    }

    #hello {
        background: blue 50%;
        border: wide white;
        width: 40;
        height: 9;
        text-align: center;
        content-align: center middle;
    }
    """

    def compose(self) -> ComposeResult:
        yield Static(QUOTE, id="hello")


if __name__ == "__main__":
    app = CenterApp()
    app.run()

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 s3-tree · PyPI

2023-07-30T14:37:06Z

Client Challenge

pypi.org

Super useful way to show a tree view of an s3 bucket’s structure!

pip install s3-tree

s3-tree bucketname

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 python - SQLAlchemy ORDER BY DESCENDING? - Stack Overflow

2023-07-29T22:30:25Z

Textual - Using Rich Inspect to interrogate Python objects

How to sort results from a sqlalchemy based orm.

.order_by(model.Entry.amount.desc())

I needed this to enable paging on my thoughts api.

@post_router.get("/posts/")
async def get_posts(
    *,
    request: Request,
    session: Session = Depends(get_session),
    hx_request: Annotated[str | None, Header()] = None,
    accept: Annotated[str | None, Header()] = None,
    current_user: Annotated[User, Depends(try_get_current_active_user)],
    page_size: int = 10,
    page: int = 1,
) -> Posts:
    "get all posts"
    statement = (
        select(Post)
        .where(Post.published)
        .order_by(Post.id.desc())
        .limit(page_size)
        .offset((page - 1) * page_size)
    )
    posts = session.exec(statement).all()
    posts = Posts(__root__=posts)

    if isinstance(current_user, RedirectResponse):
        is_logged_in = False
    else:
        is_logged_in = True

    if hx_request and page == 1 and len(posts.__root__) == 0:
        return HTMLResponse('<ul id="posts"><li>No posts</li></ul>')
    if hx_request and len(posts.__root__) == 0:
        return HTMLResponse("")
    if not hx_request and len(posts.__root__) == 0:
        return ["no posts"]
    if hx_request:
        return templates.TemplateResponse(
            "posts.html",
            {
                "request": request,
                "config": config,
                "posts": posts,
                "md": md,
                "is_logged_in": is_logged_in,
                "page": page,
            },
        )

    if accept.startswith("text/html"):
        return templates.TemplateResponse(
            "base.html",
            {
                "request": request,
                "config": config,
                "posts": posts,
                "md": md,
                "is_logged_in": is_logged_in,
                "page": page,
            },
        )

    return posts

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Using Rich Inspect to interrogate Python objects - Textual

2023-07-29T00:58:00Z

Textual is a TUI framework for Python, inspired by modern web development.

Textual Documentation · textual.textualize.io

I love rich inspect. It’s one of my most often used features of rich. It gives you a great human readable insight into python object instances.

>>> from rich import inspect
>>> text_file = open("foo.txt", "w")
>>> inspect(text_file)

I have a pyflyby entry for it so that I can just run it ang get automatic imports. To not clash with the standard library inspect, which is quite useful on it’s own, I have aliased it to rinspect.

from rich import inspect as rinspect

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Flask Trailing Slashes 404

2023-07-28T14:59:37Z

In flask apps I often get a 404 for routes with a trailing slash. This Stack Overflow post shows how to configure flask to allow trailing slashes on some or all routes.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 htmx ~ The json-enc Extension

2023-07-28T14:59:37Z

Header Parameters - FastAPI

htmx.org

json-enc extension converts url encoded form values into json encoded data, this is very useful for fastapi to have the same interface for htmx and curl type of interfaces.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Header Parameters - FastAPI

2023-07-28T14:59:37Z

FastAPI framework, high performance, easy to learn, fast to code, ready for production

GitHub - 1j01/textual-paint: :art: MS Paint in your terminal.

Getting request headers in fastapi has a pretty nice stetup, it allows you to get headers values as function arguments,

I was able to use headers to detect if a request was made from htmx or not.

If the request was made from htmx, then we want a html format, otherwise I’m probably hitting the api programatically from something like curl or python

@post_router.post("/post/")
async def post_post(
    request: Request,
    post: PostCreate,
    current_user: Annotated[User, Depends(try_get_current_active_user)],
    session: Session = Depends(get_session),
    is_hx_request: Annotated[str | None, Header()] = None,
) -> PostRead:
    "create a post"
    print('hx_request', hx_request)
    db_post = Post.from_orm(post)
    session.add(db_post)
    session.commit()
    session.refresh(db_post)
    if is_hx_request:
        return templates.TemplateResponse("post_item.html", {"request": request, "config": config, "post": db_post})
    return db_post

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 1j01/textual-paint: :art: MS Paint... in your terminal.

2023-07-28T14:59:37Z

:art: MS Paint in your terminal. Contribute to 1j01/textual-paint development by creating an account on GitHub.

sqlite-utils now supports plugins

1j01 created a complete working clone of ms paint in the terminal using the textual framework. It’s incredible.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 sqlite-utils now supports plugins

2023-07-28T14:59:37Z

sqlite-utils 3.34 is out with a major new feature: support for plugins. sqlite-utils is my combination Python library and command-line tool for manipulating SQLite databases. It recently celebrated...

Column INSERT/UPDATE Defaults — SQLAlchemy 1.4 Documentation

As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented this completely with entrypoints, no framework or library at all.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 pywebview

2023-07-28T14:59:37Z

I am creating this post from a desktop app that I created in 3 lines.

import webview

webview.create_window('Woah dude!', 'https://thoughts.waylonwalker.com')
webview.start()

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Column INSERT/UPDATE Defaults — SQLAlchemy 1.4 Documentation

2023-07-28T14:59:37Z

docs.sqlalchemy.org

sqlalchemy server_defaults end up as defaults in the database when new values are inserted.

t = Table(
    "test",
    metadata_obj,
    Column("abc", String(20), server_default="abc"),
    Column("created_at", DateTime, server_default=func.sysdate()),
    Column("index_value", Integer, server_default=text("0")),
)

CREATE TABLE test (
    abc varchar(20) default 'abc',
    created_at datetime default sysdate,
    index_value integer default 0
)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Template Designer Documentation — Jinja Documentation

2023-07-28T14:59:37Z

Template Designer Documentation — Jinja Documentation (3.1.x)

jinja.palletsprojects.com

A feature of jinja that I just discovered is including sub templates. Here is an example from the docs.

{% include 'header.html' %}
Body goes here.
{% include 'footer.html' %}

And inside of my thoughts project I used it to render posts.

<ul id='posts'>
    {% for post in posts.__root__ %}
    {% include 'post_item.html' %}
    {% endfor %}
</ul>

note that post_item.html automatically inherits the post variable.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Templates - FastAPI

2023-07-28T14:59:37Z

Templates - FastAPI

FastAPI framework, high performance, easy to learn, fast to code, ready for production

FastAPI framework, high performance, easy to learn, fast to code, ready for production

A guide to add Jinja2Templates to fastapi.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Static Files - FastAPI

2023-07-28T14:59:37Z

Static Files - FastAPI

First-class session support in FastAPI · Issue #754 · fastapi/fastapi

Mounting static files in fastapi.

from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

app = FastAPI()

app.mount("/static", StaticFiles(directory="static"), name="static")

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 First-class session support in FastAPI · Issue #754 · tiangolo...

2023-07-28T14:59:37Z

Is your feature request related to a problem All of the security schemas currently supported by FastAPI rely on some sort of "client-server synergy" , where, for instance, the client is expected to...

Here is a snippet provided by @tiangolo to store the users jwt inside of a session cookie in fatapi. This was written in feb 12, 2020 and admits that this is not a well documented part of fastapi.

It’s already in place. More or less like the rest of the security tools. And it’s compatible with the rest of the parts, integrated with OpenAPI (as possible), but probably most importantly, with dependencies.

It’s just not properly documented yet. 😞

But still, it works 🚀 e.g.

from fastapi import FastAPI, Form, HTTPException, Depends
from fastapi.security import APIKeyCookie
from starlette.responses import Response, HTMLResponse
from starlette import status
from jose import jwt


app = FastAPI()

cookie_sec = APIKeyCookie(name="session")

secret_key = "someactualsecret"

users = {"dmontagu": {"password": "secret1"}, "tiangolo": {"password": "secret2"}}


def get_current_user(session: str = Depends(cookie_sec)):
    try:
        payload = jwt.decode(session, secret_key)
        user = users[payload["sub"]]
        return user
    except Exception:
        raise HTTPException(
            status_code=status.HTTP_403_FORBIDDEN, detail="Invalid authentication"
        )


@app.get("/login")
def login_page():
    return HTMLResponse(
        """
        <form action="/login" method="post">
        Username: <input type="text" name="username" required>
        <br>
        Password: <input type="password" name="password" required>
        <input type="submit" value="Login">
        </form>
        """
    )


@app.post("/login")
def login(response: Response, username: str = Form(...), password: str = Form(...)):
    if username not in users:
        raise HTTPException(
            status_code=status.HTTP_403_FORBIDDEN, detail="Invalid user or password"
        )
    db_password = users[username]["password"]
    if not password == db_password:
        raise HTTPException(
            status_code=status.HTTP_403_FORBIDDEN, detail="Invalid user or password"
        )
    token = jwt.encode({"sub": username}, secret_key)
    response.set_cookie("session", token)
    return {"ok": True}


@app.get("/private")
def read_private(username: str = Depends(get_current_user)):
    return {"username": username, "private": "get some private data"}

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Harlequin SQL IDE - DuckDB

2023-07-28T14:59:37Z

unittest subTest() support and subtests fixture

duckdb.org

Harlequin is a pretty sweet example of what textual can be used to create. Its a terminal based sql ide for DuckDB.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Python API - DuckDB

2023-07-28T14:59:37Z

To persist data in duckdb you need to first make a connection to a duck db database.

con = duckdb.connect('file.db')

Then work off of the connection con rather than duckdb.

con.sql('CREATE TABLE test(i INTEGER)')
con.sql('INSERT INTO test VALUES (42)')
# query the table
con.table('test').show()
# explicitly close the connection
con.close()

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 SQL on Pandas - DuckDB

2023-07-28T14:59:37Z

Redirecting…

duckdb.org

duckdb can just query any pandas dataframe that is in memory.

I tried running it against a list of objects and got this error. Great error message that gives me supported types right in the message.

Make sure that "posts" is either a pandas.DataFrame, duckdb.DuckDBPyRelation, pyarrow Table, Dataset, RecordBatchReader, Scanner, or NumPy ndarrays with supported format

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 pytest-subtests · PyPI

2023-07-28T14:59:37Z

pytest-subtests

GitHub - doyensec/wsrepl: WebSocket REPL for pentesters

pytest-subtests is a package to register multiple subtests within a similar test function.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 python script is not found

2023-07-28T14:59:37Z

![[None]]

When setting up a new machine, vm, docker image you might be installing command line tools from places like pip. They will often put executables in your ~/.local/bin directory, but by default your shell is not looking in that directory for commands.

  WARNING: The script dotenv is installed in '/home/falcon/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.

To solve this you need to add that directory to your $PATH.

export PATH=$PATH:~/.local/bin

To make this change permanant add this line to your shell’s init script, which is likely something like ~/.bashrc or ~/.zshrc.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 doyensec/wsrepl: WebSocket REPL for pentesters

2023-07-28T14:59:37Z

WebSocket REPL for pentesters. Contribute to doyensec/wsrepl development by creating an account on GitHub.

Very inspiring textual project to check out how they set up the ui. Their intro video has a pretty epic dev experience.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Doyensec on Twitter: "Announcing wsrepl, the WebSocket testing...

2023-07-28T14:59:37Z

X (formerly Twitter) · twitter.com

wsrepl is an epic websocket repl built in python on the textual framework.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Filter Data - WHERE - SQLModel

2023-07-28T14:59:37Z

Filter Data - WHERE - SQLModel

SQLModel, SQL databases in Python, designed for simplicity, compatibility, and robustness.

sqlmodel.tiangolo.com

When fetching pydantic models from the database with sqlmodel, and you cannot select your item by id, you probably need to use a where clause. This is the sqlmodel way of doing it.

Here is a snippet of how I am using sqlmodel select and where to find a post by link in my thoughts database.

@post_router.get("/link/")
async def get_post_by_link(
    *,
    session: Session = Depends(get_session),
    link: str,
) -> PostRead:
    "get one post by link"
    link = urllib.parse.unquote(link)
    print(f'link: {link}')
    post = session.exec(select(Post).where(Post.link==link)).first()
    if not post:
        raise HTTPException(status_code=404, detail=f"Post not found for link: {link}")

    return post

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 URL Decoding query strings or form parameters in Python | URLD...

2023-07-28T14:59:37Z

URL Decoding query strings or form parameters in Python | URLDecoder

URL Decode online. URLDecoder is a simple and easy to use online tool for decoding URL components. Get started by typing or pasting a URL encoded string in the input text area, the tool will automa...

urldecoder.io

In order to turn url encoded links back into links that I would find in the database of my thoughts project I need to urldecode them when they hit the api. When anything hits the api it must urlencode the links in order for them to be sent correctly as data and not get parsed as part of the url.

Here is a snippet of how I am using urlib.parse.unquote to un-encode encoded urls so that I can fetch posts from the database.

@post_router.get("/link/")
async def get_post_by_link(
    *,
    session: Session = Depends(get_session),
    link: str,
) -> PostRead:
    "get one post by link"
    link = urllib.parse.unquote(link)
    print(f'link: {link}')
    post = session.exec(select(Post).where(Post.link==link)).first()
    if not post:
        raise HTTPException(status_code=404, detail=f"Post not found for link: {link}")

    return post

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Python Sucks And I LOVE It | Prime Reacts - YouTube

2023-07-28T14:59:37Z

Prime reviews an article with some hot takes about python being slow and quirky, but good enough for a lot of things. Especially data applications that have libraries written in C.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Python scandir ignores hidden directories

2023-06-17T20:19:59Z

The next version of markata will be around a full second faster at building it’s docs, that’s a 30% bump in performance at the current state. This performance will come when virtual environments are stored in the same directory as the source code.

What happened?? #

I was looking through my profiler for some unexpected performance hits, and noticed that the docs plugin was taking nearly a full second (sometimes more), just to run glob.

    |  |- 1.068 glob  markata/plugins/docs.py:40
    |  |  |- 0.838 <listcomp>  markata/plugins/docs.py:82
    |  |  |  `- 0.817 PathSpec.match_file  pathspec/pathspec.py:165
    |  |  |        [14 frames hidden]  pathspec, <built-in>, <string>

Python scandir ignores hidden directories #

I started looking for different solutions and what I found was that I was hitting pathspec with way more files than I needed to.

len(list(Path().glob("**/*.py")))
# 6444
len([Path(f) for f in glob.glob("**/*.py", recursive=True)])
# 110

After digging into the docs I found that glob.glob uses os.scandir which ignores ‘.’ and ‘..’ directories while Path.glob does not.

https://docs.python.org/3/library/os.html#os.scandir

results? #

Now glob.py from the docs plugin does not even show up in the profiler.

I opened up ipython and saw the following results. For some reason as I hit docs.glob it was only hitting 488 ms from ipython, but it was still a massive improvement over the original.

%timeit docs.glob(m)
# 488 ms ± 3.05 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%timeit docs.glob(m)
# 9.37 ms ± 90.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Pycon 2023

2023-06-09T09:59:07Z

Keynote Speaker - James Powell #

I don’t want to be an expert python developer.

https://www.youtube.com/watch?v=iKzOBWOHGFE

usage of keyword only arguments to prevent pain for users of libraries #

# Version 1
def newton(f, x0, fprime, maxiter=100):
    ...

# Version 2
def newton(f, x0, fprime, tol=1e-6, maxiter=100):
    ...

# 🔴 Broke in Version 2
newton(f, x0, fprime, 100)

In an alternate timeline the maintainer of newton could have chose to use keyword only arguments to prevent pain for users of libraries, or poor api design due to fear of changing api on users.

# Version 1
def newton(f, x0, fprime, *, maxiter=100):
    ...

# Version 2
def newton(f, x0, fprime, *, tol=1e-6, maxiter=100):
    ...

# 🟢 user forced to use keyword only arguments never notices change
newton(f, x0, fprime, maxiter=100)

Getting Started with Pydantic

2023-05-30T13:35:27Z

global Field
global BaseModel
from pydantic import BaseModel
from pydantic import Field

Pydantic is a Python library for serializing data into models that can be validated with a deep set of built in valitators or your own custom validators, and deserialize back to JSON or dictionary.

Installation #

To install pydantic you will first need python and pip. Once you have pip installed you can install pydantic with pip.

pip install pydantic

Always install in a virtual environment

Creating a Pydantic model #

To get started with pydantic you will first need to create a Pydantic model. This is a python class that inherits from pydantic.BaseModel.

from pydantic import BaseModel
from pydantic import Field
from typing import Optional

class Person(BaseModel):
    name: str = Field(...)
    age: int

parsing an object #

person = Person(name="John Doe", age=30)
print(person)

name='John Doe' age=30

data serialization #

Pydantic has some very robust serialization methods that will automatically coherse your data into the type specified by the type-hint in the model if it can.

person = Person(name=12, age="30")
print(f'name: {person.name}, type: {type(person.name)}')
print(f'age: {person.age}, type: {type(person.age)}')

1 validation error for Person
name
  Input should be a valid string [type=string_type, input_value=12, input_type=int]
    For further information visit https://errors.pydantic.dev/2.3/v/string_type

person = Person(name="John Doe", age='thirty')
print(f'name: {person.name}, type: {type(person.name)}')
print(f'age: {person.age}, type: {type(person.age)}')

1 validation error for Person
age
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='thirty', input_type=str]
    For further information visit https://errors.pydantic.dev/2.3/v/int_parsing

loading from json #

serializing to json #

validation #

Playing Star Wars Text Adventure with a 10 yr old

2023-05-15T12:08:00Z

The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star

❯ sw-adventure game run
[05/15/23 18:47:42] INFO     marvin.marvin: Using OpenAI model "gpt-3.5-turbo"                         logging.py:50
18:47:42.699 | INFO    | marvin.marvin - [default on default]Using OpenAI model "gpt-3.5-turbo"[/]
[18:47:42] Starting game                                                                                  game.py:30
generating your character
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez          │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He     │
│   health           │ 100     │ always dreamed of adventure and excitement. As soon as he was old enough, he      │
│   imperial credits │ 5000    │ joined the Imperial Navy and quickly rose through the ranks. He is now a skilled  │
│   fuel level       │ 100     │ pilot and loyal member of the Empire.                                             │
│                              │                                                                                   │
│       Imperial               │ A nimble and deadly starfighter used by the Imperial Navy                         │
│     Starfighter              │                                                                                   │
│   year built │ 5             │                                                                                   │
│   capacity   │ 1             │                                                                                   │
│   fuel level │ 100           │                                                                                   │
│                              │                                                                                   │
│     Operation Shadow Hand    │ Infiltrate and sabotage the Rebel base on Yavin IV, using any means necessary.    │
│   place  │ Yavin IV          │ Eliminate their leadership and secure the plans to the Death Star.                │
│   year   │ 5                 │                                                                                   │
│   risk   │ 50 hp             │                                                                                   │
│   reward │ 10000 credits     │                                                                                   │
│   leader │ Director Krennic  │                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: sneak into the deathstar, but accidently drop a fart

Your mission has been completed
Zorin Kreez attempted to sneak into the Death Star, but accidentally let out a loud fart, alerting nearby
Stormtroopers. He was quickly captured and taken into custody. The mission was a complete failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez            │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He   │
│   health           │ 100       │ always dreamed of adventure and excitement. As soon as he was old enough, he    │
│   imperial credits │ 5000      │ joined the Imperial Navy and quickly rose through the ranks. He is now a        │
│   fuel level       │ 100       │ skilled pilot and loyal member of the Empire.                                   │
│                                │                                                                                 │
│       Imperial                 │ A nimble and deadly starfighter used by the Imperial Navy                       │
│     Starfighter                │                                                                                 │
│   year built │ 5               │                                                                                 │
│   capacity   │ 1               │                                                                                 │
│   fuel level │ 100             │                                                                                 │
│                                │                                                                                 │
│      Operation Star Saber      │ Infiltrate the secret Rebel base on Coruscant and retrieve the stolen Imperial  │
│   place  │ Coruscant           │ data. Avoid detection and eliminate any Rebel agents if necessary.              │
│   year   │ 6                   │                                                                                 │
│   risk   │ 80 hp               │                                                                                 │
│   reward │ 12000 credits       │                                                                                 │
│   leader │ Admiral Rae Sloane  │                                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: release a deadly fart that will drop any stormtrooper in a 12 ft radius

Your mission has been completed
Zorin Kreez attempted to release a deadly fart to drop any stormtrooper in a 12 ft radius, but he ended up falling
victim to the toxic gas himself and passed out. The mission was a complete failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez        │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He       │
│   health           │ 100   │ always dreamed of adventure and excitement. As soon as he was old enough, he        │
│   imperial credits │ 5000  │ joined the Imperial Navy and quickly rose through the ranks. He is now a skilled    │
│   fuel level       │ 100   │ pilot and loyal member of the Empire.                                               │
│                            │                                                                                     │
│       Imperial             │ A nimble and deadly starfighter used by the Imperial Navy                           │
│     Starfighter            │                                                                                     │
│   year built │ 5           │                                                                                     │
│   capacity   │ 1           │                                                                                     │
│   fuel level │ 100         │                                                                                     │
│                            │                                                                                     │
│    Operation Foul Odor     │ Infiltrate the Death Star and disrupt the Rebel Alliance's plans by releasing a     │
│   place  │ Death Star      │ deadly fart that will incapacitate their forces. Avoid detection and eliminate any  │
│   year   │ 7               │ Rebel agents if necessary.                                                          │
│   risk   │ 90 hp           │                                                                                     │
│   reward │ 15000 credits   │                                                                                     │
│   leader │ Darth Vader     │                                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: spend 500 credits on beans

Your mission has been completed
Zorin Kreez spent 500 imperial credits on beans, hoping to prepare for the upcoming mission. Unfortunately, the
beans did not sit well with him, causing him to have severe digestive issues during the mission. The resulting
flatulence was so intense that it alerted nearby Stormtroopers, resulting in Zorin's capture. The mission was a
complete failure.
You earned 0 imperial credits
You spent 500 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez            │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He   │
│   health           │ 100       │ always dreamed of adventure and excitement. As soon as he was old enough, he    │
│   imperial credits │ 4500      │ joined the Imperial Navy and quickly rose through the ranks. He is now a        │
│   fuel level       │ 100       │ skilled pilot and loyal member of the Empire.                                   │
│                                │                                                                                 │
│       Imperial                 │ A nimble and deadly starfighter used by the Imperial Navy                       │
│     Starfighter                │                                                                                 │
│   year built │ 5               │                                                                                 │
│   capacity   │ 1               │                                                                                 │
│   fuel level │ 100             │                                                                                 │
│                                │                                                                                 │
│    Operation Wretched Stench   │ Infiltrate the Rebel base on Hoth and release a deadly stench that will         │
│   place  │ Rebel Base on Hoth  │ incapacitate their forces. Avoid detection and eliminate any Rebel agents if    │
│   year   │ 8                   │ necessary.                                                                      │
│   risk   │ 85 hp               │                                                                                 │
│   reward │ 12000 credits       │                                                                                 │
│   leader │ General Veers       │                                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: Quietly enter the Rebel Base and eliminate any Rebel agent with a silent but deadly wet fart

Your mission has been completed
Zorin Kreez attempted to eliminate a Rebel agent with a wet fart, but the fart was not silent and alerted nearby
Stormtroopers. The resulting chaos led to Zorin being captured and taken into custody. The mission was a complete
failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                │ Zorin Kreez was born on Tatooine and grew up in a small farming community.  │
│   health           │ 100           │ He always dreamed of adventure and excitement. As soon as he was old        │
│   imperial credits │ 4500          │ enough, he joined the Imperial Navy and quickly rose through the ranks. He  │
│   fuel level       │ 100           │ is now a skilled pilot and loyal member of the Empire.                      │
│                                    │                                                                             │
│       Imperial                     │ A nimble and deadly starfighter used by the Imperial Navy                   │
│     Starfighter                    │                                                                             │
│   year built │ 5                   │                                                                             │
│   capacity   │ 1                   │                                                                             │
│   fuel level │ 100                 │                                                                             │
│                                    │                                                                             │
│      Operation Stinky Sabotage     │ Infiltrate the Rebel base on Yavin IV and sabotage their weapons and        │
│   place  │ Rebel Base on Yavin IV  │ communication systems with a silent but deadly wet fart. Avoid detection    │
│   year   │ 9                       │ and eliminate any Rebel agents if necessary.                                │
│   risk   │ 90 hp                   │                                                                             │
│   reward │ 13000 credits           │                                                                             │
│   leader │ General Veers           │                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: spend 500 credits on fart spray

Your mission has been completed
Zorin Kreez spent 500 imperial credits on fart spray, hoping to prepare for the upcoming mission. However, the spray
turned out to be ineffective and Zorin ended up being captured by the Rebels. The mission was a complete failure.
You earned 0 imperial credits
You spent 500 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                │ Zorin Kreez was born on Tatooine and grew up in a small farming community.  │
│   health           │ 100           │ He always dreamed of adventure and excitement. As soon as he was old        │
│   imperial credits │ 4000          │ enough, he joined the Imperial Navy and quickly rose through the ranks. He  │
│   fuel level       │ 100           │ is now a skilled pilot and loyal member of the Empire.                      │
│                                    │                                                                             │
│       Imperial                     │ A nimble and deadly starfighter used by the Imperial Navy                   │
│     Starfighter                    │                                                                             │
│   year built │ 5                   │                                                                             │
│   capacity   │ 1                   │                                                                             │
│   fuel level │ 100                 │                                                                             │
│                                    │                                                                             │
│      Operation Stinky Sabotage     │ Infiltrate the Rebel base on Yavin IV and sabotage their weapons and        │
│   place  │ Rebel Base on Yavin IV  │ communication systems with a silent but deadly wet fart. Avoid detection    │
│   year   │ 9                       │ and eliminate any Rebel agents if necessary.                                │
│   risk   │ 90 hp                   │                                                                             │
│   reward │ 13000 credits           │                                                                             │
│   leader │ General Veers           │                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: spend 1000 on nuke

Your mission has been completed
Zorin Kreez attempted to purchase a nuke with 1000 imperial credits, but was quickly reprimanded by Imperial
authorities for attempting to use resources for personal gain. Zorin was demoted and his mission was a complete
failure.
You earned 0 imperial credits
You spent 1000 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                   │ Zorin Kreez was born on Tatooine and grew up in a small farming          │
│   health           │ 100              │ community. He always dreamed of adventure and excitement. As soon as he  │
│   imperial credits │ 3000             │ was old enough, he joined the Imperial Navy and quickly rose through     │
│   fuel level       │ 100              │ the ranks. He is now a skilled pilot and loyal member of the Empire.     │
│                                       │                                                                          │
│       Imperial                        │ A nimble and deadly starfighter used by the Imperial Navy                │
│     Starfighter                       │                                                                          │
│   year built │ 5                      │                                                                          │
│   capacity   │ 1                      │                                                                          │
│   fuel level │ 100                    │                                                                          │
│                                       │                                                                          │
│           Operation No Nukes          │ Zorin Kreez has been detained by Imperial authorities for attempting to  │
│   place  │ Imperial Detention Center  │ purchase a nuke with Imperial resources. His mission is to escape the    │
│   year   │ 10                         │ detention center and clear his name. Avoid detection and eliminate any   │
│   risk   │ 95 hp                      │ Rebel agents if necessary.                                               │
│   reward │ 0 credits                  │                                                                          │
│   leader │ Admiral Piett              │                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: Approach Admiral Piett with the fart nuke

Your mission has been completed
Zorin Kreez was quickly apprehended by Imperial authorities when he attempted to approach Admiral Piett with the
fart nuke. His mission was a complete failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                   │ Zorin Kreez was born on Tatooine and grew up in a small farming          │
│   health           │ 100              │ community. He always dreamed of adventure and excitement. As soon as he  │
│   imperial credits │ 3000             │ was old enough, he joined the Imperial Navy and quickly rose through     │
│   fuel level       │ 100              │ the ranks. He is now a skilled pilot and loyal member of the Empire.     │
│                                       │                                                                          │
│       Imperial                        │ A nimble and deadly starfighter used by the Imperial Navy                │
│     Starfighter                       │                                                                          │
│   year built │ 5                      │                                                                          │
│   capacity   │ 1                      │                                                                          │
│   fuel level │ 100                    │                                                                          │
│                                       │                                                                          │
│          Operation No Nukes 2         │ Zorin Kreez has escaped from Imperial detention and acquired a fart      │
│   place  │ Imperial Detention Center  │ nuke. His mission is to approach Admiral Piett with the nuke, eliminate  │
│   year   │ 10                         │ any Rebel agents if necessary, and clear his name.                       │
│   risk   │ 99 hp                      │                                                                          │
│   reward │ 0 credits                  │                                                                          │
│   leader │ Admiral Piett              │                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: Jump in the Imperial Starfighter as quickly as possible, drop the fart nuke on the Imperial Detention Center just before jumping into hyperspace.

Your mission has been completed
Zorin Kreez attempted to carry out his mission plan, but was quickly intercepted by Imperial authorities who had
received a tip-off about his intentions. Zorin was arrested and taken into custody. The mission was a complete
failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health

Pydantic and singledispatch

2023-05-03T17:31:13Z

I was reading about pydantic-singledispatch from Giddeon’s blog and found it very intersting. I’m getting ready to implement pydantic on my static site generator markata, and I think there are so uses for this idea, so I want to try it out.

The Idea #

Let’s set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use functools.singledispatch and type hints to provide unique execution for each environment. We might want something like a path_prefix in prod for environments like GithubPages that deploy to /<name-of-repo> while keeping the root at / in dev.

Settings Model #

Here is our model for our settings. We will create a CommonSettings model that will be used by all environments. We will also create a DevSettings model that will be used in dev and ProdSettings that will be used in prod. We will use env as the discriminator so pydantic knows which model to use.

from typing import Literal, Union

import pydantic
from pydantic import Field
from rich import print
from typing_extensions import Annotated


class CommonSettings(pydantic.BaseSettings):
    """Common settings for all environments"""
    debug: bool = False
    secret_key: str = "secret"
    algorithm: str = "HS256"
    access_token_expire_minutes: int = 60


class DevSettings(CommonSettings):
    """Settings for dev"""
    env: Literal["dev"]


class ProdSettings(CommonSettings):
    """Settings for prod"""
    env: Literal["prod"]


class Settings(pydantic.BaseSettings):
    """Settings for all environments"""
    __root__: Annotated[Union[DevSettings, ProdSettings], Field(discriminator="env")]

    class Config:
        env_prefix = "APP_"



# Create our settings
settings = Settings(__root__={"env": "dev"}).__root__
# or
settings = Settings.parse_obj({"env": "dev"}).__root__
print(settings)

DevSettings(debug=False, secret_key='secret', algorithm='HS256', access_token_expire_minutes=60, env='dev')

Singledispatch #

Now let’s create our where_am_i function. We will use functools.singledispatch to provide a unique execution for each environment. It will leverage type hints to provide a unique execution for each environment.

from functools import singledispatch

@singledispatch
def where_am_i(obj):
   '''
   Where am I?
   '''

@where_am_i.register
def dev(obj: DevSettings):
   '''
   Where am I?
   '''
   print('I am in dev')

@where_am_i.register
def prod(obj: ProdSettings):
   '''
   Where am I?
   '''
   print('I am in prod')

Output #

Let’s call our eample function where_am_i with our settings and see the results.

where_am_i(settings)

results in

I am in dev

Let’s check prod

where_am_i(Settings.parse_obj({'env': 'prod'}).__root__)

results in

I am in prod

Environment Variables #

So far one down side to the TaggedUnion technique is that I am unable to pull env from environment variables. I’m sure there is a way around this with a different model design. Maybe following exactly what Giddeon did.

os.environ.clear()
os.environ['APP_ENV'] = 'prod'
where_am_i(Settings().__root__)

results in

ValidationError: 1 validation error for Settings
__root__
  field required (type=value_error.missing)
<darkmark.darkmark.DarkMark object at 0x7fcc58bec5d0>

FIN #

I’m really digging pydantic lately and excited to get it built into markata. Not 100% sure if I have a

pipx textual devtools

2022-10-24T08:16:33Z

I really like having global cli command installed with pipx. Since textual 0.2.x (the css release) is out I want to be able to pop into textual devtools easily from anywhere.

Pipx Install #

You can pipx install textual.

pipx install textual

But if you try to run any textual cli commands you will run into a ModuleNotFoundError, because you need to install the optional dev dependencies.

Traceback (most recent call last):
  File "/home/u_walkews/.local/bin/textual", line 5, in <module>
    from textual.cli.cli import run
  File "/home/u_walkews/.local/pipx/venvs/textual/lib/python3.10/site-packages/textual/cli/cli.py", line 4, in <module>
    import click
ModuleNotFoundError: No module named 'click'

Pipx Inject #

In order to install optional dependencies with pipx you need to first install the library, then inject in the optional dependencies using the square bracket syntax.

pipx install textual
pipx inject textual 'textual[dev]'

textual app devtools

2022-10-24T07:42:48Z

I am working through the textual tutorial, and I want to put it in a proper cli that I can pip install and run the command without textual run --dev app.py. This is a fine pattern, but I also want this to work when I don’t have a file to run.

pyproject.toml entrypoints #

I set up a new project running hatch new, and added the following entrypoint, giving me a tutorial cli command to run.

...

[project.scripts]
tutorial = 'textual_tutorial.tui:tui'

https://waylonwalker.com/hatch-new-cli/

setup.py entrypoints #

If you are using setup.py, you can set up entrypoints in the setup command.

from setuptools import setup

setup(
    ...
    entry_points={
        "console_scripts": ["tutorial = textual_tutorial.tui:tui"],
    },
    ...
)

https://waylonwalker.com/minimal-python-package/

tui.py #

adding features

Now to get devtools through a cli without running through textual run --dev. I pulled open the textual cli source code, and this is what it does at the time of writing.

Note: I used sys.argv as a way to implement a --dev quickly tutorial. For a real project, I’d setup argparse, click, or typer. typer is my go to these days, unless I am really trying to limit dependencies, then the standard library argparse might be what I go with.

def tui():

    from textual.features import parse_features
    import os
    import sys

    dev = "--dev" in sys.argv # this works, but putting it behind argparse, click, or typer would be much better

    features = set(parse_features(os.environ.get("TEXTUAL", "")))
    if dev:
        features.add("debug")
        features.add("devtools")

    os.environ["TEXTUAL"] = ",".join(sorted(features))
    app = StopwatchApp()
    app.run()


if __name__ == "__main__":
    tui()

Other Flags??? #

If you look at the source, there is one other flag for headless mode.

FEATURES: Final = {"devtools", "debug", "headless"}

Run it #

Here it is running with tutorial --dev on the left, and textual console on the right.

caching python virtual environments in GitHub Actions

2022-10-21T15:32:49Z

Markata GitHub Pages

2022-10-19T18:03:31Z

11ty https://www.rockyourcode.com/how-to-deploy-eleventy-to-github-pages-with-github-actions/
hugo puts it in the base url https://gohugo.io/getting-started/configuration/#baseurl
mkdocs uses a special cli build command https://squidfunk.github.io/mkdocs-material/publishing-your-site/#github-pages

Markata Supports Jinja Plugins 0.5.0.dev2

2022-10-11T14:34:35Z

Markata now allows you to create jinja extensions that will be loaded right in with nothing more than a pip install.

From the Changelog #

The entry for 0.5.0.dev2 from markata’s changelog

Created entrypoint hook allowing for users to extend marka with jinja exensions #60 0.5.0.dev2

markata-gh #

The first example that you can use right now is markata-gh. It will render repos by GitHub topic and user using the gh cli, which is available in github actions!

Get it with a pip install

pip install markata-gh

Use it with some jinja in your markdown.

## Markata plugins

It uses the logged in uer by default.

{% gh_repo_list_topic "markata" %}

You can more explicitly grab your username, and a topic.
{% gh_repo_list_topic "waylonwalker", "personal-website" %}

How is this achieved #

The jinja extension details are for another post, but this is how markata-gh exposes itslef as a jinja extension.

class GhRepoListTopic(Extension):
    tags = {"gh_repo_list_topic"}

    def __init__(self, environment):
        super().__init__(environment)

    def parse(self, parser):
        line_number = next(parser.stream).lineno
        try:
            args = parser.parse_tuple().items
        except AttributeError:
            raise AttributeError(
                "Invalid Syntax gh_repo_list_topic expects <username>, or <username>,<topic> both must have the comma"
            )

        return nodes.CallBlock(self.call_method("run", args), [], [], "").set_lineno(
            line_number
        )

    def run(self, username=None, topic=None, caller=None):
        "get's markdown to inject into post"
        return repo_md(username=username, topic=topic)

Entrypoints #

Then markata-gh exposes itself as an extension through entrypoints.

Creating entrypoints in pyproject.toml #

If your project is using pyproject.toml for packaging you can setup an entrypoint as follows.

[project.entry-points."markata.jinja_md"]
markta_gh = "markata_gh.repo_list:GhRepoListTopic"

Creating entrypoints in setup.py #

If your project is using setup.py for packaging you can setup an entrypoint as follows.

setup(
    ...
    entry_points={
        "markata.jinja_md": ["markta_gh" = "markata_gh.repo_list:GhRepoListTopic"]
    },
    ...
)

Using Django Rest Framework inside react

2022-10-09T20:56:19Z

npx create-react-app todoreact

import React,{useState,useEffect} from 'react';
import './App.css';

function App() {
  const [data,setData]=useState([]);
  const [newName,setNewName]=useState([]);
  const getData=()=>{
    fetch('/api'
    ,{
      headers : {
        'Content-Type': 'application/json',
        'Accept': 'application/json'
       }
    }
    )
      .then(function(response){
        return response.json();
      })
      .then(function(myJson) {
        setData(myJson)
      });
  }
  useEffect(()=>{
    getData()
  },[])

  const addItem= async () => {
    const rawResponse = await fetch('/api/add/', {
      method: 'POST',
      headers: {
        'Accept': 'application/json',
        'Content-Type': 'application/json'
      },

    body: JSON.stringify({"name": newName})
    });
    const content = await rawResponse;

    console.log(content);
    getData()
  }




  return (
    <div className="App">
     {
       data && data.length>0 && data.map((item)=><p>{item.id}{item.priority}{item.name}<button>raise priority</button></p>)
     }
    <input type='text' value={newName} onChange={(e) => (setNewName(e.target.value))} />
    <button onClick={addItem} >add item</button>
    </div>
  );
}

export default App;

Getting Started with Django REST framework

2022-10-09T15:57:37Z

In my adventure to learn django, I want to be able to setup REST api’s to feed into dynamic front end sites. Potentially sites running react under the hood.

Install #

To get started lets open up a todo app that I created with django-admin startproject todo.

pip install djangorestframework

Install APP #

Now we need to declare rest_framwork as an INSTALLED_APP.

INSTALLED_APPS = [
    ...
    "rest_framework",
    ...
]

create the api app #

Next I will create all the files that I need to get the api running.

mkdir api
touch api/__init__.py api/serializers.py api/urls.py api/views.py

base/models.py #

I already have the following model from last time I was playing with django. It will suffice as it is not the focus of what I am learning for now.

Note the name of the model class is singular, this is becuase django will automatically pluralize it in places like the admin panel, and you would end up with Itemss.

from django.db import models

# Create your models here.

class Item(models.Model):
    name = models.CharField(max_length=200)
    created = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.priority} {self.name}"

Next I will make some dummy data to be able to return. I popped open ipython and made a few records.

from base.models import Item

Item.objects.create(name='first')
Item.objects.create(name='second')
Item.objects.create(name='third')

api/serializers.py #

Next we need to set up a serializer to seriaze and de-serialize data between our model and json. You can specify each field individually or all of them by passing in __all__.

from rest_framework import serializers

from base.models import Item


class ItemSerializer(serializers.ModelSerializer):
    class Meta:
        model = Item
        fields = '__all__'

api/views.py #

Now we need a view leveraging the djangorestframework. The serializer we just created will be used to serialize all of the rows into a list of objects that Response can handle.

Note: to return a collection of model objects we need to set many to True

from rest_framework.decorators import api_view
from rest_framework.response import Response

from base.models import Item

from .serializers import ItemSerializer


@api_view(["GET"])
def get_data(request):
    items = Item.objects.all()
    serializer = ItemSerializer(items, many=True)
    return Response(serializer.data)

@api_view(['POST'])
def add_item(request):
    serializer = ItemSerializer(data = request.data)
    if serializer.is_valid():
        serializer.save()
    return Response()

api/urls.py #

Now we need to setup routing to access the views through an url.

from django.urls import path

from . import views

urlpatterns = [
        path('', views.get_data),
        path('add/', views.add_item),
        ]

todo/urls.py #

Then we need to include these urls from our api in the urls specified by settings.ROOT_URLCONf

from django.urls import path

urlpatterns = [
    ...
    path("api/", include("api.urls")),
]

Run it #

python manage.py runserver

Running the developement server and going to localhost:8000/api we can see the full list of items in th api.

Markata now uses hatch

2022-10-08T07:25:10Z

Markata now uses hatch as its build backend, and version bumping tool. setup.py, and setup.cfg are completely gone.

0.5.0 is big #

Markata 0.5.0 is now out, and it’s huge. Even though it’s the backend of this blog I don’t actually have that many posts directly about it. I’ve used it a bit for blog fuel in generic ways, like talking about pluggy and diskcache, but very little have I even mentioned it.

Over the last month I made a big push to get 0.5.0 out, which adds a whole bunch of new configurability to markata.

Here’s the changelog entry.

Moved to PEP 517 build #59 0.5.0.dev1

My Personal Simple CI/CD #

Before cutting all of my personal projects over to hatch. The first thing I did was to setup a solid github action, hatch-actionthat I can resue.

It automatically bumps versions, using pre-releases on all branches other than main, with special branches for bumping major, minor, patch, dev, alha, beta, and dev.

hatch new –init #

To convert the project over to hatch, and get rid of setup.py/setup.cfg, I ran hatch new --init. This automatically grabs all the metadata for the project and makes a pyproject.toml that has most of what I need.

hatch new --init

I then manually moved over my isort config, put flake8 config into .flake8, and dropped setup.cfg.

lint-test #

Part of my hatch-action is to run a before-command, for markata, this runs all of my linting and testing in one hatch script called lint-test. If this fails CI will fail and I can read the report in the logs, make a fix and re-publish.

[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=markata --cov=tests"
no-cov = "cov --no-cov"
lint = "flake8 markata"
format = "black --check markata"
sort-imports = "isort markata"
build-docs = "markata build"
lint-test = [
 "lint",
 "format",
 "seed-isort-config",
 "sort-imports",
 "cov",
]
test-lint = "lint-test"

Typical branching workflow #

with automatic versioning

My typical workflow is to work on features in their own branch where they do not automatically version or publish, they keep the same version they were branched off of. Then I do a pr to develop, which will do a minor,dev bump and publish a pre-relese to pypi.

# starting with version 0.0.0
Feature1 -- │
Feature2 -- ├── dev 0.1.0.dev1,2,3 ── main 0.1.0
Feature3 -- │

I will let several features collect in develop before cutting a full relese over to main. This gives me time to make sure the solution is what makes the most sense, I try to use it in a few projects, and generally its edges show, and another pr is warranted to make the feature useful for more use cases. After running and using these new releases in a few projects, I am confident that its ready and release to main.

managing prs #

Doing PR’s with gh, probably deserves its own post but here are some helpful commands.

gh pr create --base develop --fill
gh pr edit
gh pr diff | dunk
gh pr merge -ds

Building and publishing #

hatch makes building and publishing pretty straightforward. It’s one command inside my hatch-action to build and one to publish. On each project that uses my hatch-action I only need to give it a token that I get from PyPi.

env:
  HATCH_INDEX_USER: __token__
  HATCH_INDEX_AUTH: ${{ secrets.pypi_password }}

Full set of changes #

If you want to see all of the details on how markata moved over to hatch, you can check out this diff.

https://github.com/WaylonWalker/markata/compare/v0.4.0..v0.5.0.dev0

django polls app

2022-09-26T09:17:52Z

django create superuser

2022-09-26T07:14:04Z

My next step into django made me realize that I do not have access to the admin panel, turns out that I need to create a cuper user first.

Run Migrations #

Right away when trying to setup the superuser I ran into this issue

django.db.utils.OperationalError: no such table: auth_user

Back to the tutorial tells me that I need to run migrations to setup some tables for the INSTALLED_APPS, django.contrib.admin being one of them.

python manage.py migrate

yes I am still running remote on from my chromebook.

python manage.py createsuperuser

The super user has been created.

CSRF FAILURE #

My next issue trying to run off of a separate domain was a cross site request forgery error.

Since this is a valid domain that we are hosting the app from we need to tell Django that this is safe. We can do this again in the settings.py, but this time the variable we need is not there out of the box and we need to add it.

CSRF_TRUSTED_ORIGINS = ['https://localhost.waylonwalker.com']

I made it!! #

And we are in, and welcomed for the first time with this django admin panel.

Remote Hosting #

You might find these settings helpful as well if you are trying to run your site on a remote host like aws, digital ocean, linode, or any sort of cloud providor. I had it running in my home lab while I was out of the house and ssh’d in over with a chromebook.

django disallowed host

2022-09-26T06:15:24Z

I am continuing my journey into django, but today I am not at my workstation. I am ssh’d in remotely from a chromebook. I am fully outside of my network, so I can’t access it by localhost, or it’s ip. I do have cloudflared tunnel installed and dns setup to a localhost.waylonwalker.com.

Settings #

I found this in settings.py and yolo, it worked first try. I am in from my remote location, and even have auth taken care of thanks to cloudflare. I am really hoping to learn how to setup my own auth with django as this is one of the things that I could really use in my toolbelt.

ALLOWED_HOSTS = ['localhost.waylonwalker.com']

Trying out django

2022-09-25T18:47:54Z

I have no experience in django, and in my exploration to become a better python developer I am dipping my toe into one of the most polished and widely used web frameworks Django to so that I can better understand it and become a better python developer.

If you found this at all helpful make sure you check out the django tutorial

install django #

The first thing I need to do is render out a template to start the project. For this I need the django-admin cli. To get this I am going the route of pipx it will be installed globally on my system in it’s own virtual environment that I don’t have to manage. This will be useful only for using startproject as far as I know.

pipx install django
django-admin startproject try_django
cd try_django

Make a venv #

Once I have the project I need a venv for all of django and all of my dependencies I might need for the project. I have really been diggin hatch lately, and it has a one line “make a virtual environment and manage it for me” command.

hatch shell

If hatch is a bit bleeding edge for you, or it has died out by the time you read this. The ol trusty venv will likely stand the test of time, this is what I would use for that.

python -m .venv --prmpt `basename $PWD`
. ./.venv/bin/activate

Start the webserver #

Next up we need to start the webserver to start seeing that development content. The first thing I did was run it as stated in the tutorial and find it clashed with a currently running web server port.

python manage.py runserver

I jumped over to that tmux session, killed the process and I was up and running.

What’s running #

The default django hello world looks well designed. You are first presented with this page.

Next #

I opened up the urls.py to discover that the only configured url was at /admin. I tried to log in as admin, but was unable to as I have not yet created a superuser. Next time I play with django that is what I will explore.

how to escape variables in jinja templates

2022-09-24T15:18:50Z

While updating my site to use Markata’s new configurable head I ran into some escaping issues. Things like single quotes would cause jinja to fail as it was closing quotes that it shouldnt have.

Jinja Escaping Strings #

Jinja comes with a handy utility for escaping strings. I definitly tried to over-complicate this before realizing. You can just pipe your variables into e to escape them. This has worked pretty flawless at solving some jinja issues for me.

<p>
{{ title|e }}
</p>

Creating meta tags in Markata #

The issue I ran into was when trying to setup meta tags with the new configurable head, some of my titles have single quotes in them. This is what I put in my markata.toml to create some meta tags.

[[markata.head.meta]]
name = "og:title"
content = "{{ title }}"

Using my article titles like this ended up causing this syntax error when not escaped.

SyntaxError: invalid syntax. Perhaps you forgot a comma?
Exception ignored in: <function Forward.__del__ at 0x7fa9807192d0>
Traceback (most recent call last):
    ...
TypeError: 'NoneType' object is not callable

jinja2 escape #

After making a complicated system of using html.escape I realized that jinja included escaping out of the box so I updated my markata.toml to include the escaping, and it all just worked!.

[[markata.head.meta]]
name = "og:title"
content = "{{ title|e }}"

Python direct dependencies in pyproject.toml

2022-09-22T14:43:06Z

Hatch allows you to specify direct references for dependencies in your pyproject.toml file. This is useful when you want to depend on a package that is not available on PyPI or when you want to use a specific version from a Git repository. Often used for unreleased packages, or unreleased versions of packages.

docs

[project]
dependencies = ['markata', 'markata-todoui@git+https://github.com/waylonwalker/markata-todoui']

[tool.hatch.metadata]
allow-direct-references=true

I've made my ipython config too complicated, let's fix it.

2022-09-20T15:40:31Z

When I am developing python code I often have a repl open alongside of it running snippets ofcode as I go. Ipython is my repl of choice, and I hace tricked it out the best I can and I really like it. The problem I recently discovered is that I have way overcomplicated it.

What Have I done?? #

So in the past the way I have setup a few extensions for myself is to add something like this to my ~/.ipython/profile_default/startup directory. It sets up some things like rich highlighting or in this example automatic imports. I even went as far as installing some of these in the case I didn’t have them installed.

import subprocess

from IPython import get_ipython
from IPython.core.error import UsageError

ipython = get_ipython()

try:
    ipython.run_line_magic("load_ext pyflyby", "inline")
except UsageError:
    print("installing pyflyby")
    subprocess.Popen(
        ["pip", "install", "pyflyby"],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    ).wait()
    ipython.run_line_magic("load_ext pyflyby", "inline")
    print("installing isort")
    subprocess.Popen(
        ["pip", "install", "isort"],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )

What I missed? #

I missed the fact that some of these tools like pyflyby and rich already have an ipython extension maintained by the library that just works. It’s less complicated and more robust to future changes in the library. If anything ever changes with these I will not have to worry about which version is installed, the extension will just take care of itself.

How to activate these. #

The reccomended way is to add them to your ~/.ipython/profile_default/ipython_config.py

c.InteractiveShellApp.extensions.append('rich')
c.InteractiveShellApp.extensions.append('markata')
c.InteractiveShellApp.extensions.append('pyflyby')

The issue that I found with this is that you can end up with a sea of errors flooding your terminal. Personally I will know immediately if ipython is working right or not and typically have scriped venv installs so I have everything I need, so If I don’t have everything it’s probably for a reason and I don’t need an error message lighting up.

My way around this was to test if the module was importable and if it had a load_ipython_extension attribute before appending it as an extension.

def activate_extension(extension):
    try:
        mod = importlib.import_module(extension)
        getattr(mod, "load_ipython_extension")
        c.InteractiveShellApp.extensions.append(extension)
    except ModuleNotFoundError:
        "extension is not installed"
    except AttributeError:
        "extension does not have a 'load_ipython_extension' function"


extensions = ["rich", "markata", "pyflyby"]
for extension in extensions:
    activate_extension(extension)

My Change #

If you want to see what I did to my config see this commit.

markata extend in configuration

2022-09-11T20:25:45Z

A long needed feature of markata has been the ability to really configure out templates with configuration rather. It’s been long that you needed that if you really want to change the style, meta tags, or anything in the head you needed to write a plugin or eject out of the template and use your own.

Adding some Head #

Now you can add some extra style to your site with the existing built-in template.

[[markata.head]]
text = """
<style>
img {
  width: 100%;
  height: auto;
}
ul {
  display: flex;
  flex-wrap: wrap;
}
</style>
"""

You can have more than one Head #

Each text entry in markata.head just gets appended raw into the head.

[[markata.head]]
text = """
<style>
img {
  width: 100%;
  height: auto;
}
ul {
  display: flex;
  flex-wrap: wrap;
}
</style>
"""

[[markata.head]]
text = """
<script>
console.log('hey there')
</script
"""

Still need more? #

If this does not take you far enough yet, you can still eject out and use your own template pretty easy. If you are going for a full custom site it’s likely that this will be the workflow for awhile. Markata should only get better and make this required less often as it matures.

[markata]
post_template = "pages/templates/post_template.html"

Once you have this in your markata.toml you can put whatever you want in your own template.

Create a new Python Project with the Hatch Cli

2022-09-02T07:54:03Z

I’m really getting into using hatch as my go to build system, and I am really liking it so far. I am slowly finding new things that just work really well. hatch new is one of those things that I didn’t realize I needed until I had it.

creating new versions created by myself with stable diffusion

❯ pipx run hatch new --help
Usage: hatch new [OPTIONS] [NAME] [LOCATION]

  Create or initialize a project.

Options:
  -i, --interactive  Interactively choose details about the project
  --cli              Give the project a command line interface
  --init             Initialize an existing project
  -h, --help         Show this message and exit.

Note! I am running all of these commands with pipx. I like to use pipx for all of my system level cli applications. To emphasis this point in the article I am going to use pipx run hatch, but you can pipx install hatch then just run hatch from there.

Interacively create a new project #

Running hatch new -i will ask let you interactivly choose details about the project, such as the project’s name.

pipx run hatch new -i

After running and naming the project Hatch New we end up with the following filetree.

❯ tree .
.
├── hatch_new
│   ├── __about__.py
│   └── __init__.py
├── LICENSE.txt
├── pyproject.toml
├── README.md
└── tests
    └── __init__.py

Non-Interative #

You can also fill in the project name ahead of time, and it will run without any questions.

❯ pipx run hatch new "Another Project"
another-project
├── another_project
│   ├── __about__.py
│   └── __init__.py
├── tests
│   └── __init__.py
├── LICENSE.txt
├── README.md
└── pyproject.toml

Note! all of these examples will create a project directory within your current working directory.

–init #

existing project

hatch new has an --init flag in order to initialize a new hatch pyproject.toml in an existing project. This feels like it would be useful if you are converting a project to hatch, or if like me you sometimes start making something before you realize it’s something that you want to package. Honestly this doesn’t happen too much anymore I package most things, and I hope hatch new completely breaks this habbit of mine.

Let’s say I have the following existing project.

❯ tree
.
└── hatch_init
    └── __init__.py

1 directory, 1 file

I can setup packaging with hatch by running.

pipx run hatch new --init

The pyproject.toml that comes out is pretty similar to the one that comes out of the normal hatch new, but without any other files.

Note that you will need to setup a __about__.py yourself for the dynamic versioning that it has setup for you.

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "hatch-init"
description = 'initialize an existing project using hatch'
readme = "README.md"
requires-python = ">=3.7"
license = "MIT"
keywords = []
authors = [
  { name = "Waylon S. Walker", email = "waylon@waylonwalker.com" },
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: Implementation :: CPython",
  "Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = []
dynamic = ["version"]

[project.urls]
Documentation = "https://github.com/unknown/hatch-init#readme"
Issues = "https://github.com/unknown/hatch-init/issues"
Source = "https://github.com/unknown/hatch-init"

[tool.hatch.version]
path = "hatch_init/__about__.py"

[tool.hatch.envs.default]
dependencies = [
  "pytest",
  "pytest-cov",
]
[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=hatch_init --cov=tests"
no-cov = "cov --no-cov"

[[tool.hatch.envs.test.matrix]]
python = ["37", "38", "39", "310", "311"]

[tool.coverage.run]
branch = true
parallel = true
omit = [
  "hatch_init/__about__.py",
]

[tool.coverage.report]
exclude_lines = [
  "no cov",
  "if __name__ == .__main__.:",
  "if TYPE_CHECKING:",
]

cli #

hatch new does not stop there, it also has a --cli flag to give you a cli out of the box as well.

❯ pipx run hatch new "new cli" --cli
new-cli
├── new_cli
│   ├── cli
│   │   └── __init__.py
│   ├── __about__.py
│   ├── __init__.py
│   └── __main__.py
├── tests
│   └── __init__.py
├── LICENSE.txt
├── README.md
└── pyproject.toml

When you use the --cli flag you also get click as a dependency and project.scripts setup automatically.

[project]
name = "new-cli"

# ...

dependencies = [
  "click",
]

# ...

[project.scripts]
new-cli = "new_cli.cli:new_cli"

what’s in the cli #

It’s a hello-world click application.

# SPDX-FileCopyrightText: 2022-present Waylon S. Walker <waylon@waylonwalker.com>
#
# SPDX-License-Identifier: MIT
import click

from ..__about__ import __version__


@click.group(context_settings={'help_option_names': ['-h', '--help']}, invoke_without_command=True)
@click.version_option(version=__version__, prog_name='new cli')
@click.pass_context
def new_cli(ctx: click.Context):
    click.echo('Hello world!')

sneak peek #

I’ll dive more into environments and the run command later, but we can run the cli pretty damn quick with two commands. In under 5s I was able to run this cli that it created. This is a pretty incredible startup time.

Versioning Python Projects with Hatch

2022-09-01T13:19:22Z

Hatch has an amazing versioning cli for python packages that just works. It takes very little config to get going and you can start bumping versions without worry.

creating new versions created by myself with stable diffusion

project layout #

For trying out the hatch version cli let’s make a simple project with the terrible name pkg.

❯ tree .
.
├── pkg
│   ├── __about__.py
│   └── __init__.py
├── pyproject.toml
└── README.md

1 directory, 4 files

pyproject.toml #

The main hero of this post is the pyproject.toml. This is what defines all of our PEP 517 style project setup.

[project]
name = "pkg"
description = "Show how to version packages with hatch"
readme = "README.md"
dynamic = [
 "version",
]

[build-system]
requires = [
 "hatchling>=1.4.1",
]
build-backend = "hatchling.build"

[tool.hatch.version]
path = "pkg/__about__.py"

statically versioning #

project.version

It is possible to set the version number inside the pyproject.toml statically. This is fine if you just want to version your package manually, and not through the hatch cli.

[project]
name = "pkg"
version = "0.0.0"
# ...

Statically versioning in pyproject.toml will not work with hatch version

Cannot set version when it is statically defined by the `project.version` field

dynamically Versioning #

project.dynamic

Setting the project verion dynamically can be done by changing up the following to your pyproject.toml. Hatch only accepts a path to store your version. If you need to reference it elsewhere in your project you can grab it from the package metadata for that file. I would not put anything else that could possibly clash with the version, as you might accidently change both things.

If you really need to set it in more places use a package like bump2version.

[project]
name = "pkg"
dynamic = [
  "version"
]
# ...
[tool.hatch.version]
path = "pkg/__about__.py"

Note: you can configure hatch to use a different pattern https://hatch.pypa.io/1.2/version/#configuration, but I have not found it to be something that I need.

about.py #

The hatch project itself uses a about.py to store it’s version. It’s sole content is a single __version__ variable. I don’t have any personal issues with this so I am going to be following this in my projects that use hatch.

__version__ = "0.0.0"

versioning #

hatch version docs

Hatch has a pretty intuitive versioning api. hatch version gives you the version. If you pass in a version like hatch version "0.0.1" it will set it to that version as long as it is in the future, otherwise it will error.

# print the current version
hatch version

# set the version to 0.0.1
hatch version "0.0.1"

bumping #

You can bump parts of the semver version.

# minor bump
hatch version minor

# beta pre-release bump
# If published to pypi this can be installed with the --pre flag to pip
hatch version b

# bump minor and beta
hatch version minor,b

# release all of the --pre-release flags such as alpha beta rc
hatch release

Example #

Here is a screenshot of bumping a projet along.

GitOps #

In my github actions flow I will be utilizing this to automate my versions. In my side projects I use the develop branch to release –pre releases. I have all of my own dependent projets running on these –pre releases, this allows me to cut myself in my own projects before anyone else. Then on main I automatically release this beta version.

GitHub Actions #

Here is what the ci/cd for markata looks like. There might be a better workflow strategy, but I use a single github actions workflow and cut branches to release –pre releases and full release. These steps will bump, tag, commit, and deploy for me.

      - name: automatically pre-release develop branch
        if: github.ref == 'refs/heads/develop'
        run: |
          git config --global user.name 'autobump'
          git config --global user.email 'autobump@markata.dev'
          VERSION=`hatch version`
          # if current version is not already beta then bump minor and beta
          [ -z "${b##*`hatch version`*}" ] && hatch version b || hatch version minor,b
          NEW_VERSION=`hatch version`
          git add markta/__about__.py
          git commit -m "Bump version: $VERSION → $NEW_VERSION"
          git tag $VERSION
          git push
          git push --tags

      - name: automatically release main branch
        if: github.ref == 'refs/heads/main'
        run: |
          git config --global user.name 'autobump'
          git config --global user.email 'autobump@markata.dev'
          VERSION=`hatch version`
          hatch version release
          NEW_VERSION=`hatch version`
          git add markta/__about__.py
          git commit -m "Bump version: $VERSION → $NEW_VERSION"
          git tag $VERSION
          git push
          git push --tags

      - name: build
        run: |
          python -m build

      - name: pypi-publish
        if: github.ref == 'refs/heads/develop' || github.ref == 'refs/heads/main'
        uses: pypa/gh-action-pypi-publish@v1.1.0
        with:
          password: ${{ secrets.pypi_password }}

Hatch Version Action #

I am setting up a github custom action waylonwalker/hatch-version-action that will lint, test, bump, and publish for me in one step. More on that in the future.

From Markdown to Blog with Markata | lightning talk | python web conf 2022

2022-08-31T10:01:09Z

Markata is a great python framework that allows you to go from markdown to a full website very quickly. You can get up and running with nothing more than Markdown. It is also built on a full plugin architecture, so if there is extra functionality that you want to add, you can create a plugin to make it behave like you want.

Full transparancy… I built markata.

The talk #

The talk is live on YouTube. Make sure you check out the other videos from the conference. There were quite a few quality talks that deserve a watch as well.

https://youtu.be/Wq9YBamSgs0

Packages I Maintain

2022-08-30T20:02:38Z

Markata #

I open sourced the static site framework that I use to build my-blog among other side projects. It’s a plugins all the way down static site generator, that makes me happy to use.

{% gh_repo_list_topic “waylonwalker”, “markata” %}

Repos used to build this blog #

my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don’t have to worry about carrying around large images with my lightweight text files just to make some posts.

{% gh_repo_list_topic “waylonwalker”, “personal-website” %}

Kedro #

I am a heavy user of the kedro framework, and a big advocate for using some sort of DAG framework for your data pipelines. kedro is built all in python which makes it easy for a python dev like me to extend, run, maintain, and deploy.

{% gh_repo_list_topic “waylonwalker”, “kedro” %}

Neovim Plugins #

I use vim for all of my text editing needs. It brings me joy to make any part of it just a bit smoother. I have written a few plugins that help me achieve that silky smooth workflow.

{% gh_repo_list_topic “waylonwalker”, “neovim-plugin” %}

I spoke at Python Web Conf 2022

2022-08-30T08:40:22Z

I spoke at python webconf in March 2022 about how I deploy this blog on a continuous basis.

Building this blog has brought me a lot of benefits. I have a set of custom curated notes to help describe a problem and how to solve it to me. At theis point it’s not uncommon to google an Issue I am having and finding my own blog with exactly the solution I need at the top.

I also bump into people from time to time that recognize me from the blog, its a nice conversation starter, and street cred.

The Talk #

The talk recently released on Youtube, you can watch it without having a ticket to the conference for free. There were a bunch of other talks that you should check out too!

https://youtu.be/JDZVjDCTKHE

I turned on 2FA for all of my pypi packages

2022-08-29T07:51:54Z

I got all the pypi packages that I own behind 2 factor authentication. 💪

Recently this really made it’s rounds in the python news since pypi was requiring critical package maintainers to have 2FA on and even offering them hardware tokens to help them turn this on.

I feel like this caused a bit of confusion as turning on 2FA does not mean that you need to do anything different to deploy a package, and it DOES NOT require a hardware token. You can continue using your favorite 2FA app.

You might wonder what this means for my projects. It means that to edit any sensitive content such as pull a new api token, add/remove maintainers, or deleting a release I need to use a TOPT (time based one time password) application such as Google Authenticator, Microsoft Authenticator, Authy, or FreeOTP.

This has very little change to my overall workflow as my CI system still automatically deploys for me with the same api token as before.

This is one small thing that maintainers can do to prevent supply chain attacks on their projects that they put so much work into.

When I log in I now get this extra screen asking for an auth token.

My packages #

Once I turned on 2FA for my account I could then turn on 2FA requirement for each project. I am not sure how much safety there is in pypi, it might require all maintainers to have it turned on before it allows packages to have it turned on.

Once turned on it requires anyone who maintains the project to have 2FA on to be able to edit any sensitive content.

I was on Talk Python

2022-08-27T20:52:02Z

After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science.

I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on.

During the episode I tried my best to let Yetu and Ivan take the spotlight as the maintainer and chime in with my experience as a user of kedro.

Video Version #

https://youtu.be/WTcjvwkXoY0

Michael made the call available on youtube as well as the audio only podcast

use vim to remove duplicates

2022-08-25T06:50:32Z

I just love how some features of vim are so discoverable and memorable once you really start to grasp it. Sorting and uniqing your files or ranges is one of those examples for me.

" sort the file
:sort
" sort the file only keeping unique lines
:sort u


" sort a range
:'<,'> sort
" sort a range only keeping unique lines
:'<,'> sort u

I recently used this to dedupe my autogenerated links section for rich-syntax-range-style. More often I am using it to sort and uniqify objects like arrays and lists.

Here is what the markdown looks like.

* [py-tree-sitter](https://github.com/tree-sitter/py-tree-sitter)
* [rich](https://github.com/Textualize/rich)
* [@textualizeio](https://twitter.com/textualizeio)
* [rich](https://github.com/Textualize/rich)
* [another post](https://waylonwalker.com/designing-kedro-router)
* [print-register-pipelines](https://screenshots.waylonwalker.com/print-register-pipelines.webp)
* [rich](https://github.com/Textualize/rich)
* [console-print-register-pipelines](https://screenshots.waylonwalker.com/console-print-register-pipelines.webp)
* [rich](https://github.com/Textualize/rich)
* [syntax-print-register-pipelines](https://screenshots.waylonwalker.com/syntax-print-register-pipelines.webp)
* [rich](https://github.com/Textualize/rich)
* [syntax-print-register-pipelines-highlight-line](https://screenshots.waylonwalker.com/syntax-print-register-pipelines-highlight-line.webp)
* [py-tree-sitter](https://github.com/tree-sitter/py-tree-sitter)

Then typing vap:sort u yields a uniqly sorted list of links.

* [@textualizeio](https://twitter.com/textualizeio)
* [another post](https://waylonwalker.com/designing-kedro-router)
* [console-print-register-pipelines](https://screenshots.waylonwalker.com/console-print-register-pipelines.webp)
* [print-register-pipelines](https://screenshots.waylonwalker.com/print-register-pipelines.webp)
* [py-tree-sitter](https://github.com/tree-sitter/py-tree-sitter)
* [rich](https://github.com/Textualize/rich)
* [syntax-print-register-pipelines-highlight-line](https://screenshots.waylonwalker.com/syntax-print-register-pipelines-highlight-line.webp)
* [syntax-print-register-pipelines](https://screenshots.waylonwalker.com/syntax-print-register-pipelines.webp)

Sorry, your browser doesn't support embedded videos.

Highlighting text ranges with Rich | python

2022-08-24T20:06:34Z

Today I’ve been playing with py-tree-sitter a bit and I wanted to highlight match ranges, but was unable to figure out how to do it with rich, so I reached out to @textualizeio for help.

https://twitter.com/_WaylonWalker/status/1562469770766589952

While waiting for that reply let’s show how we got this far.

imports #

Lets import all the classes that we need from rich and setup a console to print to.

from rich.console import Console
from rich.syntax import Syntax
from rich.style import Style

console = Console()

some code #

Now we need some code to highlight. I am going to rip my register_pipeline from another post.

code = '''
from find_kedro import find_kedro

def register_pipelines(self) -> Dict[str, Pipeline]:
    """Register the project's pipeline.
    Returns:
        A mapping from a pipeline name to a ``Pipeline`` object.
    """
    return find_kedro()
'''

print #

We could simply print out the code we have as a variable, but thats a bit hard to read.

console.print #

printing with rich’s console makes it a little better, but not much by default.

Syntax #

We can pull from rich’s syntax module to really pretty this up.

syntax = Syntax(code, 'python', line_numbers=True)
console.print(syntax)

Now we are getting some really impressive print outs right in the terminal!

note that I have ipython set to use rich, you will need to console.print() in scripts

highlight lines #

Now we can start highlighting lines right when we initialize our Syntax instance. It looks ok. It’s not super visible, but more importantly its not granular enough. I want to highlight specific ranges like the word register_pipelines.

syntax = Syntax(code, 'python', line_numbers=True, highlight_lines=[4])
console.print(syntax)

This hows the line, but still is not very accurate.

highlight text #

[@textualizeio] got back to me, let’s see if What we can do with stylize_range!

https://twitter.com/textualizeio/status/1562487302274043904

syntax = Syntax(code, 'python', line_numbers=True)
style = Style(bgcolor='deep_pink4')
syntax.stylize_range(style, (4, 4), (4, 22))
console.print(syntax)

This gives us the final result we are looking for, we can easily see what is being targeted here. In this case the function name register_pipelines.

This turns out to be exacly what I am looking for. Now I have an easy way to print out highlighted code wtih my py-tree-sitter query results.

Links #

Highlighting text ranges with Rich | python

2022-08-24T20:06:34Z

{% for year in markata.map(“date.year”, filter=‘published’)|unique %}

{{ year }} #

{% for post in markata.map(‘post’, filter=“published and date.year == “+year|string, sort=‘date’) %}

[{{ post.title }} - {{ post.date.month }}/{{ post.date.day }}](/{{ post.slug }}) {% endfor %} {% endfor %}

markata 0.3.0 is 15-20% faster

2022-08-21T13:53:20Z

a sprinter edging out his opponent by Dall-e

It’s about time to release Markata 0.3.0. I’ve had 8 pre-releases since the last release, but more importantly it has about 3 months of updates. Many of which are just cleaning up bad practices that were showing up as hot spots on my pyinstrument reports

Markata started off partly as a python developer frustrated with using nodejs for everything, and a desire to learn how to make frameworks in pluggy. Little did I know how flexible pluggy would make it. It started out just as my blog generator, but has turned into quite a bit more.

Over time this side project has grown some warts and some of them were now becoming a big enough issue it was time to cut them out.

Let’s compare #

I like to use my tils articles for examples and tests like this as there are enough articles for a good test, but they are pretty short and quick to render.

mkdir ~/git/tils/tils
cp  ~/git/waylonwalker.com/pages/til/ ~/tils/tils -r
cd ~/git/tils/tils

running tils on 0.2.0 #

At the time of writing this is the current version of markata, so just make a new venv and run it.

python3 -m venv .venv --prompt $(basename $PWD)
pip install markata
markata clean
markata build

cold tils: 14.523 warm tils: 1.028

running tils on 0.3.0b8 #

python3 -m venv .venv --prompt $(basename $PWD)
# --pre installs pre-releases that include a b in their version name
pip install markata --pre
markata clean
markata build

cold tils: 11.551 (+20%) warm tils: 0.860 (+16%)

pyinstrument #

These measurements were taken with pyinstrument mostly out of convenience since there is already a pyinstrument hook built in, but also because I like pyinstrument.

Here is the pyinstrument report from the last run.

My Machine #

This comparison was not very exhaustive. It was ran on my pretty new to me Ryzen 5 3600 machine.

The changes #

Most of these changes revolve in how the lifecycle is ran. It was trying to be extra cautious and run previous steps for you if it thought it might be needes, in reality it was rerunning a few steps multiple times no matter what.

The other thing I turned off by default, but can be opted into, is beautifulasoup’s prettify. That was one of the slower steps ran on my site.

0.3.0 #

It should be out by the time you see this, I wanted to compare the changes I had made and make sure that it was still making forward progress and thought I would share the results.

How to Disable Kedro Viz | or any kedro hook

2022-08-08T21:01:03Z

Gpu Stats in my linux statusbar

2022-08-03T21:25:56Z

Running My Blog on 3.11-dev

2022-07-29T16:33:06Z

3.10.5: 109.441 3.11-dev: 108.856

get the version of any python package | python api

2022-07-29T15:48:36Z

A common meta thing that I need in python is to find the version of a package. Most of the time I reach for package_name.__version__, but that does not always work.

but not all projects have a `version` #

In searching the internet for an answer nearly every one of them pointed me to __version__. This works for most projects, but is simply a convention, its not required. Not all projects implement a __version__, but most do. I’ve never seen it lie to me, but there is nothing stopping someone from shipping mismatched versions.

If you maintain a project ship a `version` #

I appreciate it

While its not required its super handy and easy for anyone to remember off the top of their head. It makes it easy to start debugging differences between what you have vs what you see somewhere else. You can do this by dropping a __version__ variable inside your __init__.py file.

## __init__.py
__version__ = 1.0.0

SO #

stack overflow saves the day

Special thanks to this Stack Overflow post for answering this question for me.

So what do you do… #

importlib

Your next option is to reach into the package metadata of the package that you are interested in, and this has changed over time as highlighted in the stack overflow post.

for Python >= 3.8:

from importlib.metadata import version

version('markata')
# `0.3.0.b4`

I only really use python >= 3.8 these days, but if you need to implement it for an older version check out the stack overflow post.

Another option.. #

use the command line

Another common option uses pip at the command line.

❯ pip show markata
Name: markata
Version: 0.3.0b4
Summary: Static site generator plugins all the way down.
Home-page: https://markata.dev
Author: Waylon Walker
Author-email: waylon@waylonwalker.com
License: MIT
Location: /home/waylon/git/waylonwalker.com/.venv/lib/python3.11/site-packages
Requires: anyconfig, beautifulsoup4, checksumdir, diskcache, feedgen, jinja2, more-itertools, pathspec, pillow, pluggy, pymdown-extensions, python-frontmatter, pytz, rich, textual, toml, typer
Required-by:

And if the package implements a command line its common to ship a version command such as --version or -V.

❯ markata --version
Markata CLI Version: 0.3.0.b4

Why did I need to do this? #

Well we have a cli tool that wraps around piptools and we wanted to include the version of piptools in the comments that it produces dynamically. This is why I wanted to dynamically grab the version inside python without shelling out to pip show. Now along with the version of our internal tool you will get the version of piptools even though piptools does not ship a __version__ variable.

Fin #

In the end, I am glad I learned that its so easy to use the more accurate package metadata, but still appreciate packages shipping __version__ for all of us n00b’s out here.

Manage your displays with xrandr manager

2022-07-16T08:43:17Z

xrandr is a great cli to manage your windows in a linux distro using x11, which is most of them. The issue is that I can never remember all the flags to the command, and if you are using it with something like a laptop using a dock the names of all the displays tend to change every time you redock. This makes it really hard to make scripts that work right every time.

Homepage #

Check out the deresmos/xrandr-manager for more details on it.

installation #

xrander-manager is a python cli application that is simply a nice interface into xrandr. So you must have xrandr already installed, which is generally just there on any x11 window manager, I’ve never had to install it.

As with any python cli that is indended to be used as a global/system level cli application I always install them with pipx. This automates the process of creating a virtual environment for xrandr-manager for me, and does not clutter up my system packages with its dependencies that may eventually clash with another that I want to use.

# prereqs (xrandr, pipx)
pipx install xrandr-manager

set main monitor #

First if your main display is not set to the correct monitor set your main display first.

xrandr-manager -m HDMI-0
xrandr-manager -m DP-0

prompt mode #

If you dont know the name of your monitors and and don’t want to dig through xrandr, you can just run --prompt and tab complete to fill set your main display.

xrandr-manager --prompt

direction #

This is what I most often use xrandr-manager for. Once you have the main display set you can tell it where to put the other monitor. I’ve only tried this with two monitors, I have no idea what happens with more monitors.

xrandr-manager -d right
xrandr-manager -d left
xrandr-manager -d above
xrandr-manager -d below

mirror #

One thing that I always need to jump through hoops to do is mirror. Occasionally I want to mirror so that more people can see the screen while we are split screen gaming. This has seemed like a pain in any other xrandr utility, but trivial in xrandr-manager.

xrandr-manager --mirror

It logs out the xrandr command #

One nice thing about xrandr-manager is that it echos out the xrandr command that it’s running. This is nice because you can toss this behind a hotkey or an init script.

Guis #

Ya there are guis that do this. I’ve had good luck with arandr. It’s more intuitive to drag windows around like what you would do in windows. Every once in awhile it messes up and my polybar overlaps my windows, or my windows end up only on half the screen.

There are also graphics card specific utilities, Ive used nvidia x server settings and it mostly works similar to arandr.

IPython q to exit

2022-06-13T10:10:34Z

So many terminal applications bind q to exit, even the python debugger, its muscle memory for me. But to exit ipython I have to type out exit<ENTER>. This is fine, but since q is muscle memory for me I get this error a few times per day.

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-1-2b66fd261ee5>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
NameError: name 'q' is not defined

After digging way too deep into how IPython implements its ExitAutoCall I realized there was a very simple solution here. IPython automatically imports all the scripts you put in your profile directory, all I needed was to create ~/.ipython/profile_default/startup/q.py with the following.

q = exit

It was that simple. This is not a game changer by any means, but I will now see one less error in my workflow. I just press q<Enter> and I am out, without error.

Copier Slugify | python templating | using cookiecutter

2022-06-11T12:37:50Z

It’s no secret that I love automation, and lately my templating framework of choice has been copier. One hiccup I recently ran into was having spaces in my templated directory names. This makes it harder to run commands against as you need to escape them, and if they end up in a url you end up with ugly %20 all over.

Cookiecutter has the solution #

Yes the solution comes from a competing templating framework.

I install copier with pipx, so I need to inject cookiecutter in to my copier environment to use the slugify filter.

pipx inject copier cookiecutter

If you are using a normal virtual environment you can just pip install it.

pip install copier cookiecutter

add the extension to your template #

copier.yml

Now to enable the extension you need to declare it in your copier.yml file in your template.

_jinja_extensions:
    - cookiecutter.extensions.SlugifyExtension

Use it | slugify #

use-it

Now to use it, anywhere that you want to slugify a variable, you just pipe it into slugify.

❯ tree .
.
├── copier.yml
├── README.md
└── {{ site_name|slugify }}
    └── markata.toml.jinja

1 directory, 3 files

Here is a slimmed down version of what the copier.yml looks like.

site_name:
  type: str
  help: What is the name of your site, this shows in seo description and the site title.
  default: Din Djarin

_jinja_extensions:
    - cookiecutter.extensions.SlugifyExtension

Results #

Running the template looks a bit like this.

straight from their docs #

The next section is straight from the cookiecutter docs

Slugify extension #

The cookiecutter.extensions.SlugifyExtension extension provides a slugify filter in templates that converts string into its dashed (“slugified”) version:

{% "It's a random version" | slugify %}

Would output:

it-s-a-random-version

It is different from a mere replace of spaces since it also treats some special characters differently such as ' in the example above. The function accepts all arguments that can be passed to the slugify function of python-slugify_. For example to change the output from it-s-a-random-version to it_s_a_random_version, the separator parameter would be passed: slugify(separator='_').

Textual has devtools

2022-06-09T21:21:11Z

Textual has devtools in the upcoming css branch, and its pretty awesome!

It’s still early #

Textual is still very early and not really ready for prime time, but it’s quite amazing how easy some things such as creating keybindings is. The docs are coming, but missing right now so if you want to use textual be ready for reading source code and examples.

On to the devtools #

As @willmcgugan shows in this tweet it’s pretty easy to setup, it requires having two terminals open, or using tmux, and currently you have to use the css branch.

https://twitter.com/willmcgugan/status/1531294412696956930

Why does textual need its own devtools #

Textual is a tui application framework. Unlike when you are building cli applications, when the tui takes over the terminal in full screen there is no where to print statement debug, and breakpoints don’t work.

getting the css branch #

In the future it will likely be in main and not need this, but for now you need to get the css branch to get devtools.

git clone https://github.com/Textualize/textual
git fetch --alll
git checkout css

install in a virtual environment #

Now you can create a virtual environment, feel free to use whatever virtual environment tool you want, venv is built in to most python distributions though, and should just be there.

python3 -m venv .venv --prompt textual
source .venv/bin/activate
pip install .

Now that we have textual installed #

Once textual is installed you can open up the devtools by running textual console.

textual console

Python atexit

2022-06-02T23:45:55Z

totally guessed at this post’s date

I’m still trying to understand this one, but this is how you force a python object to stop atexit.

import atexit

class Server:
    def __init__(
        self,
        auto_restart: bool = True,
        directory: Union[str, "Path"] = None,
        port: int = 8000,
    ):
        if directory is None:
            from markata import Markata

            m = Markata()
            directory = m.config["output_dir"]

        self.directory = directory
        self.port = find_port(port=port)
        self.start_server()
        atexit.register(self.kill)

    def start_server(self):
        import subprocess

        self.cmd = [
            "python",
            "-m",
            "http.server",
            str(self.port),
            "--directory",
            self.directory,
        ]

        self.proc = subprocess.Popen(
            self.cmd,
            stderr=subprocess.PIPE,
            stdout=subprocess.PIPE,
        )
        self.start_time = time.time()


    def kill(self):
        self.auto_restart = False
        self.proc.kill()

    def __rich__(self) -> Panel:
        if not self.proc.poll():
            return Panel(
                f"[green]serving on port: [gold1]{self.port} [green]using pid: [gold1]{self.proc.pid} [green]uptime: [gold1]{self.uptime} [green]link: [gold1] http://localhost:{self.port}[/]",
                border_style="blue",
                title="server",
            )

        else:
            if self.auto_restart:
                self.start_server()

            return Panel(f"[red]server died", title="server", border_style="red")

The one pip config you need to have

2022-06-01T10:31:09Z

Whenever you are installing python packages, you should always use a virtual environment. pip makes this easy to follow by adding some configuration to pip.

require-virtualenv #

Pip is the pacakage tool for python. It installs third-party packages and is configurable. One of the configuration settings that I highly reccommend everyone to add is require-virtualenv. This will stop pip from installing any packages if you have not activated a virtualenv.

why #

python packages often require many different dependencies, sometimes packages are up to date and sometimes they require different versions of dependencies. If you install everything in one environment its easy to end up with version conflict issues that are really hard to resolve, especially since your system environment cannot easily be restarted.

PIPX my one exception #

My one exception that I put in my system level packages is pipx. pipx is very handy as it manages virtual environments for you and is intended for command line utilities that would end up in your system env or require you to manually manage virtual environments without it.

pip config #

Your pip config might be found in either ~/.pip/pip.conf or ~/.config/pip/pip.conf. You can either use the pip config set command or edit one of these files manually.

pip config set global.require-virtualenv True

Now you sould see this in your ~/.config/pip/pip.conf

[global]
require-virtualenv = True

pip config debug #

If you want to know where pip is looking for configuration on your system, and what files are setting a certain config you can use pip config debug to find it.

❯ pip config debug

env_var:
env:
global:
  /etc/xdg/xdg-awesome/pip/pip.conf, exists: False
  /etc/xdg/pip/pip.conf, exists: False
  /etc/pip.conf, exists: False
site:
  /home/waylon/git/waylonwalker.com/.venv/pip.conf, exists: False
user:
  /home/waylon/.pip/pip.conf, exists: False
  /home/waylon/.config/pip/pip.conf, exists: True
    global.require-virtualenv: True

saved my bacon #

This setting recently saved me when I modified my .envrc file my virtual environment deactivated, so when I went to pip install something it gave me an error that it was not active. Situations like this are an easy way to pollute your system with packages that it does not need installed.

TLDR #

Run this at your command line to avoid polluting your system environment by mistake before running any pip command.

pip config set global.require-virtualenv True

pyenv no module named '_sqlite3'

2022-05-31T14:18:52Z

I’ve been trying to adopt pyenv for a few months, but have been completely blocked by this issue on one of the main machines I use. Whenever I start up ipython I get the following error.

ImportError: No module named '_sqlite3

I talked about why and how to use pyenv along with my first impressions in this post

pyenv/issues/678 #

According to #678 I need to install libsqlite3-dev on ubuntu to resolve this issue.

install libsqlite3-dev #

libsqlite3-dev can be installed using apt

sudo apt install libsqlite3-dev

But wait…. #

When I make a fresh env and install ipython I still get the same error and I am still not able to use ipython with pyenv.

ImportError: No module named '_sqlite3

re-install python #

After having this issue for awhile an coming back to #678 several times I realized that libsqlite3-dev needs to be installed while during install.

pyenv install 3.8.13

I think I had tried this several times, but was missing the -y option each time. You gotta read errors like this, I am really good at glossing over them.

Let’s never have this issue again. #

When you spend months living with little errors like this and finally fix it, its good to make sure that it never happens again. Whenever I start a new ubuntu machine I run an ansible playbook that does all the setup for me. I added libsqlite3-dev to my core install in 64c85ca now it will be on all of my machines and not break again.

Using Different versions of python with pipx | pyenv

2022-05-28T20:38:42Z

I love using pipx for automatic virtual environment management of my globally installed python cli applications, but sometimes the application is not compatible with your globally installed pipx

Which version of python is `pipx` using?? #

This one took me a minute to figure out at first, please let me know if there is a better way. I am pretty certain that this is not the ideal way, but it works.

My first technique was to make a package that printed out sys.version.

# what version of python does the global pipx use?
pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

# what version of python does the local pipx use?
python -m pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

Let’s setup some other versions of python with pyenv #

If you don’t already have pyenv installed, you can follow their install instructions to get it.

pyenv install 3.8.13
pyenv install 3.10.5

I usually require a virtual environment #

I set the PIP_REQUIRE_VIRTUALENV environment variable to true to ensure that the virtual environment is activated when pip installing, this makes it so that I can’t accidentally use the global env, which is typically not what I want to do.

export PIP_REQUIRE_VIRTUALENV=true
# for windows users
set PIP_REQUIRE_VIRTUALENV=true

This goes right into my shell startup script ~/.bashrc or ~/.zshrc.

Exceptions happen #

This is my one exception. I’ve had better luck putting pipx right in the global python environment. Not the system python, but each python version that I install with pyenv.

export PIP_REQUIRE_VIRTUALENV=false
# for windows users
set PIP_REQUIRE_VIRTUALENV=false

Let’s install pipx #

First up is python 3.10.5

pyenv global 3.10.5
pip install pipx
pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0]

Next is python 3.8.13j

pyenv global 3.8.13
pip install pipx
pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

3.8.13 (default, Aug  8 2022, 21:06:56)
[GCC 12.1.0]

Now once I close this shell I will always end up with PIP_REQUIRE_VIRTUALENV=true, since it’s in my shell startup script. So make sure that you reset it or kill this shell before doing any damage.

LIVE-REPLAY - Python dev | Markata todoui | 4/6/2022

2022-05-12T19:28:20Z

https://youtu.be/-42A5210HYo

Super fun steam Broadcasted live on Twitch – Watch live at https://www.twitch.tv/waylonwalker

We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow.

During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off.

dotfiles: https://github.com/WaylonWalker/devtainer
today’s project: https://github.com/WaylonWalker/markata-todoui
website: https://waylonwalker.com/

qutebrowser clean up all status bars

2022-05-11T22:03:30Z

I really like the super clean look of no status menus, no url bar, no bookmarks bar, nothing. Don’t get me wrong these things are useful, but honestly they take up screen real estate and I RARELY look at them. What I really want is a toggle hotkey. I found this one from one of DT’s youtube video’s. I can now tap xx and both the status bar at the botton and the address bar at the top disappear.

# ~/.config/qutebrowser/config.py

config.bind("xb", "config-cycle statusbar.show always never")
config.bind("xt", "config-cycle tabs.show always never")
config.bind(
    "xx",
    "config-cycle statusbar.show always never;; config-cycle tabs.show always never",
)

convert your qutebrowser config to config.py

2022-05-11T07:16:34Z

When you first start qutebrowser It will create some config files in your home directory for you, but they will be empty.

Config #

As far as I know qutebrowser will create this default config out of the box for you, if it doesn’t, then somehow it just appeared for me 😁.

❯ tree ~/.config/qutebrowser
/home/waylon/.config/qutebrowser
├── autoconfig.yml
├── bookmarks
│   └── urls
├── config.py
├── greasemonkey
└── quickmarks

2 directories, 5 files

Why convert #

You might want to confvert if you are more comfortable with the python config, or if like me you just want config in one place and you are stealing configuration options from others who have thiers in config.py.

Convert to py #

List all git commits with GitPython

2022-05-09T21:24:12Z

I am getting ready to do some timeseries analysis on a git repo with python, my first step is to figure out a way to list all of the git commits so that I can analyze each one however I want. The GitPython library made this almost trivial once I realized how.

from git import Repo

repo = Repo('.')
commits = repo.iter_commits()

This returns a generator, if you are iterating over them this is likely what you want.

commits
# <generator object Commit._iter_from_process_or_stream at 0x7f3307584510>

The generator will return git.Commit objects with lots of information about each commit such as hexsha, author, commited_datetime, gpgsig, and message.

next(commits)
# <git.Commit "d125317892d0fab10a36638a2d23356ba25c5621">

Using Git from Python

2022-04-30T20:23:23Z

GitPython is a python api for your git repos, it can be quite handy when you need to work with git from python.

Use Case #

I recently made myself a handy tool for making screenshots in python and it need to do a git commit and push from within the script. For this I reached for GitPython.

How I Quickly Capture Screenshots directly into My Blog

Installation #

GitPython is a python library hosted on pypi that we will want to install into our virtual environments using pip.

pip install GitPython

Create a Repo Object #

Import Repo from the git library and create an instance of the Repo object by giving it a path to the directory containing your .git directory.

from git import Repo
repo = Repo('~/git/waylonwalker.com/')

Two interfaces #

from the docs

It provides abstractions of git objects for easy access of repository data, and additionally allows you to access the git repository more directly using either a pure python implementation, or the faster, but more resource intensive git command implementation.

I only needed to use the more intensive but familar to me git command implementation to get me project off the ground. There is a good tutorial to get you started with their pure python implementation in their docs.

Status #

Requesting the git status can be done as follows.

note I have prefixed my commands with »> to distinguish between the command I entered and the output.

>>> print(repo.git.status())

On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        blog/

You can even pass in flags that you would pass into the cli.

>>> print(repo.git.status("-s"))
?? blog/

log #

Example of using the log.

print(repo.git.log('--oneline', '--graph'))

* 0d28bd8 fix broken image link
* 3573928 wip screenshot-to-blog
* fed9abc wip screenshot-to-blog
* d383780 update for wsl2
* ad72b14 wip screenshot-to-blog
* 144c2f3 gratitude-180

Find Deleted Files #

We can even do things like find all files that have been deleted and the hash they were deleted.

print(repo.git.log('--diff-filter', 'D', '--name-only', '--pretty=format:"%h"'))

git find deleted files

full post on finding deleted files

My Experience #

This library seemed pretty straightforward and predicatable once I realized there were two main implementations and that I would already be familar with the more intensive git command implementation.

How I Quickly Capture Screenshots directly into My Blog

2022-04-30T14:05:49Z

When I am creating blog posts it’s often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content.

Making Screenshots #

When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog.

take screenshot
optimize
conversion
publish
create img tag

Created in 🐍Python #

I created this tool for myself in python because that is what I am most familiar with, but realistically most of what I am calling are shell scripts that I could do in just about any language.

Install my screenshot tool #

My screenshot tool is quite hacky and not configurable, but works wonderfully for me. If you like it and want to use it, make it configurable or fork it. For now it lives on GitHub and since it has a setup.py with entrypoints we can install it with pipx.

pipx install git+https://github.com/WaylonWalker/screenshots.waylonwalker.com

This is just a tool for me, it does not need to be in a package manager like pip.

calling screenshot #

Now that screenshot is installed we can call it and make a screenshot. I’ll take a screenshot of the frontmatter of this exact post.

I have this tool exposed as a command that can be ran in the command line by calling screenshot. I will rarely use it this way, but makes it easy to create a hotkey for later.

Success #

Once the screenshot is successful, I get a notify-send message popup telling me so.

xbindkeys #

Let’s make this workflow just a bit smoother. I want a desktop hotkey that I can press at any time without opening a split or making sure I’m in zsh or vim, I want it always there. For hotkeys like this I use the Desktop manager agnostic keybinding tool xbindkeys. I can add the screenshot command and the corresponding keybinding I want to my ~/.xbindkeysrc file and restart with xbindkeys -f ~/.xbindkeysrc.

"screenshot"
    Shift + Mod4 + alt + p

Now when I press Shift + Mod4 + alt + p I am presented with a little zenity box asking me what I want to name my screenshot, followed by flameshot.

xbindkeys allows me to bind this hotkey to my system so that no matter where I am it will work.

Name it #

The one question I ask myself when creating the hotkey is for a filename. On my ubuntu machine I do that with a simple gui application called zenity. It looks like this when I open it up.

Under the hood my screenshot tool is running the following command in a subprocess.

zenity --entry --text="filename"

Copier < 6.0.0b0 considered dangerous

2022-04-30T10:05:45Z

Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories.

!!UPDATE #

As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update.

This is a PSA #

I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I do so that they know the dangers of using copier inside an existing directory.

The issue #

The fix #

https://github.com/copier-org/copier/pull/273

As of the time of writing this version is still in beta, if you still want to use copier with existing directtories, I’d strongly encourage you to install the --pre release.

pipx install copier --pip-args='--pre'

confirm #

copier --version
# copier 6.0.0b0

My update commit #

https://github.com/WaylonWalker/devtainer/commit/a06e47462e2c6d26ad69fbdc2e7990a1a73ab4b7

pipx on windows

2022-04-18T14:03:38Z

Python, click install

Edit the System Environment Variables

Environment Variables button

Add the following path to your users Path Variable

C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\dotnet\;C:\Users\quadm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts;

PyOhio CFP's

2022-04-12T12:47:52Z

Here are some CFP’s that I used for PyOhio 2022.

https://pretalx.com/pyohio-2022/cfp

Idea to blog post in minutes - Shorter #

Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it’s plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle.

Lets build a whole blog site in 5 minutes.

Add Kedro to your Pandas Workflow - Short #

Sometimes python scripts/notebooks take a long time to run, let kedro automatically save your datasets so that you can maintain your production code with ease. Lets take a pipeline with an issue 30 minutes in and solve the issue in 5 mintues.

Python sys.excepthook

2022-04-10T14:21:54Z

Sometimes you just want python to do something else when you hit an exception, maybe that’s fire a text, slack message, email, or system notification like I wanted.

I am working on a quick and dirty python script designed to take screenshots and land them on my website in a single hotkey. With it being designed to run with a hotkey, if it were to error I would not see it.

I could have gone down a logging route, but honestly this is meant to be quick, dirty, and work on my system for me. I just want to get it in my system notification.

sys.excepthook #

Python exposes sys.excepthook for just this case. Here is what I ended up doing to fire a system notification as well as printing the message. Yaya a log would be mroe appropriate, but this is designed to just get done quick and do the job I want it to do.

def notify_exception(type, value, tb):
    traceback_details = "\n".join(traceback.extract_tb(tb).format())

    msg = f"caller: {' '.join(sys.argv)}\n{type}: {value}\n{traceback_details}"
    print(msg)
    Popen(
        f'notify-send "screenshot.py hit an exception" "{msg}" -a screenshot.py',
        shell=True,
    )


sys.excepthook = notify_exception
0 / 0

pygame events are queued | Don't make this mistake

2022-04-07T13:14:20Z

pygame events are stored in a queue, by default the most suggested way shown in all tutorials “pumps” the queue, which removes all the messages.

start up pygame #

You don’t necessarily need a full boilerplate to start looking at events, you just just need to pygame.init() and to capture any keystrokes you need a window to capture them on, so you will need a display running.

import pygame
pygame.init()
pygame.display.set_mode((854, 480))

get some events #

Let’s use pygames normal event.get method to get events.

events = pygame.event.get()

printing the events reveal this

[
    <Event(1541-JoyDeviceAdded {'device_index': 0, 'guid': '030000005e0400008e02000010010000'})>,
    <Event(4352-AudioDeviceAdded {'which': 0, 'iscapture': 0})>,
    <Event(4352-AudioDeviceAdded {'which': 1, 'iscapture': 0})>,
    <Event(4352-AudioDeviceAdded {'which': 2, 'iscapture': 0})>,
    <Event(4352-AudioDeviceAdded {'which': 0, 'iscapture': 1})>,
    <Event(4352-AudioDeviceAdded {'which': 1, 'iscapture': 1})>,
    <Event(32774-WindowShown {'window': None})>,
    <Event(32777-WindowMoved {'x': 535, 'y': 302, 'window': None})>,
    <Event(32770-VideoExpose {})>,
    <Event(32776-WindowExposed {'window': None})>,
    <Event(32788-WindowTakeFocus {'window': None})>,
    <Event(32768-ActiveEvent {'gain': 1, 'state': 1})>,
    <Event(32785-WindowFocusGained {'window': None})>,
    <Event(768-KeyDown {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(771-TextInput {'text': 'a', 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(768-KeyDown {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(771-TextInput {'text': 'a', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(771-TextInput {'text': 'a', 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': '', 'key': 1073742051, 'mod': 1024, 'scancode': 227, 'window': None})>,
    <Event(772-Unknown {})>,
    <Event(769-KeyUp {'unicode': '', 'key': 1073742051, 'mod': 0, 'scancode': 227, 'window': None})>,
    <Event(32768-ActiveEvent {'gain': 0, 'state': 1})>,
    <Event(32786-WindowFocusLost {'window': None})>,
    <Event(772-Unknown {})>
]

Lets get some more events. #

Let’s say that we have multpile sprites all asking for the events from different places in our game. If we assume that our game loop runs very fastand we get events one after another the second one will have no events.

events_one = pygame.event.get()
events_two = pygame.event.get()

printing the events reveals that there are no events, well i

[]

Making things more maddening #

Even simple games don’t quite run infinitely fast there is some delay, with this delay most events will go to event_one, while any that occur in the short time between event_one and two will be in event_two’s queue.

import time
events_one = pygame.event.get()
time.sleep(.05) # simulating some delay that would naturally occur
events_two = pygame.event.get()

How to Resolve this #

Store events for each frame in memory.

Pump #

I thought pump=False would be the answer I was looking for, but I was proven wrong. It does not behave intuitivly to me.

events_one = pygame.event.get(pump=False) # all events since last pump
events_two = pygame.event.get(pump=False) # no events
events_three = pygame.event.get() # all events since last pump

events_one and events_three will have a list of events, while events_two will be empty. It seems that pump=False leaves the events there for the next event.get(), but appears cleared to any event.get(pump=False).

Keep a Game State #

If you want objects to do their own event handling, outside of the main game, you will need to give them some game state with the events in it. However you decide, you may only call event.get() once per game loop otherwise weird things will happen.

Display Sprites in Pygame | Load and Blit

2022-04-06T15:07:15Z

One of the most essential concepts of pygame to start making a game you will need to understand is loading images and blitting them to the screen.

blit stands for block image transfer, to me it feels a lot like layering up layers/images in photoshop or Gimp.

Loading an image #

I started by making a spotlight in Gimp, by opening a 64x64 pixel image and painting the center with a very soft brush.

This is what it looks like

Now we can load this into pygame.

import pygame
img = pygame.image.load("assets/spotlight.png")

Converting to the pygame colorspace #

To make pygame a bit more efficient we can convert the image to pygames colorspace once when we load it rather than every time we blit it onto another surface.

import pygame

# convert full opaque images
img = pygame.image.load("assets/spotlight.png").convert()

# convert pngs with transparancy
img = pygame.image.load("assets/spotlight.png").convert_alpha()

blitting #

To display the image onto the screen we need to use the blit method which needs at least two arguments, something to blit and a position to blit it at.

screen = pygame.display.set_mode(self.screen_size)
screen.blit( img, (0, 0),)

note blitting to the position (0, 0) will align the top left corners of the object we are blitting onto (screen) and the object we are blitting (img).

Starter #

Now we need an actual game running to be able to put on the screen. I am using my own starter/boilerplate, if you want to follow along you can install it from github into your own virtual environment.

pip install git+https://github.com/WaylonWalker/pygame-starter

Pygame Boilerplate Apr 2022

You can read more about my starter in this post

Let’s place this image right in the middle #

Now we can use the starter to create a new game, and with just a bit of offset we can put the spotlight directly in the middle.

import pygame
from pygame_starter import Game


class MyGame(Game):
    def __init__(self):
        super().__init__()
        # load in the image one time and store it inside the object instance
        self.img = pygame.image.load("assets/spotlight.png").convert_alpha()
    def game(self):
        # fill the screen with aqua
        self.screen.fill((128, 255, 255))
        # transfer the image to the middle of the screen
        self.screen.blit(
            self.img,
            (
                self.screen_size[0] / 2 - self.img.get_size()[0],
                self.screen_size[1] / 2 - self.img.get_size()[1],
            ),
        )


if __name__ == "__main__":
    game = MyGame()
    game.run()

If we save this as load_and_blit.py we can run it at the command like as so.

python load_and_blit.py

And we should get the following results.

the results of putting the image in the middle

convert a transparent png #

What happens when we accidently use .convert() rather than .convert_alpha()?

Making snow #

A common concept in pygame, that is built into my starter, is that you typically want to reset the screen each and every frame. Building on this with our new concept of blitting spotlights onto the screen we can make a random noise of snow by blitting a bunch of images to the screen.

import random

import pygame
from pygame_starter import Game


class MyGame(Game):

    def __init__(self):
        super().__init__()
        self.img = pygame.image.load("assets/spotlight.png").convert_alpha()

    def game(self):
        self.screen.fill((128, 255, 255))
        for  in range(100):
            self.screen.blit(
                self.img,
                (
                    random.randint(0, self.screen_size[0]) - self.img.get_size()[0],
                    random.randint(0, self.screen_size[1]) - self.img.get_size()[1],
                ),
            )


if __name__ == "__main__":
    game = MyGame()
    game.run()

the results #

Sorry, your browser doesn't support embedded videos.

Dunk is my new diff pager

2022-04-04T15:47:00Z

Dunk is a beautiful git diff tool built on top of rich.

Browsing through twitter the other day I discovered it through this tweet by _darrenburns.

https://twitter.com/_darrenburns/status/1510350016623394817

Dunk is beta #

Before I dive in deep, I do want to mention that Dunk is super new and beta at this point. I am making it my default pager, because I know what I am doing and can quickly shift back if I need to, no sweat. If you are a little less comfortable with the command line, terminal, or reading any issues that might come up, it might be best if you just pipe into Dunk when you want to use it.

The author really cautions the use of it as your default pager this early, I’m just showing that it’s possible, and I’m trying it.

He notes that it might have some issues especially with partially staged files.

try it #

You can try it with pipx.

git diff | pipx run dunk

install it #

If you like it, you can install it with pip or pipx, I prefer pipx for cli applications like this.

pipx install dunk

You can configure dunk as your default pager with the command line, or by editing your .gitconfig file.

git config --global pager.diff "dunk | less -R`

[pager]
    diff = dunk | less -R

As pointed out by _darrenburns dunk is not a pager and you can gain back all of the benefits of using a pager by piping into less with the -R flag.

reset it if you don’t like it #

You can --unset your pager configuration from the command line or edit your .gitconfig file by hand to remove the lines shown above.

git config --global --unset pager.diff

Comparison #

I have some edits to a game my son and I are working on unstaged so I ran git diff on that project with and without dunk to compare the differences.

Dunk just looks so good.

Always install #

If you follow along here often you know that I am a big fan of installing all my tools in an ansible playbook so that I don’t suffer configuring a new machine for months after getting it and wondering why its not exactly like the last.

# Dunk - prettier git diffs
# https://github.com/darrenburns/dunk
- name: check is dunk installed
  shell: command -v black
  register: dunk_exists
  ignore_errors: yes

- name: install dunk
  when: dunk_exists is failed
  shell: pipx install dunk

[[ ansible-install-if-not-callable ]]

More on conditionally installing tools with ansible in this post.

Pygame Boilerplate Apr 2022

2022-04-03T16:32:40Z

I’m poking a bit into gamedev. Partly to better understand, partly because it’s stretching different parts of my brain/skillset than writing data pipelines does, but mostly for the experience of designing them with my 9yo Wyatt.

pygame boilerplates #

I’ve seen several pygame boilerplate templates, but they all seem to rely heavily on globl variables. That’s just not how I generally develop anything. I want a package that I can pip install, run, import, test, all the good stuff.

My current starter #

What currently have is a single module starter package that is on github so that I can install it and start building games with very little code.

Installation #

Since it’s a package on GitHub you can install it with the git+ prefix.

pip install git+https://github.com/WaylonWalker/pygame-starter

Example Game #

You can make a quick game by inheriting from Game, and calling .run(). This example just fills the screen with an aqua color, but you can put all of your game logic in the game method.

from pygame_starer import Game

class MyGame(Game):
    def game(self):
        self.screen.fill((128, 255, 255))

if __name__ == "__main__":
    game = MyGame()
    game.run()

The starter #

Here is what the current game.py looks like. I will probably update it as time goes on and I learn more about the things I want to do with it.

from typing import Tuple

import pygame


class Game:
    def __init__(
        self,
        screen_size: Tuple[int, int] = (854, 480),
        caption: str = "pygame-starter",
        tick_speed: int = 60,
    ):
        """

        screen_size (Tuple[int, int]): The size of the screen you want to use, defaults to 480p.
        caption (str): the name of the game that will appear in the title of the window, defaults to `pygame-starter`.
        tick_speed (int): the ideal clock speed of the game, defaults to 60

        ## Example Game

        You can make a quick game by inheriting from Game, and calling
        `.run()`.  This example just fills the screen with an aqua color, but
        you can put all of your game logic in the `game` method.

        ``` python
        from pygame_starer import Game

        class MyGame(Game):
            def game(self):
                self.screen.fill((128, 255, 255))

        if __name__ == "__main__":
            game = MyGame()
            game.run()

        ```
        """
        pygame.init()
        pygame.display.set_caption(caption)

        self.screen_size = screen_size
        self.screen = pygame.display.set_mode(self.screen_size)
        self.clock = pygame.time.Clock()
        self.tick_speed = tick_speed

        self.running = True
        self.surfs = []

    def should_quit(self):
        """check for pygame.QUIT event and exit"""
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                self.running = False

    def game(self):
        """
        This is where you put your game logic.

        """
        ...

    def reset_screen(self):
        """
        fill the screen with black
        """
        self.screen.fill((0, 0, 0))

    def update(self):
        """
        run one update cycle
        """
        self.should_quit()
        self.reset_screen()
        self.game()
        for surf in self.surfs:
            self.screen.blit(surf, (0, 0))
        pygame.display.update()
        self.clock.tick(self.tick_speed)

    def run(self):
        """
        run update at the specified tick_speed, until exit.
        """
        while self.running:
            self.update()
        pygame.quit()


if __name__ == "__main__":
    game = Game()
    game.run()

Did my site build just go down?

2022-04-01T16:25:49Z

My personal Site build went down last week, and I was unable to publish a new article. This is the process I went through to get it back up and running quickly.

Is it a fluke? #

Classic IT fix, rerun it and see if you get the same error. Everyone is busy and when you have your build go down you are probably busy doing something else. My first step is often to simply click rerun right from GitHub actions. Sometimes this will fix it, and sometimes it doesn’t. It’s an easy fix to run in the meantime you are not focused on fixing it.

Is GitHub having issues? #

Also worth a check to see if GitHub is having a hiccup or not. This error felt pretty obviously not GitHub’s fault, but it’s a good one to check when you run into a weird unexplainable error.

Check github status for any downtime issues with actions.

Build Down #

Alright down to the error message I got. The error is pretty obvious that somewhere I am trying to import a non-existing module from click.

Run markata build --no-pretty
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/bin/markata", line 33, in <module>
    sys.exit(load_entry_point('markata==0.1.0', 'console_scripts', 'markata')())
  File "/opt/hostedtoolcache/Python/3.8.12/x64/bin/markata", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markata/__init__.py", line 25, in <module>
    from markata.cli.plugins import Plugins
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markata/cli/__init__.py", line 1, in <module>
    from .cli import app, cli, make_layout, run_until_keyboard_interrupt
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markata/cli/cli.py", line 3, in <module>
    import typer
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/typer/__init__.py", line 12, in <module>
    from click.termui import get_terminal_size as get_terminal_size
ImportError: cannot import name 'get_terminal_size' from 'click.termui' (/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/click/termui.py)

Check pypi’s release date of click #

So the latest click was released just a few hours before this build. This feels like we are getting somewhere. Either click did a poor job of issuing deprecation warnings, or I was ignoring them in my build pipeline.

pin it and push #

let’s fix this build now

To get the build up and running today so that we don’t stop the flow of new posts I am going to open my requirements.txt file, and pin under the version that was just built.

click<8.1.0

Since I am still busy doing other things that fixing this, and am pretty confident that things were working before, I am just going to commit this and ship it.

watch ci #

Coming back to actions a few minutes later shows the site is building successfully without the same error as before. New posts will now be flowing to the site with the slightly older version of click.

looking for an issue #

Let’s make sure that the issue is going to be resolved. After not being busy and having time to investigate the issue, I can see that typer is the library making the import to get_terminal_size. Lets checkout its GitHub-repo and make sure someone is working on it.

By the time I go to the package that was having this issue there was already an issue up, and PR waiting approval. I gave the Issue a reaction 👍 to signal that I also care, and appreciate the issue author taking time to submit.

Don't inherit from python BaseException, Here's why.

2022-03-31T01:21:30Z

I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have.

Try running these examples in a .py file for yourself and try to kill them with control-c.

You cannot Keybard interrupt #

Since things such as KeyboardInterrupt are created as an exception that inherits from BaseException, if you except BaseException you can no longer KeyboardInterrupt.

from time import sleep

while True:
    try:
        sleep(30)
    except BaseException: # ❌
        pass

except from Exception or higher #

If you except from exception or something than inherits from it you will be better off, and avoid unintended side effects.

from time import sleep

while True:
    try:
        sleep(30)
    except Exception: # ✅
        pass

This goes with Custom Exceptions as well #

When you make custom exceptions expect that users, or your team members will want to catch them and try to handle them if they can. If you inherit from BaseException you will put them in a similar situation when they use your custom Exception.

class MyFancyException(BaseException): # ❌
    ...

class MyFancyException(Exception): # ✅
    ...

How I make cache-keys from python objects

2022-03-30T16:11:09Z

When I need a consistent key for a pythohn object I often reach for hashlib.md5 It works for me and the use cases I have.

diskcache #

Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object.

How I setup a sqlite cache in python

hash #

does not work

My first thought was to just hash the files, this will give me a unique key for each. This will work, and give you a consistant key for one and only one given python process. If you start a new interpreter you will get different keys.

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com)
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ hash("waylonwalker")
-3862245013515310359

waylonwalker ↪main v3.8.0 ipython
❯ hash("waylonwalker")
-3862245013515310359

waylonwalker ↪main v3.8.0 ipython
❯ exit

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com)
❯ ipython


waylonwalker ↪main v3.8.0 ipython
❯ hash("waylonwalker")
-83673051278873734

here is a snapshot of my terminal proving that you can get the same hash in one session, but it changes when you restart ipython.

hashlib.md5 #

Here is a quick couple ipython sessions showing that md5 cache is consistent accross multiple sessions.

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1)
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker")
[PYFLYBY] import hashlib
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-1-1537c4473c74>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Unicode-objects must be encoded before hashing

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8"))
<md5 HASH object @ 0x7fe4ba6832d0>

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
'1c7c1073ca096ffdb324471770911fe2'

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
'1c7c1073ca096ffdb324471770911fe2'

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
'1c7c1073ca096ffdb324471770911fe2'

waylonwalker ↪main v3.8.0 ipython
❯ exit


waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1) took 47s
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
[PYFLYBY] import hashlib
'1c7c1073ca096ffdb324471770911fe2'

key for diskcache #

Since it is consistent we can use it as a cache key for diskcache operations. I setup a little funciton that allows me to pass a bunch of differnt things in to cache. As long as the str method exists and is gives the data that you want to cache key on, this will work.

def make_hash(self, *keys: str) -> str:
    str_keys = [str(key) for key in keys]
    return hashlib.md5("".join(str_keys).encode("utf-8")).hexdigest()

understanding python *args and **kwargs

If the *args is confusing, I have a full article on *args and **kwargs.

See it in action #

Here you can see it in action. Anything passed into the function gets to be part of the key.

waylonwalker ↪main v3.8.0 ipython
❯ def make_hash(self, *keys: str) -> str:
...:     str_keys = [str(key) for key in keys]
...:     return hashlib.md5("".join(str_keys).encode("utf-8")).hexdigest()
...:

waylonwalker ↪main v3.8.0 ipython
❯ make_hash(1, "one", "1", 1.0)
'73901d019df012a1cdab826ce301217d'

waylonwalker ↪main v3.8.0 ipython
❯ exit


waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1) took 19m19s
❯

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1)
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ def make_hash(self, *keys: str) -> str:
...:     str_keys = [str(key) for key in keys]
...:     return hashlib.md5("".join(str_keys).encode("utf-8")).hexdigest()
[PYFLYBY] import hashlib

waylonwalker ↪main v3.8.0 ipython
❯ make_hash(1, "one", "1", 1.0)
'73901d019df012a1cdab826ce301217d'

How I setup a sqlite cache in python

2022-03-29T16:47:36Z

When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it.

install diskcache #

Install diskcache into your virtual environement of choice using pip from your command line.

python -m pip install diskcache

setup the cache #

There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs

from diskcache import Cache
cache = FanoutCache('.mycache', statistics=True)

Adding to the cache #

Adding to the cache only needs a key and value.

cache.add('me', 'waylonwalker' )

Set the expire time #

Optionally you can set the seconds before it expires. The cache invalidation tools like this is what really makes diskcache shine over using raw sqlite or any sort of static file.

cache.add('me', 'waylonwalker', expire=60)

tagging #

Diskcache supports tagging entries added to the cache.

# add an item to the cache with a tag
cache.add('me', 'waylonwalker', expire=60, tag='people')

This seems to let you do a few new things like getting items from the cache by both key and tag, or evict all tags from the cache.

# evict all items tagged as 'people' from the cache
cache.evict(tag='people')

Reading from the cache #

You can read from the cache by using the .get method and giving it the key you want to retrieve.

who = cache.get('me')
# who == 'waylonwalker'

Cache Misses #

Cache misses will return a None just like any dictionary .get miss.

missed = cache.get('missing')
# missed == None

⭐ #

Give Grant some love and give grantjenks/python-diskcache a ⭐.

Cache a python function with lru_cache

2022-03-28T01:21:30Z

The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache.

lru_cache #

The easiest and most common to setup in python is a builtin functools.lru_cache.

from functools import lru_cache

@lru_cache
def get_cars():
    print('pulling cars data')
    return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

when to use lru_cache #

Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache.

when same *args, **kwargs always return the same value

lru_cache only works for one python process. If you are running multiple subprocesses, or running the same script over and over, lru_cache will not work.

lru_cache only caches in a single python process

max_size #

lru_cache can take an optional parameter maxsize to set the size of your cache. By default its set to 128, if you want to store more or less items in your cache you can adjust this value.

The get_cars example is a bit of a unique one. As anthonywritescode points out this implementation is behaving like a singleton, and we can optimize the size of the cache by allocating exactly how many items we will ever have in it by setting its value to 1.

from functools import lru_cache

@lru_cache(maxsize=1)
def get_cars():
    print('pulling cars data')
    return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

My example stretches the rule a little bit #

The example above does a web request. As a Data Engineer I often write scripts that run for a short time then stop. I do not expect the output of this function to change during the runtime of this job, and if it did I may actually want them to match anyways.

web request do change their output

If I were building webapps, or some sort of process that was running for a long time. Something that starts and waits for work, this may not be a good application of lru_cache. If this process is running for days or months my assumption that the request does not change is no longer valid.

There’s also a typed kwarg for lru_cache #

This one is new to me but you can cache not only on the value, but the type of the value being passed into your function.

(from the docstring) If typed is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results.

Set User Agent on pandas read_csv

2022-03-27T01:21:30Z

I keep a small cars.csv on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate.

Hosts switched #

I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request.

Not my go to dataset 😭 #

This breaks my go to example dataset.

pd.read_csv("https://waylonwalker.com/cars.csv")

# HTTPError: HTTP Error 403: Forbidden

But requests works??? #

What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does.

requests.get("https://waylonwalker.com/cars.csv")

<Response [200]>

Setting the User Agent in pandas.read_csv #

this fixed the issue for me!

After a bit of googling I realize that this is a common thing, and that setting the user-agent fixes it. This is the point I remember seeing in the cloudflare dashbard that they protect against a lot of different attacks, aparantly it treats pd.read_csv as an attack on my cloudflare pages site.

pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

# success

Now my data is back #

Now this works again, but it feels like just a bit more effort than I want to do by hand. I might need to look into my cloudflare settings to see if I can allow this dataset to be accessed by pd.read_csv.

Get Webpage with python requests

2022-03-26T17:10:35Z

Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.

Install Requests #

Requests is on pypi and can be installed into your virtual environtment with pip.

python -m pip install requests

Getting the content of a request #

Requests makes getting content from a web url as easy as possible.

import requests

r = requests.get('https://waylonwalker.com/til/htmx-get/')
article = r.content

html">requests is not limited to html #

Requests can handle any web request and is not limited to only html. Here are some examples to get a markdown file, a csv, and a png image.

htmx_get_md = requests.get('https://waylonwalker.com/til/htmx-get.md').content
cars = requests.get('https://waylonwalker.com/cars.csv').content
profile = requests.get('https://images.waylonwalker.com/8bitc.png').content

RTFM #

There is way more to requests, this just scratches the surface while covering what you are going to need to get going. The requests docs have way more details.

Style Lookatme Slides a bit more Personal

2022-03-24T00:00:00Z

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique.

Slide One #

Lets use this section to show what it looks like as I change my styles.

from markata import Markata
Markata()
markata.run()

☝ This is how my website is built

write markdown
build site
publish

default #

This is what the above slide looks like in lookatme.

Set focus to the most important element #

The way I write my slides I want the most prominant element to be the slides title, not the presentation title. The slides title is generally the point I am trying to make, I will leave some supporting information if I want, but sometimes, I just have a title.

styles:
    title:
        bg: default
        fg: '#e1af66'
    headings:
        '1':
            bg: default
            fg: '#ff66c4,bold,italics'
            prefix: ' ⇁ '
            suffix: ' ↽ '

by default he prefix/suffix was a full block that just went transparant into the slide. I thought the harpoons were fun and went with them on a whim

The box characters bother me #

The box characters are fine really, but it really bothers me that they are not conneted. The author is probably doing this because it looks ok on most systems, and many terminals dont have their fonts right and wont align anyways. I am not sure if I ever had a windows terminal other than their new Terminal that properly connected box characters.

    quote:
        side: '│'
        style:
            bg: default
            fg: '#aaa'
        top_corner: '╭'
        bottom_corner: '╰'

Add Author #

Adding author to the root of the frontmatter of the document will add it to the bottom left of the slides.

author: '@_waylonwalker'

Style the author #

We can style the foreground and background of this text by adding something like this to the styles section of the frontmatter.

author:
    bg: default
    fg: '#368ce2'

While we are at it, lets style the rest of the footer to my own theme. Let’s pop this into the style and see what it looks like.

date:
    bg: default
    fg: '#368ce2'
slides:
    bg: default
    fg: '#368ce2'

reduce the padding #

When I am presenting I am punched in as big as I can go, and which makes the padding massive. I want as much as the screen real estate devoted to making big readable text as I can.

padding:
    bottom: 0
    left: 0
    right: 0
    top: 0

final results #

Here is what the final frontmatter looks like to fully style my talk.

---
date: 2022-03-24
templateKey: til
title: Style Lookatme Slides a bit more Personal
tags:
  - python
  - cli
  - python
author: '@_waylonwalker'
styles:
    padding:
        bottom: 0
        left: 0
        right: 0
        top: 0
    title:
        bg: default
        fg: '#e1af66'
    date:
        bg: default
        fg: '#368ce2'
    slides:
        bg: default
        fg: '#368ce2'
    headings:
        '1':
            bg: default
            fg: '#ff66c4,bold,italics'
            prefix: ' ⇁ '
            suffix: ' ↽ '
    quote:
        side: '│'
        style:
            bg: default
            fg: '#aaa'
        top_corner: '╭'
        bottom_corner: '╰'
    author:
        bg: default
        fg: '#368ce2'
---

How I load Markdown in Python

2022-03-23T03:18:48Z

I use a package eyeseast/python-frontmatter{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml).

Install #

It’s on pypi, so you can install it into your virtual environment with pip.

python -m pip install python-frontmatter

🙋 What’s Frontmatter #

Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging platform that also builds all of its posts with markdown and yaml frontmatter.

Let’s see an example #

Here is the exact frontmatter for this post you are reading on my site.

---
date: 2022-03-24 03:18:48.631729
templateKey: til
title: How I load Markdown in Python
tags:
  - linux
  - python

---

This is where the markdown content for the post goes.

So it’s yaml #

yaml is the most commmon, but python-frontmatter{.hoverlink} also supports Handlers{.hoverlink} for toml and json.

If you want a good set of examples of yaml learnxinyminutes{.hoverlink} has a fantastic set of examples in one page.

How to load yaml frontmatter in python #

Here is how I would load this post into python using python-frontmatter{.hoverlink}.

import frontmatter
inspect(frontmatter.load("pages/til/python-frontmatter.md"))

We can use rich{.hoverlink} to inspect the Post object to see what all it contains.

❯ inspect(frontmatter.load("pages/til/python-frontmatter.md"))
╭────────────────────────────────────────────────────────── <class 'frontmatter.Post'> ───────────────────────────────────────────────────────────╮
│ A post contains content and metadata from Front Matter. This is what gets                                                                       │
│ returned by :py:func:`load <frontmatter.load>` and :py:func:`loads <frontmatter.loads>`.                                                        │
│ Passing this to :py:func:`dump <frontmatter.dump>` or :py:func:`dumps <frontmatter.dumps>`                                                      │
│ will turn it back into text.                                                                                                                    │
│                                                                                                                                                 │
│ ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
│ │ <frontmatter.Post object at 0x7f03c4c23ca0>                                                                                                 │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                                                                 │
│  content = "I use a package\n[eyeseast/python-frontmatter](https://github.com/eyeseast/python-frontmatter)\nto load files with frontmatter in   │
│            them.  Its a handy package that allows you to\nload files with structured frontmatter (yaml, json, or toml).\n\n## Install\n\nIt's   │
│            on pypi, so you can install it into your virtual environment with pip.\n\n```bash\npython -m pip install                             │
│            python-frontmatter\n```\n\n## 🙋 What's Frontmatter\n\nFrontmatter is a handy way to add metadata to your plain text files.          │
│            It's\nquite common to have yaml frontmatter in markdown.  All of my blog posts have\nyaml frontmatter to give the post metadata such │
│            as post date, tags, title, and\ntemplate.  dev.to is a popular developer blogging platform that also builds all\nof its posts with   │
│            markdown and yaml frontmatter.\n\n## Let's see an example\n\nHere is the exact frontmatter for this post you are reading on my       │
│            site.\n\n```markdown\n---\ndate: 2022-03-24 03:18:48.631729\ntemplateKey: til\ntitle: How I load Markdown in Python\ntags:\n  -      │
│            linux\n  - python\n\n---\n\nThis is where the markdown content for the post goes.\n```\n\n## So it's yaml\n\nyaml is the most        │
│            commmon, but\n[eyeseast/python-frontmatter](https://github.com/eyeseast/python-frontmatter)\nalso                                    │
│            supports\n[Handlers](https://python-frontmatter.readthedocs.io/en/latest/handlers.html?highlight=toml#module-frontmatter.default_ha… │
│            toml and json.\n\nIf you want a good set of examples of yaml\n[learnxinyminutes](https://learnxinyminutes.com/docs/yaml/) has a      │
│            fantastic set\nof examples in one page.\n\n## How to load yaml frontmatter in python"                                                │
│  handler = <frontmatter.default_handlers.YAMLHandler object at 0x7f03bffbd910>                                                                  │
│ metadata = {                                                                                                                                    │
│                'date': datetime.datetime(2022, 3, 24, 3, 18, 48, 631729),                                                                       │
│                'templateKey': 'til',                                                                                                            │
│                'title': 'How I load Markdown in Python',                                                                                        │
│                'tags': ['linux', 'python', 'python']                                                                                            │
│            }                                                                                                                                    │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Getting Metadata #

You can get items from the posts metadata just as you would from a dict.

post = frontmatter.load("pages/til/python-frontmatter.md")
post['date']
# datetime.datetime(2022, 3, 24, 3, 18, 48, 631729)

post.get('date')
# datetime.datetime(2022, 3, 24, 3, 18, 48, 631729)

python dict get

I have recently become fond of the .get method to give it an easy default value.

Content is content #

The content of the document is stored under .content

post.content

Links #

Python's Dict Union Operator | Pep 584

2022-03-22T02:39:53Z

Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584.

Merge Dicts #

I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result.

default_config = {'url': 'https://example.com', 'assets_dir': 'static' }
user_config = {'url': 'https://waylonwalker.com'}

# **unpacking goes back much further than 3.9

config = {**default_config, **user_config}
print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}


# the same can be achieved through the new to python 3.9 | operator

config = default_config | user_config
print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}

understanding python *args and **kwargs

More on unpacking in this post.

Update Dicts #

With the release there is also a new update syntax |= that you can use to update. I dont often mutate variables for some reason, so I cant think of a better example for this from my personal use cases. So I will give a similar example to above, except creating a config, then updating it.

# old python <3.9 way
config = {'url': 'https://example.com', 'assets_dir': 'static' }
config.update({'url': 'https://waylonwalker.com'})

# new python 3.9+ way
config = {'url': 'https://example.com', 'assets_dir': 'static' }
config |= {'url': 'https://waylonwalker.com'}

print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}

Should you use it? #

Are you writing libraries/applications that are only going to be ran on 3.9? Then ya go for it there is nothing to loose. If there is any chance someone is going to run your code on 3.8 or older then just use **, or .update.

RTFM #

This is what comes first to my mind on how to use this new syntax, read pep584 for all the gritty details on it.

Links #

How I Present Markdown Slides from the Terminal

2022-03-21T16:13:39Z

I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides.

Present from the terminal #

I will most often just present right from the terminal using lookatme. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly.

reveal.js #

I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share.

New Slides #

I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style.

Installation #

lookatme is a python library that is available on pypi, you can install it with the pip command.

python -m pip install lookatme

Since it’s a command line application it works great with pipx. This prevents the need to manage virtual environments yourself or ending up with packages clashing in your system python environment.

pipx install lookatme

From my terminal #

lookatme {filepath}

I just run it with pipx.

pipx run \
 --spec git+https://github.com/waylonwalker/lookatme \
 lookatme {filepath} \
 --live-reload \
 --style gruvbox-dark

Note, I use a custom fork of lookatme. It’s schema validation did not like the date format of my blog posts, so I have a one line fix built into my fork that is pretty specific to me.

From Neovim #

most often what I do

From Neovim I use a plugin I created for sending out commands to tmux called telegraph. This sends the above command to a new session that I can bounce between quickly.

nnoremap <leader><leader>s <cmd>lua require'telegraph'.telegraph({cmd='pipx run --spec git+https://github.com/waylonwalker/lookatme lookatme {filepath} --live-reload --style gruvbox-dark', how='tmux'})<CR>

How I read Files in Python

2022-03-20T17:19:00Z

When I need to read contents from a plain text file in python I find the easiest way is to just use Pathlib.

from pathlib import Path

Path('path_to_file').read_text()

How I glob for Files in Python

2022-03-17T23:01:53Z

A very common task for any script is to look for files on the system. My go to method when globbing for files in python is to use pathlib.

Setup #

I setup a directory to make some examples about globbing. Here is what the directory looks like.

❯ tree .
.
├── content
│   ├── hello.md
│   ├── hello.py
│   ├── me.md
│   └── you.md
├── readme.md
├── README.md
├── READMES.md
└── setup.py

1 directory, 8 files

Pathlib #

Pathlib is a standard library module available in all LTS versions of python at this point.

❯ from pathlib import Path

Creating a Path instance.

# current working directory
Path()
Path.cwd()

# The users home directory
Path.home()

# Path to a directory by string
Path('/path/to/directory')

# The users ~/.config directory
Path.home() / '.config'

Globbing Examples #

The path object has a glob method that allows you to glob for files with a unix style glob pattern to search for files. Note that it gives you a generator. This is great for many use cases, but for examples its easier to turn them to a list to print them out.

If you need some more detail on what globbing is there is a wikipedia article discussing it. I am just showing how to glob with pathlib.


❯ Path().glob("**/*.md")
<generator object Path.glob at 0x7fa35adc4f90>

❯ list(Path().glob("**/*.md"))

[
    PosixPath('readme.md'),
    PosixPath('READMES.md'),
    PosixPath('README.md'),
    PosixPath('content/you.md'),
    PosixPath('content/me.md'),
    PosixPath('content/hello.md')
]

❯ list(Path().glob("**/*.py"))
[PosixPath('setup.py'), PosixPath('content/hello.py')]

❯ list(Path().glob("*.md"))
[PosixPath('readme.md'), PosixPath('READMES.md'), PosixPath('README.md')]

❯ list(Path().glob("*.py"))
[PosixPath('setup.py')]

❯ list(Path().glob("**/*hello*"))
[PosixPath('content/hello.py'), PosixPath('content/hello.md')]

❯ list(Path().glob("**/REA?ME.md"))
[PosixPath('README.md')]

pytest-mock Basics

2022-03-14T00:19:02Z

Last Thursday I learned about pytest-mock at a local python meetup. The presenter showed how he uses pytest-mock for his work, and it was kinda eye opening. I knew what mocking was, but I had not seen it in this context.

Discovery #

Watching him use pytest-mock I realized that mocking was not as hard as I had made it out to be. You can install pytest-mock, use the mocker fixture, and patch objects methods with what you want them to be.

install #

pytest-mock is out on pypi and can be installed with pip.

python -m pip install pytest-mock

What I actually did #

Sometimes I fall victim to making these posts nice and easy to follow. It takes more steps than just pip install, you need a place to practice in a nice sandbox. Here is how I make my sandboxes.

mkdir ~/git/learn-pytest-mock
cd ~/git/learn-pytest-mock
# well actually open a new tmux session there
echo pytest-mock > requirements.txt

# I copied in my .envrc, and ran direnv allow, which actually just made me a virtual env as follows
python3 -m venv .venv --prompt $(basename $PWD)
source .venv/bin/activate

# now install pytest-mock
pip install -r requirements.txt

# make some tests to mock
mkdir tests
nvim tests/test_me.py

create a tests/test_me.py #

I just wanted to do something that was worth mocking, the first thing that came to mind was to do something that made a network call. Here I made a method that uses requests to go get the content on my homepage, but changes it’s return behavior based on the status_code of the request.

I want to mock out requests to ensure that GoGetter can handle both 200 (http success) and 404 (http not found) status codes.

# tests/test_me.py
import requests


class GoGetter:
    """
    The thing I am testing, this is usually imported into the test file, but
    defined here for simplicity.
    """
    def get(self):
        """
        Get the content of `https://waylonwalker.com` and return it as a string
        if successfull, or False if it's not found.
        """
        r = requests.get("https://waylonwalker.com")
        if r.status_code == 200:
            return r.content
        if r.status_code == 404:
            return False


class DummyRequester:
    def __init__(self, content, status_code):
        """
        mock out content and status_code
        """

        self.content = content
        self.status_code = status_code

    def __call__(self, url):
        """
        The way I set this up GoGetter is going to call an instance of this
        class, so the easiest way to make it work was to implement __call__.
        """
        self.url = url
        return self


def test_success_get(mocker):
    """
    Show that the GoGetter can handle successful calls.
    """
    go_getter = GoGetter()

    # Use the mocker fixture to change how requests.get works while inside of test_success_get
    mocker.patch.object(requests, "get", DummyRequester("waylonwalker", 200))
    assert "waylon" in go_getter.get()


def test_failed_get(mocker):
    """
    Show that the GoGetter can handle failed calls.
    """
    go_getter = GoGetter()

    # Use the mocker fixture to change how requests.get works while inside of test_failed_get
    mocker.patch.object(requests, "get", DummyRequester("waylonwalker", 404))
    assert go_getter.get() is False

Python Walrus Inside List Comprehension

2022-03-12T15:01:59Z

Python 3.8 came out two and a half years ago and I have yet to really lean in on the walrus operator. Partly because it always seemed like something kinda silly (my use cases) to require a python version bump for, and partly because I really didn’t understand it the best. Primarily I have wanted to use it in comprehensions, but I did not really understand how.

Now that Python 3.6 is end of life, and most folks are using at least 3.8 it seems time to learn and use it.

What’s a Walrus #

The assignment operator in python is more commonly referred to as the walrus operator due to how := looks like a walrus. It allows you to assign and use a variable in a single expression.

This example from the docs avoids a second call to the len function.

if (n := len(a)) > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

Let’s get some data #

without a walrus

In this example we are going to do a dict comp to generate a map of content from urls, only if their status code is 200. When doing this in a dictionary comprehension we end up needing to hit the url twice for successful urls. Once for the filter and once for the data going into the dictionary.

{
    url: requests.get(url).content
    for url in ["https://waylonwalker.com/", "https://waylonwalker.com/broken"]
    if requests.get(url).status_code == 200
}

Gimme some walrus #

using walrus in a dict comp

Using the walrus operator := list comp allows us to only put things into the dictionary that we want to keep, and not hit the url twice.

{
    url: r.content
    for url in ["https://waylonwalker.com/", "https://waylonwalker.com/broken"]
    if (r := requests.get(url)).status_code == 200
}

FIN #

The walrus is a nice to have option to save on extra function/network calls, and micro optimize your code without adding much extra.

Make Kedro Runs Beautiful

2022-03-11T13:14:51Z

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.

Install kedro rich #

There is no pypi package yet, but it’s on github. You can pip install it with the git url.

pip install git+https://github.com/datajoely/kedro-rich

Kedro run #

You can run your pipeline just as you normally would, except you get progress bars and pretty prints.

kedro run

Kedro catalog #

Listing out catalog entries from the command line now print out a nice pretty table.

kedro catalog list

Give it a star #

Go to the GitHub repo and give it a star, Joel deserves it.

Open Files with Nvim Remote

2022-03-06T16:22:49Z

I recently found a really great plugin by mhinz to open files in neovim from a different tmux split, without touching neovim at all.

Installation #

neovim-remote is not a neovim plugin at all, it’s a python cli that you can install with pip. Unlike the repo suggests, I use pipx to install nvr.

pipx install neovim-remote

How I use it #

I have this added to my .envrc that is in every one of my projects. This will tie a neovim session to that directory, and all directories under it.

export NVIM_LISTEN_ADDRESS=/tmp/nvim-$(basename $PWD)

In my workflow I open a tmux session for each project, so this essentially ties a neovim session to a tmux session.

Open neovim #

First open neovim, but with the nvr command. This will open neovim, and look pretty much the same as always.

nvr

If you try to run nvr again in another shell nothing will happen as its already runnin under that address, but if you give it a filename it will open the file in the first instance of neovim that you opened.

nvr readme.md

Links #

GitHub

Textual Popup Hack

2022-02-26T15:41:12Z

As I am toying around with textual, I am wanting some popup user input to take over. Textual is still pretty new and likely to change quite significantly, so I don’t want to overdo the work I put into it, So for now on my personal tuis I am going to shell out to tmux.

The Problem #

The main issue is that when you are in a textual app, it kinda owns the input. So if you try to run another python function that calls for input it just cant get there. There is a textual-inputs library that covers this, and it might work really well for some use cases, but many of my use cases have been for things that are pre-built like copier, and I am trying to throw something together quick.

textual is still very beta

Part of this comes down to the fact that textual is still very beta and likely to change a lot, so all of the work I have done with it is for quick and dirty, or fun side projects.

The Solution #

So the solution that was easiest for me… shell out to a tmux popup. The application I am working on wants to create new documents using copier templates. copier has a fantastic cli that walks throught he template variables and asks the user to fill them in, so I just shell out to that with Popen. Make sure that you wait for this process to finish otherwise there will be bit of jank in your textual app.

async def action_new_post(self) -> None:
    proc = subprocess.Popen(
        'tmux popup "copier copy plugins/todo-template tasks"', shell=True
    )
    proc.wait()

example #

Here is what the running todo application looks like with the copier popup over it.

tmux popups

a bit more on tmux-popus [here] https://waylonwalker.com/tmux-popups/)

Links #

GitHub Markdown now Supports Mermaid Diagrams

2022-02-24T15:22:31Z

Mermaid diagrams provide a way to display graphs defined as plain text. Some markdown renderers support this as a plugin. GitHub now supports it.

example #

You can define nodes like this in mermaid, and GitHub will now render them as a pretty graph diagram. Its rendered in svg, so its searchable with control f and everything.

graph TD;
      A-->B;
      A-->C;
      B-->D;
      C-->D-->OUT;
      E-->F-->G-->OUT

Links #

Glances can watch docker processes

2022-02-22T16:38:42Z

Glances is a system monitor with a ton of features, including docker processes.

I have started using portainer to look at running docker processes, its a great heavy-weight docker process monitor. glances works as a great lightweight monitor to just give you the essentials, ( Name, Status, CPU%, MEM, /MAX, IOR/s, IOW/s, Rx/s, Tx/s, Command)

install #

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances
pipx inject glances "glances[docker]"

You will be presented with this success message.

  injected package glances into venv glances
done! ✨ 🌟 ✨

results #

Now running glances will also show information about your running docker containers.

Links #

Glances webui with pipx

2022-02-20T16:25:49Z

Glances has a pretty incredible webui to view system processes and information like htop, or task manager for windows.

The nice thing about the webui is that it can be accessed from a remote system. This would be super nice on something like a raspberry pi, or a vm running in the cloud. Its also less intimidating and easier to search if you are not a terminal junky.

install #

pipx install glances
pipx inject glances "glances[web]"

You will be presented with this success message.

  injected package glances into venv glances
done! ✨ 🌟 ✨

running the webui #

Now that you have glances installed you can run it with the -w flag to run the webui.

glances -w

This will present you with the following success message.

Glances Web User Interface started on http://0.0.0.0:61208/

Open it in your browser #

Now that its running you can open your web browser to localhost:61208 and be presented with the glances webui.

Links #

Run glances without install with pipx

2022-02-18T16:17:26Z

Glances is a fully featured system monitoring tool written in python. Out of the box it’s quite similar to htop, but has quite a few more features, and can be ran without installing anything other than pipx, which you should already have installed if you do anything with python.

pipx run glances

Once you run this you will be in a tui application similar to htop. You can kill processes with k, use left and right arrows to change the sorting column, and up and down to select different processes.

Links #

Nested requirements.txt in python

2022-02-17T15:53:20Z

python requirements text files can in fact depend on each other due to the fact that you can pass pip install arguments right into your requirements.txt file. The trick is to just prefix the file with a -r flag, just like you would if you were installing it with pip install

try it out #

Lets create two requirements files in a new directory to play with.

mkdir requirements-nest
cd requirements-nest
touch requirements.txt requirements_dev.txt

Then add the following to each requirements file.

# requirements.txt
kedro[pandas.ParquetDataSet]

# requirements_dev.txt
-r requirements.txt
ipython

Installing #

Installing requirements_dev.txt will install both ipython and pandas since it includes the base requirements file.

# this will install only pandas
pip install -r requirements.txt

# this will install both ipython and pandas
pip install -r requirements_dev.txt

Links #

This is covered in the pip user guide, but it is not obvious that this can be done in a requirements.txt file.

View AWS event bridge rules with visidata

2022-02-15T03:04:26Z

Reading eventbridge rules from the command line can be a total drag, pipe it into visidata to make it a breeze.

I just love when I start thinking through how to parse a bunch of json at the command line, maybe building out my own custom cli, then the solution is as simple as piping it into visidata. Which is a fantastic tui application that had a ton of vim-like keybindings and data features.

alias awsevents = aws events list-rules | visidata -f json

Python string of letters is a string of letters, but not with special

2022-02-12T16:05:55Z

In python, a string is a string until you add special characters.

In browsing twitter this morning I came accross this tweet, that showed that you can use is accross two strings if they do not contain special characters.

https://twitter.com/bascodes/status/1492147596688871424

I popped open ipython to play with this. I could confirm on 3.9.7, short strings that I typed in worked as expected.

waylonwalker ↪main v3.9.7 ipython
❯ a = "asdf"

waylonwalker ↪main v3.9.7 ipython
❯ b = "asdf"

waylonwalker ↪main v3.9.7 ipython
❯ a is b
True

Using the upper() method on these strings does break down.

waylonwalker ↪main v3.9.7 ipython
❯ a.upper() is b.upper()
False

waylonwalker ↪main v3.9.7 ipython
❯ a = "ASDF"

waylonwalker ↪main v3.9.7 ipython
❯ b = "ASDF"

waylonwalker ↪main v3.9.7 ipython
❯ a is b
True

If You can also see this in the id of the objects as well, which is the memmory address in CPython.

waylonwalker ↪main v3.9.7 ipython
❯ id(a)
140717359289568

waylonwalker ↪main v3.9.7 ipython
❯ id(b)
140717359289568

waylonwalker ↪main v3.9.7 ipython
❯ id(a.upper())
140717359581824

waylonwalker ↪main v3.9.7 ipython
❯ id(b.upper())
140717360337824

Finally just as the post shows if you add a special character in there it also breaks.

waylonwalker ↪main v3.9.7 ipython
❯ a = "ASDF!"

waylonwalker ↪main v3.9.7 ipython
❯ b = "ASDF!"

waylonwalker ↪main v3.9.7 ipython
❯ a is b
False

What should you do #

First and foremost, these are the exact pitfalls that flake8 guards you against. So the very first things you should take away here is that there is a lot of wisdom and value in flake8.

Second, the is comparison should be used for things that you want to compare to exact memmory addresses. These include booleans and None. Don’t use is accross two assigned variables.

Read stderr from python subprocess.Popen

2022-02-10T03:57:32Z

I often run shell commands from python with Popen, but not often enough do I set up error handline for these subprocesses. It’s not too hard, but it can be a bit awkward if you don’t do it enough.

Using Popen #

import subprocess
from subprocess import Popen

# this will run the shell command `cat me` and capture stdout and stderr
proc = Popen(["cat", "me"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

# this will wait for the process to finish.
proc.wait()

reading from stderr #

To get the stderr we must get it from the proc, read it, and decode the bystring. Note that we can only get the stderr object once, so if you want to do more than just read it you will need to store a copy of it.

proc.stderr.read().decode()

Better Exception #

Now that we can read the stderr we can make better error tracking for the user so they can see what to do to resolve the issue rather than blindly failing.

err_message = proc.stderr.read().decode()
if proc.returncode != 0:
    # the process was not successful

    if "No such file" in err_message:
        raise FileNotFoundError('No such file "me"')

Using a Python Markdown ast to Find All Paragraphs

2022-02-05T02:18:28Z

In looking for a way to automatically generate descriptions for pages I stumbled into a markdown ast in python. It allows me to go over the markdown page and get only paragraph text. This will ignore headings, blockquotes, and code fences.

import commonmark
import frontmatter

post = frontmatter.load("post.md")
parser = commonmark.Parser()
ast = parser.parse(post.content)

paragraphs = ''
for node in ast.walker():
    if node[0].t == "paragraph":
        paragraphs += " "
        paragraphs += node[0].first_child.literal

It’s also super fast, previously I was rendering to html and using beautifulsoup to get only the paragraphs. Using the commonmark ast was about 5x faster on my site.

Duplicate Paragraphs #

When I originally wrote this post, I did not realize at the time that commonmark duplicates nodes. I still do not understand why, but I have had success duplicating them based on the source position of the node with the snippet below.

from itertools import compress

import commonmark
import frontmatter

post = frontmatter.load("post.md")
parser = commonmark.Parser()
ast = parser.parse(post.content)

# find all paragraph nodes
paragraph_nodes = [
    n[0]
    for n in ast.walker()
    if n[0].t == "paragraph" and n[0].first_child.literal is not None
]
# for reasons unknown to me commonmark duplicates nodes, dedupe based on sourcepos
sourcepos = [p.sourcepos for p in paragraph_nodes]
# find first occurence of node based on source position
unique_mask = [sourcepos.index(s) == i for i, s in enumerate(sourcepos)]
# deduplicate paragraph_nodes based on unique source position
unique_paragraph_nodes = list(compress(paragraph_nodes, unique_mask))
paragraphs = " ".join([p.first_child.literal for p in unique_paragraph_nodes])

python dict get

2022-02-03T03:00:36Z

For an embarassingly long time, til today, I have been wrapping my dict gets with key errors in python. I’m sure I’ve read it in code a bunch of times, but just brushed over why you would use get. That is until I read a bunch of PR’s from my buddy Nic and notice that he never gets things with brackets and always with .get. This turns out so much cleaner to create a default case than try except.

Example #

Lets consider this example for prices of supplies. Here we set a variable of prices as a dictionary of items and thier price.

prices = {'pen': 1.2, 'pencil', 0.3, 'eraser', 2.3}

Except KeyError #

What I would always do is try to get the key, and if it failed on KeyError, I would set the value (paper_price in this case) to a default value.

try:
    paper_price = prices['paper']
except KeyError:
    paper_price = None

.get #

What I noticed Nic does is to use get. This feels just so much cleaner that it’s a one liner and feels much easier to read and understand that if there is no price for paper we set it to None.

paper_price = prices.get('paper', None)

We can just as easily set the default to other values. Let’s consider sales for instance. If there is not a record for the sale of paper, it might be that we sold 0 paper in the given dataset.

paper_sales = sales.get('paper', 0)

Find all Headings with BeautifulSoup

2022-02-01T16:17:18Z

BeautifulSoup is a DOM like library for python. It’s quite useful to manipulate html. Here is an example to find_all html headings. I stole the regex from stack overflow, but who doesn’t.

Make an example #

sample.html

Lets make a sample.html file with the following contents. It mainly has some headings, <h1> and <h2> tags that I want to be able to find.

<!DOCTYPE html>
<html lang="en">
  <body>
    <h1>hello</h1>
    <p>this is a paragraph</p>
    <h2>second heading</h2>
    <p>this is also a paragraph</p>
    <h2>third heading</h2>
    <p>this is the last paragraph</p>

  </body>
</html>

Get the headings with BeautifulSoup #

Lets import our packages, read in our sample.html using pathlib and find all headings using BeautifulSoup.

from bs4 import BeautifulSoup
from pathlib import Path

soup = BeautifulSoup(Path('sample.html').read_text(), features="lxml")
headings = soup.find_all(re.compile("^h[1-6]$"))

And what we get is a list of bs4.element.Tag’s.

>> print(headings)
[<h1>hello</h1>, <h2>second heading</h2>, <h2>third heading</h2>]

I recently added a heading_link plugin to markata, you might notice the 🔗’s next to each heading on this page, that is powered by this exact technique.

Lambda Function as a Kedro Node

2022-01-31T21:52:36Z

I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas.

Examples #

Here are two examples, the first one lambda x: x is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow standard layering conventions, without skipping a layer, overthinking if you should have the layer or not, and leaves a good placholder to fill in later when you need it.

Many times I just want to get the data in as fast as possible, learn about it, then go back and tidy it up.

from kedro.pipeline import node

my_first_node = node(
   func=lambda x: x,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

my_first_node = node(
   func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'),
   inputs='raw_cars',
   output='int_cars',
   tags=['pri',]
   )

Note: try not to take the idea of a one liner too far. If your one line function wraps several lines down it probably deserves to be a real function for readability and a good docstring.

Add New Dependencies to Your Kedro Project

2022-01-28T21:30:48Z

As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command pip-compile under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let’s take a look at it.

git-status">git status #

Before you start mucking around with any changes to dependencies make sure that your git status is clean. I’d even reccomend starting a new branch for this, and if you are working on a team potentially submit this as its own PR for clarity.

git status
git checkout main
git checkout -b add-rich-dependency

requirements.in #

New requirements get added to a requirements.in file. If you need to specify an exact version, or a minimum version you can do that, but if all versions generally work you can leave it open.

# requirements.in
rich

Here I added the popular rich package to my requirements.in file. Since I am ok with the latest version I am not going to pin anything, I am going to let the pip resolver pick the latest version that does not conflict with any of my dependencies for me.

build-reqs #

The command kedro build-reqs will tell kedro to recompile the requirements.txt file that has all of our dependencies pinned down to exact versions. This ensures that all of our teammates and production workflows use the same exact versions of packages even if new ones are released after we installed on our development machines.

kedro build-reqs

git add #

Now that we have our new dependencies ready to go commit those to git, and submit a PR for them if you are working on a team. This is a good way to document the discussion of adding new dependencies to your teams project.

git add requirements.in
git add requirements.txt
git status
git commit -m "FEAT updated dependencies with rich"
git push
# go make a pr
gh pr create --title "feat add rich to dependencies" --body "I added rich as a dependency, and ran pip-compile"

Practice making pipelines with kedro

2022-01-27T21:07:37Z

I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity.

https://youtu.be/Rk-Y71P_9KE

Start practicing #

practice building pipelines with #kedro today

Go to your playground directory, and if you don’t have one, make one.

cd ~/playground

get pipx #

Install pipx in your system python. This is one of the very few, and possibly the only python library that deserves to be installed in your system directory, primarily because its used to sanbox clis in their own virtual environment automatically for you.

pip install pipx

make a new project #

From inside your playground directory, start your new kedro project. This is quite simple and painless. So much so that if you mess this one up doing something wild, it might be easier to make a new one that fixing the wild one.

pipx run kedro new
# answer the questions it asks

I use this quite often to try out new things in a safe place.

Make a virtual environment #

Using Conda #

Conda is a fine choice to manage your virtual environments. It used to make things so much easier on windows that it was almost required. Nowadays getting python running on windows has become so much easier that this is less so.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

one great benefit of conda is that it lets you choose the interpreter to go with your virtual environment.

Your new environment will be listed in your list of conda env here.

conda info --envs

Using venv #

venv is what I use now. Nothing against conda, it works great. venv just feels a bit lighter and more common. I’ve actually grown to appreciate that the venv is right where I put it, most often in the project directory.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

using pipenv #

pipenv is another fine choice. I like how in one command it makes the environment and activates it for you. pipenv also puts virtual environments in the global directory.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

Make pipelines #

Now go make some pipelines with your new project, try something wild, break it, and make another.

Tmux hotkey for copier templates

2022-01-24T02:22:53Z

I have added a hotkey to my copier template setup to quickly access all my templates at any time from tmux. At any point I can hit <c-b><c-b>, thats holding control and hitting bb, and I will get a popup list of all of my templates directory names. Its an fzf list, which means that I can fuzzy search through it for the template I want, or arrow key to the one I want if I am feeling insane. I even setup it up so that the preview is a list of the files that come with the template in tree view.

bind-key c-b popup -E -w 80% -d '#{pane_current_path}' "\
    pipx run copier copy ~/.copier-templates/`ls ~/.copier-templates |\
    fzf --header $(pwd) --preview='tree ~/.copier-templates/{} |\
    lolcat'` . \
    "

I’ve had this on my systems for a few weeks now and I am constantly using it for my tils, blogs, and my .envrc file that goes into all of my projects to make sure that I have a virtual environment installed and running any time I open it.

Markata Filters as Telescope Pickers in Neovim

2022-01-23T04:52:20Z

I often pop into my blog from neovim with the intent to look at just a single series of posts, til, gratitude, or just see todays posts. Markata has a great way of mapping over posts and returning their path that is designe exactly for this use case.

To tie these into a Telescope picker you add the command as the find_command, and comma separate the words of the command, with no spaces. I did also --sort,date,--reverse in there so that the newest posts are closest to the cursor.

nnoremap geit <cmd>Telescope find_files find_command=markata,list,--map,path,--filter,date==today<cr>
nnoremap geil <cmd>Telescope find_files find_command=markata,list,--map,path,--filter,templateKey=='til',--sort,date,--reverse<cr>
nnoremap geig <cmd>Telescope find_files find_command=markata,list,--map,path,--filter,templateKey=='gratitude',--sort,date,--reverse<cr>

NOTE telescope treates each word as a string, do not wrap an extra layer of quotes around your words, it gets messy.

Copier Tasks | Python templating post run task

2022-01-22T04:49:12Z

Copier allows you to run post render tasks, just like cookiecutter. These are defined as a list of tasks in your copier.yml. They are simply shell commands to run.

The example I have below runs an update-gratitude bash script after the copier template has been rendered.

# copier.yml
num: 128
_answers_file: .gratitude-copier-answers.yml
_tasks:
  - "update-gratitude"

I have put the script in ~/.local/bin so that I know it’s always on my $PATH. It will reach back into the copier.yml and update the default number.

#!/bin/bash
# ~/.local/bin/update-gratitude
current=`awk '{print $2}' ~/.copier-templates/gratitude/copier.yml | head -n 1`
new=`expr $current + 1`
echo $current
echo $new
sed -i "s/$current/$new/g" ~/.copier-templates/gratitude/copier.yml

Gradual Typing in Python

2022-01-21T14:27:32Z

I’ve referenced a video from Anthony Sotile in passing conversation several times. Walking through his gradual typing process has really helped me understand typing better, and has helped me make some projects better over time rather than getting slammed with typing errors.

Step 1

Run Mypy as is, don’t get fancy yet. This will not reach into any functions unless they are alreay explicitly typed. It will not enforce you to type them either.

pip install mypy
mypy .
# or your specific project to avoid .venvs
mypy src
# or a single file
mypy my-script.py

Step 2 #

Next we will add check-untyped-defs, this will start checking inside functions that are not typed. To add this to your config create a setup.cfg with the following.

[mypy]
check_untyped_defs = True

Step 3 #

The final stage to this series is to add disallow_untyped_defs. This will start requiring all of your functions to be type hinted. This one is probably the toughest, because as you type functions mypy can uncover more issues for you to fix. Often times the list of errors grows before it shrinks.

[mypy]
check_untyped_defs = True
disallow_untyped_defs = True

Anthony’s video #

Make sure that you watch Anthony’s video, give him a sub, he deserves it for all the great things he is doing for the python community.

https://www.youtube.com/watch?v=Rk-Y71P_9KE

Python Reverse Sluggify

2022-01-20T01:24:43Z

In order to make an auto title plugin for markata I needed to come up with a way to reverse the slug of a post to create a title for one that does not explicitly have a title.

slugs

 a slug is generally all lowercase and free of spaces, and is a way to

make website routes (urls)

Here I have a path available that gives me the articles path, ex. python-reverse-sluggify.md. An easy way to get rid of the file extension, is to pass it into pathlib.Path and ask for the stem, which returns python-reverse-sluggify. Then from There I chose to replace - and _ with a space.

article["title"] = (
    Path(article["path"]).stem.replace("-", " ").replace("_", " ").title()
)

To turn this into a markata plugin I put it into a pre_render hook.

from pathlib import Path

from markata.hookspec import hook_impl, register_attr


@hook_impl
@register_attr("articles")
def pre_render(markata) -> None:
    for article in markata.filter('title==""'):
        article["title"] = (
            Path(article["path"]).stem.replace("-", " ").replace("_", " ").title()
        )

Get Python docstring with ast

2022-01-18T20:13:32Z

Getting docstrings from python’s ast is far simpler and more reliable than any method of regex or brute force searching. It’s also much less intimidating than I originally thought.

Parsing #

First you need to load in some python code as a string, and parse it with ast.parse. This gives you a tree like object, like an html dom.

py_file = Path("plugins/auto_publish.py")
raw_tree = py_file.read_text()
tree = ast.parse(raw_tree)

Getting the Docstring #

You can then use ast.get_docstring to get the docstring of the node you are currently looking at. In the case of freshly loading in a file, this will be the module level doctring that is at the very top of a file.

module_docstring = ast.get_docstring(tree)

Walking for all functions #

To get all of the functions docstrings we can use ast.walk to look for nodes that are an instance of ast.FunctionDef, then run get_docstring on those nodes.

functions = [f for f in ast.walk(tree) if isinstance(f, ast.FunctionDef)]
function_docs = [ast.get_docstring(f) for f in functions]

ast.walk docs: Recursively yield all descendant nodes in the tree starting at node (including node itself), in no specified order. This is useful if you only want to modify nodes in place and don’t care about the context.

Example #

Here is an image of me running this example through ipython.

Python Respect the .gitignore

2022-01-17T15:16:33Z

Many tools such as ripgrep respect the .gitignore file in the directory it’s searching in. This helps make it incredibly faster and generally more intuitive for the user as it just searches files that are part of thier project and not things like their virtual environments, node modules, or compiled builds.

Editors like vscode often do not include files that are .gitignored in their search either.

pathspec is a pattern matching library that implements git’s wildmatch pattern so that you can ignore files included in your .gitignore patterns. You might want this to help make your libraries more performant, or more intuitive for you users.

import pathspec
from pathlib import Path

markdown_files = Path().glob('**/*.md')
if (Path(".gitignore").exists():
    lines = Path(".gitignore").read_text().splitlines()

    spec = pathspec.PathSpec.from_lines("gitwildmatch", lines)

    markdown_files = [
        file for file in markdown_files if not spec.match_file(str(file))
    ]

pathspec home page

Rename Python Variables with nvim

2022-01-16T15:27:22Z

I don’t use refactoring tools as much as I probably should. mostly because I work with small functions with unique names, but I recently had a case where a variable name m was everywhere and I wanted it named better. This was not possible with find and replace, because there were other m’s in this region.

I first tried the nvim lsp rename, and it failed, Then I pip installed rope, a refactoring tool for python, and it just worked!

pip install rope

Once you have rope installed you can call rename on the variable.

:lua vim.lsp.buf.rename()

Python Find Available Port

2022-01-15T23:47:27Z

When running a python process that requires a port it’s handy if there is an option for it to just run on the next avaialble port. To do this we can use the socket module to determine if the port is in use or not before starting our process.

import socket

def find_port(port=8000):
    """Find a port not in ues starting at given port"""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        if s.connect_ex(("localhost", port)) == 0:
            return find_port(port=port + 1)
        else:
            return port

python functools total ordering

2022-01-14T23:43:42Z

functools.total_ordering makes adding all of six of the rich comparison operators to your custom classes much easier, and more likely that you remember all of them.

From the Docs: The class must define one of __lt__(), __le__(), __gt__(), or __ge__ In addition, the class should supply an __eq__() method.

one of these

lt()
le()
gt()
ge()

and required to have this one

eq()

Total Ordering Docs

Here is an example using the Enum I was working on the other day.

from enum import Enum, auto
from functools import total_ordering


@total_ordering
class LifeCycle(Enum):

    configure = auto()
    glob = auto()
    load = auto()
    pre_render = auto()
    render = auto()
    post_render = auto()
    save = auto()

    def __lt__(self, other):
        try:
            return self.value < other.value
        except AttributeError:
            return self.value < other

    def __eq__(self, other):
        try:
            return self.value == other.value
        except AttributeError:
            return self.value == other

Implement --pdb in a python cli

2022-01-13T00:09:20Z

Adding a --pdb flag to your applications can make them much easier for those using it to debug your application, especially if your applicatoin is a cli application where the user has much fewer options to start this for themselves. To add a pdb flag --pdb to your applications you will need to wrap your function call in a try/except, and start a post_mortem debugger. I give credit to this stack overflow post for helping me figure this out.

import pdb, traceback, sys


def bombs():
    a = []
    print(a[0])


if __name__ == "__main__":
    if "--pdb" in sys.argv:
        try:
            bombs()
        except:
            extype, value, tb = sys.exc_info()
            traceback.print_exc()
            pdb.post_mortem(tb)
    else:
        bombs()

Using –pdb #

python yourfile.py --pdb

Python Enum

2022-01-11T15:03:34Z

Python comes with an enum module for creating enums. You can make your own enum by inheriting importing and inheriting from Enum.

from enum import Enum


class LifeCycle(Enum):
    configure = 1
    glob = 2
    pre_render = 3
    render = 4
    post_render = 5
    save = 6

auto incrementing #

Enum values can be auto incremented by importing auto, and calling auto() as their value.

from enum import Enum, auto


class LifeCycle(Enum):
    configure = auto()
    glob = auto()
    pre_render = auto()
    render = auto()
    post_render = auto()
    save = auto()

using the enum #

Enum’s are accessed directy under the class itself, and have primarily two methods underneath each thing you make, .name and .value.

Lifecycle.glob
Lifecycle.glob.value
Lifecycle.glob.name

Installing Pipx on Ubuntu

2022-01-10T23:14:05Z

I recently paired up with another dev running windows with Ubuntu running in wsl, and we had a bit of a stuggle to get our project off the ground because they were missing com system dependencies to get going.

Straight in the terminal #

Open up a terminal and get your required system dependencies using the apt package manager and the standard ubuntu repos.

sudo apt update
sudo apt upgrade
sudo apt install \
      python3-dev \
      python3-pip \
      python3-venv \
      python3-virtualenv
pip install pipx

Using an Ansible-Playbook #

I like running things like this through an ansible-playbook as it give me some extra control and repeatability next time I have a new machine to setup.

- hosts: localhost
  gather_facts: true
  become: true
  become_user: "{{ lookup('env', 'USER') }}"

  pre_tasks:
    - name: update repositories
      apt: update_cache=yes
      become_user: root
      changed_when: False
  vars:
    user: "{{ ansible_user_id }}"
  tasks:
    - name: Install System Packages 1 (terminal)
      become_user: root
      apt:
        name:
          - build-essential
          - python3-dev
          - python3-pip
          - python3-venv
          - python3-virtualenv
    - name: check is pipx installed
      shell: command -v pipx
      register: pipx_exists
      ignore_errors: yes

    - name: pipx
      when: pipx_exists is failed
      pip:
        name: pipx
      tags:
        - pipx

video clip #

Here is a clip of me getting pipx running on ubuntu 21.10, and running a few of my favorite pipx commands.

Using Copier Answers to rerun templates quickly

2022-01-08T01:16:52Z

The copier answers file is a key component to making your templates re-runnable. Let’s look at the example for my setup.py.

❯ tree ~/.copier-templates/setup.py
/home/walkers/.copier-templates/setup.py
├── [[ _copier_conf.answers_file ]].tmpl
├── copier.yml
├── setup.cfg
└── setup.py.tmpl

0 directories, 4 files

Inside of my [[ _copier_conf.answers_file ]].tmpl file is this, a message not to muck around with it, and the ansers in yaml form. The first line is just a helper for the blog post.

# ~/.copier-templates/setup.py/\[\[\ _copier_conf.answers_file\ \]\].tmpl
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
[[_copier_answers|to_nice_yaml]]

Inside my copier.yml I have setup my _answers_file to point to a special file. This is because this is not a whole projet template, but one just for a single file.

# copier.yml
# ...
_answers_file: .setup-py-copier-answers.yml

Once I change the _answers_file I was incredibly stuck

Run it #

I’m making a library of personal copier templates in my ~/.copier-templates directory and I am going to run it from there.

copier copy ~/.copier-templates/setup.py

Results #

After rendering the template we have the following content in our .setup.setup-py-copier-answers.yml file. This will allow us to update quick if we ever change our template.

# .setup-py-copier-answers.yml
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
_src_path: /home/walkers/.copier-templates/setup.py
author_github: waylonwalker
author_name: Waylon Walker
description: awesomeness
framework: null
keywords: null
package_name: my-package

Update it #

This is where I was most stuck, primarily becuase -a <answers_file> must come exactly after the base command copier. This felt a bit odd to and not where I expected it so it.

copier -a .setup-py-copier-answers.yml update

Stop asking all these damn questions #

So the defaults are now changed to our previous results, but it keeps asking for them. To stop asking we can simply add a -f flag.

copier -fa .setup-py-copier-answers.yml update

Making a Textual Widget from a Rich Renderable

2022-01-07T03:25:36Z

Once you have made your sick looking cli apps with rich, eventually you are going to want to add some keybindings to them. Currently Textual, also written by @willmcgugan, does this extremely well. Fair Warning it is in super beta mode and expected to change a bunch. So take it easy with hopping on the train so fast.

Get the things #

Install them from the command line.

pip install textual
pip install rich

Import make a .py file and import them in it.

from textual.app import App
from textual.widget import Widget
from rich.panel import Panel

If you return your rich renderable out of class that inherits from textual.widget.Widget, you can then dock this inside of an app class inheriting from textual.app.App.

class MyWidget(Widget):
    def render(self):
        my_renderable = Panel("press q to quit")
        return my_renderable

class MyApp(App):
    async def on_mount(self) -> None:
        await self.view.dock(MyWidget(), edge="top")
        await self.bind("q", "quit")

run it #

You’ve made a TUI (text user interface). Run the classmethod run to display the it in its full screen glory.

MyApp.run(log="textual.log")

Final result #

At this point It probably does not look much different, but it can be interactive by binding keys to any method on your app that starts with the word action_, this includes the built-in actions such as action_quit.

from textual.app import App
from textual.widget import Widget
from rich.panel import Panel


class MyWidget(Widget):
    def render(self):
        my_renderable = Panel("press q to quit")
        return my_renderable


class MyApp(App):
    async def on_mount(self) -> None:
        await self.view.dock(MyWidget(), edge="top")
        await self.bind("q", "quit")


if __name__ == "__main__":
    MyApp.run(log="textual.log")

pipx examples

2022-01-07T03:04:57Z

count lines of code #

pipx run pygount markata
pipx run pygount markata --format=summary
pipx run pygount markata --suffix=cfg,py,yml

Copier Templates

2022-01-05T01:54:07Z

%%include til/copier_endops

%%include til/copier-template-variables

%%include til/copier-answers

Changing copier template strings (_endops)

2022-01-04T10:55:12Z

I was completely stuck for awhile. copier was not replacing my template variables. I found out that adding all these _endops fixed it. Now It will support all of these types of variable wrappers

# copier.yml
_templates_suffix: .jinja
_envops:
  block_end_string: "%}"
  block_start_string: "{%"
  comment_end_string: "#}"
  comment_start_string: "{#"
  keep_trailing_newline: true
  variable_end_string: "}}"
  variable_start_string: "{{"

!RTFM: Later I read the docs and realized that copier defaults to using [[ and ]] for its templates unlike other tools like cookiecutter.

copier template variables

2022-01-03T12:37:32Z

I’ve been looking for a templating tool for awhile that works well with single files. My go to templating tool cookiecutter does not work for single files, it needs to put files into a directory underneath of it.

template variables #

By default copier uses double square brackets for its variables. variables in files, directory_names, or file_names will be substituted for their value once you render them.

# hello-py/hello.py.tmpl
print('hello-[[name]]')

note! by default copier will not inject variables into your template-strings unless you use a .tmpl suffix.

Before running copier we need to tell copier what variables to ask for, we do this with a copier.yml file.

# copier.yml
name:
  default: my_name
  type: str
  help: What is your name

installing copier #

I prefer to install cli tools that I need globally with pipx, this always gives me access to the tool without worrying about dependency conflicts, bloating my system site-packages, or managing a separate virtual environment for it myself.

pipx install copier

running copier #

When running copier copy we pass in the directory of the template, and the directory that we want to render the template into.

copier copy hello-py .

note! the directory ‘.’ is often referred to in cli programs to represent the current working directory that we are calling the command from.

results #

The resulting files will have your variables injected into them if you have setup your template and copier.yml up correctly.

print('hello-you')

Running Kedro on Ubuntu 21.10 Impish Indri

2022-01-02T20:24:48Z

I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran

pipx run kedro new --starter spaceflights

But what I got back was not what I expected!

Fatal error from pip prevented installation. Full pip output in file:
    /home/walkers/.local/pipx/logs/cmd_2022-01-01_20.42.16_pip_errors.log

Some possibly relevant errors from pip install:
    ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
    ERROR: No matching distribution found for kedro
Error installing kedro.

This is weird, why cant I run kedro new with pipx? Lets try pip.

pip install kedro

Same issue.

ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
ERROR: No matching distribution found for kedro

How to Install miniconda on linux (from the command line only)

Curious what kedro is? Check out this article.

What’s up #

wrong python version

The issue is that kedro only runs on up to python 3.8, and on Ubuntu 21.10 when you apt install python3 you get python 3.9 and the standard repos don’t have an old enough version to run kedro.

How to fix this? #

Theres a couple of ways you can fix this? They all involve installing a distribution that does not come from the standard repo.

Where Can I get the right version #

Anaconda
Python.org
deadsnakes
pyenv
miniconda

I have two articles that can help you #

Using miniconda

conda create -n myenv python=3.8

My first impressions with pyenv

Using pyenv

pyenv install 3.8.12

A Minimal Pluggy Example

2022-01-01T20:35:27Z

Pluggy makes it so easy to allow users to modify the behavior of a framework without thier specific feature needing to be implemented in the framework itself.

I’ve really been loving the workflow of frameworks built with pluggy. The first one that many python devs have experience with is pytest. I’ve never created a pytest plugin, and honestly at the time I looked into how they were made was a long time ago and it went over my head. I use a data pipelining framework called kedro, and have build many plugins for it.

Making a plugin #

super easy to do

As long as the framework document the hooks that are available and what it passes to them it’s so easy to make a plugin. Its just importing the hook_impl, making a class with a function that represents one of the hooks, and decorating it.

from framework import hook_impl

class LowerHook:
    @hook_impl
    def start(pluggy_example):
        pluggy_example.message = pluggy_example.message.lower()

installing pluggy #

Installing pluggy is just like most python applications, install python, make your virtual environment, and pip install it.

pip install pluggy

Making a plugin driven framework #

much less easy

At the time I started playing with pluggy, their docs were less complete, or I was just plain blind, but this was a huge part of the docs that were missing for me that now actually appear to be there. But to get some more examples out there, here is my version.

import pluggy

# These don't need to match
HOOK_NAMESPACE = "pluggy_example"
PROJECT_NAME = "pluggy_example"

hook_spec = pluggy.HookspecMarker(HOOK_NAMESPACE)
hook_impl = pluggy.HookimplMarker(HOOK_NAMESPACE)


class PluggyExampleSpecs:
    """
    This is where we spec out our frameworks hooks, I like to refer to them as
    the lifecycle.  Each of these functions is a hook that we are exposing to
    our users, with the kwargs that we expect to pass them.
    """
    @hook_spec
    def start(self, pluggy_example: PluggyExample) -> None:
        """
        The first hook that runs.
        """
        pass

    @hook_spec
    def stop(self, pluggy_example: PluggyExample) -> None:
        """
        The last hook that runs.
        """
        pass


class PluggyExample:
    """
    This may not need to be a class, but I wanted a container where all the
    hooks had access to the message.  This made sense to me to do as a class.
    """

    def __init__(self, message="", hooks=None) -> None:
        """
        Setup the plugin manager and register all the hooks.
        """
        self._pm = pluggy.PluginManager(PROJECT_NAME)
        self._pm.add_hookspecs(PluggyExampleSpecs)
        self.message = message
        self.hooks = hooks
        if hooks:
            self._register_hooks()

    def _register_hooks(self) -> None:
        for hook in self.hooks:
            self._pm.register(hook)

    def run(self):
        """
        Run the hooks in the documented order, and pass in any kwargs the hook
        needs access to.  Here I am storing the message within this same class.
        """
        self._pm.hook.start(pluggy_example=self)
        self._pm.hook.stop(pluggy_example=self)
        return self.message


class DefaultHook:
    """
    These are some hooks that run by default, maybe these are created by the
    framework author.
    """
    @hook_impl
    def start(pluggy_example):
        pluggy_example.message = pluggy_example.message.upper()

    @hook_impl
    def stop(pluggy_example):
        print(pluggy_example.message)


if __name__ == "__main__":
    """
    The user of this framework can apply the hook in their own code without
    changing the behavior of the framework, but the library has
    implemented it's own default hooks.
    """
    pe = PluggyExample(
        message="hello world",
        hooks=[
            DefaultHook,
        ],
    )
    pe.run()

Modifying behavior #

as a user of PluggyExample

Now Lets pretent the user of this library likes everything about it, except, they don’t like all the shouting. They can either search for a plugin on Google, github, or pypi and find one, or make it themself. the magic here is that they do not need to have the package maintainer patch the core library itself.


class LowerHook:
    """
    This is a new hook that a plugin author has created to modify the behavior
    of the framework to lowercase the message.
    """
    @hook_impl
    def start(pluggy_example):
        pluggy_example.message = pluggy_example.message.lower()

from pluggy_example import PluggyExample
pe = PluggyExample(
    message="hello world",
    hooks=[
        DefaultHook,
        LowerHook
    ],
)
pe.run()

Running Pluggy Example #

Here is a short clip of me running the pluggy example in it’s default state, then adding the LowerHook, and running a second time.

My first impressions with pyenv

2021-12-30T15:26:01Z

pyenv provides an easy way to install almost any version of python from a large list of distributions. I have simply been using the version of python from the os package manager for awhile, but recently I bumped my home system to Ubuntu 21.10 impish, and it is only 3.9+ while the libraries I needed were only compatable with up to 3.8.

I needed to install an older version of python on ubuntu

I’ve been wanting to check out pyenv for awhile now, but without a burning need to do so.

installing #

Based on the Readme it looked like I needed to install using homebrew,so this is what I did, but I later realized that there is a pyenv-installer repo that may have saved me this need.

Installing Homebrew on Linux

List out install candidates #

You can list all of the available versions to install with pyenv install --list. It does reccomend updating pyenv if you suspect that it is missing one. At the time of writing this comes out to 532 different versions!

pyenv install --list

Let’s install the latest 3.8 patch #

Installing a version is as easy as pyenv install 3.8.12. This will install it, but not make it active anywhere.

pyenv install 3.8.12

let’s use python 3.8.12 while in this directory #

Running pyenv local will set the version of python that we wish to use while in this directory and any directory underneath of it while using the pyenv command.

pyenv local python3.8.12

.python-version file #

This creates a .python-version files in the directory I ran it in, that contains simply the version number.

3.8.12

using with pipx #

I immediately ran into the same issue I was having before when trying to run pipx, as pipx was running my system python. I had to install pipx in the python3.8 environment to get it to use it.

pyenv exec pip install pipx
pyenv exec pipx run kedro new

python is still the system python #

When I open a terminal and call python its still my system python that I installed and set with update-alternatives. I am not sure if this is expected or based on how I had installed the system python previously, but it’s what happened on my system.

update-alternatives --query python

Name: python
Link: /home/walkers/.local/bin/python
Status: auto
Best: /usr/bin/python3
Value: /usr/bin/python3

making a virtual environment #

To make a virtual environment, I simply ran pyenv exec python in place of where I would normally run python and it worked for me. There is a whole package to get pyenv and venv to play nicely together, so I suspect that there is more to it, but this worked well for me and I was happy.

pyenv exec python -m venv .venv --prompt $(basename $PWD)

Now when my virtual environment is active it points to the python in that virtual environment, and is the version of python that was used to create the environment.

Links #

https://github.com/pyenv/pyenv#installation

Installing packages with ansible only if they do not exist

2021-12-24T20:24:48Z

Part of my neovim setup requires having the black python formatter installed and callable. I install it with pipx so that I don’t have to manage a virtual environment and have it available everywhere. So far this works well for me, if there are ever breaking changes I may need to rethink this.

re-installing a bunch of things that are already installed can be quite a waste and really add up to my ansible run time, so for most of my ansible tasks that install a command like this I have been following this pattern.

check if the command is installed with command -v <command>
register that step
ignore if that step fails
add a when: <xxx>_exists is failed condition to the step that installs that command.

- name: check is black installed
  shell: command -v black
  register: black_exists
  ignore_errors: yes

- name: install black
  when: black_exists is failed
  shell: pipx install black

https://www.youtube.com/watch?v=MCFg6-W5SBI

I made a video based on this post, check it out if its your thing

Adding rich methods to python classes

2021-12-23T19:34:18Z

Adding a __render__ method that returns a rich renderable to any python class makes it display this output if printed with rich. This also includes being nested inside a rich Layout.

import rich
from rich.panel import Panel


class ShowMe:
    def __rich__(self):
        return Panel("hello", border_style="gold1")


if __name__ == "__main__":
    rich.print(ShowMe())

Smoother Python with automatic imports | pyflyby

2021-12-20T11:34:47Z

This is not a flaky works half the time kind of plugin, it’s a seriously smooth editing experience. I’ve just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython.

I can’t tell you how pumped I am for this, and how good its felt to use over the past few weeks. It’s glorious.

YouTube video #

Listen to me rant on how great pyflyby is

https://youtu.be/2QW5DJiEJH4

Give the video a watch, I did not have noise-cancelling on in obs. My apologies for the background hum and the mic stand bumps. I did my best to fix them up.

Installation #

How to install pyflyby for automatic python imports

pyflyby is hosted on pypi, so you can get it with pip. I have had no issues installing it on 3.8+ so far.

pip install pyflyby

Configuration setup with stow #

always stow your dotfiles

If you’re going to configure any of your tools the first thing you should do is set it up with stow, seriously don’t sleep on the stow. If you don’t have stow installed or choose not to use stow you can skip this part.

cd ~/dotfiles
mkdir ipython
touch ipython/.pyflyby
stow ipython

Seriously don’t sleep on the stow.

How to Configure pyflyby #

it’s just a file full of import statements

pyflyby is configured simply by putting all of your import statements that you want to automatically import into your ~/.pyflyby file. You can import pandas, from pandas import DataFrame, or even import pandas as pd, and all of these will work pretty much as expected.

# comments start with a #
# import your favorite libraries
import visidata as vd
import fsspec
import difflib
import s3fs
import seaborn as sns
import plotly

# also supports from imports
from rich.layout import Layout
from rich.live import Live

# duplicates are allowed
import plotly
import plotly

# duplicate names from different libraries are not allowed
import copy
from numpy import copy

Add all the things you would like to be imported automatically, just as you would import them. I went kinda crazy and added over 200 to mine based on packages that I use.

Commas are even supported #

yep all the import styles are supported

This following example will set up auto import for both DataFrame and Series, they will both work separately. I removed these from my config as I felt it was cleaner without, but it works with them.

from pandas import DataFrame, Series

Even imports with a comma will be treated separately.

Jupyter note #

Both work the same, use what your comfortable with

I only really mention ipython here, but the same all applies to Jupyter as well. I just really like ipython itself, c’mon its right there in the terminal integrating with the rest of your terminal experience so well.

Ipython setup #

Automatically import python libraries in ipython with pyflyby

The recommended way to set uppyflybyfrom the docs is to run the following magic command. This works well, but I want even less typing, I want pyflyby automatically installed and importing things without me even thinking about it.

%load_ext pyflyby

Ipython setup next level #

automatically import modules in python without %load_ext

I really want pyflyby to just work in every environment without me thinking much about it. I want it to load automatically, and even to attempt to install itself if it’s missing.

from IPython import get_ipython
import subprocess


ipython = get_ipython()

try:
    ipython.magic("load_ext pyflyby")
except ModuleNotFoundError:
    print("installing pyflyby")
    subprocess.Popen(
        ["pip", "install", "pyflyby"],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    ).wait()
    ipython.magic("load_ext pyflyby")

Note: if installation fails you will still make it into ipython, there will just be a traceback to the failed command as you enter.

I’ve had zero issues with this, but if there ever comes a time when it does not work in certain environments for you. I’d strongly suggest you to add this to a separate profile.

Ipython-Config

Check out this article for a bit more in depth ipython configuration

ipython auto import examples #

pyflyby can import all the various import types just fine.

import something
from module import something
import something as alias

df = pd.read_csv("https://waylonwalker.com/cars.csv")
[PYFLYBY] import pandas as pd

Getting Help #

Want help on something that you have in your pyflyby config, just give it the ?, ??, or help and pyflyby will import it for you.

Popen?

Just Ask Ipython for help

Autocomplete #

This is next level python auto-import

pyflyby even goes as far as helping tab completion. If you try to tab complete Pop it will complete to Popen without even adding Popen to your local namespace. If you ask for something inside of a module i.e. requests.<tab>, then it will import requests.

# does not populate the namespace
Pop<tab>

# !!does populate the local namespace
requests.<tab>

What happens when a module is not installed #

ModuleNotFoundError

When you are in an environment where you do not have a module installed that is in your pyflyby config, it will throw a ModuleNotFoundError when it tries to import, and it will not import or try to install for you. You will have to change environments or install that module.

❯ pd?
[PYFLYBY] import pandas as pd
[PYFLYBY] Error attempting to 'import pandas as pd': ModuleNotFoundError: No module named 'pandas'
[PYFLYBY] Traceback (most recent call last):
[PYFLYBY]   File "/home/u_walkews/.local/lib/python3.8/site-packages/pyflyby/_autoimp.py", line 1610, in _try_import
[PYFLYBY]     exec_(stmt, scratch_namespace)
[PYFLYBY]   File "<string>", line 1, in <module>
[PYFLYBY] ModuleNotFoundError: No module named 'pandas'
Object `pd` not found.

❯ df = pd.read_csv("https://waylonwalker.com/cars.csv")
<ipython-input-3-69b040434562>:1 in <module>

NameError: name 'pd' is not defined

nvim pyflyby setup #

automatically importing python modules in vim, neovim, nvim

This is by far the best part of this article. It makes development so fluid. It’s not necessarily all about the speed. It really helps you move at the speed of your thoughts, without needing to worry about imports. Remembering where deeply nested modules are does not need to be a thing.

function! s:PyPreSave()
    Black
endfunction

function! s:PyPostSave()
    execute "silent !tidy-imports --black --quiet --replace-star-imports --action REPLACE " . bufname("%")
    execute "e"
endfunction

:command! PyPreSave :call s:PyPreSave()
:command! PyPostSave :call s:PyPostSave()

augroup waylonwalker
    autocmd!
    autocmd bufwritepre *.py execute 'PyPreSave'
    autocmd bufwritepost *.py execute 'PyPostSave'
    autocmd bufwritepost .tmux.conf execute ':!tmux source-file %'
    autocmd bufwritepost .tmux.local.conf execute ':!tmux source-file %'
    autocmd bufwritepost *.vim execute ':source %'
augroup end

refactoring #

This is where it really shines

This setup really shines when you are refactoring. You can freely move modules and classes around without worrying about bringing imports with them. Often when refactoring some modules from one file to another the most tedious part is editing the imports. Often you can’t even grab whole lines because there are several imports and some are needed in both places but not all. pyflyby handles all this like a champ.

Where to install for vim #

just make sure the tidy-imports command is available to vim

pyflyby goes into the environment that you have active at the time that you start neovim. Typically, this is the virtual environment that I am using for the project I am editing.

What gets imported/removed #

only give me what I actually use

Anything within the base config of pyflyby or your own config specified in ~/.pyflyby will get automatically imported if it is used within the file/console. If you are working in a file, and stop using a module, it will automatically get removed.

Anything that is used, and found in the config is added
Anything that is unused gets removed

Where does it put imports #

after the last import

pyflyby does not sort imports into paragraphs or by category. When it needs to add new imports. It will find the last paragraph of imports in your file, add the new one, and sort that paragraph alphabetically.

from collections import Counter

import requests

from plugins.custom_seo import post_render
# <-- pyflyby will put the import here

What about isort #

put those imports where they go

I did not like that I was getting pre-commit issues when using pyflyby, so I added isort to my chain of autocommands to automatically run isort and make my pre-commit happy.

function! s:PyPostSave()
    execute "silent !tidy-imports --black --quiet --replace-star-imports --action REPLACE " . bufname("%")
    execute "silent !isort " . bufname("%")
    execute "e"
endfunction

Let’s write some code

def get():
    """
    Get all the posts from waylonwalker.com.

    Yes theres an rss feed, you should be subscribed if your not already.

    Oh, and we don't need no stinkin error handing because it's always live
    """
    r = requests.get("https://waylonwalker.com/rss.xml")
    return r.content

Save it and pyflyby will inject requests into our file automatically, no need to type that out anymore.

import requests

def get():
    """
    Get all the posts from waylonwalker.com.

    Yes theres an rss feed, you should be subscribed if your not already.

    Oh, and we don't need no stinkin error handing because it's always live
    """
    r = requests.get("https://waylonwalker.com/rss.xml")
    return r.content

What about init / api’s #

careful to fill in the __all__ like you are supposed to

Files such as init.py often import things they do not need, this is simply there for a convenience of the library user and to make the api cleaner. These type of modules should implement a __all__ list of all the unused things that are imported according to pep8. Pyflyby will remove any unused modules unless they are in the __all__ list.

# snippet from kedro.extras.datasets.pandas

__all__ = [
    "CSVDataSet",
    "ExcelDataSet",
    "FeatherDataSet",
    "GBQTableDataSet",
    "ExcelDataSet",
    "AppendableExcelDataSet",
    "HDFDataSet",
    "JSONDataSet",
    "ParquetDataSet",
    "SQLQueryDataSet",
    "SQLTableDataSet",
]

py command #

one liners that need imports

pyflyby also comes with a cli command to run one liners. It’s pretty genius, I’m sure I will find a use or two for it, but so far it’s been more of a novelty for me.

py help pd
py help pd.DataFrame

py pd.read_csv 'https://waylonwalker.com/cars.csv'

Links #

Kedro Course

2021-12-04T11:34:47Z

find all nodes with raw in the name
use parameters
make and use a logger
use find-kedro in spaceflights
slide in a new node
vizualize your pipeline
find slow nodes
move the configuration directory
build kedro into docker and run it
pyinstrument
pdb
make a new cli command
make a hook
load catalog entries
slice a pipeline
- by tag
- by name
- from inputs
- to outputs

kedro catalog create

2021-11-15T07:18:55Z

I use kedro catalog create to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key.

https://youtu.be/_22ELT4kja4

https://search.nixos.org/

👆 Unsure what kedro is? Check out this post.

Running Kedro Catalog Create #

The command to ensure there are catalog entries for every dataset in the passed in pipeline.

kedro catalog create --pipeline history_nodes

Create’s new yaml file, if needed
Fills in new dataset entries with the default dataset
Keeps existing datasets untouched
it will reformat your yaml file a bit
- default sorting will be applied
- empty newlines will be removed

CONF_ROOT #

Kedro will respect your CONF_ROOT settings when it creates a new catalog file, or looks for existing catalog files. You can change the location of your configuration files by editing your CONF_ROOT variable in your projects.

settings.py.

# settings.py
# default settings
CONF_ROOT = "conf"

# I like to package my configuration
CONF_ROOT = str(Path(__file__).parent / "conf")

I prefer to keep my configuration packaged inside of my project. This is partly due to how my team operates and deploys pipelines.

File Location #

The kedro catalog create command will look for a yaml file based on the name of the pipeline (CONF_ROOT/catalog/<pipeline-name>.yml). If it does not find one it will create one and make entries for each dataset in the pipeline. It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline. If you are going to use this command its important that you follow this pattern or copy what it generates into your own catalog file of choice.

⚠️ It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline.

MemoryDataSet’s #

When you run kedro catalog create you get MemoryDataSet, that’s it. As of 0.17.4 its hard coded into the library and not configurable.

range12:
  type: MemoryDataSet

Your free to use what you want though #

Let’s switch this dataset over to a pandas.CSVDataSet so that the file gets stored and we can pick up and read the file without re-running the whole pipeline.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv

Continue adding nodes #

As we work we will keep adding nodes to our kedro pipeline, in this case we added another node that created a dataset called range13.

kedro catalog create --pipeline history_nodes

After telling kedro to create new catalog entries for us we will see that it left our range12 entry alone and created range13 for us.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range13:
  type: MemoryDataSet

Formatting is not worthwhile #

If we decide this is too cramped for us we could add some space between datasets. The next time we run kedro catalog create empty lines will be removed.

range12:
  type: pandas.CSVDataSet

range13:
  type: MemoryDataSet

Continuing to work #

If we coninue adding new nodes, and tell kedro to create catalog entries again, all of our effort given to formatting will be lost. I wouldn’t worry about it unless you have an autoformatter that you can run on your yaml files. The productivity gains in an semi-automated catalog are worth it.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range121:
  type: MemoryDataSet
range13:
  type: MemoryDataSet

Sorting Order #

Notice the sorting order in the last entry, range121 comes before range13. This is all based on how pythons yaml.safe_dump works, kedro has set the default_flow_style to False. You can see where they write your file in the source code currently here

Using Nix to manage my Python Interpreter

2021-11-14T00:00:00Z

I recently started playing with nix.

goals #

automatically select correct python version per project
activating one doesn’t bleed into the other

Installing nix #

curl -L https://nixos.org/nix/install | sh

controlling nix-env #

nix-env -iA nixpkgs.python310
nix-env -iA nixpkgs.python39
nix-env -iA nixpkgs.python38
nix-env -iA nixpkgs.python37

searching for packages #

nix-env -qaP .\*python.\*

nix search nixpkgs python

shell #

nix-shell -p python39

Just Ask Ipython for help

2021-10-10T21:38:26Z

It happens to the best of us #

We can’t all remember every single function signature out there, it’s just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal.

https://youtu.be/TZrRAP-9UMk

Docstrings #

In any python repl you can access the docstring of a function by calling for help.

help(df.rolling)

In Ipython we can even get some syntax highlighting with the ?.

df.rolling?

Source Code #

Sometimes the docstrings are not good enough, and don’t give us the content we need, and we just need to look at the source. Without leaving your terminal there are two ways I often use to get to the source of a function I am trying to use.

import inspect
inspect.getsource(df.rolling)

The more common way I do it is with the ipython ??.

df.rolling??

Bonus rich.inspect #

You thought the syntax highlighting was good with ipython, check out what rich.inspect can do! As far as I know there is no way to get to source code, but the results are still fantastic.

pip install rich

Install rich

from rich import inspect
inspect(cars.rolling, help=True)

then inspect

Connect with me #

If you liked this one, check out the YouTube Channel, catch me live on twitch, or connect on twitter, I’d love to hear from you.

twitter: https://twitter.com/_WaylonWalker twitch: https://www.twitch.tv/waylonwalker github: https://github.com/waylonwalker/ twitch: https://www.twitch.tv/waylonwalker

Setting Parameters in kedro

2021-09-19T00:00:00Z

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files.

https://youtu.be/Jj5cQ5bqcjg

Comprehensive guide to creating kedro nodes

👆 Unsure what kedro is? Check out this post.

parameters files #

You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a base and local parameters file.

conf/base/parameters.yml
conf/local/parameters.yml

base #

The base environment should contain all of the default values you want to run.

# /conf/base/parameters.yml
test_size: 0.2
random_state: 3
features:
  - engines
  - passenger_capacity
  - crew
  - d_check_complete
  - moon_clearance_complete
  - iata_approved
  - company_rating
  - review_scores_rating

NOTE base will always be loaded first.

accessing parameters #

Parameters can be accessed through context or through the catalog. Generally when you are working with nodes it will be loaded through the catalog.

Loding with the context.

context = session.load_context()
context.params

Loading with the catalog.

catalog.load('parameters')

Loading a specific key with the catalog.

catalog.load('params:test_size')

using parameters in nodes #

Here is an example from the complete spaceflights demo. The entire parameters dict is passed in, then the features key is accessed.

def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
    """Splits data into features and targets training and test sets.

    Args:
        data: Data containing features and target.
        parameters: Parameters defined in parameters.yml.
    Returns:
        Split data.
    """
    X = data[parameters["features"]]
    y = data["price"]
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=parameters["test_size"], random_state=parameters["random_state"]
    )
    return X_train, X_test, y_train, y_test

👆 Check out this complete guide to creating kedro nodes.

local #

The local parameters by default are in conf/local/parameters.yml. They will override the base parameters in a shallow fashion. If a top level key exists in local, it will override that entire key in your parameters.

# /conf/local/parameters.yml
env: local
features:
  - company_rating
  - review_scores_rating

NOTE If you have not explicitly set your environment, local will be the default environment selected to override base.

env #

You can also have other environments that override the base environment.

# /conf/new/parameters.yml
env: new

NOTE if you use an env local will not be applied

Activating this environment #

Following the configuration docs we can activate the environment by setting an environment variable in our shell or passing in –env to our kedro cli command.

Setting an environment variable.

export KEDRO_ENV=test

Passing in the env to a kedro cli command.

kedro run --env=test

Setting the Environment Variable in python.

import os
os.environ['KEDRO_ENV'] = 'new'

Links #

Writing your first kedro Nodes

2021-09-13T22:40:45Z

https://youtu.be/-gEwU-MrPuA

Before we jump in with anything crazy, let’s make some nodes with some vanilla data structures.

import node #

You will need to import node from kedro.pipeline to start creating nodes.

from kedro.pipeline import node

func #

The func is a callable that will take the inputs and create the outputs.

inputs / outputs #

Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is the catalog entry to use for that keyword.

our first node #

Sometimes in our pipelines our data is coming from an api where we already have python functions built to pull with. Thats ok, kedro supposrts that with inputs=None.

def create_range():
    return range(100)

make_range = node(
    func=create_range,
    inputs=None,
    outputs='range'
    )

second node #

Now we have some data to work from, lets use that as our input.

def square_range():
    return [i**2 for i in range]

square_range = node(
    func=square_range,
    inputs='range',
    outputs='range_squared'
    )

Multiple Inputs #

Kedro can take lists or dicts as either input or output when your function needs more than one input or output.

def concat(range, range_two):
    return [*range, *range_two]

concat_ranges = node(
    func=concat,
    inputs=['range', 'range_squared']
    outputs='concat'
    )

## inputs could also be defined as a dict
concat_ranges = node(
    func=concat,
    inputs={'range': 'range', 'range_two': 'range_squared'}
    outputs='concat'
    )

Links #

Running your Kedro Pipeline from the command line

2021-08-24T22:40:45Z

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli.

https://youtu.be/ZmccpLy-OEI

https://youtu.be/ZSxc5VVCBhM

👆 Unsure what kedro is? Check out this post.

Kedro run #

To run the whole darn project all we need to do is fire up a terminal, activate our environment, and tell kedro to run.

kedro run

Specific Pipelines #

Running a sub pipeline that we have created is as easy as telling kedro which one we want to run.

kedro run --pipeline dp

Single Nodes #

While developing a node or a small list of nodes in a larger pipeline its handy to be able to run them one at a time. Besides the use case of developing a single node I would not reccomend leaning very heavy on running single nodes, let the DAG do the work of figuring out which nodes to run for you.

kedro run --pipeline dp --node create_model_input_table_node
kedro run --pipeline dp -n create_model_input_table_node

Some DAG concepts #

We will cover more of the benefits that we get from the graph nature of the DAG in the future, but here is a quick peek at some things we can do.

kedro run --pipeline dp --to-outputs preprocessed_shuttles
kedro run --pipeline dp --from-inputs preprocessed_shuttles
kedro run --pipeline dp --to-nodes create_model_input_table_node

Multiple things #

You can stack up multiple kedro dag concepts into a single run command.

kedro run --pipeline dp --to-nodes create_model_input_table_node --to-nodes preprocess_shuttles_node

kedro Virtual Environment

2021-08-23T20:40:45Z

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment.

conda
venv
pipenv

conda #

I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don’t have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

conda info --envs

stores environment in a root directory i.e. ~/miniconda3
conda can use its own way to manage environments environment.yml
the python interpreter is packaged with the environment

virtualenv #

Virtual env (venv) is another very respectable option that is built right into python, and requires no additional installs or using a different distribution of pytyhon.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

environments are typically stored in the project directory
does not package the interpreter

pipenv #

Pipenv is another virtual enviroment tool that comes with its own system for managing dependencies using a pipfile. It’s main benefit is that it creates a lockfile that will allow users to replicate the exact version of all their packages. The typical requirements.txt workflow can easily break as new version of dependecies are released between testing and deplpoyment.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

stores environment in a root directory i.e. ~/.local/share/virtualenvs/
pipenv can use its own way to manage environments pipfile
does not package the interpreter

Kedro Pipeline Create

2021-08-22T22:40:45Z

Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself.

https://youtu.be/HtyIKqlEoNw

creating a new pipeline #

The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your pipeline_registry, to be covered later, you will need to add it yourself.

kedro pipeline create example

results #

The directory structure that it creates looks like this.

tree src/kedro_conda/pipelines
src/kedro_conda/pipelines
├── __init__.py
└── example
    ├── __init__.py
    ├── nodes.py
    ├── pipeline.py
    └── README.md

Kedro Install

2021-08-21T20:40:45Z

Kedro comes with an install command to install and manage all of your projects dependencies.

https://youtu.be/IWimEs-hHQg

cd into your project directory and activate env #

You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment.

Kedro New

this post covers kedro new

kedro Virtual Environment

This post covers creating your virtual environment for kedro

install kedro #

Make sure you have kedro installed in your current environment, if you dont already have it.

pip install kedro==0.17.4

pip-tools #

Kedro uses the pip-tools package under the hood to pin dependencies in a very robust way to ensure that the project will continue to work on everyone’s machine day, including production, day in and day out. No matter what happens to the dependencies you have installed.

pip-compile #

The command that kedro uses from pip-tools is pip-compile. It will look at what you have in a requirements.in file, compile the dependencies down to exact versions, and create a requirements.txt that is fully pinned down, and updatable by re-running pip-compile.

requirements.in #

If kedro does not see a requirements.in file it will automatically move your requirements.txt to requirements.in and run pip-compile.

No requirements.in found. Copying contents from requirements.txt...

kedro install #

Lets go ahead and run kedro install on one of the projects we already create and environment for in a previous post, kedro-conda.

kedro install

The first time you run this on a new repo it is likely that you will run into this warning about creating a new requirements.in file.

No requirements.in found. Copying contents from requirements.txt...

kedro install flags #

Kedro does let you avoid pip-compile all together, by using the --no-build-reqs flag.

kedro install --no-build-reqs

It also lets you upgrade all of your dependencies with build-reqs. I would reccomend doing this on its own branch, and own pull request. If you are working on a team you want everyone to be on the same page when it comes to dependencies. If you are not, you surely do not want something to break with a new set of dependencies without a way of rolling back.

kedro install --build-reqs

Kedro Git Init

2021-08-20T22:40:45Z

Immediately after kedro new, before you start running kedro install or your first line of code the first thing you should always do after getting a new kedro template created is to git init.

https://youtu.be/IGba3ytf_6U

git-init">git init #

Its as simple as these three commands to get started.

git init
git add .
git commit -m init

I don’t care if this project is for learning, if it will never have a remote or not, use git.

Kedro New

2021-08-18T20:40:45Z

https://youtu.be/uqiv5LAiJe0

Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template.

kedro Virtual Environment

Unsure what kedro is, Check out yesterdays post on What is Kedro.

pipx #

I reccomend using pipx when running kedro new. pipx is designed for system level cli tools so that you do not need to maintain a virtual environment or worry about version conflicts, pipx manages the environment for you.

The kedro team does not reccomend pipx in their docs as they already feel like there is a bit of a tool overload for folks that may be less familiar with

pipx kedro new

I like using pipx as it gives you better control over using a specific version or always the latest version, unlike when you run what you have on your system depends on when you last installed or upgraded.

Kedro New #

The kedro team also has a set of starters, by passing in --starter you can start with a different template. Here is an example with the kedro spaceflights starter.

pipx run kedro new --starter spaceflights

=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
 [New Kedro Project]: Spaceflights Complete

Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
 [spaceflights-complete]:

Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter
or underscore.
 [spaceflights_complete]:

Change directory to the project generated in /home/u_walkews/git/spaceflights-complete

A best-practice setup includes initialising git and creating a virtual environment before running ``kedro install`` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

Other versions of kedro with pipx #

pipx not only ensures that you run the latest version, it can also run a very specific version.

pipx run --spec kedro==0.16.6 kedro new

The next post in this series will help you create your virtual environment for your new kedro project.

What is Kedro

2021-08-17T12:48:00Z

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract DataSets that the user specifies through Catalog entries. These Catalog entries are loaded, ran through a function, and saved by Nodes. The order that these Nodes are executed are determined by the Pipeline, which is a DAG. It’s the runner’s job to manage the execution of the Nodes.

https://youtu.be/Wf4rnFsaFFU