Blog Archive

/carry/

2026-03-27T16:35:40Z

EDC #

I try to keep a pretty light every day carry, but it never works out, keyfobs and headphone cases end up causing more bulk than I’d like, but My EDC is no where near the bulk I had as a kid with my cargo pants decked out with everything I could possibly need.

I hold no attachment to anything in my EDC. Nothing on my person has sentimental value. Anything I carry can be lost, stolen, or destroyed at any point in time. I pick things of sufficient usable, utilitarian, quality sufficient to work. No extra fluff.

Google Pixel 6
Ridge Wallet
Olight Perun - Since April 2022
Civivi Qubit - Since March 2024 after my last one was donated to the Indianapolis Stadium
OpenFit Shokz

Rules #

lightweight
replacable
no emotional attachments
utilitarian
everything serves a purpose

My kit #

Photo taken March 2023

Phone #

Google Pixel 6

I will not get finance a new phone for the foreseeable future again. The last time I feel like I got a phone that felt like an upgrade was 2018 when I went to the Pixel 3. The pixel 6 and pixel 8 my wife has feel no different whatsoever to me. It’s become more of a tool that does its job, there’s no major upgrades really driving me to create more ewaste with new devices.

Wallet #

Ridge Wallet

A necessary evil that I still need. I keep as few of cards as necessary in it. This would be the worst thing in my kit to loose. Only because the time it would take to replace all of them. This one was a gift to replace my last leather wallet, and I see no reason for this one to ever need replaced. It’s much lighter and smaller, would not change that aspect.

Flashlight #

Olight Perun

When I became a homeowner I added a dedicated flashlight to my EDC. Its light and out of the way. It’s the only thing I’ve ever found useful to fit in the coin pocket of jeans, so I don’t even notice it. Compared to using a phone, its more powerful, easier to position on me with the clip on my shirt/pants, or magnetically to a metal surface for inspections/house work.

Knife #

Civivi Qubit

I’ve carried a knife since as long as I can remember, feels like 10 but probably more like 14 years old. Never anything fancy, a few knock off leathermans, a few Buck, Case, KaBar that I found on a bridge, and now 3 Civivis. I use them till they break or more commonly they get lent and not returned, or lost. This is the most utilitarian thing I keep, almost always kept razor sharp, used, abused, and put away dirty. Never used for prying, but used to cut anything it will cut no matter how hard it is on it. Its not to look pretty, show off, its purely a tool that I hold no attachment to.

Headphones #

OpenFit Shokz

Not my first choice in headphones, but they work. Headphones are probably the item of biggest churn and ewaste for me. Eventually I might pony up for something that is more durable and repairable, but headphones just don’t last forever before they stop working for me. This latest pair was a pair my wife chose to try and did not like. They sat in a drawer for a year or so until I started carrying them.

I Built A Tmux Session Switcher

2026-03-04T20:47:37Z

I’ve been thinking about this for awhile now. For years now, fuzzy pickers and last session have been my go to. They have served me well. I can typically only keep so much in my head anyways. I’m often doing a hub and spoke pattern between main project, notes, and infra repo, maybe two projects. Don’t get me wrong, I regularly run with a dozen or more sessions running at a time, but only two to three are in my immediate context at any point anyways.

The Design #

harpoon for tmux

press a hotkey followed by one more keystroke, currently any left hand letter SIMPLE, FAST, thats of utmost importance, what I want are sessions that I can can be assigned in order of importance from middle row, top row, bottom row.

I added this binding to my tmux config. Now I can press c-a a to go to the first session, c-a s to go to the second session. c-a and pause to think j/k to navigate, space to pick up a session and move it, x to kill it.

bind-key -n c-a popup -E '~/go/bin/tgo'

Enter the agents #

Now with agents doing more and more work, and cooking for longer periods this is not cutting it, I’m often swapping through a lot more sessions, I’ve got more projects sitting in more phases between research, implementation, fully orchestrated agents, and sessions that need their hand held to get things right.

Image of the tgo tool in action, I’d show you a video, but its not really that interesting, it just flashes up and goes to the next session as fast as you can press the key.

Made with gpt-5.3-codex and gpt-5.1-codex-mini #

Idk how much it cost, it barely touched my $20/month weekly token allowance. I’ve been thinking about it for awhile and decided today was the day to pull the trigger on something and get it started, worst case I throw it away. It took the agents about 7 minutes to implement a working version with broken ci that it later fixed.

I’ve typed far more characters and put more thought into this post than the tool itself, that where we are these days I guess.

See the session that made tgo Ses_3444b09b1ffexter3elfg5qqtg

Should you use it? #

You can its there for anyone to use, go for it, there’s an easy to download binary.

It’s all vibe coded slop that you could probably do a better job doing in a cave with scraps anyways. Take inspiration and make one that works for you.

/verify

2026-02-24T10:36:57Z

Inspired by @mollywhite’s verify slashpage. This page serves as the system of record for my online identity. The best places to follow me are:

My Website: waylonwalker.com
YouTube: Waylon Walker
YouTube Gaming: Waylon Walker Gaming
Twitch: Waylon Walker
Twitter: @_WaylonWalker
LinkedIn: Waylon Walker
GitHub: WaylonWalker
Dev.to: Waylon Walker
Bluesky: @waylonwalker.com

First W In Brotato

2026-02-11T20:48:08Z

After having brotato and doing a few runs every once in a while I finally beat the most basic balanced run in the game! Wyatt wanted to play tonight and its such an easy game to jump in do a few runs and move on without getting overly invested.

Watching back I cannot believe how lucky I got, barely scraping by with 1hp at this point

Your browser does not support the video tag.

the last 80s of the game

/top4

2026-02-11T14:00:00Z

Definitive ranked lists of my top 3 favorites plus an honorable mention.

Terminal Tools #

neovim - modal editing changed how I think about text
tmux - terminal sessions that survive disconnects and allow me to hop between projects at the speed of thought.
k9s - S tier tui interface that all tuis should strive for

Honorable mention: ipython

Python Tools #

pandas - This is what got me out of corporate spreadsheets and back into code/software.
kedro - data pipelines with opinions I agree with
fastapi - my favorite python web framework

Honorable mention: typer - fast cli apps

Games to Play with Kids #

multiplayer

Minecraft - infinite creativity, modding potential
Wobbly Life - Open World Co-op
Stardew Valley - cozy, collaborative farming

Honorable mention: Terraria - 2d world builder

Games to Play Alone #

singleplayer

Hollow Knight/Hollow Knight Silksong - S tier 2d platformer with adventure, mystery, lore, boss fights.
Session Skate Sim - The best skate sim
Animal Well - epic 2d platformer with the most incredible graphics at 320p ever created and the deepest puzzles Ive seen

Honorable mention: Portal

Inspired by topfour.net and slashpages.net

/yep

2026-02-11T12:00:00Z

Inspired by @fyrio’s yep slashpage, a list of S tier things I enjoy, use, recommend, want to know more about, or seek out in no particular order, updated as I think about it.

Seealso

/nope

Inspired by @baty's nope slashpage, a list of ** things I don't like, don't care, avoid, overhyped, or won't do in no particular order, updated as I think...

Feb 11, 2026

/yep #

coffee
small web 1.0
RSS
minecraft
Hollow Knight
Silksong
terminals
python
vim keybindings
self-hosting
open source
running kubernetes in my basement
mechanical keyboards
markdown
data engineering
Woodworking
Fingerboarding
Darts
Skateboarding
Biking
Trampoline
3d printing

/nope

2026-02-11T12:00:00Z

Inspired by @baty’s nope slashpage, a list of F tier things I don’t like, don’t care, avoid, overhyped, or won’t do in no particular order, updated as I think about it.

Seealso

/yep

Inspired by @fyrio's yep slashpage, a list of ** things I enjoy, use, recommend, want to know more about, or seek out in no particular order, updated as I...

Feb 11, 2026

/nope #

roblox
mobile games
telemetry
Windows
VSCode
allow notifications
subscription hell
ads
social media
clickbait
WYSIWYG editors particularly ones that use proprietary non text formats
politics
short form video

Pm Not Babysitter

2026-02-06T14:04:25Z

Stop babysitting your agents, treat them like a real team and they will reward you.

Back in December I saw theo make a comment that code is now cheap, its the run rate of models, He quoted a study, not sure that he fully even believed it, but it claimed that the average developer after all meetings, training, emails, planning and extra shit in their day averages out 10 well tested lines of code per day. Opus 3.5 made him 10k loc (lines of code) that day.

We have all agreed for decades that lines of code is not a proxy to productivity or quality. Often more code means more risk, more review, more infrastructure. This has become MUCH different. Lines of code are still far from any sort of good metric. That aside, your agents are not doing 10k lines with you babysitting them, and in fact its very likely that the product quality is MUCH worse as you babysit them.

You need a tool for planning and tracking, otherwise you are playing babysitter rather than Product Manager (PM).

Agent Management Is Exhausting

2026-01-26T14:38:35Z

The state of development in early 2026 is all wrapped around learning how to manage many agents running in parallel. Everyone’s trying to figure out the workflow.

The Plan Is Everything #

The secret I’ve discovered is a good, well-defined plan. This could be a markdown file or a GitHub issue. Agents are actually great at writing these for you. They’ll include reproduction steps, outline changes needed, and structure the work.

This is your opportunity to step in. Read the plan. Look for hallucinations. Spot where it’s going off track. Edit the plan before the agent starts coding.

I had one today where it laid out reproduction steps beautifully, but I could add context about network requests that completely changed the approach. This editing phase is what most people are missing right now. Skip it and you’ll watch your agent solve the wrong problem with impressive efficiency.

The Pace Problem #

Here’s what nobody warned me about: managing these things is exhausting.

Depending on the day, agents move so damn fast. I can barely research, find, and raise issues as fast as Claude can implement features and fixes. It’s like trying to speedrun a Minecraft seed when you just figured out how to craft a pickaxe.

A Different Kind of Work #

This stretches a different part of my brain than I’m used to using. I’m learning new skills around:

Issue tracking and management at high velocity
Knowing which models handle which tasks best
Spotting when a session is about to go sideways from context bloat or bad compaction

I had a session yesterday where the context got poisoned with a wrong assumption. The agent spent 20 minutes building on that false premise before I caught it. That’s 20 minutes of perfectly executed code solving the wrong problem entirely.

Worth It, But Hard #

I’m not going to hype-bro you and say this is easy or that you’re wrong to be skeptical. If you have the opportunity to work with agents, it’s worth learning proper techniques.

It’s not easy. It requires actual management skills, not just prompting skills. And yeah, I’m tired.

Stop Using Boomer Ai

2026-01-26T13:12:29Z

I was listening to these guys talk about migrating off of boomer ai the other day. Introducing the term boomer ai to describe using chat, copy, paste instead of agents. Something magical happened to the tooling and models around december, they got really good. The chatgpt $20 plan hooked into opencode is good, the Free models in Opencode Zen (Big Pickle and Kimi K2.5 Free) are really good. Neither of these quite match up to the speed and quality of the larger plans, but they are good. good enough to throw away your boomer ai techniques and start using agents. Agents are the future, and they are here now. If you are still using chat, copy, paste, you are doing it wrong. Stop using boomer ai and start using agents. You will be amazed at how much better your results will be.

https://www.youtube.com/watch?v=3dmPKuTWbsI

Mentions

2026-01-24T19:11:15Z

I can now just mention people from my markata Waylon Walker [[ blogroll ]] like @simonwillison.net or @swyx.io

/now [[ upto ]] [[ doing ]]

What Your Coding Tool Says About You

2026-01-21T16:25:41Z

open code - libre free as in beer and speech
Copilot - corporate 9-5er
Cursor - You sip on Philz coffee with your macbook
Claude Code - Agentic Workflows or Bust
Jetbrains - I didn’t know you wrote java
Vim/neovim - definite neck beard
VsCode - What else is there?

📝 Git Notes

2026-01-15T09:05:21Z

See old revisions of one file #

git log --oneline -- <file>
git log -n 2 --oneline -- <file>

Checkout an old revision of a file #

git checkout <commit> -- path/to/file

fuzzy pick a file and check out an old revision #

#!/usr/bin/env bash
set -euo pipefail

file="${1:-}"

if [[ -z "${file}" ]]; then
  file="$(git ls-files | fzf --prompt="select file > ")" || exit 0
fi

if [[ -z "${file}" ]]; then
  exit 0
fi

if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  echo "Not a git repository." >&2
  exit 1
fi

if ! git ls-files --error-unmatch -- "${file}" >/dev/null 2>&1; then
  echo "File is not tracked by git: ${file}" >&2
  exit 1
fi

choice="$(
  git log --follow --pretty=format:'%h %ad %s' --date=short -- "${file}" |
    fzf --ansi --no-sort --reverse \
        --preview-window=down:70% \
        --prompt="checkout revision > " \
        --preview "git show --color=always {1}^..{1} -- '${file}' 2>/dev/null || git show --color=always {1} -- '${file}'"
)"

if [[ -z "${choice}" ]]; then
  exit 0
fi

commit="$(awk '{print $1}' <<<"${choice}")"
git checkout "${commit}" -- "${file}"

Dont Trust Users Tokens

2026-01-14T19:49:43Z

User states: Upon picking up an old project and trying to install pip says “cannot find a version to satisfy”

I’ve got this, I’ve had this a hundred times before it’s a python version, a rogue package, maybe a yank from the pinned deps. I pop open the project get us on the same commit. I get a different error, make a few updates and we are good, except the user gets the same error from the start.

They never saw the error I did, and my fix did not magically resolve their error. We circle all the things it could be for hours. I consistently wipe my venv, and recreate with ease, send them the commands I ran to no avail. Something is up and I can’t put my finger on it. We’ve checked all the things and inched as close as we can to running everything exactly the same.

Os
python version
Network vpn
uv version

Nothing makes any sense. Finally I throw in the towel, is it the artifact server. I forge a token and give him one to borrow.

BAM it works, like magic. The first sign of progress. Then he mentions.

Huh that’s odd cause I just got mine this morning

Failing to mention this any earlier that getting a new token for a service and it completely borked it! I get it though, the error was very oddly presented and not easy to see why

There it was all along, looking back in the logs I see his redacted token going to the wrong registry. One that does not include our packages, everything makes sense now. The reason it couldn’t find a version to satisfy was not a python version, os version, package conflict, it was that it couldn’t find a fucking version of the thing to begin with.

All this to say, don’t trust users tokens, save your time and just get fresh ones with them.

if you read this it’s all cool, like I said it’s all part of the job, no harsh feelings, at least we got a good story out of it right!

Ai

2026-01-11T13:48:43Z

Last updated Jan 2026.

Seealso

Looking for all posts tagged with ai see [[ tag/ai ]]

Ai is a tool I use a lot for code generation, research, image generation, and debugging. The words I publish on this site are my own unless explicitly stated from the top. There’s only one or two posts in this category.

Words are mine #

The core of what this blog is, is my thoughts ideas, sharing experiences. The words are the important part. They are not perfect, I often do not spell or grammer check, and what is here is from a flow state of writing and very often not refactored.

No AI unless explicitly stated

code gen #

Up till around 2023 all of the code to create the site was hand written by me. I have my own site generator that I maintain markata.

From 2023 through 2025 snippets of code, sometimes entire plugins or modules were created in chat apps like chatgpt. There was no agentic coding, windsurf, cursor, claude code, opencode, none of it. It was all integrated into the site by hand.

Starting around Dec 2025 I was finding freely available agents through tools like opencode to be more and more useful, making less mistakes, doing better at getting context and making changes. The flow is relying on heavy planning and issue reporting.

image gen #

I really got into stable diffusion as it came out. I thought it was fascinating to generate different kinds of images of things that I would otherwise have no ability to create, for a long time I was generating images for every post. I’ve moved on from doing it for every post, but still do it occasionally.

I’m not an artist, sometimes I like to include art that I would not otherwise be able to. Most of the art you see on a post is AI generated, and may not be called out.

Techbrophobic #

I feel this term strongly. I find ai tooling useful, but the industry and hype bros bring a bad taste.

The tools do not not do everything right, you still have to think, watch its output. You have to put the work in to do proper software engineering. Its not a magic pill, but it has the ability to implement ideas, plans, very quickly.

Also the whole industry creeps me out. The theft that happened to get here was wrong. AI shovoled into everything as slop sucks. The stock market pump is sus.

Og-Sample

2026-01-09T08:19:29Z

I’m making an effort to make my og images better yet again, I’m going for that next 10% better. I really like my og images, but there are some title sizes that overflow. This page is a page to help debug. How I make these og images is for another day.

Script #

I use my own static site generator markata. I can use it to generate a list of posts wrapped in their og image. I use itertools to do a groupby so that I can do roughly every 5 characters larger, and see a wide variety of sizes.

from markata import Markata
from itertools import groupby

markata = Markata()
lens = [{'length': len(post.title), 'title': post.title, 'slug': post.slug} for post in m.posts]
lens_sorted = sorted(lens, key=lambda x: x["length"])

groups = {
    k: list(g)
    for k, g in groupby(lens_sorted, key=lambda x: (x["length"] // 5) * 5)
}

posts = [g[0] for g in groups.values()]
ogs = [ f'[![{post["title"]}](https://shots.waylonwalker.com/shot/?url=https://waylonwalker.com/{ post["slug"] }/og/&height=600&width=1200&scaled_width=1200&scaled_height=600&format=jpg&v=4)](https://waylonwalker.com/{ post["slug"] }/og/)' for post in posts]
print("\n\n".join(ogs))

OG-Sample #

shots #

I also do Posts tagged: shots posts that change the og template quite a bit by adding the image to the page. I’ll do the same script, but add a filter to the images.

from markata import Markata
from itertools import groupby

markata = Markata()
lens = [{'length': len(post.title), 'title': post.title, 'slug': post.slug} for post in m.filter('templateKey=="shots" and "family" not in tags')]
lens_sorted = sorted(lens, key=lambda x: x["length"])

groups = {
    k: list(g)
    for k, g in groupby(lens_sorted, key=lambda x: (x["length"] // 5) * 5)
}

posts = [g[0] for g in groups.values()]
ogs = [ f'[![{post["title"]}](https://shots.waylonwalker.com/shot/?url=https://waylonwalker.com/{ post["slug"] }/og/&height=600&width=1200&scaled_width=1200&scaled_height=600&format=jpg&v=4)](https://waylonwalker.com/{ post["slug"] }/og/)' for post in posts]
print("\n\n".join(ogs))

2026 Resolutions

2026-01-08T10:48:12Z

It’s that time of year, Coming back to work out of a big break and thinking about big changes.

Daily Notes #

Keep up with daily notes, maybe not here, I tend to have more targeted notes here with full blog posts, but for work daily notes is POG and needs to be leaned on. LLM’s do really good at ingesting markdown and reminding me of things that I need to do, or did, or need to follow up on.

Reader #

Social media is changing, quickly becoming enshitified, I enjoy interacting with some of the people I’ve met online, reading their opinions, and learning from their experiences. I don’t need their hot takes, don’t care about their political takes. I like boring posts that typically fade out of whats picked up on the algorithm. I like a good 5-10 minute read or a long form podcast talk. I will be aggressively collecting more rss feeds to read and keep up with.

(Neo)Vim #

I’ve had these bad habits in my config for years, from day one of using vim. It’s time to kill these. jjgww. I killed them this morning, we will see how long I keep typing out jj all over my editor before I fix it mentally.

-- bad habits
vim.keymap.set("i", "jj", "<esc>")
vim.keymap.set("i", "jk", "<esc>")
vim.keymap.set("i", "JJ", "<esc>")
vim.keymap.set("i", "jJ", "<esc>")
vim.keymap.set("i", "Jj", "<esc>")

Tip

I need to lean on <C-o> a little more to run single commands in normal mode from insert mode. The pattern that is hitting me hard right now is jjgwwA insert mode, format, insert at end of line.

Distrobox #

I was a heavy user of distrobox as my primary dev environment on Bazzite and I need to get back there. I have a lot of stuff installed on this current system. I’ve been sloppy. I am not sure I was planning on staying on arch/hyprland long, I did it to do it with my son, his got borked a month ago and he is back to bazzite, I think I’ll be back there and want to be ready to float freely between distros.

clean dotfiles
work from a distrobox image that I make
keep working projects backed up

Vibe Coding #

Agents are the future, people that I really trust are on both sides of the argument. Some saying they stopped coding months ago and claude does everything for them. Some in between, some are completely against.

It feels like the tooling has really taken off over the past few months, the models are getting better, but not at the same rate they were. The tools are really making this possible.

My experience is quite mixed, I’m often impressed with what it does, and appalled at how bad it does other things. I just want it to do what I mean with the fewest words possible. I’ve been working on deeper plans more and more, still with mixed results. With larger plans the models seem to do more work, but still not getting that great experience every time.

I really need to get my workspaces workflow down, have models working in a separate workspace, and be willing to throw away shit code quickly and try again from another angle.

The future is a deep understanding on what needs to be done, how do to it, and not the syntax. The tools are getting better. You need to know less and less about them, less and less about models as the auto routers get better. The core skills of architecture, design, and implementation are more important than ever.

Last year I was able to vibe out some POC work, but still not letting the agents touch my production code. It wasn’t there for me, I probably did not have access to the really good stuff either. Now I think what I have access to is catching up to Claude.

sandbox them
quick to POC
quick to move on from broken attempts
learn to lead
learn to let go of syntax opinions
learn what makes good products/tools/apis hold the vibes to this.

There’s something here, very respectable people are betting on this, but its not working for me, I don’t think I have the right tools. I already have it a day, and got a few hundred LOC out while theo is claiming opus can do 10k.

Not Everything will be a side hustle #

I’ve tried to start too many side hustles, none of them work out. Thinking of them as a side hustle steals the joy of the hobby. I have too much going on outside of work for adding things like this not to add stress to already stressful situations, I need to reduce stress and find the joy and happiness in side projects and not turn everything into something more.

Keebs #

This one is fresh on the mind only because I just built out a new keyboard for work. I want to have a good robust build at home, work, and potentially have a good backup. I have a good design that is very cheap and fairly easy to build, but is quite time consuming. I am going to put some work in early this year to get a couple of clones.

More Offline #

Looking back at 2025, I actually did fairly well of doing a bit more offline. It’s been a hard transition out of pandemic times for our family, we had a lot of extra health precautions that lasted longer than most other people.

This year I should do more of this. More bike rides, more skate, more basement hangouts, more art.

New dart board in the basement

More creativity. I designed and built fingerboard obstacles to work like legos with 3/4" plywood last November.

building a fingerboard park in the basement

More building

building set for Jolly Holiday with Rhiannon

building boxes for Jolly Holiday with Wyatt

More bike rides, fewer nice days wasted, they are relaxing and get good energy out.

Bike Rides

Developer Vs Artist Ai

2025-12-19T09:44:00Z

The other day I was watching [thePrimeTimeagen]https://youtube.com/@theprimetimeagen?si=jVcp23FbfQSFZfDc) and he talked about devs loving ai and artists revolting. There was some discussion in chat about art being more creative and prime quickly squashed that. He ended with being oddly confused why developers are jumping on board and artists are not. Both had their art stolen to build out the models.

my own vibes

I'm writing this from my phone without further research, all vibes, personal experience, and thoughts,  no research.

Good Tools #

First I want to argue that artists have had some form of ai in their tools for years. Idk, probably not ai as we know it today but functionally similar. Content aware fill. This is a Photoshop feature from Adobe, as far as I know it’s one of the special things you get from Adobe that you don’t get from the FOSS alternatives easily.

This is an example of a good took that is well loves by the community and widely used, if you put ai in it to make it better no one would care, if it made things look oddly ai it would get rejected.

taking the brush #

Here is the critical difference that I see between the communities. Software development has made great tools for itself in a wide spectrum, from hands off vibe code no code to better auto complete. There’s something for everyone and every situation. This feels very obvious the minute you try to look at how to do it. In fact we started with better autocomplete. This autocomplete is really good at recognizing patterns in your document and continuing your current thought for you.

Art is a complete 180. It started with prompt engineering. Hands off let the tool do everything, you do nothing but prompt it. If it’s wrong reprompt. Diffusion is the worst because it generates completely new results every time nothing like what you had before. The newer stuff from openai and nano banana are not diffusion and do much better text and edits, but it’s not the same.

hands off is frustrating #

Anyone who has tried to get something specific from ai art or code gen without touching it themselves knows the frustration that you ask it to do one thing that takes 3 prompts to get it to understand what your asking and by the time it’s right it’s wrecked something else. It’s not fun, it’s soul sucking and ripe for meme content.

freedom of expression #

Software development has made a really good effort to make really great tools for itself. We have a wide spectrum that allow you to prototype fast, get something complex working in hours not month, but riddles with issues. Allowing us to get better scoped work from PMs who have put their hands on a semi working prototype. We have the ability to slice out parts and refactor with ease, and when things really matter we can right the code ourselves with ai there to take away the boilerplate repetition that finds it’s way into some modules.

I’m sure there is better ai art tools out there that allow artists to keep their brush in hand and enhance their skills, but I have not seen them. Ive seen in painting, but its not discoverable as it’s first experience, is hard to setup, might cost a lot, and doesn’t quite fit in to where you work with it. It’s still all or nothing prompt engineering, just a bit more focused in.

ai video #

I haven’t even thought much about things like sora throughout this. I can say for sure I’m enjoying editors for small time creators having the ability to throw in short clips to enhance the story. Especially for story telling style content that is impossible to get real video of, and would cost a fortune to build set, costume, and cast. It makes no sense for this scale. The McDonald’s and coke videos im on the fence on. They seem fine, they don’t feel inspiring, great, or terrible to me.

fin #

Developers naturally started with great tools here, and artist were given tools for developers to do art with, nothing tools for them.

Please tell me I’m wrong, artists have great tools you just haven’t seen them. I’d love to be wrong.

The Right Reasons To Run Kubernetes In Your Homelab

2025-12-10T09:48:14Z

Running kubernetes in your homelab is a fantastic way to learn, explore, express yourself, and run services that you use.

The Right Reasons To Run Kubernetes In Your Homelab #

There are not many

You want to learn kubernetes
You like kubernetes
You want to learn to scale

There are also The Wrong Reasons To Run Kubernetes In Your Homelab

You want to learn kubernetes #

Homelabbing is a such a great way to learn new skills, deploy real apps that you use. Create new custom apps for your specific use cases that no one else has. You should absolutely run kubernetes in your homelab if you want to learn it.

I would recommend to start locally, pull up kind, minikube, or k3d and start from your local machine before putting it on a server.

When you decide you are ready for a server, you probably don’t need any crazy hardware. You can probably run on some old retired Dell Optiplex or an old desktop someone is throwing out as it no longer runs windows.

You like kubernetes #

Hell Yeah Brother, 100% no better reason to run kubernetes at home than because you enjoy it. I’m with you. There’s nothing quite like having git ops kick in and deploy new services, updates, watching deployments rollover with zero downtime. Watching your cluster heal itself when a node goes down. Never ssh-ing in to do deployments. Still owning your entire hardware.

You want to learn to scale #

This is probably a stretch reason, maybe not a good one, there are probably better ways, but here we go.

Don’t claim that you need scale in your homelab, you don’t. But it sure is fun to run a cluster of nodes, and load balancing services that run across them. Solving these hard problems to scale across machines is hard. There’s no way around it, there’s a lot to think about. Doing so in a low stakes environment that you have skin in the game is a great way to learn.

I run kubernetes in my homelab #

I run it and I really like it

What flavor of autism did you guys get, I got the kind where I run kubernetes in my basement.

Here are some things I really like about it, and Yes I know you can achieve most of these without kubernetes.

I don’t have to ssh, hardly ever.
I can see everything I’m running, and its defined in a manifest
k9s is amazing, and I use it all the time.
- shell into running pods
- restart deployments
- scale deployments
- trigger cronjobs
- watch logs
- I can scale our minecraft server to 0 in seconds if we are in a different season of life
ArgoCD is amazing
- I ❤️ gitops
Ingress just works
Longhorn
- makes snapshots and backups easy
- makes multi node easy
zero-downtime deployments
self healing health checks

I mostly do very simple things, deployments with a container, a volume and ingress. Probably things that you could easily run on fly.io. Theres nothing really fancy. I just like how easy this setup works for me.

One Year Of Shots

2025-12-09T12:44:39Z

I’ve been running my shot scraper api for a year now. It creates og images for my website and thumbnails for my [[ reader ]] using a headless chrome instance.

25870 shots
73 shots per day on average
12-09-2025 first shot taken

Histogram #

a histogram of shot counts by day

You can see in the histogram that I’ve had a few big spike days, This has been mostly for days that I’ve integrated into a new service or changed the endpoint. On February 13, 2025 I swapped over from using the post to using template specific to open graph images.

-content = "https://shots.waylonwalker.com/shot/?url={{ config.url }}{{ post.slug }}&height=600&width=1200&scaled_width=1200&scaled_height=600"
+content = "https://shots.waylonwalker.com/shot/?url={{ config.url }}{{ post.slug }}/og/&height=600&width=1200&scaled_width=1200&scaled_height=600"

Image Comparison

Original Post Image

originally I simply used an image of the post itself

New OG Image

In Feb 2025 I made OG specific templates to use for the OG images.

Swapping to og images

Collage #

For fun I made a collage of all the shots. It’s cool to see all of these together, I remember a lot of the thumbnails and posts. Many of them from my rss reader.

collage of all shots, click to see full size

I tried to make a video collage, but turns out it takes a long time to show all 25k shots in video form. It also turned out to be a bit of a strobe as I don’t yet have it figured out how to dark mode in headless chrome.

video collage of shots

video collage of 600 shots, I tried to include all, but even at 20fps its a 20 minute video.

The Wrong Reasons To Run Kubernetes In Your Homelab

2025-12-06T09:46:47Z

Running kubernetes in your homelab is complex, time consuming, there are almost no docs to help you (homelab focused docs for things you want to install), and nothing is copy paste. You have to make everything happen yourself.

The Wrong Reasons To Run Kubernetes In Your Homelab #

I run compose and think kubernetes is the next logical step
Techno Tim runs it
I heard it’s what cool kids do
Kubernetes BTW
Talos Linux looks cool
I found a cool helm chart on GitHub
I need scale

There are also The Right Reasons To Run Kubernetes In Your Homelab.

I run compose and think kubernetes is the next logical step #

No it’s not. It’s much different than running docker, compose, swarm. It’s meant for scale, it’s complex, it’s made for enterprise, not your local development or your homelab. It can do these things, it can do them quite well, but it’s not the target audience.

Techno Tim runs it #

I heard it’s what cool kids do

You need to rethink who the cool kids are, touch some grass. Tim also does it for his job, he likes it, he knows it, he wants to lean on it and learn more.

Kubernetes BTW #

Kubernetes does not make you look cool, it makes you look like you are trying to over optimize and over engineer your life. It’s not worth it, in fact nothing in life is worth worrying about what everyone else thinks of you.

Talos Linux looks cool #

Talos is an S tier OS wherever you deploy it. It is a secure, minimal, kubernetes first OS. They also have some really great people working there putting Talos in some really cool places like backpack or Apple Power Mac

I found a cool helm chart on GitHub #

No you didn’t. Everything in homelab is compose first. A few things have a k8s option, but almost nothing is k8s first.

I need scale #

No. You’re homelab does not need scale. If you think it does, you have some real shit hardware, some bad optimizations, or somehow you have a startup you need to launch cause you got more users than most.

Another Big Cloud Outage Nov 2025

2025-11-18T18:39:00Z

Today I woke up to finding out that cloudflare hade a widespread outage. My [[Reader]] uses tailwind cdn for styles and it was down. Otherwise it was not so impactful to me and felt kike they were quick to have it up.

I’m not really researching here, just jotting thoughts down from a parking lot waiting for pickup.

It feels like we are seeing a lot of these lately. They feel much more frequent. It feels like a whole industry was sold on 9’s and reliability of big cloud that we just aren’t getting.

There’s a huge push to go back to self hosting, racking and stacking. I think this is great. I love it. I’m a big proponent for ownership and self hosting. It’s not the right move for everything and everyone, and is certainly not something to make a knee jerk reaction about in the moment of frustration.

There’s a lot of things that are just impossible to do yourself, cdn caching, edge compute, ddos protection.

These companies are not magic they are vulnerable to changes just like you and I. It really feels like more and more of these are due to misconfigurations, and small bugs introduced. As we see big tech downsize and lean more on ai that likes to do big code changes I dont see it getting better soon. Theres a lot of things we can all armchair quarterback about here, better testing, review, canary deployment, staged rollouts, rigorous review. All great things. I can hear Uncle Bob talking about rigor, giving a shit, and following principles.

We are at odds of reliability and speed. This critical infrastructure runs so many important things in our lives it feels like it deserves a professional engineer signoff on changes. Documentation of changes and testing done between changes. This would all but hault forward progress, taking us back to the level of physical components and manufacturing.

How do you decide what needs this rigor and how to regulate it when companies are incentized by number go up.

The answer right now is that we can’t and if you are managing critical infrastructure you need to take these outages into account in your disaster recovery plan and understand what you are willing to allow go down on failure.

3d-Printed Corner Clamp

2025-11-11T21:13:07Z

Getting ready to batch out 18 apple boxes for the local theater. Need to step up my woodworking tool game here quick on a low budget. Whipped this up up and built the prototype box , went really well. We have 4 in the arsenal now, might do 4 more if we need more assembly capacity. Pretty proud of the first 3d printed thread project here. The design for good 3d prints can be quite different with its anisotropic strength and hollow sections being nearly weightless when compared to traditional manufacturing methods. Its so fun to be able to do it for almost no cost right in my home office.

3d-printed corner clamp printed in black pla.

Isometric view of my corner clamp v1 that supports up to 3/4" sheets and includes slots for dowell points on 3/4" and 1/2" material.

Techbrophobic

2025-11-09T20:13:33Z

I just heard someone drop the this term and it kinda fits a lot of shit on the internet right now. Arguing that its OK to question AI, its OK to like it, its OK to question if it needs to be in every goddamn thing we do, question its morality on training and the slop being pushed at us all the time.

I’m not Technophobic I’m Techbrophobic

I heard this and it kinda hit with a lot of things that I’ve resonated with lately. Tech bros of today have been compared to Steve Jobs in a lot of ways. Whether its style or the way he was so good at marketing, but this feels different. When Jobs launched the iPhone as this next great thing, He fucking made the thing.

No broken promises of being sold something with hopes that it will do more tomorrow.

No pushing around insane amounts of money with the hope to become profitable years down the line.

No fear pushing that if you are not doing X today your business will be dead in 6 months.

Giving us the promise that it was about to create an entire industry of careers through building software that brought us real value, real entertainment, Not the promise of that.

This tech made it easier to do the mundane, no longer carrying out dated maps in your glove box, everyone now has a flashlight, a calculator, a camera, a way to video call your mom.

I’m not a complete Luddite, I see value in being able to lean on AI for generating code, asking questions, helping change the tone of your messages.

There are so many issues with AI currently even looking past the stolen works, copyright, and power consumption. The messaging does not pass the vibe check. High up folks are getting the impression that they can remove humans from the workforce. Deploy the same features with fewer headcount. It’s doing our art, writing our code, communicating with our colleagues. It’s not doing protein folding and cancer research. No matter how many promises we get its not driving our vehicles.

I do believe there is a world in which it makes us more efficient. It fills in the gaps of our knowledge. It helps us learn. It helps solve complex problems. It allows humans to flourish and create things they never could have.

Mcat Anything

2025-11-05T20:10:02Z

I’ve long looked for a way to cat anything in the terminal. I’m am terminally in the terminal. I manage all of my projects, code, website, notes, files, servers, infrastructure, almost everything from the terminal. I occasionally open a file manager, mostly at home, only so that I can browse images.

Compounding my issue, I’m a tmux user. It works great for me, and I barely have to think about it at this point. The keybindings are second nature to me. I can go between server, terminal, nvim, and between projects instantly, no loader, no lag, no animation, it just works for everything that really matters to me for really getting things done.

Mcat #

mcat is a new tool that seems like it can cat anything in the terminal, code, files, images, markdown, markdown with images, and even video, without leaving tmux!

mcat static/8bitcc.png
curl https://r.jina.ai/https://waylonwalker.com/store/ | mcat --theme dracula --md-image all
curl https://r.jina.ai/https://waylonwalker.com/shots/ | mcat --theme dracula
mcat ~/git/dropper/data/01b21044-0bf8-4b06-9db1-a002c0519df6.mp4
mcat ~/git/dropper/data/0e659c05-1c12-4524-aa54-ef52ba680865.webm

1fa2e162-deeb-4bb5-b1c5-961632abd452.webm

But it doesn’t always work #

The keen eyed of you will notice the blank screen at the start of the demo above, not sure what happened, but resizing the terminal fixed something. Sometimes video comes up as an image, sometimes it core dumps, once even it crashed my system…

The hope is there, I’ll continue to use it occasionally, but right now it feels like a novelty that is fun to look at when it works. For now I’m going to stick to my kitty/tmux combo that works even though they seem to hate each other and I’m going to be happy just getting shit done.

Missing Thoughts

2025-11-05T07:23:35Z

No one is perfect, this is why we have things like checkpoints or gates in the form of pull requests, linting, type checking, and tests. What happens when you work on small side projects by yourself that try to be content focused? What happens when you end up building a lot of the tech under that site and build it on the bleeding edge of all the tech you make? They are likely missing these things and occasionally there are some periods of regression. This is one reason I really like the term digital garden to describe one’s small corner of the internet where they share their thoughts.

There will be regressions

The Signs #

There were signs, signs I did not notice

Chat

is your rss feed broken?

I’m not seeing anything show up in my rss reader

Do I not put thoughts in my rss feed, I swore I did.

Chat

my fault, Turns out I must have already clicked it in my reader.

great, glat it’s working

…But it wasn’t

Later this week comes the next sign that I also choose to ignore… Google search has unindexed a number of pages due to a soft 404. I don’t look at all of the emails from search console, but I did happen to see this one, and thought “Huh, that’s odd” and went on with my day without another thought on the issue.

Huh, that’s odd

The Discovery #

I didn’t realize I even had an issue until later that week when I went to my [[ archive ]] myself and noticed the shape of it looks off. When I opened up my feed something felt off, it was only the big posts. Thoughts show up as a different kind of card

Then it hit me.

Something just felt off

Thoughts are Gone!

The Fix #

I popped open my config and immediately knew what happened. I turned off some plugins locally for faster build iteration and that change found its way into production…. because I have nothing to check myself on other than me. The plugin is now back and thoughts are flowing from thoughts.waylonwalker.com to waylonwalker.com, no problem.

❯ git log -p
commit adeb9812f7ecf4d9a68c6aa5e01e549e4dd91285 (HEAD -> main, origin/main, origin/HEAD)
Author: Waylon S. Walker <waylon@waylonwalker.com>
Date:   Mon Nov 3 19:56:40 2025 -0600

    thoughts were missing

diff --git a/markata.toml b/markata.toml
index 59539bab..a55a494d 100644
--- a/markata.toml
+++ b/markata.toml
@@ -31,7 +31,7 @@ hooks = [
     'plugins.chartjs',
     'plugins.md_video',
     'plugins.post_model',
-    # 'plugins.thoughts',
+    'plugins.thoughts',
     # "plugins.wikilink_hover",
     # "plugins.wa_wikilink_hover",
     "plugins.tippy_wikilink_hover",

Owning Everything has a cost #

You see here I am a one man show. I own the content, the build, the infra that does the build. I love it, but there is so much to do for one persons side project passion project. Did I bite off more than I can chew? Did I choose the wrong abstraction level for my needs? Should I have better tests in place? Or is this just a digital garden that has ebs and flows, growing weeds and pruning them every few months?

Rules

2025-10-30T08:38:48Z

There is no such thing as magic
Be ready to roll back live deployments
If CI was too fast be suspicious

First 3d Printed Threads

2025-10-24T14:19:49Z

Working on an upcoming project that requires some threaded screws. Trying to keep a low budget on this one with as much to come off of the printer as I can. It might become a slant3d portals product if it works out. I always like making test prints for stuff like this especially to see how the feel is off of the printer that is going to print the final product and take much longer. First try was a success.

b485b759-719a-4aa0-aa8d-f98e0a5e1ac3-1080p.mp4

What worked #

I started out looking up standard half inch thread pitch and size, but ran out of time to get the exact profile of a half inch bolt, so I will need to fix that later. Th

The print orientation is critical for strength here. This part is a full 1/2: so it should be strong either way, but to make sure we are printing the bolt horizontally to get nice long print layers. To do this we have to give it a bit of a flat spot on the top and bottom. This does not hurt performance, if anything it probably helps give some room for poor tolerances.

.5" od
13tpi pitch
non-standard profile… kids needed me and I called it good enough to run a test.
0.1mm offset on all surfaces
external threads printed horizontally
internal threads printed vertically
Chamfer all lead-in/lead-out

3d Printing Dovetails Experiment

2025-10-12T20:34:35Z

I hit an issue with 3d printing oversized parts that I have not hit before. I’m working on some jigs for an upcoming woodworking project that will involve a lot of repetition. We want to utilize some dowel joinery and jigs for consistency. These parts will be up to 20in in length this is much larger than my print bed.

I’ve fit things together before #

Here’s where I went wrong, I wasn’t really thinking through my previous applications. They’ve all been slip fit, primarily print in place joints that need to move. My go to offset for print in place on my printer is 0.2mm, sometimes 0.1mm depending on the scale.

knife sharpener double hinge first try

A live hinged [[ knife-sharpener-double-hinge-first-try ]].

And in the hinges of [[ a-box-of-caps-and-a-macropad ]].

Experimenting for feel #

Fitment like this is a lot dependent on the tolerences of your printer and the feel you are going for. I went to school as a mechanical engineer and theres a lot of science behind press fit joints, that’s not happening in my house on my desktop printer. Most of us don’t have that kind of ability to gauge our outputs so I’m doing like a cook in the kitchen and going for feel.

Here are all of the experiements lined up.

Fillets #

I again, I feel like I should have known better. 3d printers generally have 0.4mm nozzles, common sizes you can get 0.2mm, 0.4mm, 0.6mm, and 0.8mm. This leaves us with the sharpest corner we can do at 0.2mm.

Diagram courtesy of gpt-5.

Potentially overkill #

In order to make sure that the flat edges of the dovetails are what are making contact and not corners binding I made the inside fillet 0.6mm and the outside 0.4mm. This 0.4mm radius (0.8mm diameter) is double my nozzle size, and will cause the printer to follow a smooth curve rather than start and stop and potentially overflow a bit. This difference in fillet size will also leave a little room for error and allow the important flat faces to mate together.

Here are the test results after filleting the edges of the dovetails to compensate for printer errors.

Fillets

Before Fillets

After Fillets

TLDR What works for me #

After all of this, turns out that common sense kinda just works here, 0mm offset works really well on my printer, and leaving room in the corners to keep them from binding up helps a bit, but is probably unnecessary.

0mm offset
0.6mm inside fillet
0.4mm outside fillet

python extras are for shipping

2025-10-03T20:18:23Z

Python has two ways of adding optional dependencies to your projects pyproject.toml file dependency-groups and optional-dependencies.

dependency-groups #

for development

Dependency grooups are used when working on the project, they do not ship with the project, users cannot select to install them with the project. These are for things like running tests, linting, or docs. You might want to run these in ci, or keep your dev machines tight. For the most part you can probably keep these in dev. Depending on your team, fluency, and tolerance for slower installs extra packages. Adding too many tight groups might make it hard for the team to remember all the groups and which one to use and end up with them using --all-groups anyways.

optional-dependencies #

for users

Optional dependencies are for shipping. These are for your users, not your development team. This is used for dependencies that are clearly not needed for all or main use cases. It is annoying to use projects that you need to add optionals to just to use at all so use them a bit sparingly.

fastapi">example fastapi #

fastapi provides a very simple example. fastapi itself provides almost everything you need with optional-dependencies for standard, standard-no-fastapi-cloud-cli, and all. These primarily add support for uvicorn websockets and multipart forms.

example pandas #

Pandas is a very good example here. As a data processing library there are a lot of different sources for data that you might want to use, but you probably won’t need most of them, and often don’t need them to just get a DataFrame going. They offer a long list of optional dependencies such as pyarrow, aws, gcp, postgresql and many more.

example kedro-datasets #

Very similar to pandas kedro-datasets uses a similar pattern to pandas but at a higher level. A dataset is an abstraction of a datasource that get defined in the catalog, and primarily provide the framework with load and save methods for datasets. There are many optionals for data providers like pandas, but also many for each dataframe abstraction like pandas, polars, dask, databricks.

kedro-datasets

let’s build a package #

Let’s make a package called learn-uv that implements these dependency features.

mkdir /tmp/learn-uv
cd /tmp/learn-uv
uv init --package
uv add httpx
# add development dependencies
uv add --dev pytest
uv add --group dev ipython
# add documentation dependencies to the docs group
uv add --group docs markata

# add optional dependencies
# these are dependencies that a user could install
# python -m pip install 'learn-uv[data-science]'
uv add --optional data-science pandas

# add extra dependencies
# this adds the jupyter extra from rich
# similar to `python -m pip install 'learn-uv[jupyter]'`
uv add --extra jupyter rich

This will result in the following pyproject.toml.

[project]
name = "learn-uv"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
authors = [
    { name = "Waylon S. Walker", email = "hello@waylonwalker.com" }
]
requires-python = ">=3.13"
dependencies = [
    # uv add httpx
    "httpx>=0.28.1",
    # uv add --extra jupyter rich
    "rich[jupyter]>=14.1.0",
]

[project.scripts]
learn-uv = "learn_uv:main"

[project.optional-dependencies]
data-science = [
    # uv add --optional data-science pandas
    "pandas>=2.3.3",
]

[build-system]
requires = ["uv_build>=0.8.22,<0.9.0"]
build-backend = "uv_build"

[dependency-groups]
dev = [
    # uv add --group dev ipython
    "ipython>=9.6.0",
    # uv add --dev pytest
    "pytest>=8.4.2",
]
docs = [
    # uv add --group docs markata
    "markata>=0.9.1",
]

Using the project #

Let’s start off with one common point of confusion. uv tool run or its alias uvx does not install any dependencies or anything from the project.

uv tool run ipython

✅ ipython
❌ learn_uv
❌ httpx
❌ pytest
❌ markata
❌ pandas
❌ rich
❌ ipywidgets (from rich[jupyter])

Using uv run is project aware, installs the dependencies from the pyproject.toml.

uv run ipython

✅ ipython
✅ learn_uv
✅ httpx
✅ pytest
❌ markata
❌ pandas
✅ rich
✅ ipywidgets (from rich[jupyter])

uv run --group docs markata

✅ ipython
✅ learn_uv
✅ httpx
✅ pytest
✅ markata
❌ pandas
✅ rich
✅ ipywidgets (from rich[jupyter])

Installing the optional dependency extra data-science gives me pandas in this project.

uv run --extra data-science ipython

✅ ipython
✅ learn_uv
✅ httpx
✅ pytest
✅ markata
✅ pandas
✅ rich
✅ ipywidgets (from rich[jupyter])

everything #

There are many combinations of include/exclude that you can use, check uv run --help to see all of the options. Here is the nuclear option to include everything.

uv run --all-groups --all-extras ipython

Potential `uv` bug #

At the time of writing once a dependency group is used it persists in the virtual environment for the rest of the project. This is not the case with extras it only includes the extras you ask for on each run. The state is clean and consistent with extras, but groups depend on what you have ran before in the project.

❯ uv --version
uv 0.8.22 (ade2bdbd2 2025-09-23)

example

# install docs group
uv sync --group docs
# docs dependencies are available
uv run markata
# sync resets the environment
uv sync
# docs dependencies are not available
uv run markata
error: Failed to spawn: `markata`
  Caused by: Permission denied (os error 13)

Kraft-Coordinates

2025-10-02T16:52:02Z

Handy reference for coordinates in the kraft world.

Home #

Overworld:-208 71 -291 Nether:-26 9 -36

Ocean Monument #

Overworld: 209 62 -752 Nether:26 1 -94

Shots Get An Upgrade To the Main Feed

2025-09-20T20:19:57Z

Today I updated my jinja templates and feeds to include shots posts in the main feed. These posts will prominently show the image in the feed with a small post for each one.

Shots are instagram style imge posts inspired by https://justin.searls.co/shots/.

zmk two hand hold down

2025-09-17T20:04:36Z

I don’t know about you, but I don’t re-flash my keyboard enough to ever remember where I put the bootloader. Sometimes its the last thing I think about in a refactor and I end up cornering myself into a place where I cant get into that layer anymore. I’ve started putting hardware switches on my newer builds, but some older builds don’t have a hardware one, so it requires disassembly and jumping the microcontroller. Even when I have one though, I gotta flip my board over and its annoying sometimes, so I prefer to have a keystroke for it.

Two Hand hold down #

What I’ve landed on recently is the idea of a two hand hold down combo for the bootloader. These combos are ones that there is no way I can hit without picking my hands up from their normal homerow position and pressing four keys simultaneously with pointer and thumbs.

Here are some example layouts from keymap-editor

40% layout - similar to corne

Here it is on my Son’s 3x5 macropad

Here it is on my 3x3 macropad

Timing #

You can even give it a long prior timeout to really make sure that its not an accidental hit.

Why Make a Website in 2025

2025-09-16T19:37:23Z

Inspired by Jim and Dave

It’s Fun #

If it’s not something that you enjoy, you might as well move on there are far better ways to spend your time in 2025. Only the weirdos read this shit anymore, the masses have long moved on to curated social media feeds, and on to chatting with llms. I enjoy spending some time in the digital garden every once in awhile tweaking templates, creating markata plugins to do interesting things. I enjoy expressing my thoughts, or at least shouting them into the ether.

I’ll be the first to admit that I often go to an llm for first pass at many things that I could look up in a guide or 😲 the docs.

I enjoy reading others #

This year I have made extra effort to add more and more people into my [[reader]]. I enjoy reading content from people I chose to follow and can remove whenever I want. I enjoy reading actual human thoughts and not ai generated content all day.

I even will create Thoughts based on these posts that I find interesting, and want to further share to others. I want to give these people that took time to make something a little boost.

Not Perfection #

The last thing we need in 2025 is perfection. We have enough ai generated slop that is far from perfectly accurate that sits behind a veil of mostly clean look, semi consistent feel, and a narrow set of tone. I see it in gen z there is something about the imperfect that is coming out. Not perfectly cut video or audio. I just heard Gary Vee mention this morning that a siren going off in the background of NYC has a somewhat natural human feel to it now.

Ownership #

Dave points out that the first thought for most people trying to make a business or pop off as an influencer is TikTok, then a few similar apps. If you are trying to reach a large audience and swim in the big pond with the big fish thats where you need to go.

But you don’t own that content anymore. Some posts will pop, some will be lost to the algorithm for nothing more that the first few eyes that got it swiped past too fast. The whole platform can go, or choose to not show your content, its their platform. You don’t pay them for anything. They are allowed to do what they want.

If you want some ownership to keep your content alive, curate a small, focused, core audience, also put that on a website.

Not for the faint of heart #

It’s not easy to manage hundreds of posts, images, builds, servers. Theres probably a hundred ways to make it easier, but if you really want full control, full DIY, it takes time to do.

Learn a skill #

This can go a few different ways, One is that you can make a blog or site about a topic, deep dive into this topic. Create blogs, videos, images, about this topic. Create real content that real people find interesting.

OR a bit meta, you can learn more about tech. This site is built with markata, its a static site generator that I built myself mostly to make this site. I’ve learned a ton about markdown, html, ast, python, plugin architecture in doing so. I build this site in kubernetes, another skill that I wanted to develop, and owning my own services that I have real skin in the game for is a great way to do so.

Just Build Websites #

As the opening to shoptalkshow says, just make websites. Who the fuck cares why or how. The important thing is that you built something, maybe it is for an audience of one or one million, doesn’t matter, keep the internet alive and just build something.

store

2025-09-10T20:38:00Z

I’ve been 3d printing for years, and create a lot of my own designs. Many of them are hyper specific to me, but for the ones that I think others might find useful I will be putting up here for sale, you can buy prints that will show up to your door in a few days. I’m leaning on the slant3d print farm so this depends on your proximity to them. The prints are typically black petg, if you would like a different color reach out to me and I will see what other options we have, or for an additional fee I can print it myself and ship out special colors.

These are all designs that I made and actually use myself, they bring me joy knowing that I made just the thing that I wanted to exist and if you buy something I hope that it brings you this joy as well.

Skateboard Wall Mount #

I’ve used these skateboard wall mounts for years, I have 5 in my office featuring new decks in the queue, and probably 10 in the garage to display used decks, and fully assembled boards for me and my kids. I use these for both regular popscicle boards and longboards, but if you are putting up a thicker longboard go with the thicker size.

Mic stand Clips #

These clips will hold your cables on your mic stand nicely and look nice while doing it. I’ve got them on both of my mic stands, there are two versions, one for a smaller stand with usb cable, and one for a larger stand with an xlr cable make sure you measure your mic stand before buying.

13mm stand | 6mm xlr cable

10mm stand | 4mm usb

Cat Door #

In our house we have a cat liter box in the basement, so we need to leave the door open for our cat. But if our dog goes down there he seems to only cause trouble. So I modeled this door prop to leave the door just enough open for the cat, but keep the dog out.

cat door

Kicker Ramp #

kicker ramp

manny pad

When To Vibe Code

2025-09-02T19:44:20Z

I enjoyed this post from Theo and think it deserves re-iterated, revisited, and to remind myself of some of these things.

https://youtu.be/6TMPWvPG5GA?si=guQem4R8dLOMBntP&t=1356

The skill/read spectrum #

The first diagram describes that there has become a spectrum of agentic coding from vibe coding where you don’t ready anything, to looking at everything in detail, across a group of people who don’t have a clue what the code says to people who could do it way better if they took the time.

The importance spectrum #

He argues here that its ok to bounce between A,B, and D, but C becomes dangerous. I’d argue that he brought up a 3rd spectrum that is important later on, “how critical is this”. I think the I don’t know, don’t care, didn’t read, but the thing did its job is quite fine, but don’t know, don’t care, mission critical is the main issue we are seeing with agentic coding, primarily in the didn’t read but critical Zone.

The Rules #

This is the list that prompted this post, I think it serves as a good reminder when you should care a bit more.

You still need to know how code works if you want to be a coder #

If you want to get good, you need to put in the reps, do the practice, learn to debug the strange looking error messages and not just pass them to ChatGPT.

There’s a lot of code worth having that is not worth writing or reading #

As professionals we have a lot of code that we read more than we write, will get executed millions of times over the next 10 years that its in service, pay close attention to this and probably set aside the llm.

We also write a lot of that is ran only a few times, maybe its a special report, or a shell script to bootstrap something. It might be a POC service that you build out only ever on your machine, you get your idea out in front of users or try it yourself and find the mistakes before you spend weeks building it yourself.

you can’t be mad at vibe coding AND be mad at left-pad #

This one is interesting that he was very strongly for. The idea is that no one ever reads the code from libraries they use, or the diffs of new versions (outside of google where everything is literally vendored into the monolith). If you don’t care about all of those dependencies, you cannot get mad at vibe coding.

I’ll argue here a smidge, but I agree with the premiss. Left-pad was probably written by someone smarter than average. It was definitely used by far more people and projects, therefore was battle tested to no end. This was the real reason this stuff got in everywhere. People are probably not good enough to catch all the odd edge cases for js, especially if front end is not their specialty.

Vibe code is a type of legacy code #

Vibe coding is a type of legacy code, but as a type of debt that we opt into, not one that has accumulated over time and we have opted not to take care of. Both are types of code that no one reads, no one remembers existed.

When someone needs to change legacy code they don’t, they take a hammer to it and rewrite it from scratch.

If the tools are better than you, stop using them #

Theo has a good example here. Critial business components to his businesses have switched to using Effect. It solves async issues with ease, but melts the brain of anyone seeing it for the first time. He admits that he does not code as much anymore and thus he sucks at it, but knows that if he is ever to learn it one bit he must turn off the LLM and put in the work to do it himself.

You will not get any better at this by copy pasting from ChatGPT. It is too easy to put your brain aside, paste error messages, and copy the reponse in. Or worse with agents tell them fix this. Unlike the days of Stack Overflow, you had to understand your problem well enough to search it. You had to understand the answers enough to integrate the solution, you had to understand if the solution was even for your problem in the first place. There was much more brain work that had to happen even then when the answers were basically given to you.

Fin #

Lastly Theo does not bring this rule, up, but if you are writing text for humans to read, you shall not copy paste from ChatGPT without thoroughly editing and reading for yourself first, this is considered Ai Slop and you should be ashamed.

Keycap Cut Away

2025-08-30T16:03:01Z

I was curious how/if my custom keycap design was hitting my switches. So I set out to find out what the fitup inside of this assembly looks like, but not theoretically, a fully sliced view into their fit up in the flesh.

The set up #

To setup for this cut, I flooded the edge of a 2x4 with hot glue, and inserted the cap such that the step was tangent with the edge. This way I could use the edge as a guide to cut one side off and leave the stem in tact. I took a handsaw to it and filed it smooth.

Removal was applying some isopropyl alcohol and it popped right off.

Results #

Actually the results aren’t that interesting. I found that the pictures turned out really good, but nothing was touching and it sits in the stem exactly as I thought it did. I had a thought, what if the cap is not allowing the switch to reach it’s full travel, and had no way to know, but now I know.

Key Up

Key Down

Knife Sharpener Small Upgrade

2025-08-29T16:17:00Z

I’ve used this knife sharpener that I printed for a few years now. I thought that it was based on the Russian designed TSPROF, but in looking through the history it looks very similar to the USA Edge Pro Inc Apex designs that goes back to the 1990’s. The angle isn’t quite holding like it used to. I’ve got a lot of ideas for my own model, but for now I’m going to print some spacers to help get repeatable angles.

setting the angle on my sharpener

Where I want to place a fixed height collar

16deg 31.5mm
17deg 34.8mm
18deg 38.0mm
19deg 40.6mm
20deg 43.8mm
21deg 47.4mm
22deg 50.2mm
23deg 52.0mm
24deg 54.3mm

so I may have went to the next level #

30335f07-9cac-4e66-b908-f0e3cfbf7582.mp4

I'm Out On Agents

2025-08-27T10:10:51Z

Its the year 2025 and we are only a few years into having 6 months to live before ai takes our jobs, and the big push right now is agents, managing agents. I will fully concede to I’m not doing it right, or a future state gets better than where we are right now, but right now they kinda suck.

Transparency

I’m sitting offline right now as I write this, These are my feels, no research, no links, no ai, just vibes.

Chat #

Chat is what really kicked off ai uses and goes back as old as computers, but it always sucked. Then chatgpt rocked the world with the biggest launch day in history and showed us that it could actually be pretty good. Unethically trained on everything they could get their hands on, burning cities worth of electricity to train, and keep training to stay ahead of the competition. It does a damn good job. There are tells, and if you see enough of it there is a lot that turns to slop, but if you had never seen it before, there is no way you would assume that it was not a computer.

It does a damn good job at being average, it can do what seems like everything not related to security and authentication at a pretty average level. This is its super power. Whenever you are trying to bridge between something you know and something new, you can get a pretty good answer, and likely spot the bs in what you know.

Agents #

Now that our models have gotten better, hardware has gotten bigger, better, and a lot more of it, we can really expand context windows really wide. With that brought the use of agents, these tools get context from sources on their own and often are given read/write access to your computer. Depending on the model these things will branch out to make small changes that look no worse than a formatter on every goddam line of your codebase. Except they are not a formatter, they are not backed by ast checks. They do not have any guarantees.

They take the fun out of creation.

The emphasis now becomes on the code review. All you have to do is ask it to makes changes for you. Bring in your expertise of what changes should be next, or even just punt to asking it what comes next. I’ve yet to talk to someone that is diligent enough to read everything it spits out in excruciating detail to the point that it does not cause significant issues.

You see here is the thing, its average. With a little bit of context it can do average work. It is not an expert. On the surface this feels fine, making crud endpoints has been done for decades, and average developers crush these things every day no problem.

It’s Average at best

You know what is not average? Your knowledge of the use case you are solving. You may not feel like an expert, but given that there are likely about 5 people working in your codebase, you are a fukin expert at that thing. The average person off the street takes time to onboard, often months, or years for someone to really understand the business you are working in. These things don’t have that.

I’m not letting it in shit that I care avout #

I was early to the game to codeium, even used the predecessor for awhile, I was early to chat gpt, I was early to windsurf and the possibilities that agentic ides brought. I will be late on letting agents touch my production code bases. The few times I’ve tried for changes that seem easy enough, but more work than what I want to do at the time, I’ve regretted it. It’s only left behind a mess that it cant deal with anymore, runs in circles trying to solve any problem, and I’m left with shit that feels like a house of cards that breaks anytime you touch it.

I will continue to POC #

As much of a mess as these things, make they are super useful to vibe code ideas, move quickly and try different approaches to a problem. They let you make a proof of concept that you can get in front of team mates, bosses, or users.

Honestly I still prefer the chat interface. It feels like a nice balance of using my brain, and knowing that I am punting on something. Do I need to know the whole ffmpeg interface to grab a thumbnail for my webapp, no, would I actually like to learn it someday, yes, but I don’t need to right now I just need a goddamn thumbnail.

For the most part I am still slotting these things into the codebase myself, but occasionally I have it do an entire module, and even more rare I pull out agents and have it do all the work. The further away from the code I get, the less I care about it and just want it done.

I will laugh at this post in 2 years #

The way I see it agentic coding is not here to stay, it sucks. There is one two ways to go this was a blip in the radar that we laugh at for giving ai companies all our data for free so they could fuck up our products. Or they actually get pretty good and we all become architects that peek at the code if we really need to. I think for the second to become true its going to take a lot of time, consumer hardware will need to catch up, local llms will need to work a lot better than they do now. Even if we still need to call out to the big boys for some heavy work getting the context right for them locally would make a huge difference. Currently local llms are too slow and dumb.

So will this bubble pop or explode, we can only wait to find out.

trying forgejo

2025-08-12T11:57:42Z

WIP

networks:
  forgejo:
    external: false
services:
  server:
    image: codeberg.org/forgejo/forgejo:11
    container_name: forgejo
    environment:
      - USER_UID=1000
      - USER_GID=1000
    restart: always
    networks:
      - forgejo
    volumes:
      - ./forgejo:/data
    ports:
      - '3000:3000'
      - '2222:22'
  docker-in-docker:
    image: docker:dind
    container_name: docker_dind
    privileged: true
    command: ["dockerd", "-H", "tcp://0.0.0.0:2375", "--tls=false"]
    restart: unless-stopped
    networks: [forgejo]
  runner:
    image: data.forgejo.org/forgejo/runner:4.0.0
    container_name: forgejo-runner
    user: "1001:1001"
    depends_on:
      - docker-in-docker
    environment:
      DOCKER_HOST: tcp://docker-in-docker:2375
    volumes:
      - ./runner-data:/data:Z,U # will hold .runner + cache
    command: /bin/sh -c "while :; do sleep 1; done"
    restart: unless-stopped
    networks: [forgejo]

blender donut tutorial 2025

2025-07-25T08:29:27Z

First Launch #

my setup #

Add #

shift A

bump resolution scale #

edit -> preferences -> interface -> resolution scale 1.25

orbit without middle mouse button #

viewport shading rendered #

command palettes are overrated

2025-07-06T12:37:40Z

Command palettes are slow, and overrated, you should treat yourself better. You probably installed VSC*** out of the box and your co-workers see you using the mouse and reprimanded you as they should. Mouse usage is not OK if you are a software dev, you should have the cheap ass free mouse that came with your cousins dell machine five years ago and only use if for emergencies. If you want to be fast you cannot do that by moving cursors to imprecise locations and clicking with your hand. You are not a caveman, put down the stones and get with the damn times. You need to be moving with precision.

Stage One, the command palette #

So you are taking your first few baby steps away from that Logitech MX Master and you need to get shit done, during these infant months the command palette is your friend. Use it you will be 10x faster than Razer Naga Ron from accounting. If you are in an IDE like VSC*** or a JEttedBrains editor they come with a command palette for running commands and fuzy finding files, use it. If you are in nvim, move on you probably don’t need this, unless you are still teething on VSC***, during that tim use :Telescope commands.

Close the fn Tree #

You don’t need a goddamn file tree open all the goddamn time, its taking up screen space and filling your brain with useless shit. They can be a helpful tool to move, rename, refactor files, or familiarize yourself with a codebase, but you don’t use one if you want to walk someday, so just like the command palette we are going to give it up for fuzzy find. If you are in nvim you are going to want to use :Telescope find_files those other big brained ides have things, look up the keybinding.

Now that we are crawling #

As you start to get your legs under you and you can crawl away from mamas teet note down the commands that you use all the time, we want to get command palette usage way down. Not to zero, without some sort of fuzzy command picker (even :<tab>) you are probably trying to remember too much and allocating too many brain cells to editing text, don’t do that. Resist over-correcting.

assign hotkeys #

Now you need to get yourself some hotkeys going, this is for the hot shit that you use several times a minute. You should be able to do things like swap between the current file and the last file in one keystroke without wasting a single brain cycle, it should be automatic. Go to definition, go to reference, no searching, searching is slow, find_files is slow. These are tools for exploration we are tyring to get real work done here.

Here’s a few things you should be able to do if you want to keep up with your Grandma, she’s been slayin these keys for years.

go to definition
go to reference
find_files
find and replace
rename
git add this file, commit, push
go to the last file
go up and down the jumplist
go up and down the quickfix list
list open files
harpoon all the files you go to regularly
format
move around your files

fly #

Over time you should be using your command pallete less and less, this is not designed to run every goddamn command through. Note the ones you use a lot and add keybinds. If you are using one of those editors that don’t make it clear what the fuck your running when you execute a command ask gippity, it can probably knock out that binding in 30s just fine.

inspiration #

The decline of “vim btw” by Sylvan Franklin. It’s a pretty incredible video, he is crushing these half satire dry humor tech content videos. He nails that with the death of vimBTW we lost the craft of knowing your editor in and out. We lost the art of flying through text. Now we have plugin kitties that say vimBtw, but they really aren’t using any vim features past hjkl.

[[ thoughts-724 ]]

https://www.youtube.com/watch?v=RmnqdAidVeE

just fucking use kubernetes

2025-07-01T12:46:33Z

You want to run containers?

JUST FUCKING USE KUBERNETES.

Obvious satire

If you don't like harsh language this is not the post for you.  Obviously

ripping off motherfuckingwebsite.

ThIs is AI SLoP

If you don't like if you can fuck off to the next post, I'm having fun here,

but satire is not my strong suit and needed some help.

Seealso

“But it’s complicated!” #

Shut up. Close twitter and fucking do something. Life is complicated. You know what else is complicated? Email. DNS. Life. Kubernetes is the least painful way to orchestrate containers at scale. Docker Compose is for your laptop.

Swarm is dead.
Nomad is just sad.
Systemd units? Get out of here.

“But my app is small!” #

SO IS YOUR AMBITION. You could write a bunch of bash scripts and hope they work on prod. You could SSH into servers and handcraft your infra like it’s 2011. Or you could just fucking use Kubernetes and sleep at night.

I can just throw my script in crontab #

Tell that to your boss when your cronjob failed 16 times in the last week without anyone noticing. kubernete makes it fucking simple, want retry ask for it. Hanging script, activeDeadlineSeconds that bitch. Connecting to six other services in your shitty ass infrastructure this shit retries automatically.

“I don’t need autoscaling!” #

Cool. Tell that to your boss when the CEO tweets your link and the site goes down harder than your last date.

“But YAML is ugly!” #

So is your Terraform, your Ansible, your Prometheus config, your custom CI/CD scripts written in Bash, and the spaghetti you called a monolith before you went microservices and made it worse.

“Kubernetes is too heavy!” #

Compared to what?

Your handcrafted, artisanal, single-node LXC setup running on an Intel NUC from 2014?

Heard of k3s? k0s? No. These fuckers will have you running kubernetes running on your grandpas goddamn gateway 2000 right next to AOL messenger without splooging out the the ashes of his Marlboro Reds.

SSH? #

You don’t need no goddam ssh to install your 200lb gorilla editor so you can hand edit your init files and carefully contruct your init system. This is kubernetes, you use the fucking api, all you need is a connection and a kubeconfig. This motherfucker runs containers so you can keep your bitch-ass editor where it belongs, off the fucking host machine!

“What if it’s overkill?” #

What if YOU are underkill?

“How do I do zero-downtime deploys?” #

Probes my dude, you fucking probe your shit. Rolling out a new deployment kubernets won’t cut over to your broke ass release if that shit don’t pass. No more writing janky scripts that SSH into prod and run git pull while praying to the CI/CD gods.

“What if I still fuck it up - How do I roll back?” #

k9s is your best friend, pop that shit open find your broke ass deployment, jump owner to the replicaset and roll that bitch back to the working shit.

I need to scale #

This shit is built in, add a goddamn replica or 6 for fuck sake, need autoscaling use the HPA. This aint your granpas hand fucking crafted pet server, its fucking cattle. Load balancing just fucking happens, don’t think about it just use it, and it will work for your six goddamn friends that actually use your shit.

I want gitops #

Let me introduce you to argocd, this fucker uses helm so one fucking command and your ass is on the beach while your dev team deploys their own shit.

USE KUBERNETES #

It fucking works.
Everyone else is using it.
There are like 500 open-source projects built just to make it easier.
It runs on your laptop, your server, the cloud, and inside your dreams.
It will make your resume better.

Not convinced? #

Here’s your alternative stack:

A bash script that restarts Docker when it dies.
A Makefile that deploys via SCP.
A cron job that prays to the log gods.
A wiki page explaining how to debug your hand-rolled bullshit.
You. Crying.

So yeah #

Save yourself. Save your team. Save your infrastructure.

JUST #

FUCKING #

USE #

KUBERNETES #

(or don’t, and become a DevOps cautionary tale)

Seealso

markata parallel render

2025-07-01T08:05:58Z

  _._   __/__   __  __ _/_   Recorded: 07:53:56  Samples:  71681
 /_//_/// /_\ / //_// / //_'/ //     Duration: 92.741    CPU time: 91.748
/_/                      v4.5.1

Program: /home/waylon/git/waylonwalker.com/.venv/bin/markata build --pdb

92.740 Markata.run  markata/__init__.py:443
`- 92.714 HookCaller.__call__  pluggy/_hooks.py:479
      [2 frames hidden]  pluggy
         92.714 PluginManager._hookexec  pluggy/_manager.py:106
         |- 38.207 wrapper_register  markata/hookspec.py:265
         |  |- 26.105 render  plugins/link_collector.py:59
         |  |  |- 10.012 BeautifulSoup.__init__  bs4/__init__.py:122
         |  |  |     [14 frames hidden]  bs4, html
         |  |  |- 5.599 <listcomp>  plugins/link_collector.py:181
         |  |  |- 4.050 <listcomp>  plugins/link_collector.py:173
         |  |  |- 3.466 Markata.map  markata/__init__.py:565
         |  |  |`- 2.092   markata/__init__.py
         |  |  `- 0.942 BeautifulSoup.find_all  bs4/element.py:2008
         |  |        [2 frames hidden]  bs4
         |  |- 9.107 render  markata/plugins/render_markdown.py:260
         |  |`- 8.902 result_iterator  concurrent/futures/_base.py:612
         |  |        [5 frames hidden]  concurrent, threading, <built-in>
         |  `- 2.079 glob  markata/plugins/glob.py:112
         |`- 1.909 <listcomp>  markata/plugins/glob.py:161
         |        `- 1.908 Future.result  concurrent/futures/_base.py:428
         |              [3 frames hidden]  concurrent, threading, <built-in>
         |- 25.708 render  markata/plugins/post_template.py:639
         |`- 25.612 render_article  markata/plugins/post_template.py:404
         |     |- 22.223 <dictcomp>  markata/plugins/post_template.py:421
         |     |  `- 22.200 render_template  markata/plugins/post_template.py:429
         |     |`- 22.185 Template.render  jinja2/environment.py:1269
         |     |        `- 21.210 root  templates/post.html:4
         |     |`- 20.937 root  templates/base.html:4
         |     |              `- 20.455 block_content  templates/post.html:17
         |     |                 |- 11.212 root  templates/recent.html:4
         |     |                 |`- 10.853 root  templates/feed_sm_partial.html:4
         |     |                 |     `- 8.392 Environment.getattr  jinja2/environment.py:480
         |     |                 |`- 8.328 Feed.posts  markata/plugins/feeds.py:329
         |     |                 |           `- 8.300 Feed.map  markata/plugins/feeds.py:352
         |     |                 |`- 7.963 Markata.map  markata/__init__.py:565
         |     |                 |                 `- 7.786 <listcomp>  markata/__init__.py:628
         |     |                 |`- 6.465   markata/__init__.py
         |     |                 `- 8.486 call  jinja2/runtime.py:260
         |     |`- 8.468 Feed.map  markata/plugins/feeds.py:352
         |     |                       `- 8.394 Markata.map  markata/__init__.py:565
         |     |`- 8.326 <listcomp>  markata/__init__.py:628
         |     |                             `- 7.086   markata/__init__.py
         |`- 3.015 Cache.set  diskcache/core.py:749
         |           [3 frames hidden]  diskcache
         |- 7.788 save  markata/plugins/feeds.py:462
         |  `- 7.779 create_page  markata/plugins/feeds.py:493
         |     |- 4.530 Template.render  jinja2/environment.py:1269
         |     |`- 2.437 root  templates/feed.html:4
         |     |     `- 2.360 root  templates/base.html:4
         |     |`- 2.231 block_content  templates/feed.html:17
         |     |           `- 2.118 root  templates/feed_partial.html:4
         |     |`- 1.298 Environment.getattr  jinja2/environment.py:480
         |     |                 `- 1.290 Feed.posts  markata/plugins/feeds.py:329
         |     |`- 1.290 Feed.map  markata/plugins/feeds.py:352
         |     |                       `- 1.276 Markata.map  markata/__init__.py:565
         |     |- 1.325 Feed.map  markata/plugins/feeds.py:352
         |     |`- 1.317 Markata.map  markata/__init__.py:565
         |     `- 1.237 Cache.set  diskcache/core.py:749
         |- 7.029 pre_render  markata/plugins/auto_description.py:185
         |`- 6.840 set_description  markata/plugins/auto_description.py:137
         |     `- 5.791 get_description  markata/plugins/auto_description.py:107
         |        |- 4.485 MarkdownIt.parse  markdown_it/main.py:256
         |        |     [8 frames hidden]  markdown_it
         |`- 1.066 MarkdownIt.__init__  markdown_it/main.py:33
         |- 5.730 pre_render  markata/plugins/analytics.py:102
         |  |- 2.397 savefig  matplotlib/pyplot.py:1238
         |  |     [4 frames hidden]  matplotlib
         |  `- 2.004 heatmap  seaborn/matrix.py:355
         |        [10 frames hidden]  seaborn, matplotlib
         |- 1.704 post_render  plugins/permalink_aria.py:40
         |`- 1.588 process_html_content  plugins/permalink_aria.py:13
         |- 1.619 save  markata/plugins/publish_source.py:83
         |  `- 1.216 Post.dumps  markata/plugins/post_model.py:333
         |`- 1.203 Post.yaml  markata/plugins/post_model.py:254
         |        `- 1.044 dump  yaml/__init__.py:248
         |              [3 frames hidden]  yaml
         `- 1.183 post_render  plugins/wikilink_hover.py:104
            `- 1.094 do_hover_links  plugins/wikilink_hover.py:126

[07:55:38] save complete                                                                            __init__.py:465
           teardown running                                                                         __init__.py:462
           teardown complete                                                                        __init__.py:465
           lifetime cache hit rate 0.17%                                                            __init__.py:471
           lifetime cache hits/misses 36/21118                                                      __init__.py:476
           run cache hit rate 0.17%                                                                 __init__.py:482
           run cache hits/misses 36/21118                                                           __init__.py:487

Map Cache Statistics:
Total calls: 17197
Cache hits: 15317
Cache misses: 1880
Hit rate: 89.1%
Cache size: 1880

principal-engineer-at-meta

2025-06-30T20:22:08Z

Jake Bolam principal engineer at Meta, has some of the best career advice for those looking to become principal or just be better at their craft. This video was such a banger I had to bring it in as a full post, and not just a thought. It was a random YouTube auto play, something that I probably wouldn’t have clicked on given title an thumbnail, but turned out to be very impactful. Jake is such a smart guy with a lot of great insights, and I can tell he thinks really quick on his feet, he just pulled all of these things out of his head on the fly.

YT Algorithm Gold

I don't know what it is about this title and thumbnail, but it gives me "ex

google, ex facebook, ex microsoft, $100M engineer" vibes in a cringy and not satire kind of way. I would have never clicked on it, it autoplayed after a podcast and it hit, immediately I’m like who are these guys? subd and started this post it was so good.

Long On Boarding #

Jake had a super long period of on boarding at meta, he came in as a seasoned leader yet took many months to get going. This was a phase during or near the end of the COVID-19 pandemic and his team was so swamped they were unable to give him time to help him. He felt his on boarding was taking longer than he wanted, and found backend work at Instagram. Moving to Instagram he jumped in and starting making impact quick and found his way moving up the ladder serving as tech lead for several teams.

I grabbed a ton of clips from this video, but did not for this section as it was kinda long.

Always open for a Chat #

As busy as Jake is, he leaves “Always open for a chat” on his internal profile. He does not care about rank into account, the intern can ask for a coffee chat. His key metric is how well is this person receiving information. If they are coming back with the same questions every 3 months he is going to start telling them he does not have the time.

I’ve had a fair amount of personal experience with this. Early in my career I experience quite a bit of higher ranks feeling untouchable, they were impossible to get a hold of, blew you off, and had more meetings than anyone could bear. It made me really look outside the company for inspiration and mentorship most of the time.

There’s always time for coffee #

While I don’t have “Always open for a chat” on my profile, I’ve had many of these types of conversations with my peers. It might be about something career related or more often how to implement containers, cloud, and good practices into small teams.

I remember a course in college shared the famous quote “There’s always time for coffee” and it stuck with me. The demonstration goes that no matter how much you fill your day with big rocks and tiny sand particles, there’s always a little more room for coffee, and he dumps coffee into a jar that he has already filled up.

These conversations often become very impactful. They can lead to having good relationships or job opportunities.

If we are not getting enough feedback, move faster #

Take Down Prod Sometimes #

This was an interesting take. None of us really want to take down prod. No one is advocating for major outages, but this is is not 2005 shipping out software on CD’s anymore. This take 100% depends on what you do and where you work. Obviously some sectors cannot take any downtime; nuclear power, cloud systems supporting nuclear power, navigation systems, idk theres a bunch of stuff. I bet if you are reading this that aint you. You’re probably writing some backend dashboard for the marketing team, or building out a website to upload cat photos on … Wait, thats literally what jake is doing here.

Idea being if you don’t occasionally cause some small production issues, you probably are not taking enough risk. You are moving too slow, getting feedback too late, your competitors just ran past you.

Go where you are rare #

I find this interesting. I’ve found myself within non-software companies doing sofware and data analysis most of my career, the amount of value you can bring by being that one guy that knows some pandas, containerization, how to run a vm is massive in an org that uses excel as its primary database. The rest of the org generally has massive knowledge in the business, but greatly slowed down by their ability to find value in the data.

Do work that you don’t get credit for #

I’ve worked with a lot of people in my 15 years of professional work, and I can tell you the worst ones to work with are the ones that focus too much on value. Every ounce of effort they bring needs to be backed by dollars comming into the business. I’ve seen this shake out a number of ways. You got the guy who sits on his ass all of January waiting for goals to be set, you got the guy who holds everything he knows close to his chest so that he is the one that can take the glory, and you got the guy who wont ever; refactor his code, cleanup, lint, update dependencies, and so on; because that does not have direct line to dollars coming into the business.

The clip goes a bit further than this, and hints at things that are going to enable you and your org to move faster. You might shave off 2 minutes off of ci, or docker builds. You might give everyone an easy way to run dev containers with production like dependencies in a snap. You might give them a way to clone prod data into a sandbox. These type of things provide no dollars to the company, it’s likely that few will notice but damn they add up to an efficient running organization.

Work Diary - the value of writing things down #

This one hits home, for far too long I’ve been in between note taking systems and am finally getting [[ marakta ]] setup to build out a really good zettelkasten style work notes. I’ve kinda had this on my blog for a long time, but not fully. I think the piece that I am missing here is the dumping ground. I don’t yet have a daily notes implementation that lets me just dump idea onto a page that I care little about, but is the process of starting something bigger, crosslinking between people, meetings, tasks, and launchdocs.

I hadn’t heard of the term launchdoc before this, but I took it and I am using it. I generally do an ok doc in the changelog of my products and spice it up to go to the announcement chat channel, but I don’t fully keep record of it, pulling the changelog into a launchdoc gives me that chance to spice it up and have the language it needs to go right into chat.

csv

2025-06-26T20:44:33Z

name,age,city
Alice,30,New York
Bob,25,San Francisco
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago

name,age,city
Alice,30,New York
Bob,25,San Francisco
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago
Charlie,35,Chicago

graph TD
  A-->B
  A-->C
  A-->D
  D-->E

perfect

2025-06-26T13:55:17Z

Perfect is a made up word that humans use to describe something that is above average, or works really well for them. The idea of perfection is fleeting, as you think more deeply about something, you can continue to chase the idea of perfection to unimaginable senses. Sometimes perfect simply means good enough. Could there be something better, Always, but at what cost. If I spent 10 more minutes on this post would it be better, maybe, but I might fuck it up. If I spent my lifetime studying how humans read and think, sole focused on how it pertains to this post, ya it would get better. When I use this word perfect it’s not meant in the most literal sense of the word, but perfect to me, maybe good enough given the constraints I have, its the best thing I’ve got.

just

2025-06-23T19:02:29Z

I just never quite understood why the word just can send people over the top. I get it when you don’t know someone, you don’t have history with them, and they come in saying you are doing something wrong.

When you say “just,” you’re skipping over all the invisible complexity. You’re assuming the problem is simple, and that the person asking for help hasn’t already considered the obvious.

You’re not seeing the constraints:

Legacy code Business requirements Team conventions Time, budget, or technical debt Platform limitations

~https://www.neverjust.net/

If I’ve worked with someone for more than 6 months, we have established patterns for problems, libraries we use, and they are deep in the weeds of trying to fix something, I want to ask “Why don’t you just do the same thing we do everywhere else?”

I don’t need a snarky ass response, I don’t need you to get bent out of shape about it. I am communicating that I do not know the damn constraints to this problem. I am communicating I am ignorant to the pain you have suffered trying to do something for days that I thought should be copy paste pr up in 10 minutes.

What I want to hear is “Well here’s the story of why this particular one is complex, and let me take you on my travels.” or “I’m about there and don’t have time right now just understand that this is different, we can discuss later.”

My just should not cause you pain, it should not cause you to feel belittled, If we have worked together for any amount of time and have respect for each other, you should understand that this is not coming from a place of belittlement, rather its an open door for you to explain why this thing is not what it seems at surface level.

Just use another library #

~https://www.neverjust.net/

Here Sam asks “Why don’t you just use another library?”, and Alex replies “If it were that easy, I would have done it already.”

Honestly this example is a little vague and non-specific. On boarding a new library to a team can be a big ask. It becomes something that the team has to integrate and depend on, and take on supply chain risk. This example would be better with something more specific, if the team often uses a mix of SQL, pandas, and occasionally pulls out polars for bigger tasks, Sam might legitimately ask “why don’t you just use <insert pandas, polars, SQL>”.

This communicates that Sam see’s the obvious we use polars when things get tough, or we use SQL when data transfer gets too slow.

You are not Dumb #

I want to hear the story. In no way am I calling you dumb, in fact I’m probably claiming ignorance to the problem at hand. I see that you have a lot more time invested than me and I’m offering help, not asking for snarky ass remarks back Alex.

In fact take it as a sign that I am this comfortable with you and that you are so far from dumb that I don’t have to risk being taken out of context.

Or maybe you are #

Or maybe Sam is trying to belittle Alex here, if that’s the case we need to have a conversation and work some shit out or find somewhere else to be, Alex should not be belittled like this because Sam is full of himself.

Suggestions #

~https://www.neverjust.net/

The suggestions are fine, but they feel like they are asking me to hunt around for the problem rather than you recognizing my ignorance to the issue, and let Sam tell me “Why don’t you just use another library?”

Final Thoughts #

In recognizing how much pain this causes others I do try to limit my use of just, it comes out from time to time, though I try not to. I try to remain kind, humble, and helpful. And typically in the first few weeks or months interaction with new teams/people I kinda walk on eggshells for a bit. I don’t know you or where you are, if I make it sound like you are dumb and turns out that’s what people think of you I’m gonna feel kinda bad. Just understand that not everyone is trying to attack you, communication via text is hard, give folks a break before jumping them.

Don’t be an Ass #

This post is not trying to say that you can freely belittle your team, go forth and be the irritating guy on the team that comes off like a know it all. I’m saying when you are on the receiving end try not to take it with such offense and as an opportunity to explain without the need to dance around the issue.

kraft

2025-06-01T20:32:39Z

Kraft is the family gaming server that we started early 2025.

Repopulating #

Villagers are really hard to get gong. We have this huge villager tower on the server, we nearly every bed filled, and one day we logged in and there were three. Not sure what happened until I witnessed one of them jump maybe 3 blocks down and off himself. I checked my last two villagers and they were each on one or two hearts left. We almost lost every single villager on the server.

villager-tower-needs-repopulated.mp4

To get villagers to breed and make baby villagers you need beds, which we already have, and food. I gave my guys some food and they started making baby villagers immediately, crisis averted as we start to get the first few fresh full health villagers on the server.

repopulatingv4.mp4

my nvim spellcheck setup

2025-05-31T20:11:59Z

I’ve gone too long without a proper spellcheck setup in nvim. I know it’s there, I just don’t use it, I don’t have the right keybinds, like I do with vim date, to make it work, and its clunky.

Default keybinds #

z= show spell suggestions
zg add word to dictionary
zw remove word from dictionary
]s jump to next misspelled word
[s jump to previous misspelled word

I really struggle with bracketed keybinds, they don’t flow for me. I have to shift into it and hit two keys, you cant just pop through them with intent, it always feels clunky to me.

Custom keybinds #

I barely use F-keys in my keymap so that was free game. On my keyboard I have F1-F9 in a numpad layout on my right hand, so F4-F6 are home row, these are super easy to pop through and update. I really refrain from using such high real estate keys like this unless it’s for something good, and I do a lot of writing in nvim, so fingers crossed I use the heck out of it.

jump to next misspelled word
jump to previous misspelled word
show spell suggestions

I still use zg and zw, they seem fine to me.

The Setup #

In my keymap.lua file I added these to the end, they are working so far and hopefully I use spellcheck more on my posts now that I’ve made it easy.

set("n", "<f4>", "]s")
set("n", "<f5>", "[s")
set("n", "<f6>", "<cmd>Telescope spell_suggest<cr>")

One Failure #

I went down a long rabbit hole before this trying to populate the quickfix with spelling errors, I tried looking for existing plugins, tried to get ai to give me a good prototype to start with, and everything was over complicated. So far I’m really liking this setup.

wants

2025-05-27T21:01:14Z

Inspired by mara.town/wants

want but do not enjoy #

New hardware that cannot be repaired
Disciplining Children
Nice landscaping which requires regular maintenance

enjoy but do not want #

breakfast
Fancy things
Manual Labor in moderation
Vacations to far away places

Linux Is About Choice

2025-05-27T13:37:00Z

This Luke Smith video came across my feed Linux, Bitcoin: When Tech Projects Become “Too Popular…” Don’t forget the goal.. It’s interesting to hear his perspective about Linux, FOSS, Free Software being the end goal, and that we are loosing sight of the goal. This sentiment really aligns with the early FOSS movement from Stallman, but was this ever the goal?

Taken over by apathetic interests #

Luke talks about these projects getting taken over by people with no passion for the original goal of freedom and privacy. They want the projects to grow, get bigger, and become mainstream. This feels exactly the opposite of anything Luke would want, so my bias alarm goes off here. Honestly I do see some of the grossness of projects like this that were grassroots, for freedom and privacy get taken over for money grabs. I’m completely out on bitcoin so I cannot make any comment there, but I Truly believe that the Linux kernel is not a money grab as Luke makes the new face of bitcoin sound.

Corporate backing is OK #

I’m going to take a counter point here from Luke, that it’s OK for Companies to be made, and money to be made on FOSS. While the Nats relicense has caused a Kerfuffle, I appreciate their side of the relicense, as a they discussed on The Changelog: NATS and the CNCF kerfuffle | Derek Collison shares NATS/Synadia story. They had a struggle to make money as a small project in the Linux Foundation. Large companies would not pay for their service as they have already paid into the foundation. They chose to relicense new versions of the server so such that it would become FOSS after 2 years. This gives them time to pay for their development.

Rug pulls, price gouging, and other bad behaviors aside Funding is overall good for the FOSS community. It is a good thing that Torvalds is able to make a living from the work he has done on the kernel.

FOSS hardware #

Luke mentions FOSS hardware in the video. I think this is great and really plays to the longevity of a product and limiting e-waste. I think the main goal here is not necessarily openness, but repairability. At least for me. I could not care to build out ram or hard drives from scratch, but what I do care is that I can repair my hardware. I don’t want my investments into hardware to go to waste and be thrown away because of a component failure. I get that we are largely there on the mobile side, and that really sucks.

Choice #

I really feel like the goal of Linux is choice, its not locked into FOSS. FOSS is a great option, and is really only there for Linux, and not at all an option for Windows or MacOS. I’d argue that at the beginning of the Linux project there was no choice your only real option was closed source all the way around. These days not only do we have the freedom to choose an OS, but we can choose how we want our Linux. If you want to deploy Linux at a large company, and you are willing to pay for support from someone like Red Hat or Canonical that’s an option, but its not the only option.

macOS Power User Asks DistroTube: Which Linux Never Breaks?

In this interview DT talks about getting new hardware sent to him, and they ask do you want us to put Linux on it? His answer, No, it doesn’t matter, I wont use it anyways. You see DT is not here purely for the Linux, he is here for the choice. His choice is to pick every last config option, and tailor every installed executable to fit his needs. His preferences happen to be in the full Free as in Freedom realm.

Infinite Linux #

Linux has infinite options of configuration, hundreds of distros, hundreds of desktop environments. You can Choose a Floating window manager, Tiling window manager, or go full TTY with no graphical environment. With MacOS and Windows you get it the way they decide you get it with a level of configuration that pales in comparison to what you get on Linux.

Immutable Linux #

A very recent and modern example of this is immutable or cloud native distros, the idea is that they are more like an appliance, giving you less freedom and control than you typically would have on a normal bare metal install of linux, bootc is just really taking off this year. I’m currently running one of these distro’s called bazzite and I am really loving it. I have a couple of desktop applications that I install from the discover store as flatpaks, but mosty I live in a fully custom distrobox that I put every ounce of ricing and configuration that I want into. For me it has been ultra reliable, rather than rolling my own thing that is a combination of packages unique to me, I have something that is rock hard in the community and I run versioned images of my devtainer that make it easy for me to roll back on if something were to break.

For me this is feedom, I get to fully control the environment I really care about, my terminal. I get a window manager with 9 workspaces that I can hotkey to, and I never miss a game night with my kids due to some weird graphics issue.

Freedom #

In the end Linux was born of this freedom, and is an important part. While its great that there are players here that are able to make money, be good citizens and give back, there are bad players. Even if all the players are good not everyone is going to agree and money will only spoil their hurt feelings even more. If Linux looses this freedom of choice and becomes something that you only get littered with Closed source software, or on paid platforms, or imposes any level of restrictions on to you its lost, and we need a new revolution to rise in its ashes.

Fin #

I’m trying to write more free flowing articles philosophical in nature. There are probably grammatical and spelling misttakes. There are probably things I will re-read in 6 months and not agree with, but these are my free flowing thoughts right now. I think we are missing really good opinion pieces on the internet these days. Too much AI generated, “Hello World”, shoulda read the docs BS. I subscribe to a lot of other good folks that give their real opinions you can find them on my blogroll.

tinyapps

2025-05-22T14:31:29Z

I’m working on replacing my usage of google inline search apps with real apps, these are ones that I create and host on my own homelab. The first three that I created are mostly chatgpt based, with a bit of hand edit after the fact, uploaded to minio and become an app on my k8s-pages

renamed

The original title of this post was "Replacing Google Search Apps With Self Hosted Web Apps"

I’m leaning on web wakelock to keep the screen on while these apps are running, primarily clos, timer, and stopwatch.

Clock #

A large displya clock.

Timer #

A simple timer that counts down from thet set time.

Stopwatch #

This is the one that inspired it all, I need to run a few stopwatches at work, and chose to just do it right in the google search with a few tabs running.

Dice #

A simple dice roller, this one is the one that I decided to start adding ? for help.

UUID #

It displays a uuid, thats it. ctrl + c to copy.

b64 #

Today i spent some time on b64, it is a base64 decoder/encoder. Just start tying to enter text, or paste, escape to deselect the text box, d to decode, e to encode, ? for help. It took a bit to get all of the keymaps right with the differnt modes and make sure that for instance you don’t keep typing in the input box while in help mode, or decode/encode while in the input box.

PocketCal #

This one is not mine, it’s made by Cassidy Williams), but is within the ethos and deserves a mention here. Its a single page calendar that is a static site completely rendered on the front end, data is stored in the url as you interact with it. None of your data goes to a server.

qrcode #

Today I wanted to finish an article that I was on afk and It killed me to get it from my work computer to my phone, so I made a qrcode generator. Paste in or type in your message and it shows up as a qr code live. This one needed an event listener for window resize since the library creates a rigid canvas qr code that does not fit on the screen once you create it then resize the window.

I even added query params to pass hex codes in for color and bg.

m9a devlog 1

2025-05-11T09:33:44Z

It’s sad to see textualize.io close the doors, but textual is still alive and maintained as a n open source project. I tried to use it very early, and struggled, this was before docs and tutorials really existed, before a lot of the widgets and components existed. Then as we all do I got busy and moved on to other things in life and did not have the capacity to build TUIs.

I like tuis #

I like tuis, I like staying in the terminal. I use lf daily to move files around when I want something more than mv and cp. I use k9s hourly to monitor and manage my kubernetes cluster.

Are they worth the effort?? #

As awesome as tui’s are, they are more effort to build, and less automatable. I feel like the first stage into automation of a project really needs to be a good cli, and this is often good enough for the project and I move on.

m9a (em - nine - ah) #

inspired by k9s

Like I said I really like k9s and use it all the time, It really makes running kubectl commands a breeze and much less verbose. I don’t know how useful this will be, but as a learning exercise I am working on a k9s experience for my blog generator markata.

m9a-1-trimmed.webm

Learning #

So far this is just for learning and not quite the most useful thing, I am not sure if there is a way to do it, but I am interested in the idea of some sort of framework (maybe just widgets) that can more easily turn pydantic objects into this kind of tui. I don’t quite know how it would work, or if it could work, for now just exploring the idea, and I think I hit a fairly crude clone of k9s so far.

hollow knight home row layout

2025-05-05T20:44:36Z

I just made it past 100% in my main hollow knight run, so now I will allow myself to get silksong when it comes out. I did this with a little bit of YT guidance, but mostly just figuring it out. I only just discovered the ⭐ ReznoRMichael hollow-knight-completion-check which got me an extra 2% for a few items I must have got and not saved on, because I was sure I had them.

Controller #

Hollow Knight is a game that can be played with keyboard or controller, You can use analog stick for movements, but they just translate to dpad, there really are no analog moves in the game. This makes it ripe for playing on pure keyboard. I really favor controller when there are more than one analog (throttle, brake, steering for example).

On controller I’ve switched to only using d-pad as I feel like it gives me the most crisp of controls. It is really easy to miss a pogo on analog by hitting slightly left or right.

My Keyboard #

My daily driver keyboard is a custom built 40% monoblock keyboard I absolutely love using it, it perfectly plops down on a laptop when I am in laptop mode and really fits me quite well for my wrok as a software/infrastructure engineer.

WASD #

We’ve got to talk about pinky stagger

I can use wasd perfectly fine on it, I have a layer that has no holds on any keys so that they are all nice and crisp and tailored to gaming operations that would require a hold.

Now what I haven’t ran into before is that trying to game with wasd and use the right side is very uncomfortable with this steep of a pinky stagger. Generally with keyboard an mouse games I have the keyboard a little slanted and move my fingers to wasd just fine, but it just does not work well with both hands on the keyboard.

Enter asdf #

Today is day one, I’m not quite sure how long it will last, but I am giving asdf a try for arrow keys. This will work much more akin to a fight stick where each finger has a primary direction instead of using one finger for both up and down.

vim keys, but mirrored

I chose to mirror vim keys with down towards the middle.  I had it the other

way, but when I played, I kept pressing it the other way.

a - left
s - up
d - down
f - right

power keys

I wanted to put the most common keys on the strongest finger (feels like

left pinky breaks this mantra, but whatever).

j - attack
k - dash
l - focus/cast
h - quick cast
e - inventory
u - super dash
m - map
g - dream nail

This seems weird #

So Ya, I know it’s weird. I did some searching wasn’t able to find much other than this one reddit post talking about using asdf. oddly one person reccommended sdfg, this souds so much like vim, which makes a lot of sense, but brings back moving fingers off of home row position or using one finger for down and right, which aint gonna work.

First thoughts #

I’m about 20 minutes in and struggling on False Knight, I could be a bit more patient, but I’m trying to work on pogos on all the mobs on the way in.

Holding up or down #

The ability to just hold up or down as you run through the map with enemies coming from above or below feels superior, and its going to be nice, I really just got to put some time in to get use to the layout, maybe even make some changes.

I realize now that Hollow Knight rarely needs quick swiching between up and down and would probably be fine on a wasd layout if i were to be using a split keyboard or a bit less pinky stagger that would work out, but for now I am giving this a shot.

My Son’s experience #

So my son is playing along with me, new save as well. He went with wasd and mouse, while this sounds insane to me to use mouse for clicking and no movement the natural feel that it has, has him blazing through faster than he does on controller, doing pogos he’s never done before.

False Knight down #

The recording somehow stuck on the menu screen and did not record video, but I got this screenshot of the victory.

Vengeful Spirit #

Day 2 I managed to make it to the Ancestral Mound and get the Vengeful Spirit cast. Everything is starting to feel much better and more natural. Surprisingly my main issue is with up and down since they are used a bit less, left with my pinky has not been any issue.

hollow-knight-keeb-run-vengeful-spirit.mp4

Hornet Battle 1 #

I made it successfully through the first hornet battle I was able to snag this clip of a few nicely timed pogos. Its getting better, I am still really struggling between down srike and up strike.

hollow-knight-keeb-run-hornet-pogos.mp4

Mothwing Cloak #

I got so lucky with the finish of hornet, I thought she hit me while healing, kinda lost where we were tanked a hit and almost took one last hit again and beat her to the final punch.

hollow-knight-keeb-run-hornet-finish.mp4

Gruz Mother #

After completing gruz mother I went to grab my macropad to click the clip button when her babies came out, and it was at this moment my son told me I was done for, there was no recovering from this. I was already at 2 hearts, and a mistake left me with one. You can see me fumble for the up key in the clip as I get repositioned fully on back on the board.

hollow-knight-keeb-run-gruz-mother.mp4

Now we have Sly the merchant unlocked and can buy items from him.

City #

I made it to the city, but on my way, it looks like one of the smokey layers were getting put on top of the game instead of layer in behind. I’ve never seen this, so I really have no idea what it is. During the thorny area it was in and out and right as I entered the city it locked on top and did not leave.

Soul Master #

I made my way over to the soul master, dying several times along the way to a few Soul Twisters, its time to battle.

hollow-knight-keeb-run-soul-meeting-master.mp4

After beating his second room, (this clip makes him look way too easy as he wasn’t targeting me fore some reason at the end), I picked up desolate dive, which brings a new key combo to get familiar with down+cast.

hollow-knight-keeb-run-soul-master-finish.mp4

Moss Knight #

I took a break for awhile on the game, my son got bored and moved on. He tends to switch out games a lot faster than me. I came back to it tonight after a few months off and just got some geo from the moss knights for awhile. I got this decent clip of clearing them, I’m no pro, but getting better at timing on the keeb layout.

hollow-knight-keeb-moss-knight.mp4

Lumafly Lantern #

While I had a lot of geo from the Moss Knights I finished out getting 1800 geo to get the lumafly lantern from sly’s shop at dirtmouth.

hollow-knight-keeb-lumafly-lantern.mp4

Royal Waterway #

I decided to head back to the city of tears and into the royal waterway to fight the dung defender and collect Isma’s tear.

hollow-knight-keeb-royal-waterway-unlock.mp4

hollow-knight-keeb-royal-waterway-entering.mp4

hollow-knight-keeb-royal-waterway-bench.mp4

The Dung Defender battle, great lore with this guy, lots of bouncy movement, but overall not too hard of a boss. He is not very accurate with attacks, so you have time to just sit in the corners and heal.

hollow-knight-keeb-dung-defender.mp4

Once the Dung Defender was defeated, I continued on up to the Kings Station and unlocked it.

hollow-knight-keeb-kings-station.mp4

I could not remember where ismas tear was and I ended up all the way down in deep nest, the climb out is a bit of a challenge, here I am going right by the route you take with crystal dash to get isma’s tear.

climbing-out-of-deep-nest.mp4

Isma’s Tear #

I managed to find my way through the royal waterway and where you crystal dash over to isma’s tear. This battle took two attempts, because the first I had no health left and walked in with two masks.

hollow-knight-keeb-ismas-tear-battle.mp4

Now the tear coats our shell and allows us to swim through the acid, and we can get a lot more places, and some places are just easier to get without precision parkour over the acid.

hollow-knight-keeb-ismas-tear.mp4

Thorny Mask #

I wandered around the world a bit today and found a few small items, nothing of major concequence.

hollow-knight-keeb-thorny-mask-shard.mp4 hollow-knight-keeb-spore-shroom.mp4 hollow-knight-keeb-thorny-wanderers-journal.mp4

Challenging Mantis Lords #

It felt like time to challenge the Mantis Lords. For the reason to prove to myself that I am getting better at this layout.

I drew my nail and ready for the challenge.

hollow-knight-keeb-challenging-mantis-lords.mp4

First try I made it past the first one, and deep into the second one, I think If I would have remembered the healing window for them I would have had a shot at a first try on them.

hollow-knight-keeb-mantis-lords-1.mp4

After three tries I had it. Maybe its not apples to apples with my first ever run years ago on steam deck, but it took months. I would have about 20 minutes a day waiting for kids pickup at school and I would just keep grinding at them for what added up to be likely several hours.

hollow-knight-keeb-defeating-mantis-lords.mp4

After defeating them I’m let into the treasure room where I get a bench, Hallownest seal, a big loot box, and the mark of pride. This is actually one of my favorite charms as it gives you an extra 25% range to your nail attacks. This opens up the window for pogos and reach attacks without taking damage.

hollow-knight-keeb-mark-of-pride.mp4

fix feed descriptions

2025-05-05T10:32:04Z

Today I fixed a bug in markata that has been occurring for a few months where the description for posts come out as None if coming from cache, the issue was a pretty simple check and pull properly from cache. This fixes all the descriptions in feeds and metadata on the post.

Better description #

While in there we went ahead and improved our get_description to more accurately return plain text without escaped characters, remove cutoff words, and add an elipsis if the description cuts off the text.

More description #

While I was there I made longer form posts, til, blog-post use the super description of 500 characters instead of the regular 120 character description.

Before #

After #

portal-platformer-devlog-1

2025-04-28T19:56:58Z

Here is the current state of my platformer yet to really be named, I want to make something in between hollow knight and portal.

Starting #

I made one once in make code arcace on a pybadge. It was quick and dirty, but fun to work on. It had the basic of blocks that I could move, blocks i could put a portal onto, and a goal for each level. Some levels you can just walk through and some levels required you to really think about where to place the portal.

History #

So this version of the game is a least 2 years in the making, I open it every few months give it a day or two and move on. Its mostly something that I work on with my son. He really likes to jump around on projects so its hard to make real progress on something, but we are hitting an age where he is able to come back to projects a little better.

All of this is built in python, and mostly before vibe coding was a thing, its mostly me trying to get out ideas as quick as my son is spitting out the the next idea.

Coyote #

It includes a few frames of coyote so it feels a bit more like most games.

!! Note If you are unfamiliar with the term coyote in platformers it allows you to jump for a few frames after falling off a platform, like wiley coyoyte in Luney Toons.

Wall slide/jump #

Wall slide and jump work, but so is wall climb as an unintended side effect. When you are touching a wall, your fall speed is halved.

Levels and loader #

There is a crude level loader that loads json levels with pydantic. No editor yet, just hand editing levels with json.

Checkpoints #

It makes checkpoints, when you die, you go back to your last checkpoint. Checkpoints can be invisible, and have a link that turns them into a door to another level.

66c5b984-5450-48ee-9978-00fec4815807.mp4

no art #

There is no art yet, just the skin of a platformer, levels, checkpoints, and coyote. No portals, but there is a pointer with a janky box that covers my lighting.

backup distrobox image

2025-04-09T17:35:50Z

Today I’m upgrading my distrobox, but don’t want to end up in a situation where I can’t get anything done becauase I work out of my distrobox.

distrobox ls
distrobox stop devtainer
distrobox create --clone devtainer --name devtainer-20250409
distrobox enter devtainer

Changing k8s Storage Class - Migration Job

2025-04-04T09:57:50Z

I’m setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I’m all in on gitops so option 1 was not an option. So changing storageclass it is.

Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute.

Migration Job #

This migration job will create a new pvc with the new storageclass and move the data from the old pvc to the new pvc.

Existing Pods

 This migration job will not work if you have a pod using the old pvc.  You

will need to shutdown the pod and delete it.

# old pvc with longhorn storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
# new pvc with longhorn-backup storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn-backup
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
# migration job to move the data to the new pvc
apiVersion: batch/v1
kind: Job
metadata:
  name: pvc-migration
  namespace: waylonwalker-com
spec:
  template:
    spec:
      containers:
      - name: pvc-migration
        image: ubuntu:22.04
        command: ["/bin/bash", "-c"]
        args:
          - |
            echo "Starting migration..."
            cd /data
            cp -av source/. destination/
            echo "Migration complete!"
        volumeMounts:
        - name: source-vol
          mountPath: /data/source
        - name: dest-vol
          mountPath: /data/destination
      restartPolicy: Never
      volumes:
      - name: source-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn
      - name: dest-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn-backup

Apply the manifests and wait for the job to complete.

kubectl apply -f pvc-migration.yaml

Cleanup #

I had chatgpt create me a script to help me find what is using the pvc so that it can be deleted.

#!/bin/bash

NAMESPACE="waylonwalker-com"
PVC_NAME="site-pvc-longhorn-new"

echo "⏳ Checking if PVC exists..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" || {
  echo "✅ PVC already deleted."
  exit 0
}

echo "🔍 Describe PVC..."
kubectl describe pvc "$PVC_NAME" -n "$NAMESPACE"

echo -e "\n🔗 Checking if any pod is using this PVC..."
kubectl get pods -n "$NAMESPACE" -o json | jq -r \
  --arg PVC "$PVC_NAME" \
  '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName == $PVC) | .metadata.name'

echo -e "\n🧹 Checking finalizers..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq '.metadata.finalizers'

echo -e "\n🔎 Checking associated VolumeAttachment..."
PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}')
echo "🔗 PVC is bound to PV: $PV_NAME"

kubectl get volumeattachment -A -o json | jq \
  --arg PV "$PV_NAME" \
  '.items[] | select(.spec.source.persistentVolumeName == $PV) | {name: .metadata.name, node: .spec.nodeName, attached: .status.attached}'

echo -e "\n🚀 Done."

I had still had cronjob pods completed, so I had to delete them first.

🔗 Checking if any pod is using this PVC...
pvc-migration-ndv92
waylonwalker-com-cronjob-29057840-8s92p
waylonwalker-com-cronjob-29057850-4rvm9
waylonwalker-com-cronjob-29057860-6g89j

kubectl delete pod pvc-migration-ndv92 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057840-8s92p -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057850-4rvm9 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057860-6g89j -n waylonwalker-com

homelab drive ids

2025-03-26T11:22:07Z

ls -l /dev/disk/by-id/

Drive Bay 1 ata-ST4000VN008-2DR166_ZDHBZSWZ

+————————————————————————-+ | [ Power] [ Reset ] | +————————————————————————-+ | [ BAY 5 ] 3TB WD30EFRX WMC4N0D3J9R7 ext4 /mnt/sdf4 | +————————————————————————-+ | [ BAY 4 ] 14TB EXOS ZTM09R9N zfs main pool mirror /mnt/main | +————————————————————————-+ | [ BAY 3 ] 14TB EXOS ZTM0AALS zfs main pool mirror /mnt/main | +——————————————-+ | [ BAY 2 ] 4TB IRONWOLF ZDHBZV3N zfs tank pool mirror /mnt/tank | +————————————————————————-+ | [ BAY 1 ] 4TB IRONWOLF ZDHBZSWZ zfs tank pool mirror /mnt/tank | +————————————————————————-+

THPS 1 Downtown Gaps

2025-03-15T18:46:01Z

Previously THPS 1 Skate Park Gaps

Big Ass #

THPS1-5-BigAss.mp4

Billboard Grind #

THPS1-5-BillboardGrind.mp4

BS Gap #

THPS1-5-BSGap.mp4

BS Grind #

THPS1-5-BSGrind.mp4

Burly Deck Gap #

THPS1-5-BurlyDeckGap.mp4

Car Ollie #

THPS1-5-CarOllie.mp4

THPS1-5-CarPlant.mp4

Cheesy Deck Gap #

THPS1-5-CheesyDeckGap.mp4

Death Grind #

THPS1-5-DeathGrind.mp4

Deck Gap #

THPS1-5-DeckGap.mp4

Dirty Rail #

THPS1-5-DirtyRail.mp4

Glass Gap #

THPS1-5-GlassGap.mp4

Kicker Gap #

THPS1-5-KickerGap.mp4

Kicker 2 Edge #

THPS1-5-Kicker2Edge.mp4

Kicker 2 Street #

THPS1-5-Kicker2Street.mp4

Rail 2 Rail #

THPS1-5-Rail2Rail.mp4

Roof 2 Roof #

THPS1-5-Roof2Roof.mp4

Secret Tunnel Entrance #

THPS1-5-SecretTunnelEntrance.mp4

Sucky Room Gap #

THPS1-5-SuckyRoomGap.mp4

T 2 T Gap #

THPS1-5-T2TGap.mp4

Transfer #

THPS1-5-Transfer.mp4

Truck Gap #

THPS1-5-TruckGap.mp4

Tunnel Gap #

THPS1-5-TunnelGap.mp4

Wimpy Gap #

THPS1-5-WimpyGap.mp4

Mechanic Gap #

THPS1-5-MechanicGap.mp4

Box Office Gap #

THPS1-5-BoxOfficeGap.mp4

Skyway Gap #

THPS1-5-SkywayGap.mp4

Super Skyway Gap #

THPS1-5-SuperSkywayGap.mp4

THPS 1 Skate Park Gaps

2025-03-08T14:36:23Z

Previously THPS 1 Mall Gaps Next THPS 1 Downtown Gaps

Acid Drop #

THPS1-4-AcidDrop.mp4

Across The Park #

THPS1-4-AcrossThePark.mp4

Around The Bend #

THPS1-4-AroundTheBend.mp4

HP Transfer #

THPS1-4-HPTransfer.mp4

Light Grind #

THPS1-4-LightGrind.mp4

Over The Box #

THPS1-4-OverTheBox.mp4

Over The Pipe #

THPS1-4-OverThePipe.mp4

Over the Rafters #

THPS1-4-OverTheRafters.mp4

Pool 2 Walkway #

THPS1-4-Pool2Walkway.mp4

Pool Hip #

THPS1-4-PoolHip.mp4

Pool Rail Trans #

THPS1-4-PoolRailTrans.mp4

Rafter Rail #

THPS1-4-RafterRail.mp4

Transfer #

THPS1-4-Transfer.mp4

Walkway Rail Trans #

THPS1-4-WalkwayRailTrans.mp4

Wall Gap #

THPS1-4-WallGap.mp4

Wall Gap Jr #

THPS1-4-WallGapJr.mp4

Whoop Gap #

THPS1-4-WhoopGap.mp4

Trailmakers Pioneers

2025-03-06T20:27:37Z

Trailmakers 2.0 Pioneers was released today as an update to trailmakers. It’s a brand new campaign mode that requires you to mine resources to get materials to craft your vehicles rather than salvage parts like you so in the Stranded Campaign. There also seems to be a more linear play through of the story with the frogitts, unlike stranded where it let you do anything completely open world, as long as you had the equipment to go there.

Four Frogitts #

Out of the gate Wyatt came in with four frogitts to rescue, I don’t know how he found them all, I’ve only been able to find the first one that the master gives you to bring home as a tutorial.

four-frogitts.mp4

First Boss #

After Wyatt left for bed I played a little more and beat the first of what felt like a boss level enemy, he had much different attacks. I did not think I would make it as I got hit hard from his first explosion attack before I realized what it did.

first-boss.mp4

Here is where he was on the map.

First Mega Drill #

So this first mega drill was a mistake we walked right into. Master Sapo says, “Hey go investigate the Drill, but be careful, you are not powerful enough for it.” Little did we know that getting too close will fully start the battle.

mega-drill-1-1.mp4

mega-drill-1-2.mp4

mega-drill-1-3.mp4

Leapsville Froggit rescue #

Back to leapsville we were presented with two froggit rescues. We later found out that the settlement missions will repeat forever and cycle through a few missions.

leapsville-froggit-rescue-1.mp4

leapsville-froggit-rescue-2.mp4

After doing a few missions we completed leapsville.

completing-leapsville.mp4

Finding Sulfur #

Finding sulfur in Trailmakers was a little tricky, first trying to find where it was, then realizing that it was hidden in this bulbous plant.

finding-sulfur.mp4

Derogs Pond Froggit Rescue #

derogs-pond-froggit-rescue-2.mp4

Meadows Intercity Bus Service #

meadows-intercity-bus-service-2.mp4 meadows-intercity-bus-service-3.mp4

Mega Drill Mudflats #

We came into this mega drill thinking we were prepared, but I was wrong. I equipped shotguns thinking it would be kinda like the last one, at the last second Master Sapo tells you that you need an EMP, which we got equiped. Once we got in we realized that getting close to anything is really hard and all the shotguns were useless, and to top it off all the Botnak attacks make it impossible to reconfigure.

mega-drill-mudflats.mp4

I really think the devs should make a small free zone where you don’t get attacked where you respawn.

Huge Trophy Truck #

I like making Trophy Trucks with big suspension. They seem like they glitch out more than they help, but I think they are fun, here is a huge jump I landed hard in mine.

huge-trophy-truck-jump.mp4

Emperical Extractions #

When pulling the scientists out of the their reserch to bring them back I did not have enough boost to lift them, we needed to take a pitstop to get them up to the treetops.

emperical-extractions-adding-boost.mp4

emperical-extractions-landing.mp4

THPS 1 School Gaps

2025-03-01T10:39:28Z

On the road to Gap Master achievement I got all School Gaps in THPS1. This is a super fun level, I’ll have to admit that I had to look up at least a quarter of them, and could not find them myself, mostly due to the naming, all of the rail tricks were actually ledges and I was looking for big rails. None of the warehouse gaps were manuals so I had no idea that the downhill and accross the gym would be manual tricks.

Previously THPS 1 Warehouse Gaps next THPS 1 Mall Gaps

Ditchin Class #

THPS1-2-DitchinClass.mp4

Dumpster Rail Gap #

THPS1-2-DumpsterRailGap.mp4

Funbox Table Transfer #

THPS1-2-FunBoxToTableTransfer.mp4

Garbage Ollie #

THPS1-2-GarbageOllie.mp4

Gimme Gap #

THPS1-2-GimmeGap.mp4

Hall Pass gap #

THPS1-2-HallPassGap.mp4

All-Access Gap #

THPS1-2-AllAccessGap.mp4

Huge Rail #

THPS1-2-HugeRail.mp4

Kicker Gap #

THPS1-2-KickerGap.mp4

Long Ass Rail #

THPS1-2-LongAssRail.mp4

Mini Gap #

THPS1-2-MiniGap.mp4

Over a Footbridge #

THPS1-2-OverAFootBridge.mp4

Over the Air Condidtioner #

THPS1-2-OverTheAirConditioner.mp4

Park Gap #

THPS1-2-ParkGap.mp4

Planter Gap #

THPS1-2-PlanterGap.mp4

Playground Rail #

THPS1-2-PlaygroundRail.mp4

Rail to Rail Transfer #

THPS1-2-RailToRailTransfer.mp4

Roof To Awning Gap #

THPS1-2-RoofToAwningGap.mp4

Roof To Roof Gap #

THPS1-2-RoofToRoofGap.mp4

Funbox To Rail Transfer #

THPS1-2-FunboxToRailTransfer.mp4

Downhill Manual #

THPS1-2-DownhillManual.mp4

Accross the Gym #

THPS1-2-AccrossTheGym.mp4

Roof Hop #

THPS1-2-RoofHop.mp4

Down the Awning #

THPS1-2-DownTheAwning.mp4

Perfectly Balanced #

THPS1-2-PerfectlyBalanced.mp4

Gigantic Rail #

THPS1-2-GiganticRail.mp4

Ridiculous Rail #

THPS1-2-RidiculousRail.mp4

Got the badge #

THPS 1 Mall Gaps

2025-03-01T10:39:28Z

This is one of the most memorable levels for me in THPS1 from when I had originally played it. Surprisingly there aren’t as many gaps in it as I thought there would be, there are definitely a lot that you can make up on your own. Anyways here’s my playthrough of all the mall gaps in THPS1 on my way to gap master.

Previously THPS 1 School Gaps Next THPS 1 Skate Park Gaps

Huge Stair Set Off A Mezzanine #

THPS1-3-HugeStairSetOffAMezzanine.mp4

Coffee Grind #

THPS1-3-CoffeeGrind.mp4

The Flying Leap #

THPS1-3-TheFlyingLeap.mp4

For the Whole Atrium #

THPS1-3-ForTheWholeAtrium.mp4

Fountain Gap #

THPS1-3-FountainGap.mp4

Exit Stage Right #

THPS1-3-ExitStageRight.mp4

Exit Stage Left #

THPS1-3-ExitStageLeft.mp4

Over A Stair Set #

THPS1-3-OverAStairSet.mp4

Over A Huge Stair Set #

THPS1-3-OverAHugeStairSet.mp4

Planter Gap #

THPS1-3-PlanterGap.mp4

Rail Combo #

THPS1-3-RailCombo.mp4

Skater Excalator Gap #

THPS1-3-SkaterEscalatorGap.mp4

Accross The Light Beam #

THPS1-3-AcrossTheLightBeam.mp4

The Long Rail #

THPS1-3-TheLongRail.mp4

The Short Rail #

THPS1-3-TheShortRail.mp4

THPS 1 Warehouse Gaps

2025-02-22T20:46:03Z

On the road to Gap Master achievement I got all Warehouse Gaps in THPS1. This is this classic warehouse level that everyone thinks of when THPS is mentioned. Such a good level, compact, easy to play, with some fun gaps that might take a few tries to hit.

Next up THPS 1 School Gaps

Big Rail #

THPS1-1-BigRail.mp4

Channel Gap #

THPS1-1-ChannelGap.mp4

Deck 2 Rail #

THPS1-1-Deck2Rail.mp4

Face Plant #

THPS1-1-FacePlant.mp4

High Rail #

THPS1-1-HighRail.mp4

Holy Shi #

THPS1-1-HolyShi.mp4

Kicker 2 Ledge #

THPS1-1-Kicker2Ledge.mp4

Kicker Gap #

THPS1-1-KickerGap.mp4

Monster Grind #

THPS1-1-MonsterGrind.mp4

Over The Pipe #

THPS1-1-OverThePipe.mp4

Secret Room #

THPS1-1-SecretRoom.mp4

Taxi 2 Ledge #

THPS1-1-Taxi2Ledge.mp4

Taxi Gap #

THPS1-1-TaxiGap.mp4

Transfer #

THPS1-1-Transfer.mp4

Transition Grind #

THPS1-1-TransitionGrind.mp4

fragmention

2025-02-15T09:32:43Z

This post is still WIP. …..

https://indieweb.org/fragmention##Challenges

I’ve been digging through David Bushell’s blog over the past day, he has some really good ideas about blogging and webdev. One really interesting post I came accross is url-fragment-text-directives. I’ve long had id’s linked on my headings, though sometimes broken, or now showing the link, I’ve done my best to include them. Fragmentions extend this to allow any text to be linkable like this.

fragmentioner ui: https://github.com/kartikprabhu/fragmentioner/tree/master?tab=readme-ov-file fragmentioner js: https://github.com/chapmanu/fragmentions

Examples #

https://resilientwebdesign.com/#This%20is%20a%20web%20book

fixed long standing nvim startup error

2025-02-13T08:41:39Z

Here’s the diff, this is it.

  local M = {}


  M.setup = require("waylonwalker.setup")
  M.settings = require("waylonwalker.settings")
+ M.lazy = require("waylonwalker.lazy")
  M.options = require("waylonwalker.options")
  M.globals = require("waylonwalker.globals")
  M.keymap = require("waylonwalker.keymap")
- M.lazy = require("waylonwalker.lazy")
  M.autocmds = require("waylonwalker.autocmds")
  M.util = require("waylonwalker.util")
  M.plugins = require("waylonwalker.plugins")
  M.snippets = require("waylonwalker.snippets")

  return M

The error #

On first install of my dotfiles I’m presenting with this flashbang of an error filling the screen with red background. Its kinda hard to read, I’m not deep into lua and reading their tracebacks. It pops up in this pager that if I scroll too far it quits and the error is gone before I know what it is or how it got there.

For the longest time it just felt like it randomly showed up without much warning.

I sent ai at the issue #

I tried some chatgpt and windsurf, both gave me overconfident answers that all did nothing. They just sent me in loops for way too long.

I fixed it #

What did it take??

Just sitting down and thinking about what the problem was and setting up a good test workflow. Yesterday I worked out testing nvim installs and I was immediately able to replicate the error over and over. Unlike before where it felt random, I now have a good problem statement that I can replicate.

clean install
start nvim
Lazy pops up
FLASHBANG!! Error that treesitter is not installed

What was confusing for so long was that treesitter was the first thing in lazy, and it appeared that lazy was running before the error.

The fix #

Once I really thought about not installed, it clicked. It must be the order of operations. I popped open my init.lua and there it was, lazy running after things that use treesitter. A little d+d+k+k+k+k+k+p and it was fixed. running just testnvim the next time there was no flashbang!

  local M = {}


  M.setup = require("waylonwalker.setup")
  M.settings = require("waylonwalker.settings")
+ M.lazy = require("waylonwalker.lazy")
  M.options = require("waylonwalker.options")
  M.globals = require("waylonwalker.globals")
  M.keymap = require("waylonwalker.keymap")
- M.lazy = require("waylonwalker.lazy")
  M.autocmds = require("waylonwalker.autocmds")
  M.util = require("waylonwalker.util")
  M.plugins = require("waylonwalker.plugins")
  M.snippets = require("waylonwalker.snippets")

  return M

Setting up 4G Backup with Google Fi and Netgear LM1200

2025-02-12T13:23:35Z

I connected my home network to have 4G bakcup with Google Fi and Netgear LM1200. Goodle Fi offers free data-only sim cards that you can order from their service. It takes a couple of days, and a new sim arrives in the mail free of charge. It does pull data from your account, so if you are not on an unlimited plan be careful of how much you let go through the sim.

I’ve owned this for a few years now, but it’s been disconnected for a good six months or so. I’m not s[e what happened, but it stopped recognizing the old sim card. no need to point out the coffee stains at the end, its definitely not related

Follow the provided instuctions #

Activating the sim asked for a confirmation code shipped with the sim card, then brought me to this page.

Clicking Have a Different Device? brought up instructions to set up the APN in the LM1200.

Restart #

First thing for me was that the sim was not recognized, restarting the LM1200 did the trick to recognize it as a Google Fi sim, and I started setting up by adding the APN as instructed from Google.

Setup LM1200 #

Once restarted the LM1200 was recognize the sim right away and I was able to add the APN details.

Connected #

After Saving these I was immediately connected as a backup.

Testing Failover #

Backups do not exist without testing. I pulled the WAN cable from the LM1200 and after a few seconds it swapped over to the backup.

The dashboard shows fully connected, and it popped up that it sent an alert to my phone, but that did not work for me. I might hae something misconfigured.

I ran a speed test and got a pretty respectable 38 Mbit/s up and 12 Mbit/s down.

I reconnected the WAN cable and it went back to the primary, and it sat at orange for about 10 seconds before switching back to wired connection.

Fully back on wired.

Now its all back up and running giving me a super cheap 4G backup with pretty low effort. Hopefully its reliable, I’ve upgraded about everything else on my network since last running this, I think it will behave much better. I think some other issues were actually causing me to think I was not getting connection and this fully got pulled from the network to try to fix it.

Why not 5G? #

The devices are way more expensive and this gives me all I need for a backup.

ahrefs-cleanup-2024

2025-02-12T12:56:40Z

This post is a big work in progress, expect it to keep getting better.

Initial Score #

404 Not Found, generate a page for each tag #

Title too long #

404 Not Found, comma separated tags #

Another hit on 404’s caused by tags, was tag parsing from thoughts into posts, this cause links to the full comma separated list of tags rather than one per tag.

You can see on the website the whole dang set of tags was being treated as a single tag.

Broken images #

I’m not fully sure what happened to all of these gifs in my old host and why they didn’t show up, but I moved them all and they are solid now.

I burned all of my January Credits #

So I had an issue with my self hosted shots not making them fast enough for ahrefs, and essentially ddosing myself on one of their runs. The long wait times for images burnt all of my credits.

Here was the next successful run with a HUGE improvement in overall score.

The trend for errors is looking quite good.

md files were Missing #

I had several links out to the raw source of some pages generally hosted at {slug}.md, but I had turned it off due to it causing builds to fail.

Taking another look at the error it was obvious what was going on, thoughts never had a file extension and Pathlib was throwing isADirecotryError because the path was already a directory for the index.html, adding a .md to the path for the thoughts plugin fixed it.

I made this fix in commit e0bbc on my site repo.

Broken links to tags #

I wrote a cli plugin to automatically generate feeds for my tags, but I have not ran it in awhile so I had a number of posts trying to link tags to feeds that did not exist. I fixed this in commit [5d2e22](https://github.com/WaylonWalker/waylonwalker.com/commit/5d2e227a03acb1104e4214507872c61ae2a47d64].

fix: Open Graph URL not matching canonical #

[[ thoughts-565 ]] fixes an error introduced while fixing og tags.

dst session one

2025-02-09T09:40:05Z

Wyatt is also documenting his run, while we aren’t playing in the same world we are playing together.

https://wyattbubbylee.com/dst-forever-world/

Loading into the world #

I always select the character Woodie. He comes with an axe called Lucy she has some dark humor (but not too dark), I like her. She also does not wear out so you always have a weapon and way to chop wood with you.

dst-1-1.mp4

First encounter with a Tall Bird #

On the first day I made a look around my home biome, and ran into a Tall Bird

dst-1-2.mp4

Tall Bird Stats #

Here are the tall bird stats from the wiki

Setting up base #

Just after the first night started I found my way back to the center of the map and made a stone campfire. I really like to use the stone campfires as they last forever, you just have to add wood. I was lucky enough to find stone by the Tall Bird to make this one on first day.

dst-1-3.mp4

Opening up the garden #

On day two I had enough materials gathered to make the garden digamajig to prep the soil for gardening. I planted the few seeds that I stole from the birds right away.

dst-1-4.mp4

Getting a hat #

I had the grass for a hat, so I figured why not. I’m not sure if it really helps this time of year, but I thought it looked good on Woodie.

dst-1-5.mp4

Getting into it with a Clockwork Bishop #

As I was gathering resources with Lucy, minding my own business BAM I got hit with an attack from a Clockwork Bishop, man these things hit hard and have really good aim. I took three hits before getting away, leaving me with very low heath at the end of this play session.

dst-1-6.mp4

Here is the clockwork bishop stats from the wiki

Session Two #

To kick off the second session, I noticed that when you have a torch you can light your cooking pot, what I did not realize was that this burns your cooking pot to smitherines. Nothing left but a frame of useless ashes.

dst-2-1.mp4

The TreeGuard #

After that terrible mistake I was off with Lucy collecting wood when I awoken the TreeGuard.

dst-2-2.mp4

The dude is slow, has terrible aim, but he is so persistent. No matter how far you get away he just shows up trying to kill you.

I needed to get my cooking pot back, so the next step was to get charcoal. So while I was collecting charcoal I did my best to try to take down the TreeGuard by burning him. But to no avail, he lives on.

dst-2-3.mp4

Once back at camp with my new cooking pot, he snuck up on me with his gigantic sluggish form, one wrong keystroke on mybehalf and I walked straight into his attack and I was done for.

dst-2-4.mp4

After that embarrasing death I made my way over to a touchstone I found and Woodie is back from the dead, but without his beloved Lucy, to get her we will need to go

dst-2-5.mp4

markdown split panel

2025-02-03T13:21:23Z

Today I was playing with markdown split panels. I want to be able to compare and constrast occasionually, today the inspiration hit to do this using admonitions.

Mobile Users 🔄

You will need to rotate your device to see the side by side feature.

The Markdown #

This is what I am going for, one admonition that is easy to remember, that nests inside of itself , and I can put as much markdown on the inside that I want.

!!! vsplit I Have two opinions

    !!! vsplit Left Opinion

        supporting arguments

        * lorem ipsum
        * ipsum dolor

        - [x] lorem ipsum
        - [ ] ipsum dolor

    !!! vsplit Right Opinion

        supporting arguments

        * lorem ipsum
        * ipsum dolor

        - [ ] lorem ipsum
        - [x] ipsum dolor

Here is the result of that markdown.

I Have two opinions

Left Opinion

supporting arguments

lorem ipsum
ipsum dolor

lorem ipsum
ipsum dolor

Right Opinion

supporting arguments

lorem ipsum
ipsum dolor

lorem ipsum
ipsum dolor

Vsplit

Hello World

Here is a hello world application written in the typer cli framework for cli.

Vsplit

    ``` python

#!/usr/bin/env -S uv run –quiet –script

///

requires-python = “>=3.12”

dependencies = [

“typer”,

]

///

import typer

app = typer.Typer()

@app.command() def hello(name: str = “World”): “““Prints a greeting message.””” typer.echo(f"Hello, {name}!")

if name == “main”: app()

Make MinIO Access Key

2025-02-02T19:18:05Z

Today in my homelab I wanted to setup a new service that needed a MinIO access key. So I created a new user and a new access key with the MinIO CLI rather than poking through the ui like I have before.

Global Level vs User Level #

The MinIO CLI has two levels of access, global and user level. Most of the commands in this post will have several ways to do similar tasks that would potentially work. We are going to prefer to use the user level commands for more control. For some commands such as listing Keys it is handy to use the global level.

The Policy #

First we are going to make a new policy file named mypages_rw_policy.json.

{
"Version": "2012-10-17",
"Statement": [
    {
    "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::mybucket"
    ]
    },
    {
    "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListMultipartUploadParts",
        "s3:AbortMultipartUpload"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::mybucket/*"
    ]
    }
]
}

Set the Minio Alias #

Before we can create new access keys we will need to start by settin up an alias in minio that has admin rights to the minio server.

# default values
export MINIO_ACCESS_KEY=minioadmin
export MINIO_SECRET_KEY=minioadmin

mc alias set myminio https://myminio.example.com $MINIO_ACCESS_KEY $MINIO_SECRET_KEY

Check to see if your alias exists.

mc alias list

The Script #

Now we are going to pick a SECRETKEY and a NEWUSERNAME, create the policy, create the user, attach the policy to the user and add the user to the alias.

#!/bin/bash
NEWUSERNAME=MYPAGESUSER
NEWPASSWORD=mysupersecretkey
echo USERNAME: $NEWUSERNAME
echo PASSWORD: $NEWPASSWORD

# create a new policy for read/write to the bucket
mc admin policy create myminio mybucket-readwrite mypages_rw_policy.json

# create a new user
mc admin user add myminio $NEWUSERNAME $NEWPASSWORD

# attach the policy to the user, giving them read/write to the bucket
mc admin policy attach myminio mybucket-readwrite --user $NEWUSERNAME

# add the user to the alias
mc config host add myminio https://minio.wayl.one $NEWUSERNAME $NEWPASSWORD

# create a new access key for the user with thier permissions
mc admin user svcacct add                       \
myminio MYPAGESUSER                     \
--name mypagesRWKey                       \
--description "MYPAGESUSER Key for myminio" \
--expiry 2025-03-01

NEWSECRETKEY
3e11************************************************************
Access Key: IL4*****************
Secret Key: M3D*************************************
Expiration: 2025-03-01 06:00:00 +0000 UTC

Attention

* This is the secret key, do not share it with anyone.

This secret key will only be displayed once here, make sure you copy it to a secure location now.

Give it a test #

Now we can test that it works, by creating a file and copying it into the bucket.

# set up to work with the aws cli
export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=IL4*****************
export AWS_SECRET_ACCESS_KEY=M3D*************************************
export AWS_ENDPOINT_URL=https://myminio.example.com

# create a test file
echo "You How" > hi-hello.txt
# upload the file
aws s3 cp hi-hello.txt s3://mybucket/hi-hello.txt
# test the file exists
aws s3 ls s3://mybucket
# output
# 2025-02-02 19:25:02          8 hi-hello.txt

Note

I am using the aws cli to test, I installed it with pip.

pipx install awscli

Managing Access Keys #

You can list all of the access keys for a user, or all users.

# for one user
mc admin accesskey ls myminio/ MYPAGESUSER

# for all users
mc admin accesskey ls myminio/ --all

The output will show you all of the access keys for each user.

User: MYPAGESUSER
  Access Keys:
    IL4*****************, expires: 3 weeks from now, sts: false

You can also get a list of the service accounts for a user with this command.

mc admin user svcacct ls myminio/ MYPAGESUSER

   Access Key        | Expiry
IL4***************** | 2025-03-01 06:00:00 +0000 UTC

Note

You cannot see all of these keys from the web ui, the cli seems to be the

only way to display all access keys, including access keys for other users.

Creating an RO Access Key #

I ran into errors when trying to create a new key with exactly the same permissions as the user, I’m not sure if adding a policy that does not match the user is allowed or not.

I made a new policy that has read only access to the bucket as mypages_ro_policy.json

{
"Version": "2012-10-17",
"Statement": [
    {
    "Action": [
        "s3:GetBucketLocation",
        "s3:ListBucket"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::mypages"
    ]
    },
    {
    "Action": [
        "s3:GetObject",
        "s3:ListMultipartUploadParts"
    ],
    "Effect": "Allow",
    "Resource": [
        "arn:aws:s3:::mypages/*"
    ]
    }
]
}

This command will use the above policy to create a new read only access key.

mc admin user svcacct add \
  myminio MYPAGESUSER \
  --name mypagesRWKey \
  --description "MYPAGESUSER READ ONLY Key for myminio" \
  --expiry 2025-03-01 \
  --policy mypages_ro_policy.json

The output will show you the access key and secret key.

Access Key: KDM*****************
Secret Key: 8Ww*************************************
Expiration: 2025-03-01 06:00:00 +0000 UTC

Attention

* This is the secret key, do not share it with anyone.

This secret key will only be displayed once here, make sure you copy it to a secure location now.

Removing a service account #

If you want to remove a service account, you can use the rm command to remove the Access Key, by alias and Access Key.

mc admin user svcacct rm myminio/ QH6*****************

Getting info #

You can get the info for a user or service accounts using the info subcommands.

⬢ [devtainer] ❯ mc admin user info minio-wayl-one/ MYPAGESUSER
AccessKey: MYPAGESUSER
Status: enabled
PolicyName: mypages-readwrite
MemberOf: []

⬢ [devtainer] ❯ mc admin user svcacct ls minio-wayl-one/ MYPAGESUSER
   Access Key        | Expiry
KDM***************** | 2025-03-01 06:00:00 +0000 UTC
IL4***************** | 2025-03-01 06:00:00 +0000 UTC

⬢ [devtainer] ❯ mc admin user svcacct info myminio/ IL4*****************
AccessKey: IL4*****************
ParentUser: MYPAGESUSER
Status: on
Name: mypagesRWKey
Description: MYPAGESUSER Key for myminio
Policy: implied
Expiration: 3 weeks from now

⬢ [devtainer] ❯ mc admin user svcacct info myminio/ KDM*****************
AccessKey: KDM*****************
ParentUser: MYPAGESUSER
Status: on
Name: mypagesRWKey
Description: MYPAGESUSER READ ONLY Key for myminio
Policy: embedded
Expiration: 3 weeks from now

markata 0.8.0

2025-01-31T20:17:00Z

I realize that I never did a post on markata 0.8.0, so here it is. 0.8.0 was released on Jan 2, 2024, just over a year ago at this point. This was the release that we got pydantic support, and multi feeds.

Pydantic Support Was kinda big #

Markata is leaning on pydanttic for configuration and Post models. These models are filled with validators such that you can give it an empty markdown post and it will figure out some pretty sane default values for the frontmatter. From there you can progressivly enhance your post with more information like title, date, tags, slug, description.

validators are awesome!! for instance I don’t set the description on many of my posts by hand, I let the auto_description grab the first bit of text from the post most of the time.

Multi Feeds #

Markata==0.8.1 brought multiple feeds into the mix, and started to take place of rss and sitemap. Feeds are a list of posts that are configured with a slug, filter, sort, reverse, description, and a template. These feeds then become objects you can use to access posts, as well as html pages, rss feeds and sitemaps.

Jinja Templates #

The final major change within this series is the change out from string templates to proper jinja templates with partial templates that you can include. This has made maintaing templates much easier, as well as the ability to customize. Markata will load templates from both your local templates directory then from its built in templates directory if it does not find a template locally.

[[markata.feeds]]
slug = 'python'
filter = "date<=today and 'python' in str(tags).lower()"
sort = "date"
reverse = true
description = 'A feed of all my python posts'

Directly from the release notes #

The rest of the post is details directly from the release notes.

0.8.2 #

Fix: markata installs setuptools required by one dependency
Fix: cleaup cli output
Fix: speed up cli starup with some lazy imports
Fix: all cache.adds were replaced with cache.set
Fix: Updated to new typer format requiring name=
Fix: teardown only runs if a the build process was started, i.e. some clis will not need to teardown

0.8.1 #

Feeds have partials #

The feeds plugin now has configurable partial_template that can be used to render only the inside of the feeds page. This is indended to allow you to load small feeds into a page with htmx.

Better Jinja Templates #

Markata now fully supports jinja templates with a loader that will load from your templates directory, the markata built-in templates, and from a dynamically generated templates directory in your .markata.cache directory.

cli #

You can list out your templates and configuration with the following command

markata templates show

Variables #

The following variables are available within jinja templates for post templates. This is now consistent accross all three built in plugins that render jinja templates.

post_template

__version__ - the version of markata
markata - the markata instance
config - the markata config
body - the body of the post
post - the current post object

feeds

Similarly from within rendering feeds.

__version__ - the version of markata
markata - the markata instance
config - the markata config
posts - the list of posts
post - a pseudo post object with title, slug, description, and date for template consistency
feed - the current feed object

jinja_md

Similar to posts from within jinja_md to render a markdown post as a template.

__version__ - the version of markata
markata - the markata instance
body - the body of the post
config - the markata config
post - the current post object

Feeds cli #

The feeds cli will help show which templates each feed will be using.

❯ markata feeds show
                                          Feeds 6
┏━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃            Feed ┃ posts ┃ config                                                        ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ project_gallery │ 2     │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Project Gallery                                        │
│                 │       │ slug: project-gallery                                         │
│                 │       │ name: project_gallery                                         │
│                 │       │ filter: 'project-gallery' in str(path)                        │
│                 │       │ sort: title                                                   │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│            docs │ 10    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Documentation                                          │
│                 │       │ slug: docs                                                    │
│                 │       │ name: docs                                                    │
│                 │       │ filter: "markata" not in slug and "tests" not in slug and ... │
│                 │       │ sort: slug                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│         autodoc │ 17    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: AutoDoc Python Modules.                                │
│                 │       │ slug: autodoc                                                 │
│                 │       │ name: autodoc                                                 │
│                 │       │ filter: "markata" in slug and "plugin" not in slug and "te... │
│                 │       │ sort: slug                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│             all │ 73    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: All Markata Modules                                    │
│                 │       │ slug: all                                                     │
│                 │       │ name: all                                                     │
│                 │       │ filter: True                                                  │
│                 │       │ sort: date                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│    core_modules │ 17    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Markata Core Modules                                   │
│                 │       │ slug: core_modules                                            │
│                 │       │ name: core_modules                                            │
│                 │       │ filter: 'plugin' not in slug and 'test' not in slug and ti... │
│                 │       │ sort: date                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│         plugins │ 42    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Markata Plugins                                        │
│                 │       │ slug: plugins                                                 │
│                 │       │ name: plugins                                                 │
│                 │       │ filter: 'plugin' in slug and 'test' not in slug               │
│                 │       │ sort: date                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
└─────────────────┴───────┴───────────────────────────────────────────────────────────────┘

0.8.0 #

pydantic support

Pydantic Support #

Now plugins are configured through a pydantic Config object.

breaking changes #

There are a number of breaking changes going into 0.8.0. Use caution when upgrading.

glob config is now under markata.glob #

- [markata]
- glob_patterns = "pages/**/*.md"
+ [markata.glob]
+ glob_patterns = "pages/**/*.md"

Feeds are now a list #

Feeds are now a list of Objects within the configuration that you choose from whether its toml or yaml. Also templates_dir is now configurable, and once you have a templates dir it is better to specify templates by name relative to your templates_dir.

[markata]
templates_dir = "pages/templates"

[markata.feeds.published]
template="archive_template.html"
card_template = "feed_card.html"
filter="date<=today and templateKey in ['blog-post', 'til'] and status.lower()=='published'"
sort="date"

old

[[markata.feeds.published]]
template="pages/templates/archive_template.html"
card_template = "pages/templates/feed_card.html"
filter="date<=today and templateKey in ['blog-post', 'til'] and status.lower()=='published'"
sort="date"

new

markata.summary.filter_count is now a list #

The old way was to set up a dict, where the keys were the name, now its a list of Objects with an explicit name field.

[markata.summary.filter_count.drafts]
filter="published == 'False'"
color='red'

Old

[[markata.summary.filter_count]]
name='drafts'
filter="published == 'False'"
color='red'

markata 0.8.0

2025-01-31T20:17:00Z

Pydantic Support Was kinda big #

Markata is leaning on pydantic for configuration and Post models. These models are filled with validators such that you can give it an empty markdown post and it will figure out some pretty sane default values for the frontmatter. From there you can progressively enhance your post with more information like title, date, tags, slug, description.

validators are awesome!! for instance I don’t set the description on many of my posts by hand, I let the auto_description grab the first bit of text from the post most of the time.

Multi Feeds #

Markata==0.8.1 brought multiple feeds into the mix, and started to take place of RSS and sitemap. Feeds are a list of posts that are configured with a slug, filter, sort, reverse, description, and a template. These feeds then become objects you can use to access posts, as well as html pages, RSS feeds and sitemaps.

Jinja Templates #

The final major change within this series is the change out from string templates to proper jinja templates with partial templates that you can include. This has made maintaining templates much easier, as well as the ability to customize. Markata will load templates from both your local templates directory then from its built in templates directory if it does not find a template locally.

[[markata.feeds]]
slug = 'python'
filter = "date<=today and 'python' in str(tags).lower()"
sort = "date"
reverse = true
description = 'A feed of all my python posts'

Directly from the release notes #

The rest of the post is details directly from the release notes.

0.8.2 #

Fix: markata installs setuptools required by one dependency
Fix: cleanup cli output
Fix: speed up cli startup with some lazy imports
Fix: all cache.adds were replaced with cache.set
Fix: Updated to new typer format requiring name=
Fix: teardown only runs if the build process was started, i.e. some cli’s will not need to teardown

0.8.1 #

Feeds have partials #

The feeds plugin now has configurable partial_template that can be used to render only the inside of the feeds page. This is indented to allow you to load small feeds into a page with htmx.

Better Jinja Templates #

cli #

You can list out your templates and configuration with the following command

markata templates show

Variables #

The following variables are available within jinja templates for post templates. This is now consistent across all three built in plugins that render jinja templates.

post_template

__version__ - the version of markata
markata - the markata instance
config - the markata config
body - the body of the post
post - the current post object

feeds

Similarly from within rendering feeds.

__version__ - the version of markata
markata - the markata instance
config - the markata config
posts - the list of posts
post - a pseudo post object with title, slug, description, and date for template consistency
feed - the current feed object

jinja_md

Similar to posts from within jinja_md to render a markdown post as a template.

__version__ - the version of markata
markata - the markata instance
body - the body of the post
config - the markata config
post - the current post object

Feeds cli #

The feeds cli will help show which templates each feed will be using.

❯ markata feeds show
                                          Feeds 6
┏━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃            Feed ┃ posts ┃ config                                                        ┃
┡━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ project_gallery │ 2     │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Project Gallery                                        │
│                 │       │ slug: project-gallery                                         │
│                 │       │ name: project_gallery                                         │
│                 │       │ filter: 'project-gallery' in str(path)                        │
│                 │       │ sort: title                                                   │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│            docs │ 10    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Documentation                                          │
│                 │       │ slug: docs                                                    │
│                 │       │ name: docs                                                    │
│                 │       │ filter: "markata" not in slug and "tests" not in slug and ... │
│                 │       │ sort: slug                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│         autodoc │ 17    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: AutoDoc Python Modules.                                │
│                 │       │ slug: autodoc                                                 │
│                 │       │ name: autodoc                                                 │
│                 │       │ filter: "markata" in slug and "plugin" not in slug and "te... │
│                 │       │ sort: slug                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│             all │ 73    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: All Markata Modules                                    │
│                 │       │ slug: all                                                     │
│                 │       │ name: all                                                     │
│                 │       │ filter: True                                                  │
│                 │       │ sort: date                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│    core_modules │ 17    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Markata Core Modules                                   │
│                 │       │ slug: core_modules                                            │
│                 │       │ name: core_modules                                            │
│                 │       │ filter: 'plugin' not in slug and 'test' not in slug and ti... │
│                 │       │ sort: date                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
│         plugins │ 42    │ DEFAULT_TITLE: All Posts                                      │
│                 │       │ title: Markata Plugins                                        │
│                 │       │ slug: plugins                                                 │
│                 │       │ name: plugins                                                 │
│                 │       │ filter: 'plugin' in slug and 'test' not in slug               │
│                 │       │ sort: date                                                    │
│                 │       │ reverse: False                                                │
│                 │       │ rss: True                                                     │
│                 │       │ sitemap: True                                                 │
│                 │       │ card_template: card.html                                      │
│                 │       │ template: feed.html                                           │
│                 │       │ rss_template: rss.xml                                         │
│                 │       │ sitemap_template: sitemap.xml                                 │
│                 │       │ xsl_template: rss.xsl                                         │
│                 │       │                                                               │
└─────────────────┴───────┴───────────────────────────────────────────────────────────────┘

0.8.0 #

pydantic support

Pydantic Support #

Now plugins are configured through a pydantic Config object.

breaking changes #

There are a number of breaking changes going into 0.8.0. Use caution when upgrading.

glob config is now under markata.glob #

- [markata]
- glob_patterns = "pages/**/*.md"
+ [markata.glob]
+ glob_patterns = "pages/**/*.md"

Feeds are now a list #

[markata]
templates_dir = "pages/templates"

[markata.feeds.published]
template="archive_template.html"
card_template = "feed_card.html"
filter="date<=today and templateKey in ['blog-post', 'til'] and status.lower()=='published'"
sort="date"

old

[[markata.feeds.published]]
template="pages/templates/archive_template.html"
card_template = "pages/templates/feed_card.html"
filter="date<=today and templateKey in ['blog-post', 'til'] and status.lower()=='published'"
sort="date"

new

markata.summary.filter_count is now a list #

The old way was to set up a dict, where the keys were the name, now its a list of Objects with an explicit name field.

[markata.summary.filter_count.drafts]
filter="published == 'False'"
color='red'

Old

[[markata.summary.filter_count]]
name='drafts'
filter="published == 'False'"
color='red'

markata search

2025-01-30T20:21:36Z

A side effect of Markata DidYouMean is that we are able to implement some rudimentary search based on slug, title, tags, and description.

I made a video of the early work on using this side effect to search for content on markata.dev.

Replay markata-search-1.mp4

This was first implemented to solve 404 pages giving users a way to try to find the page that they tried to access. But it turned out to be a decent way to search through the docs.

Markata DidYouMean

2025-01-29T20:28:36Z

Coming in Markata 0.9.1 is far better documentation. i.e. Documentation that actually exists for everything. As part of poking around I realized that I often go to look up the docs for a plugin and forget that the path is /markta/plugins/feeds, sometimes I might try /markata/feeds or /plugins/feeds.py or /feeds or I might even forget the plugin name exactly and try something like feed and get a 404. So I added a didyoumean plugin to markata that takes care of this.

I made a quick recording of this early feature, pay close attention to the url as it automatically updates to the correct page.

markata-didyoumean.mp4

Happy Path #

direct forward

If you have one post called /markata/plugins/feeds, and it is the only post called feeds, any combination of /markata/feeds or /plugins/feeds or /feeds will all automatically redirect with an html page (not a server 3xx) to the /markata/plugins/feeds post.

Here is the snippet that does the redirect.

<div class="container mx-auto px-4 py-8">
    <h1 class="text-2xl font-bold mb-4">Redirecting...</h1>
    <p class="mb-4">You will be redirected to <a href="/markata/plugins/feeds" class="text-blue-500 hover:underline">/markata/plugins/feeds</a></p>
    <script>window.location.href = "/markata/plugins/feeds";</script>
</div>

You won’t see it unless you are on a really bad network, or you have js disabled, but this is what it would look like if you ever saw it. Notice there is a backup link if you have js disabled.

Multiple Similar Posts #

list of options

If you have multiple posts called didyoumean from different directories, in the video I made a clashing post at /markata/cli/didyoumean with /markata/plugins/didyoumean. If you go to /didyoumean it will notice that there are multiple options and present you with a list of all of the potential pages that match.

Finally No Matches #

404.html

Last ditch effort is to implement a 404 page. This page will know all of the possible paths in your project and give you a list of all the similar pages more like a traditional didyoumean plugin.

cold builds

2025-01-27T20:28:43Z

Here are a list of some cold builds from my site. I’ve ran this site for a long time and would like to have some references to go back to, and wish I had kept a few profiles of cold builds laying around to compare with. The time is now lets keep some cold build links around for reference.

01/27/2025 #

https://ec314b08.waylonwalker-com.pages.dev/_profile/ - 429s - markata==0.9.0.dev5 has md_video open cache on every post issue
https://27f117fd.waylonwalker-com.pages.dev/_profile/ - 112s- markata==0.9.0.dev5 after fixing md_video issue
https://e6b8f64a.waylonwalker-com.pages.dev/_profile/ - 16s - markata==0.9.0.dev5 1 post not skipped, this one
https://10b778b4.waylonwalker-com.pages.dev/_profile/ - 5.65s - markata==0.9.0.dev5 fully hot cache build with no changes

pesos

2025-01-25T17:20:55Z

Pesos is the act of Publish Elsewhere Syncicate to Own Site. It is an indieweb concept that I recently started applying to my own site.

here does it skip again

Note

See <https://indieweb.org/PESOS> for more information, they have a ton of

information about the indieweb

In short it is the concept of pulling data from other sites that you use and republishing it to your own site. This gives a single source of information for you, and protection against sites and apis changing or rug pulling. Other people might have a lot more use cases for this, but I already begin a lot of my data right on my site.

GitHub stars #

I am using the github api to get a list of my stars and then create posts in the github repo for my blog. This allows me to keep track of things I star on GitHub in my own way, and share them out with my rss feeds.

hover z-index and positioning

2025-01-23T15:54:07Z

I broke my sick wikilink hover recently in a refactor, today I did some diving in to figure out what happened.

Before #

As you can see in the screenshot below, the link is in a list of links, and when the hover image pops up it sits behind all of the other text. The z-index of the list-item is supposed to be raised above the others on hover.

Manually setting z-index to 20 in the inspector I noticed this message from devtools, “The position: static property prevents z-index from having an effect. Try setting position to something other than static.”, looking back at some of my refactoring I had relative in an old template and it was lost.

After #

After properly setting position to relative on the list-item, the hover image is raised above the others.

analytics

2025-01-22T08:36:27Z

I’ve been posting on this site since 2016, when layoffs were rolling through the company I worked for at the time. Starting a personal blog and a pile of side projects felt like one of the best things I could do for my resume, so off I went. This site is built on markata, more about that in the /colophon.

The old version of this page embedded static SVGs from my Python Markata build. Those files are gone in the markata-go site, so this page now renders the yearly posting history directly from the current content set.

Post Contributions All Time Monthly #

Contribution Graph Error: Invalid JSON configuration

invalid character '%' looking for beginning of object key string

Post Contributions in 2026 #

Post Contributions in 2025 #

Post Contributions in 2024 #

Post Contributions in 2023 #

2023 was a very busy year for me and I started slowing down. About mid year I felt like I had a lot that I wanted to get out, but felt like I couldn’t, because I did not have the time to blog, so I added Thoughts on 7/22/2023 with this first thought.

Notice the huge uptick that started immediately as shots was released

Post Contributions in 2022 #

Post Contributions in 2021 #

At the End of 2021 I started posting [[ til ]]s daily for a few months. This is the point when I really started lowering the barrier to entry to make a blog post. A blog post did not need to be a super long essay, but could be the size of a tweet.

Post Contributions in 2020 #

Post Contributions in 2019 #

2019 was a huge learning year for me. I was very busy leading a migration to the cloud, containerized data pipeline orchestration, and setting up new projects and templates using Kedro

Post Contributions in 2018 #

This was the year I really started reaching for the terminal kicking off the year with Vim Notes and rounding out with 📝 Bash Notes. I had been watching luke smith for a awhile, and started managing my first linux server at work. I was stuck with windows at the time, but wsl was a new thing that let me run linux in the terminal.

Post Contributions in 2017 #

2017 was the year of getting started, I was coming up on 5 years into my career, and layoffs were happening hard at the time. I remember fist landing 5 years before and being told in the companies long history they have never laid off engineers, it was a very safe place to be. This was a wake up call that it might all turn around and I would take nothing with me. Blogging became my way to document things I was learning, it was making be better at communicating, and giving me a reason to take a deeper dive into interesting topics.

Post Contributions in 2016 #

2016 was the year that I created my github account, and really got serious about career switching from Mechanical Engineering to Software Engineering. All of my 2016 posts are back-dated github stars.

I fixed lists in my blog

2025-01-21T16:08:01Z

A small improvement to my blog today. I was checking out my /now post and noticed that lists have left padding but no discs or numbers. I popped open my tailwind/app.css added these lines and rebuilt.

#post-body li {
  @apply list-disc;
}
#post-body ol {
  @apply list-decimal;
}

And rebuild, which I actually do with ⭐ casey just, but showing the command here for clarity.

npx tailwindcss --input tailwind/app.css --output static/app-39.css

WHY the applies #

Before #

After #

a night of rounds

2025-01-16T21:11:56Z

Tonight my son and I played rounds, I pulled clips of all the final rounds of each set of rounds. We play hardly any games with guns in them, this is one of the few exceptions, it feels quite whimsical, fun, and strategic, and not at all realistic or first person. The strategy part really brings me back to the fps games I would play with friends in my teens.

Round 1 - Orange #

I thought I had him on this one. I had this rare power up that allows me to shot through walls. It came down to the very last round and he had too much firepowerfor me.

Sorry, your browser doesn't support embedded videos.

Round 2 - Blue #

The final round of round 2 was Super fast. I felt like I nerfed my homing power up by adding fastball to it and it didn’t really home in anymore. What made it so quick, I think he had glass cannon and I laid down two well placed shots.

Sorry, your browser doesn't support embedded videos.

Round 3 - Orange #

Round three he got some really good cards, but too much firepower and stacking bouncy and ricochet made it too much to control and he friendly fired both rounds.

Sorry, your browser doesn't support embedded videos.

Round 4 - Blue #

Round 4 I stacked up radiance and saw and just went for it. Radiance emits the power of the sun while reloading, so I went in guns blazin, pulled the shield for saw and ran at him.

Sorry, your browser doesn't support embedded videos.

Round 5 #

Round 5 I stacked up a multi shot, Big bullet, and timed explosion. I didn’t let off the firepower and there was no chance.

Sorry, your browser doesn't support embedded videos.

wondercraft session one

2025-01-13T15:06:46Z

My son and I are starting a new Minecraft server today called wondercraft.

setup #

We are running a few mods on this server notably FTB Ultimine for less grindy resource gathering, Sodium for shaders, and Universal Shops.

We are runninng on Minecraft 1.20.1, I think this is because it was the latest supported by Universal Shops, its at least one major release old at this point.

I grabbed a few shaders, the ones that you see in this post are rethinking voxels.

spawn #

We initially spawned in a plains biome near an acacia and forest biome. We ended up going straight for the forest, only coming back for some sheep and wheat seeds.

Night one Zombie fight #

On the first night we had the obligitory zombie fight before getting beds and bases set.

Salmon Wrangler #

I was the designated food provider. I setup a small wheat farm with about 64 seeds, then went out to hunt down some salmon for the immediate food need. I ended up with about a stack of salmon at the end of the session.

Setting up Shop #

So as mentioned above we are running a mod for shops called universal shops. This mod is new to me and my son wanted to run it inspired by watching hermitcraft. We both decided that diamonds were the goal currency, but we wanted to set up some shops early so we settled on Oak logs as the currency. My job was to setup food based shops while he set up ore based shops.

My earnings after the first day.

First Death #

I forgot that I turned my sound off for a call, got lost on the chase of the salmon and did not leave myself enough time to get to the top.

Sorry, your browser doesn't support embedded videos.

Second Death #

Playing sloppy again I was walking around at night with only three hearts left. My son wanted me to check out his shop that he just built out and I got hit by a skelly on the way up, after the first hit I was sitting at 1 heart with no where to go.

Sorry, your browser doesn't support embedded videos.

Third Death #

This one is just plain embarrasing, I was lost in the flow of hunting salmon again, not paying any attention. I came up for air, but didn’t even gie time for more than a couple bubbles to come back before diving back in and died almost instantly.

Sorry, your browser doesn't support embedded videos.

Note to self #

Next time run Minecraft in full screen so you don’t dox your start menu.

nvim-manager

2025-01-12T21:14:19Z

I recently built a cli application as a nearly-one-shot-app called nvim-manager. It manages your nvim dotfiles install.

Why #

How is nvim manager any better #

nvim-manager allows you to install pinned versions of your dotfiles, your friends dotfiles, and distros in ~/.config. This allows you to have stable versions that will not break installed while you change things.

I’m sure most of us have experienced the pain of installing one plugin, only to update all of your plugins and break something.

Or, you have small changes on every machine you use, because they are all just a bit different and now you have big merge conflicts to deal with.

All of this aside you can install a distro to get you by, or a known working version of your own dotfiles.

So all these versions in ~/.config #

ya, thats the magic of NVIM_APPNAME, I can boot up any of these intalled working versions in an instant with NVIM_APPNAME=nvim-waylonwalker-v0.0.1 nvim. I can still cowboy up and break my main install, but as long as I am diligent to keep these installs untouched I will always have a version to fall back to in that moment of need.

So what do I need to do? #

start tagging your dotfiles as you hit stable versions
Install curl https://i.jpillora.com/waylonwalker/nvim-manager | bash
Add some env variables to your shell startup (~/.bashrc or ~/.zshrc for example)
Install that pinned version of your dotfiles nvim-manager install v0.0.1
Install a distro nvim-manager install --distro astronvim

Here are those environment variables I was talking about, set them to use your dotfiles repo, name it what you like, and set your NVIM_APPNAME if you want to default to a stable nvim and force yourself to unset NVIM_APPNAME to live on the edge.

Install nvim-manager #

apt install curl git unzip
curl https://i.jpillora.com/MordechaiHadad/bob | bash
bob install nightly
bob use nightly
ln -s ~/.local/share/bob/nvim-bin/nvim ~/.local/bin
curl https://i.jpillora.com/waylonwalker/nvim-manager | bash

Install your own dotfiles #

Setup with the following environment variables.

# nvim-manager
export NVIM_MANAGER_REPO=https://github.com/WaylonWalker/devtainer
export NVIM_CONFIG_PATH=nvim/.config/nvim
export NVIM_MANAGER_INSTALL_DIR=$HOME/.config
export NVIM_MANAGER_PREFIX="nvim-waylonwalker"
export NVIM_APPNAME=${NVIM_MANAGER_PREFIX}-v0.0.1

nvim-manager install v0.0.1
nvim

Note I like installer by jpillora, I self host it for my own security, but feel free to download from GH if it makes you feel safer.

Ubuntu Container Speedrun #

Here is a speedrun to getting nvim up and running in an ubuntu container.

set -euxo
mkdir -p ~/.local/bin
export PATH=$PATH:~/.local/bin
apt update
apt install curl git unzip -y
curl -LsSf https://astral.sh/uv/install.sh | sh
curl https://i.jpillora.com/MordechaiHadad/bob | bash
mv bob ~/.local/bin
bob install nightly
bob use nightly
ln -s ~/.local/share/bob/nvim-bin/nvim ~/.local/bin
curl https://i.jpillora.com/waylonwalker/nvim-manager | bash
mv nvim-manager ~/.local/bin
export NVIM_MANAGER_REPO=https://github.com/WaylonWalker/devtainer
export NVIM_CONFIG_PATH=nvim/.config/nvim
export NVIM_MANAGER_INSTALL_DIR=$HOME/.config
export NVIM_MANAGER_PREFIX="nvim-waylonwalker"
export NVIM_APPNAME=${NVIM_MANAGER_PREFIX}-v0.0.1
nvim-manager install v0.0.1
nvim-manager install --distro lazyvim
nvim-manager install --distro astronvim
nvim-manager install --distro nvchad
nvim-manager install --distro kickstart
nvim-manager install --distro lunarvim

# plugins like treesiter need gcc and make
apt install gcc make -y
export TZ="America/Chicago"
export DEBIAN_FRONTEND=noninteractive
apt update
apt install tzdata -y
ln -fs /usr/share/zoneinfo/$TZ /etc/localtime
dpkg-reconfigure -f noninteractive tzdata
# Some of the mason installs need npm
apt install nodejs npm -y
# plugins like telescope require ripgrep
apt install fzf ripgrep -y

Give it a Star #

I’d appreciate a star if you find this app useful.

https://github.com/waylonwalker/nvim-manager

/now

2025-01-12T21:07:12Z

This page is meant to be simple view into what is happening in my life right now. Inspired by @NowNowNow. and Derek Sivers.

Last Update 2026-04-19

markata-go

Last Update 2025-10-29

Silksong has my attention, it’s great
Working on new features for dropper
- Vibe coding sucks
- Images will soon be able to be resized and be more efficient over the network.
Trying to branch out and add more than software to the blog. Things like design and 3d printing.

Last Update 2025-05-22 14:27:47

Working on [[ninesui]]
Working on [[krayt]]
learning observability with grafana
Fairly happy with markata, at the moment, less focus and some small fixes
The blog is fully k8s, love the experience
replacing Google apps with self hosted apps [[replacing-google-search-apps-with-self-hosted-web-apps]]
Doing a second run through hollow knight home row layout
Also second run of animal well
put a pause on the thps gap project Posts tagged: thps-gaps
working on a split version of my keyboard

Last update Fri Jan 31 08:43:10 PM CST 2025

Heavy Focus in on markata, upcoming markata search and Markata DidYouMean
Moving from Arch to Bazzite, I’m fully here but making everything just that much smoother in a container based world.
kubernetes - trying to get better
k8s in the homelab - All of my services I run in my homelab are currently k8s, trying to make it all smooth and consistent
k8s in the blog, I’m building this static site in k8s and its never been better.
learning agentic workflows - I’m trying to build out a few side projects minimally touching the code and only instructing windsurf
mostly-one-shot-apps - I just built nvim-manager as a mostly one-shot-app (i.e. the app was nearly v1 out of the first prompt)
markata plugins with chatgpt project using rag - analytics - tag aggregator
Family focus - we beat trailmakers a night of rounds

one-shot-s3-cleanup

2025-01-12T11:18:25Z

So I made a mistake in an app I am working on and ended up creating thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails… etc. I was able to delete them all, but I wanted to make a one-shot script to do it.

One Shot App #

I got the idea of the one shot app from Simon Willison and replicated his setup in a chatgpt project

Initial Prompt #

a mistake was made in my s3 bucket and I've created a bunch of extra files
write a script that deletes all files that contain _thumb_thumb

⬢ [devtainer] ❯ aws s3 ls s3://dropper
2024-12-29 14:32:32      16158 02271f4c-be18-4bea-b23e-d00f9fe42b9f.webp
2025-01-11 14:20:49       2878 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb.webp
2025-01-11 14:21:17       2858 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb_thumb.webp
2025-01-11 14:21:44       2856 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb_thumb_thumb.webp
2025-01-11 14:21:44       2856 02271f4c-be18-4bea-b23e-d00f9fe42b9f_thumb_thumb_thumb_thumb.webp
2024-12-27 10:25:36       2812 06422c09-d0da-44ec-9339-786864ebccf2.webp
2025-01-11 14:20:49       2710 06422c09-d0da-44ec-9339-786864ebccf2_thumb.webp
2025-01-11 14:21:17       2652 06422c09-d0da-44ec-9339-786864ebccf2_thumb_thumb.webp
2025-01-11 14:21:45       2632 06422c09-d0da-44ec-9339-786864ebccf2_thumb_thumb_thumb.webp
2025-01-11 14:21:45       2632 06422c09-d0da-44ec-9339-786864ebccf2_thumb_thumb_thumb_thumb.webp
2024-12-29 20:44:14     146060 074edddb-8b46-4d94-9b55-d70a7bb74366.webp
2025-01-11 14:20:50      12476 074edddb-8b46-4d94-9b55-d70a7bb74366_thumb.webp
2025-01-11 14:21:17      12400 074edddb-8b46-4d94-9b55-d70a7bb74366_thumb_thumb.webp

create a typer application to do this job
include a --dry-run flag
make the search for _thumb_thumb editable

s3_cleanup #

Here is the result of the s3_cleanup script. It ended up being 5 or 6 passes to get everything I wanted with soft-delete enabled. Nice preview outputs in the dry run. This was 90% created by chatgpt 4o, with some slight hand edits by me along the way.

#!/usr/bin/env -S uv run --quiet --script
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "boto3",
#     "typer",
#     "rich"
# ]
# ///

import boto3
from datetime import datetime, timezone
from rich.console import Console
from rich.table import Table
import traceback
import typer

app = typer.Typer()
console = Console()

DEFAULT_SOFT_DELETE_DIR = "soft-delete/"


def human_readable_size(size: int) -> str:
    """Convert bytes to a human-readable format."""
    for unit in ["B", "KB", "MB", "GB", "TB"]:
        if size < 1024:
            return f"{size:.2f} {unit}"
        size /= 1024
    return f"{size:.2f} PB"


def calculate_file_age(last_modified: datetime) -> str:
    """Calculate the age of the file from the current time."""
    now = datetime.now(timezone.utc)
    age = now - last_modified
    days = age.days
    if days > 0:
        return f"{days} days"
    hours, remainder = divmod(age.seconds, 3600)
    minutes = remainder // 60
    if hours > 0:
        return f"{hours} hours"
    return f"{minutes} minutes"


@app.command()
def clean(
    bucket_name: str = typer.Argument(..., help="The name of the S3 bucket."),
    substring: str = typer.Argument(
        ..., help="The substring to search for in file names."
    ),
    dry_run: bool = typer.Option(
        False, help="If set, no files will be deleted; only listed."
    ),
    soft_delete: bool = typer.Option(
        False,
        help="If set, files will be moved to a soft delete directory instead of permanently deleted.",
    ),
    soft_delete_dir: str = typer.Option(
        DEFAULT_SOFT_DELETE_DIR, help="The directory to move soft-deleted files to."
    ),
    max_age: int = typer.Option(None, help="Maximum age of files to include, in days."),
    min_age: int = typer.Option(None, help="Minimum age of files to include, in days."),
):
    """Delete or soft-delete files in an S3 bucket that contain a specific substring in their names."""
    s3 = boto3.client("s3")

    try:
        # List all objects in the bucket
        response = s3.list_objects_v2(Bucket=bucket_name)

        if "Contents" not in response:
            console.print("[bold yellow]No files found in the bucket.[/bold yellow]")
            return

        now = datetime.now(timezone.utc)

        def file_within_age_range(obj):
            file_age = (now - obj["LastModified"]).days
            if max_age is not None and file_age > max_age:
                return False
            if min_age is not None and file_age < min_age:
                return False
            return True

        files_to_process = [
            obj
            for obj in response["Contents"]
            if substring in obj["Key"] and file_within_age_range(obj)
        ]

        if not files_to_process:
            console.print(
                "[bold yellow]No files matching criteria found in the bucket.[/bold yellow]"
            )
            return

        if dry_run:
            console.print(
                "[bold blue]Dry run mode: The following files would be processed:[/bold blue]"
            )
            table = Table(title="Files to be Processed")
            table.add_column("File Name", style="cyan", no_wrap=True)
            table.add_column("Size", style="magenta", justify="right")
            table.add_column("Age", style="green", justify="right")
            table.add_column("Action", style="yellow", no_wrap=True)

            total_size = 0
            for obj in files_to_process:
                human_size = human_readable_size(obj["Size"])
                file_age = calculate_file_age(obj["LastModified"])
                action = (
                    f"[bold yellow]-> {soft_delete_dir}{obj['Key']}[/bold yellow]"
                    if soft_delete
                    else "[bold red]DELETE[/bold red]"
                )
                table.add_row(obj["Key"], human_size, file_age, action)
                total_size += obj["Size"]

            console.print(table)
            console.print(
                f"\n[bold green]Summary:[/bold green] {len(files_to_process)} files, Total Size: {human_readable_size(total_size)}, Operation: {'Soft Delete' if soft_delete else 'Delete'}"
            )

        else:
            for obj in files_to_process:
                file_key = obj["Key"]
                if soft_delete:
                    copy_source = {"Bucket": bucket_name, "Key": file_key}
                    new_key = f"{soft_delete_dir}{file_key}"
                    s3.copy_object(
                        CopySource=copy_source, Bucket=bucket_name, Key=new_key
                    )
                    s3.delete_object(Bucket=bucket_name, Key=file_key)
                    console.print(
                        f"[bold yellow]Moved to soft-delete:[/bold yellow] {file_key} -> {new_key}"
                    )
                else:
                    s3.delete_object(Bucket=bucket_name, Key=file_key)
                    console.print(f"[bold red]Deleted:[/bold red] {file_key}")

            console.print(
                f"[bold green]Operation completed. {len(files_to_process)} files processed. Operation: {'Soft Delete' if soft_delete else 'Delete'}[/bold green]"
            )

    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] {e}", style="red")


@app.command()
def clear_soft_delete(
    bucket_name: str = typer.Argument(..., help="The name of the S3 bucket."),
    soft_delete_dir: str = typer.Option(
        DEFAULT_SOFT_DELETE_DIR, help="The soft delete directory to clear."
    ),
    dry_run: bool = typer.Option(
        False, help="If set, no files will be deleted; only listed."
    ),
):
    """Remove all files in the soft delete directory."""
    s3 = boto3.client("s3")

    try:
        response = s3.list_objects_v2(Bucket=bucket_name, Prefix=soft_delete_dir)

        if "Contents" not in response:
            console.print(
                f"[bold yellow]No files found in the soft delete directory '{soft_delete_dir}'.[/bold yellow]"
            )
            return

        files_to_delete = [obj for obj in response["Contents"]]

        if dry_run:
            console.print(
                "[bold blue]Dry run mode: The following files would be deleted from the soft delete directory:[/bold blue]"
            )
            table = Table(title="Files to be Deleted")
            table.add_column("File Name", style="cyan", no_wrap=True)
            table.add_column("Size", style="magenta", justify="right")
            total_size = 0

            for obj in files_to_delete:
                human_size = human_readable_size(obj["Size"])
                table.add_row(obj["Key"], human_size)
                total_size += obj["Size"]

            console.print(table)
            console.print(
                f"\n[bold green]Summary:[/bold green] {len(files_to_delete)} files, Total Size: {human_readable_size(total_size)}"
            )
        else:
            for file_key in files_to_delete:
                file_key = file_key["Key"]
                s3.delete_object(Bucket=bucket_name, Key=file_key)
                console.print(
                    f"[bold red]Deleted from soft-delete:[/bold red] {file_key}"
                )

            console.print(
                f"[bold green]Soft delete directory '{soft_delete_dir}' cleared. {len(files_to_delete)} files removed.[/bold green]"
            )

    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] {e}", style="red")
        console.print("[bold red]Operation failed.[/bold red]")

        console.print(traceback.format_exc())


@app.command()
def undo_soft_delete(
    bucket_name: str = typer.Argument(..., help="The name of the S3 bucket."),
    soft_delete_dir: str = typer.Option(
        DEFAULT_SOFT_DELETE_DIR, help="The soft delete directory to restore from."
    ),
    dry_run: bool = typer.Option(
        False, help="If set, no files will be restored; only listed."
    ),
):
    """Restore all files from the soft delete directory to their original locations."""
    s3 = boto3.client("s3")

    try:
        response = s3.list_objects_v2(Bucket=bucket_name, Prefix=soft_delete_dir)

        if "Contents" not in response:
            console.print(
                f"[bold yellow]No files found in the soft delete directory '{soft_delete_dir}'.[/bold yellow]"
            )
            return

        files_to_restore = [obj for obj in response["Contents"]]

        if dry_run:
            console.print(
                "[bold blue]Dry run mode: The following files would be restored:[/bold blue]"
            )
            table = Table(title="Files to be Restored")
            table.add_column("File Name", style="cyan", no_wrap=True)
            table.add_column("Size", style="magenta", justify="right")
            table.add_column("Original Location", style="green", no_wrap=True)
            total_size = 0

            for obj in files_to_restore:
                human_size = human_readable_size(obj["Size"])
                original_key = obj["Key"][len(soft_delete_dir) :]
                table.add_row(obj["Key"], human_size, original_key)
                total_size += obj["Size"]

            console.print(table)
            console.print(
                f"\n[bold green]Summary:[/bold green] {len(files_to_restore)} files, Total Size: {human_readable_size(total_size)}"
            )
        else:
            for obj in files_to_restore:
                file_key = obj["Key"]
                original_key = file_key[
                    len(soft_delete_dir) :
                ]  # Remove the soft delete prefix
                copy_source = {"Bucket": bucket_name, "Key": file_key}
                s3.copy_object(
                    CopySource=copy_source, Bucket=bucket_name, Key=original_key
                )
                s3.delete_object(Bucket=bucket_name, Key=file_key)
                console.print(
                    f"[bold green]Restored:[/bold green] {file_key} -> {original_key}"
                )

            console.print(
                f"[bold green]Restoration completed. {len(files_to_restore)} files restored.[/bold green]"
            )

    except Exception as e:
        console.print(f"[bold red]Error:[/bold red] {e}", style="red")


if __name__ == "__main__":
    app()

full help text output #

Here is the help text for all of the commands in the script.

dropper on  main [!?] is 📦 v1.0.0 via  v22.13.0  v3.11.10 on  (us-east-1)  NO PYTHON VENV SET took 7s
⬢ [devtainer] ❯ ./scripts/s3_cleanup.py --help

 Usage: s3_cleanup.py [OPTIONS] COMMAND [ARGS]...

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                         │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.  │
│ --help                        Show this message and exit.                                                       │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ clean               Delete or soft-delete files in an S3 bucket that contain a specific substring in their      │
│                     names.                                                                                      │
│ clear-soft-delete   Remove all files in the soft delete directory.                                              │
│ undo-soft-delete    Restore all files from the soft delete directory to their original locations.               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

⬢ [devtainer] ❯ ./scripts/s3_cleanup.py clean --help

 Usage: s3_cleanup.py clean [OPTIONS] BUCKET_NAME SUBSTRING

 Delete or soft-delete files in an S3 bucket that contain a specific substring in their names.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    bucket_name      TEXT  The name of the S3 bucket. [default: None] [required]                               │
│ *    substring        TEXT  The substring to search for in file names. [default: None] [required]               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --dry-run            --no-dry-run                 If set, no files will be deleted; only listed.                │
│                                                   [default: no-dry-run]                                         │
│ --soft-delete        --no-soft-delete             If set, files will be moved to a soft delete directory        │
│                                                   instead of permanently deleted.                               │
│                                                   [default: no-soft-delete]                                     │
│ --soft-delete-dir                        TEXT     The directory to move soft-deleted files to.                  │
│                                                   [default: soft-delete/]                                       │
│ --max-age                                INTEGER  Maximum age of files to include, in days. [default: None]     │
│ --min-age                                INTEGER  Minimum age of files to include, in days. [default: None]     │
│ --help                                            Show this message and exit.                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

⬢ [devtainer] ❯ ./scripts/s3_cleanup.py clear-soft-delete --help

 Usage: s3_cleanup.py clear-soft-delete [OPTIONS] BUCKET_NAME

 Remove all files in the soft delete directory.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    bucket_name      TEXT  The name of the S3 bucket. [default: None] [required]                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --soft-delete-dir                    TEXT  The soft delete directory to clear. [default: soft-delete/]          │
│ --dry-run            --no-dry-run          If set, no files will be deleted; only listed. [default: no-dry-run] │
│ --help                                     Show this message and exit.                                          │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

⬢ [devtainer] ❯ ./scripts/s3_cleanup.py undo-soft-delete --help

 Usage: s3_cleanup.py undo-soft-delete [OPTIONS] BUCKET_NAME

 Restore all files from the soft delete directory to their original locations.

╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *    bucket_name      TEXT  The name of the S3 bucket. [default: None] [required]                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --soft-delete-dir                    TEXT  The soft delete directory to restore from. [default: soft-delete/]   │
│ --dry-run            --no-dry-run          If set, no files will be restored; only listed.                      │
│                                            [default: no-dry-run]                                                │
│ --help                                     Show this message and exit.                                          │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

we beat trailmakers

2025-01-10T19:53:14Z

Today we broke through the atmosphere in trailmakers ready for pickup from those back at mission conrol.

I haven’t posted a lot of gaming content to by blog, idk why, but I think I am going to start, especially for the ones that are good memories with my son.

Today we hit 100% of the required parts required to build the space ship and leave the atmosphere of the planet that the story is set in. Trailmakers is a game where you design and build your own vehicles. The campaign mode standed is a story where you are in a high tech freighter ship that crash lands on a distant planet. Your crew back home can come as far as the planet your on but cannot break through the atmosphere. Your job is to collect all the wreckage and gain all the parts required for a space ship build to take you out of the atmosphere to meet them.

There were a couple really heavy salvages to gather up, this one is high in the floating islands and is the cockpit fo the space ship.

I thought this was a cool shot of one of our builds with the suspension fully squatted out with a heavy load on the back.

Multiple Salvages #

It took us way to long to realize that we had enough power cores to add a bunch of tractor beams to do multiple salvages at once. Each tractor beam only has the ability to pick up one salvage at a time.

Some wild moments #

This clip brought both of us to curled over laughter, I wish I had recording on to get a full clip of it, this screenshot is the best I got. What happened was my son was being super silly and blew us up, but somehow the seat of the last build stayed in while we respawnd and our new vehicle picked it up. Not only that, but the broken seat was still usable and he was able to sit in it!

acheivements status #

Currently I am quite a ways from getting 100% of the achievements for the game. I think I will finish 100% of the stranded campaign because it is super fun, but probably wont 100% it.

I’m trying out video #

I turned on steam recording and played for a minute to test out hosting video here.

I accidentally entered deep space #

/start

2025-01-03T10:49:47Z

Welcome to waylonwalker.com, my small corner of the internet. I currently have 3606 posts published, here are some links to help you get started around here.

Feeds #

I have quite a few different feeds that you can browse or subscribe to in your rss reader, you can find them on my feeds page.

Slash posts #

[[ slashes ]]

Slash pages are some evergreen pages that I will do my best to keep up to date, they are typically not targeted to a specific moment in time, but designed to be ever living.

Waylon Walker - Husband, dad of two, and hobbyist builder of things on the internet.
Ai - Last updated Jan 2026.
analytics - I've been posting on this site since 2016, when layoffs were rolling through the company I worked for at the time. Starting a personal blog and a pile of…
/carry/ - I try to keep a pretty light every day carry, but it never works out, keyfobs and headphone cases end up causing more bulk than I'd like, but My EDC is no…
/colophon - Colophon a page that describes how the site > is made, with what tools, supporting what technologies
feeds - I have several feeds that you can consume my content with. Many posts will cross between them, so if you would like to follow a certain stream of content in…
markata - This post is a work in progress.
/nope - Inspired by @baty's nope slashpage, a list of ** things I don't like, don't care, avoid, overhyped, or won't do in no particular order, updated as I think…
/now - This page is meant to be simple view into what is happening in my life right now. Inspired by @NowNowNow. and Derek Sivers.
/start - Welcome to waylonwalker.com, my small corner of the internet. I currently have posts published, here are some links to help you get started around here.
Thoughts - These are generally my thoughts on a web page or some sort of url, except a rare few don't have a link. These are dual published off of my…
tinyapps - I'm working on replacing my usage of google inline search apps with real apps, these are ones that I create and host on my own homelab. The first three that…
/top4 - Definitive ranked lists of my top 3 favorites plus an honorable mention.
/verify - Inspired by @mollywhite's verify slashpage. This page serves as the system of record for my online identity. The best places to follow me are:
wants - Inspired by mara.town/wants
/yep - Inspired by @fyrio's yep slashpage, a list of ** things I enjoy, use, recommend, want to know more about, or seek out in no particular order, updated as I…

slow nfs performance

2025-01-02T20:23:10Z

I’m running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I’m already learning why this is not reccommended.

Slow #

I’ve been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don’t care if they take a bit longer to keep my master node freed up for more critical work.

I just started trying to build this site in a cronjob, It was taking 20 minutes to build, and something I noticed was that markata was taking minutes to run glob ( search for files ), normally this happens in a few ms and I never notice this step.

I just moved into the master node and the results were wild at ~30x faster

Permissions #

I have seen where you can get diffent permissions on the nfs share based on user id. Since I’m homelabbing here I only have one user per machine. As you step into enterprise level VMs with tighter controls and dozens of users for all the different services that might run on it.

I’ve ran into maybe one issue where I was root in one place and not another, other than that it’s been fine.

And it only got better #

As the cache was warm subsequent runs only got better.

I just checked again and we are now 80x faster

Conclusion #

I don’t have the answers yet, it might be my network, it might be nfs settings, it might be ext4 filesystem. I have some things to try, but what I do know is that it is not as easy as I thought it would be just to have the same file system mounted on both ends and share data between nodes.

I might even go with a complete alternative and use minio as a storage backend, and sync the files in on each run, this will add some latency to do the sync each time though.

/colophon

2025-01-02T10:21:23Z

Colophon a page that describes how the site is made, with what tools, supporting what technologies

Author #

All posts on this site are written by Waylon Walker, the typical content has changed and evolved over time. I go back and make a few corrections, but for the most part things stay pretty much as they were published originally.

see more in Waylon Walker

tech #

This site is a static site build with my own static site generator markata, Thoughts or as Simon Willison calls it a link blog posts are pulled in as a regular posts, all is hosted on cloudflare pages.

markata
Thoughts
cloudflare pages

see more about these components in about this site

Analytics #

I do not track users, I respect the privacy of my readers and do not track their information. I do track analytics on my own writing a post rate. Its more of an interesting history of the site.

meta #

Some evergreen pages that are more about me or this site from the Posts tagged: meta feed.

setting up ucore-zfs

2024-12-27T08:55:50Z

I just setup my oldest hardware on the newest hotest server distro ucore-zfs. This is a gateway FX6860 manufactured in 2010.

Immutable is the future #

My current boot log shows that I first started daily driving bazzite back in August 2024. I’ve been hapily using it since my arch install was plaugued with a crippling display driver error, or something that would lock the display for minutes every 30s or so, it became unusable. I switched because this is what I put my son on and it was working great for him.

waylon@razorcrest:~$ journalctl --list-boots
IDX BOOT ID                          FIRST ENTRY                 LAST ENTRY
-19 7e6e154d2609407da24fa12814eadbd7 Thu 2024-08-29 16:15:15 CDT Thu 2024-08-29 17:37:25 CDT

Four months later and I am really loving the immutable distro experience. My base system gets fresh reliable updates, and I barely install anything directly on it, a handful of things are snaps or flatpaks from the discover store, but my main workflow is now in distrobox. It has been rock solid reliable, and just works.

The Hardware #

This gem is running a an intel i7-2600 (4) @ 3.80 ghz with 16gb of ram. I’ve maxed out the ram that the motherboard will allow me. I may have even forgot about this limitation and ordered a 2x32gb setup for it and it did nothing. That’s now sitting in my new k3s master node.

Here’s a B&H photo post of the machine, she is big and heavy but still working.

I’m a big fan of keeping these old machines running and avoiding the e-waste pile, great for running a home lab. Admittedly this is one is probably on its last leg, dell optiplexes are pretty cheap and run circles around this one, so this one is become my experimental setup for trying new things like core-os.

Get Password Hash #

We will need to create a password hash for the root user to put into our ignition file.

podman run -ti --rm quay.io/coreos/mkpasswd --method=yescrypt
Password:
$y$j9T$0ZsoVynV7y0Z7/l6588Ba1$VZT0uCGP0CnYSX/EArCvYMuo3q.gnyOnk1RO6.HDNDB

Get SSH Pub Key #

Generate an ssh key using the ssh-keygen command.

$ ssh-keygen -t ed25519 -C "waylon@waylonwalker.com"
Generating public/private ed25519 key pair.
Enter file in which to save the key (/var/home/core/.ssh/id_ed25519):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /var/home/core/.ssh/id_ed25519
Your public key has been saved in /var/home/core/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:xVJAVreKVILOnxTDxK88RyMwhdDCBMnjMSGU7rsAqwQ waylon@waylonwalker.com
The key's randomart image is:
+--[ED25519 256]--+
|oo+*oo OBo+ .    |
| o* o *.== . .   |
|.. + + oo+o .    |
| ..   o.o++.     |
|E      +S=..     |
|oo      * .      |
|o..      o       |
|+.               |
|...              |
+----[SHA256]-----+

Now copy your public key into the ignition file from your local machine

$ cat ~/.ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPY73r4EU9wm/26/rTpx/uvAyInmbQ/k+l04eadSahD0 waylon@waylonwalker.com

ucore-autorebase.butane #

I got my ignition file from ucore/ucore-autorebase. Put my secret values into it and used it.

https://github.com/ublue-os/ucore/blob/main/examples/ucore-autorebase.butane

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPY73r4EU9wm/26/rTpx/uvAyInmbQ/k+l04eadSahD0 waylon@waylonwalker.com
      password_hash: $y$j9T$0ZsoVynV7y0Z7/l6588Ba1$VZT0uCGP0CnYSX/EArCvYMuo3q.gnyOnk1RO6.HDNDB
storage:
  directories:
    - path: /etc/ucore-autorebase
      mode: 0754
systemd:
  units:
    - name: ucore-unsigned-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to unsigned OCI and reboot
        ConditionPathExists=!/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/signed
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        StandardOutput=journal+console
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/unverified
        ExecStart=/usr/bin/systemctl disable ucore-unsigned-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target
    - name: ucore-signed-autorebase.service
      enabled: true
      contents: |
        [Unit]
        Description=uCore autorebase to signed OCI and reboot
        ConditionPathExists=/etc/ucore-autorebase/unverified
        ConditionPathExists=!/etc/ucore-autorebase/verified
        After=network-online.target
        Wants=network-online.target
        [Service]
        Type=oneshot
        StandardOutput=journal+console
        ExecStart=/usr/bin/rpm-ostree rebase --bypass-driver ostree-image-signed:docker://ghcr.io/ublue-os/ucore:stable
        ExecStart=/usr/bin/touch /etc/ucore-autorebase/signed
        ExecStart=/usr/bin/systemctl disable ucore-signed-autorebase.service
        ExecStart=/usr/bin/systemctl reboot
        [Install]
        WantedBy=multi-user.target butane

Creating an ignition file #

podman run --interactive --rm --security-opt label=disable \
        --volume ${PWD}:/pwd --workdir /pwd quay.io/coreos/butane:release \
        --pretty --strict ucore-autorebase.butane >transpiled_config.ign

Getting zfs #

Now this is where I realized I went wrong and wished I would have paid attention to the autorebase.butane file, it did not use the zfs flavor ucore. Luckily they make it wildly easy to rebase between these base images.

I needed to run this to rebase into the zfs flavor.

/usr/bin/rpm-ostree rebase --bypass-driver ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable-zfs

This was the output.

core@falcon-FX6860:~$ /usr/bin/rpm-ostree rebase --bypass-driver ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable-zfs
==== AUTHENTICATING FOR org.projectatomic.rpmostree1.rebase ====
Authentication is required to switch to a different base OS
Authenticating as: CoreOS Admin (core)
Password:
==== AUTHENTICATION COMPLETE ====
Pulling manifest: ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable-zfs
Importing: ostree-unverified-registry:ghcr.io/ublue-os/ucore:stable-zfs (digest: sha256:8ebae90f6844949044c026d7ba05c035956992b68e13bdcbd9158a37beda571e)
ostree chunk layers already present: 51
custom layers needed: 2 (492.9 MB)
Fetching layer sha256:67f3c0e0e0fe (269.7 MB)... done
Fetching layer sha256:25992805e895 (223.1 MB)... done
Staging deployment... done
Upgraded:
  cockpit-bridge 330-1.fc41 -> 331-1.fc41
  cockpit-networkmanager 330-1.fc41 -> 331-1.fc41
  cockpit-selinux 330-1.fc41 -> 331-1.fc41
  cockpit-storaged 330-1.fc41 -> 331-1.fc41
  cockpit-system 330-1.fc41 -> 331-1.fc41
Added:
  groff-base-1.23.0-7.fc41.x86_64
  kmod-zfs-2.2.7-1.fc41.x86_64
  libnvpair3-2.2.7-1.fc41.x86_64
  libuutil3-2.2.7-1.fc41.x86_64
  libzfs5-2.2.7-1.fc41.x86_64
  libzpool5-2.2.7-1.fc41.x86_64
  lm_sensors-libs-3.6.0-20.fc41.x86_64
  lzop-1.04-15.fc41.x86_64
  mbuffer-20241007-1.fc41.x86_64
  perl-AutoLoader-5.74-512.fc41.noarch
  perl-B-1.89-512.fc41.x86_64
  perl-Capture-Tiny-0.48-21.fc41.noarch
  perl-Carp-1.54-511.fc41.noarch
  perl-Class-Struct-0.68-512.fc41.noarch
  perl-Config-IniFiles-3.000003-14.fc41.noarch
  perl-Data-Dumper-2.189-512.fc41.x86_64
  perl-Digest-1.20-511.fc41.noarch
  perl-Digest-MD5-2.59-5.fc41.x86_64
  perl-DynaLoader-1.56-512.fc41.x86_64
  perl-Encode-4:3.21-511.fc41.x86_64
  perl-Errno-1.38-512.fc41.x86_64
  perl-Exporter-5.78-511.fc41.noarch
  perl-Fcntl-1.18-512.fc41.x86_64
  perl-File-Basename-2.86-512.fc41.noarch
  perl-File-Path-2.18-511.fc41.noarch
  perl-File-Temp-1:0.231.100-511.fc41.noarch
  perl-File-stat-1.14-512.fc41.noarch
  perl-FileHandle-2.05-512.fc41.noarch
  perl-Getopt-Long-1:2.58-2.fc41.noarch
  perl-Getopt-Std-1.14-512.fc41.noarch
  perl-HTTP-Tiny-0.090-1.fc41.noarch
  perl-IO-1.55-512.fc41.x86_64
  perl-IO-Socket-IP-0.43-1.fc41.noarch
  perl-IO-Socket-SSL-2.089-1.fc41.noarch
  perl-IO-stringy-2.113-15.fc41.noarch
  perl-IPC-Open3-1.22-512.fc41.noarch
  perl-MIME-Base32-1.303-21.fc41.noarch
  perl-MIME-Base64-3.16-511.fc41.x86_64
  perl-NDBM_File-1.17-512.fc41.x86_64
  perl-Net-SSLeay-1.94-7.fc41.x86_64
  perl-POSIX-2.20-512.fc41.x86_64
  perl-PathTools-3.91-511.fc41.x86_64
  perl-Pod-Escapes-1:1.07-511.fc41.noarch
  perl-Pod-Perldoc-3.28.01-512.fc41.noarch
  perl-Pod-Simple-1:3.45-511.fc41.noarch
  perl-Pod-Usage-4:2.03-511.fc41.noarch
  perl-Scalar-List-Utils-5:1.68-1.fc41.x86_64
  perl-SelectSaver-1.02-512.fc41.noarch
  perl-Socket-4:2.038-511.fc41.x86_64
  perl-Storable-1:3.32-511.fc41.x86_64
  perl-Symbol-1.09-512.fc41.noarch
  perl-Sys-Hostname-1.25-512.fc41.x86_64
  perl-Term-ANSIColor-5.01-512.fc41.noarch
  perl-Term-Cap-1.18-511.fc41.noarch
  perl-Text-ParseWords-3.31-511.fc41.noarch
  perl-Text-Tabs+Wrap-2024.001-511.fc41.noarch
  perl-Time-Local-2:1.350-511.fc41.noarch
  perl-URI-5.30-1.fc41.noarch
  perl-base-2.27-512.fc41.noarch
  perl-constant-1.33-512.fc41.noarch
  perl-if-0.61.000-512.fc41.noarch
  perl-interpreter-4:5.40.0-512.fc41.x86_64
  perl-libnet-3.15-512.fc41.noarch
  perl-libs-4:5.40.0-512.fc41.x86_64
  perl-locale-1.12-512.fc41.noarch
  perl-mro-1.29-512.fc41.x86_64
  perl-overload-1.37-512.fc41.noarch
  perl-overloading-0.02-512.fc41.noarch
  perl-parent-1:0.242-1.fc41.noarch
  perl-podlators-1:6.0.2-2.fc41.noarch
  perl-vars-1.05-512.fc41.noarch
  pv-1.8.14-2.fc41.x86_64
  python3-cffi-1.17.0-1.fc41.x86_64
  python3-ply-3.11-25.fc41.noarch
  python3-pycparser-2.20-18.fc41.noarch
  python3-pyzfs-2.2.7-1.fc41.noarch
  sanoid-2.2.0-1.fc41.ucore2.noarch
  sysstat-12.7.6-2.fc41.x86_64
  zfs-2.2.7-1.fc41.x86_64
  zfs-dracut-2.2.7-1.fc41.noarch
Changes queued for next boot. Run "systemctl reboot" to start a reboot

For some reason I double checked, and runnign it a second time gave me this.

core@falcon-FX6860:~$ sudo rpm-ostree rebase ostree-image-signed:docker://ghcr.io/ublue-os/ucore:stable-zfs
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/ucore:stable-zfs
Staging deployment... done
Changes queued for next boot. Run "systemctl reboot" to start a reboot

ZFS #

Once I rebooted I had all of my necessary zfs utilities that I needed.

connecting nfs #

The last thing I wanted to do was to mount an nfs share from my master node so that they can share a storage backend. I found that nfs was already in the host and ready to go.

sudo mkdir /mnt/vault/nfs
sudo mount -t nfs <nfs server ip>:/mnt/vault/nfs /mnt/vault/nfs

thoughts 0.0.4

2024-12-18T19:04:27Z

This is such an improvement to the backend of my website it warrants a blog post of celebration. For far too long I’ve been dealing with a tiny ass edit form on thoughts. I tend to not edit them, and try to get them right in one go. This is kinda the point of a thought, its a quick post meant to be the size of a tweet, but sometimes I’m leaving thoughts on a video or long post and want to make sure I have a good save point, but I just keep the thing in draft and hope I don’t loose if for far too long.

Results #

Let’s see this change in action!!

before #

This is the tiny ass form nested deeply in the flow of the feed. When I made it I naively just swapped out the post itself with the edit form, and swapped the post back in after edit.

thoughts is built with HTMX btw so all html is rendered in the backend and swapped by htmx client side.

after #

Now the edit is a full page modal with a nice blurry backdrop effect to the rest of the content. This feels pretty similar to making a new post on twitter.

How #

How did I do this with htmx? I had to break out of the mindset my brain was in with swapping in place and letting the edit form take over the entire screen.

First the empty #modal-container was added to the top of every page.

<div id="modal-container"></div>

Then each post that gets added to the page already had an edit button, but now the target is set to #modal-container, and the swap is set to innerHTML so that we keep the #modal-container in place.

<button class="h-8 w-8 p-1 text-center" hx-get="{{ config.root }}/edit-thought/{{ post.id }}"
        hx-target="#modal-container" hx-swap="innerHTML" title="Edit">
    {% include 'edit.svg' %}
</button>

Now the edit post that is returned from the server is turned into a full height and width modal with a nice backgrop blur over the content.


<div id="modal-container">
    <div class="fixed inset-0 z-50 overflow-y-auto" aria-labelledby="modal-title" role="dialog" aria-modal="true">
        <!-- Background backdrop -->
        <div class="fixed inset-0 bg-black/30 backdrop-blur-sm"></div>

        <!-- Modal panel -->
        <div class="flex min-h-screen items-center justify-center p-4">
            <div class="relative w-full max-w-4xl transform rounded-xl bg-zinc-900 p-6 shadow-2xl transition-all">
... similar to the original edit form in here
            </div>
        </div>
    </div>
</div>

One notable change to the original edit form is clearing the modal container on submit. It is done with an hx-on::after-request event and one line of js.

<form id="websiteForm"
      hx-patch="{{ config.root }}/post/html/"
      method="POST"
      name="newPost"
      hx-target="#post-{{ post.id }}"
      hx-swap="outerHTML"
      hx-on::after-request="document.getElementById('modal-container').innerHTML = ''">
    ```

Similarly on the Cancel button.

``` html
<button class="rounded-lg border border-black bg-zinc-950 px-6 py-3 text-lg hover:bg-zinc-900"
        type="button"
        onclick="document.getElementById('modal-container').innerHTML = ''">
    Cancel
</button>

Hotkeys #

Now I struggled to get this right with htmx, and it seemed like things were getting more complicated by trying to get the buttons to trigger using htmx triggered keyboard events, I ended up just using javascript. Its still in the same file, so locality of behavior is barely an issue on this one anyways, and the js was just working.

<script>
document.addEventListener('keydown', function(e) {
    if (e.key === 'Escape') {
        document.getElementById('modal-container').innerHTML = '';
    }
    if (e.key === 'Enter' && e.ctrlKey) {
        e.preventDefault();
        htmx.trigger('#websiteForm', 'submit');
    }
});
</script>

Other small change #

When the server is returns an empty list of posts the post container would change size and cause a layout shift. Now the empty posts element, while not great does not cause layout shift.

<ul id="posts" class="min-h-screen"><li>No posts</li></ul>

New-caps-for-porta-john

2024-08-05T19:06:00Z

Today I dropped some new caps in the porta-john. I need to do a bunch of posts on zmk and my journey building the porta-john, I’ve been rocking it for at least two weeks now pretty solid and I am absolutely loving it. More on that later.

KLP-Lame-Keycaps #

The model of the caps are KLP-Lame by braindefender , they are super rounded and smooth. They look really good on camera and feel good to the finders.

I printed these at home on my ender3 s1 pro with a black and purple silk filliment. I printed all 42 at once using a Printing 42 keys at once technique. They do have just a bit of a rough texture that you can feel on your figertips. I printed these on super quality .12mm layer height, while my last set was printed on draft quality .32mm layer height. They look crazy different in print quality with the lines of the new caps barely showing, the feel isn’t nearly as drastic.

The Result #

Check out the final shots as I set up the porta-john with these sweet caps.

And here they are all set up.

Full Disclosure #

I am building out a keyboard business, so if you are interested in getting some of these caps for yourself or an entire board custom made to your liking hit me up. If you’re not quite ready, I would be stoked if you keep in touch by sigining up for the email newsletter https://ticklemykeys.com.

blogging in 2024

2024-08-05T10:02:09Z

Over the past years couple of years blog traffic has been tanking, for some time I thought it might have just been me, but then i heard it from Dave Rupert on Shop Talk show, and it made sense. LLMs and AI is slowly killing the internet as we know it.

My metrics #

I dug into my emails from the google search console team and found two results nearly one year apart. Impressions are only 60% of its value a year ago while clicks are only 35% of where they were a year ago. I’m getting less impressions and even less clicks.

May 2023

June 2024

Suspicions #

What I think is happening is a large portion of users are getting their answers from ChatGPT, Copilot, and llms like this. A lot of my posts have been filling gaps in the internet, notes for myself when I struggle to integrate x in y because there was a gap of how to do so in the docs and stack overflow. Some small fraction of users have moved on to duck duck go and other non global warming search engines with AI in their results.

Clicks are down even further, than impressions. I can’t tell for sure but my gut feeling is that people are using the AI generated results. I know I use them quite often, but listening to Justin and Autumn at the end of the latest ship it podcast https://changelog.com/shipit/115, they both claim that they get no benefits from it and don’t know anyone who does. In fact Justin has taken the most ethical approach to move to duck duck go since it doesn’t kill the earth with each search by injecting AI results. Personally I do find it quite helpful, sometimes the answer is there, or an idea that triggers me to think of the idea.

Ok blogging is dead why keep going #

Just as always blogging has been for me. It’s a great place to write down your thoughts for later reference, and to help cement them into your brain. Beyond getting famous and clicks blogging is useful to you first and foremost, that should be the number one reason.

How do we keep the internet of old #

RSS feed readers is a great way to keep creators that you like and want to support in your feed regardless of what the higher powers of search and social media think of them. This year I built my own [[reader]], and I use it every single day to read articles out of my own custom curated feed.

I have also created Thoughts to help share and lift up creators that make good posts. I have a crome extension that allows me to create a blog post tied to a link right within chrome. I use this several times per day to save some pages for later with a small note.

Trying Obsidian

2024-07-31T00:00:00Z

I am giving obsidian a try, this is a test post to learn the flow. Something that has been really hard for me for a long time is images, I don’t include a lot of images just because it takes so much longer just to get the post out. I store them in a separate repo, I need to resize and compress them for the web so they aren’t so big.

Images are easy #

This is my current wallpaper.

I took he screenshot and just pasted it in.

I have more about my process in this post about Obsidian Image Converter.

It’s still just markdown #

I don’t know why it took me so long to understand this but obsidian is just markdown files. I pretty much just plugged in my existing blog and it picked up all the tags and was ready to run. I now get some nice visualizations to help me identify posts that were not ever tagged or maybe left unfinished.

It leans hard on wikilinks #

I just finished moving my backend over to md-it-python, which comes with some good wiliklink support. I even make a sick wikilink hover feature that previews wikilinks on hover.

Inside of obsidian when you start a wikilink it starts searching for posts to link. I get this feature in using the marksman lsp, and it’s nice to see that it works pretty similarly, but with much less setup and configuration.

creating new files #

The same keybinding that will take you to a new fill will create a Obsidian-new-file in the root directory of your project. This is very handy to create a bunch of new links and files quickly. This feature does not work the greatest for my blog currently as it does not create them from a template, and I use templates to start all of my posts. see Obsidian Using Templater Like Copier. I am now seeing why these plugins push so much automation into the templating system by directory.

New keybindings #

Until I really got myself in and working I didn’t realize all of the vim features that I would really need, Obsidian-go-to-definition was one of the very first ones. Obsidian has a feature to take you to the note under the cursor by pressing alt+enter, but that was not obvious at first.

Renaming posts #

Obsidian makes it so easy to rename a post, and it renames the file for you without leaving the editor, needing to close the file or anything. You can just go to the top of the file and change the name without needing to find it in a file tree or anything. Its a very nice and clean feature.

I miss my gqap #

There are still some vim features like Vim-gq that I use constantly to reflow long lines into readable paragraphs no matter where I open them that I cannot find a way to do in obsidian, and I miss it. ## Recent Posts.

Note

ok, so vim mode inside of obsidian really does work, and it does this, its

just less obvious because of obsidians softwraps in the editor. And I don’t think it works on indented lines like these, in this note.

missing seemless publish #

I run pre-commit on my posts to clean up the whitespace and line endings. Generally this does not do much, but aparantly obsidian is clashing with my setup, and not running pre-commit correctly so I need to leave my editor to publish a post.

With neovim I have a hotkey to commit verbosely with fugitive, gic, and to push gpp, and I use this all the time as I am writing.

animal well codes

2024-07-11T20:05:36Z

fish room #

fish bubbles ul d dl l dl ur ul

jellyfish d l d l d l u r

egg room #

bar dir

given bar codes top r mid dr bot dl

calculated bar codes num dir

1 r 2 dr 3 d 4 dl 5 l 6 ul 7 u 8 ur

bar code direction num bar 1 top 2 mid 3 top mid 4 bot 5 top bot 6 mid bot 7 top mid bot 8 empty

egg room decode 6 3 3 6 3 1 3 4 2 4 3 2 8 3 5 8 _ 3 5 7 5_ __ 5 2 5 6 3 1 3 3 1 3 6 4 4 6 4 2 4 6 8 6 4 8 7 3 7 6 5 7 6 __

egg room decode dir ul dd __ dd ul dd rr dd __ dl dr dl dd __ dr ur__ dd ll ur __ dd ll uu ll __ ____ ll dr ll ul dd rr dd __ dd rr dd ul dl __ dl ul dl dr dl ul ur ul dl ur uu __ dd uu ul __ ll uu ul ____

kind cluster

2024-07-02T08:01:20Z

kind{.hoverlink} is a very useful tool to quickly standup and teardown kubernetes clusters. I use it to run clusters locally. Generally they are short lived clusters for trying, testing, and learning about kubernetes.

Kind is Kubernetes in Docker, its very fast to get a new cluster up and running. Other than checking a box in docker desktop it is the easiest way currently to get a cluster up and running. I’ve used docker desktop for k8s before I really developed on k8s and it was buggy at the time and sometimes started and sometimes didn’t, when it didnt I had no idea how to fix it. I’d suggest kind as the best option to get a cluster up and running locally.

Not Production #

If you are looking for a production ready cluster this is not it. I really like k3s{.hoverlink}. At the time that I chose k3s it was the most lightweight option that easily supported multi-node clusters.

Starting a kind cluster #

The first step, and maybe only one that you need is to create a cluster and give it a name. This command will edit your $KUBECONFIG file, and set the kind cluster as your default cluster to interact with.

kind create cluster --name <CLUSTER_NAME>

Using podman as a backend #

I use podman as my docker engine, kind works with docker and podman, but docker by default, in order to switch to podman you need to set an environment variable.

export KIND_EXPERIMENTAL_PROVIDER=podman

This will tell kind to use podman as the backend provider instead of docker.

Loading images #

If your images are not publically available from a registry, you can load them in kind using the kind load docker-image command.

kind load docker-image $REPOSITORY:$TAG --name <CLUSTER_NAME>

Note

 the CLUSTER_NAME is the name that you gave kind when you started the kind cluster.

Argocd #

Argocd is a great way to setup gitops workflows in kubernetes. compared to just hand-rolling kubectl apply, argo holds the state and is able to not only apply new and change, but cleanup removed things.

setting up a kind cluster with argocd installed

You can stand up argocd in kind for learning argo or getting a nice visual. But often when I use kind its overkill. The cluster is not long lived, I don’t care if things are not cleaned up, and I want to quickly apply changes without a commit and push to a git repo.

What I'm learning in 2024

2024-06-09T10:06:30Z

2024 has been a learning fueled year, Diving deep into things I never would have previously thought I would. It’s been a bit of a mix of the 🔥hot twitter trends, and exactly what tech twitter tells you not to do. It just goes to show community is great, the tech community is filled with strong opinions, but you need to think about what really makes sense for you, your career and your customers (or lack there of).

tech #

k8s
tailwind
fastapi
htmx
jinja
opnsense

successful one day builds #

play-outside
reader
thoughts
thoughts chrome-extension

Kubernetes #

Damn did I sleep on k8s for way took long. This is like exactly what I’ve needed for a lot of things. It’s a perect example of what happens when you listen to the tech community tell you.

Looking for a Heroku replacement, What I found was shocking!

I started looking for something to make my homelab deployments easier at home. Previously I needed to ssh into my server and fuss around with docker compose. This just did not feel right for me to go into production and fiddle with it like this.

In my search I keep seeing kubernetes crop up as the best solution. I held off for a long time, until I finally decided to give it a go. I’m not sure if I got lucky or what but my first few applications that I migrated in went smooth as could be.

I chose to run k3s as this seemed like the lowest power consumption, easiest to manage, and still allowed me to run multiple nodes. At this point I was unsure if I would use multiple nodes, but I did not want to take away the option out of the gate.

I wrote a follow up article on my experience 6 monthts in.

kubernetes 6 months in

tailwind #

Again I wrote off tailwind for a long time. This time tech twitter really likes it and tells me I should use it, but I already know css and I just don’t see the point of needing to use a framework.

It wasn’t until thePrimeagen roughly said all css classes are shitty, everyone writes shitty css classes, and you might as well use the same shitty css classes everywhere that it really hit me, and I gave it a shot.

A Case For Tailwindcss

What I really found was that having a designer lay out all of the rhythm, spacing, and colors for you is really powerful.

I now have this site fully styled with tailwind, and even use md-it-attrs that allow me to pop extra classes right in markdown and style posts like [this]

fastapi #

I’ve been using fast api for a little while now but using it more and more in 2024. It has a lot of really great ideas with dependency management, and pydantic for moving data in and out of routes.

Here are some of the posts I’ve made about fastapi, theres defitely more to come here, I have some Ideas that I am honing down around meeting users where they are so that routes return the appropriate content type based on their explicitly requested content type or assumed by user agent.

how to host static content with fastapi

[[ fastapi-jinja-url-for-with-query-params ]]

htmx #

I have posts about htmx going back to Ease into htmx with htmx-get on 3/25/2022. I’ve been interested for awhile, but just didn’t really have the platform to use it till recently. My public content has been all static built content for a long time, but now I am building applications for myself that are server rendered such as Thoughts and One Day Build - Play Outside. I’ve had more use cases for using htmx.

The last pre-release of markata from 6/15/2022 now supports feed partials. This gives me a really easy way to pop things like recent posts on the bottom of every single page. I’m sure I can do this with jinja, but htmx makes it really easy to do and understand. If its up and running you will now see a Recent posts seciton below the article on this page powered by htmx.

jinja #

I’ve been using jinja for years now, but its mostly been some template variables here, some loops there. Now that I am putting fastapi and htmx to work I am really learning how to setup and design templates properly. Building up from a good re-usable base, including partials that so that pages can be rendered as full pages or partials.

just #

I have long been using my shell history to re-run complex shell commands that I use to build projets, compile tailwind, build docker images, run docker in local development. Most of the time they are not complex, but sometimes are something that would take me a few minutes to remember if I lost my shell history without it being documented

I’ve started storing all these things that I can in just. If its something that I get from my shell history I start thinking about how it would belong in a justfile. I’ve tried a few others over the years but they always seem tied to a language or build tool, just is language agnostic and runs in any terminal. Worst case, someone does not want to install it they can copy paste out of the justfile.

Biggest benefit is that it communicates to others what I do when I am working on a project. There are some good competitors like runbook that run readmes, but just has been working great for me and easy to use.

Refactoring one line links into wikilinks

2024-05-31T20:33:10Z

Previously I had setup a feature of my website to expand one line links into a card. This was not a standard, even to the point that some formatters wrap the links with , thus breaking my custom plugin. Moving to the wikilink standard will allow my markdown posts to work accross more site builders without custom integrations.

Expand One Line Links

What is a wikilink #

Wikilinks are standard to a lot of wikis written in markdown.

markdown-it-wikilinks

The wikilink syntax is a slug wrapped in double square brackets.

[[ slug ]]

Marksman lsp will even autocomplete these for you, its pretty sweet.

Note

 I recently implemented hover for wikilinks and and am pretty stoked about the

result. Check this one out sick wikilink hover.

Vim Quickfix #

You could use vimgrep to fill your quickfix list will all of the one line links but I am less familiar with vimgrep and kept missing posts for some reason, I think it was something in my file glob missing some directories.

I chose to use cexpr to fill my quickfix list using a command that outputs a vimgrep format filename:line:col:msg

:cexpr system('rg ^<https -t md --vimgrep .')

This filled my quickfix list with all of the one line links.

Vim cdo #

Now all I needed to do was to run a substitution command on every line in the quickfix list. This one features the one eyed fighting kirby that I learned from the primeagen [[ thoughts-200 ]].

:cdo s/\(^https:\/\/waylonwalker.com\/\)\(.*\)/[[ \2 ]]

This converts all of the full links into a slug wrapped in double square brackets.

More #

There was a bit more to the full refactor, for instance some had a til/ preix, some were for youtube, and some were not pointed to my site.

about this site

2024-05-31T14:07:02Z

I registered waylonwalker.com and started making content for it in 2017 after a big industry downturn in 2016 that left me scared for what would happen if I were laid off. The company I was working for at the time did it’s first major downsizing in history sending many really good engineers out to look for new opportunities in a world flooded with many in the same situation. This was very similar to what happened to the tech industry in 2024.

This was very similar to what happened to the tech industry in 2024.

See Waylon Walker for more about me.

What is this site? #

It’s a blog
Digital Garden
Learn in public
TIL
portfolio
my personal corner of the internet

It’s [NOT]{.text-red-500 .text-6xl .font-bold} Perfect #

There will be mistakes
I will learn
Grammar will be fine, but never perfect
Code will run on my machine, but not guaranteed to be perfect
There will be days when the whole site is broken

It started as a blog #

This site started out as a [big B]

It’s a digital garden #

Somewhere along the line I found a really great community of people who create content in ways that they does not require that level of depth and polish on each and every post they make. Ideas flow out and content gets made over time.

Maggie Appleton{.hoverlink} and the idea of digital gardening{.hoverlink} has been very inspiriational. She makes really great content with amazing drawings in them. She publishes early and often and labels content with the state that it is in, early and often. Some posts are budding, and some are evergreen. She also distinguishes between notes and essays.

Learn in public #

Shawn Wang{.hoverlink} (swyx) Has been very inpsiring with ideas around Learn In Public{.hoverlink}.

Whatever your thing is, make the thing you wish you had found when you were learning. Don’t judge your results by “claps” or retweets or stars or upvotes

just talk to yourself from 3 months ago. I keep an almost-daily dev blog written for no one else but me.

Try your best to be right but don’t wait til your the expert to share what you are learning. If something works for you, you are likely mostly right even if you don’t have all the right words for it yet. Don’t be afraid to be wrong.

TIL #

On to one of the most inspirational pieces that I have taken ideas from @jbrancha #til repository{.hoverlink}. He has been posting daily Tils for Years, and they do pretty damn well on google despite what SEO experts say his results end up in my google search quite often.

I’ve made many posts on this site in the TIL format. I don’t strictly stick to a schedule or a character length quite like Josh Does, but generally they are short, sweet, single ideas.

Thoughts #

I created a sub site Thoughts to serve as a sort of list of bookmarks, its a place for me to comment, quote, and link to other pages on the internet.

Portfolio #

This is my place to create and share content that becomes part of my portfolio. I network with others, share content across social accounts and make it public. This is an example of my work in writing, documenting, mentoring, and web development.

Static Site #

This site has gone from html, to gatsby, and finally I built my own static site generator markata. It’s been through a few different hosts, but at the end of the day the content is all markdown{.hoverlink} with frontmatter.

For instance the beginning of this post looks like this.

---
date: 2024-05-31 14:07:02
templateKey: blog-post
title: about this site
tags:
  - meta
published: True

---

Own your Content #

Throughout this process I have become a proponent of owning your own content on the internet. If you own the domain, the content, and host it yourself you control when it gets published, deleted, and updated. No one can take this away from you execept for you. It’s a very powerful tool that I highly recommend everyone to use. You can get your own domain for $12/year or less, and that is all you need to pay.

sick wikilink hover

2024-05-30T21:48:49Z

Today I set up some sick wikilink hover effects using tailwind see A Case For Tailwindcss. When you hover over them they show an image preview of the link that you are going to. I cant find where I have seen this but it comes from some docs sites.

I’ll finish this article later, just excited to see it up.

markata

2024-05-30T13:45:54Z

This post is a work in progress.

Markata is the static site generator that I created to build my website about this site. I built it for me and I enjoy using it. I know everying it can do and I can extend it to do more easily. I have set it up for some friends to also use it and am proud that it helps them publish their content.

It’s a meme to create your own static site generator to make your website. Yes its funny, I don’t recommend it if your not ready for the level of work that comes with it, but at the end of the day it’s very rewarding and a great way to learn.

Static Sites were all the rage #

JAMStack was 🔥

Gatsby and Next.js hit the scene as the next generation of static site builders and were getting big around the time I started building my site in 2017. They were based on react. I dove into react and learned it enough to build my website, but I really lacked the depth of knowledge in the js ecosystem to really work on it effectively. For instance when it got slow, it was hard for me to profile and find out why. What I really wanted was my site written in python, which I knew the ecosystem for very well, but I did not find the existing site generators easy to extend to do the things I needed. Naively I thought it would be easier to just build my own than learn how to make one do what I wanted it to. Not invented here syndrome hitting hard.

In their Defense

I really lacked the depth of knowledge in the js ecosystem to really work on

it (gatsby) effectively.

Plugins all the way down #

I started building this as I dove deeper into the Kedro framework for Data Engineering pipelines. They use a plugin framework for allowing users to extend it called pluggy. I had a great experience extending kedro using pluggy and wanted to build something with based on pluggy when I had started markata.

A Minimal Pluggy Example

my linked in work history

2024-05-30T11:14:16Z

My linked in work history is empty. I made up a position about content developer that tracks how long I’ve been blogging. I think i did this because LinkedIn requires it. Either way this is public knowledge and fine sharing.

If you have taken any security class for your job seriously you have already been told not to share your work with most companies to the public, this is private information that only opens you up for social engineering attacks against that company.

I care about privacy and security #

I care about the security of these companies I work for and their reputation, so I refuse to publically share it.

Need to know #

If somehow you need to know where I work it’s my choice to tell you. I don’t need to advertise to every social engineering hacker where I work on the platform that they go to get that information from.

Your LinkedIn is Garbage

2024-05-30T11:02:39Z

Your linkedin link sits at the top of your resume, its one of the first things I see when I open your resume, but yet it gives me no more information that the damn resume you sent me.

Save that space on your resume for something useful.

So you want that /in/me on your resume #

Fine if you want it on your resume make it actually useful for someone reading your resume.

Actually post something #

If I am reading resume’s and I actually take the time to look at your linkedin I want to see you post something. Take a side, make an opinion and post it. Learn something new, make a post about it. If you have a blog and you make a good post share it there.

Your work history belongs on your resume #

Any security 101 tells you that you should not share your work history on linked in. You should not share photos of you at your workplace that include sensitive information such as your badge.

Your work history on LinkedIn is for hackers to steal and use for social engineering.

If you took security seriously you would not include you work history at all, and any hiring manager who gives a shit would see that and should understand. Reality is that they probably don’t and it probably raised more questions than answers for them.

my linked in work history

Congrats on your work anniversary #

All these comments and messages are auto generated, both sides know it and its garbage. If you actually care about someones work history or achievements take 30s to craft a message of your own to them, that actually means something and is not garbage.

TLDR #

If you are not going to take the time and effort to make something worth sharing, take your garbage profile off your resume.o

One Day Build - Play Outside

2024-04-10T12:32:20Z

Inspired by Adam Savage and his One Day builds on youtube. I often build things, and want to make them generally useful for others and over configure out of the gate. This project is purely for me inspired by a need I have.

play-outside

!How-To #

This post will not directly show how to make a weather app, but document the process that I went through to make mine. It will show the tools that I used to make it, and the final result.

The Situation #

It often goes in our house ask dad while he is busy and he will probably just say yes without thinking much. This happens a lot when kids ask to go outside. I think sure, go for it, you will figure it out. Then my wife walks in and asks where they are, followed by, did you even check the weather, its -11 degrees outside right now.

I need a tool for this decision making process

Lungs #

You we have a family of not the most heathly lungs, we have my wife with lung cancer, one lung missing, and kids with asthma. We need to account for temperature, humidity, wind chill, and air quality before heading outside and seeing the repercussions of it later.

Final result #

So this is what I built, its a web app that checks the weather and air quality in your area and determines if its safe to go outside. It will even recommend limiting your time, or wearing a coat.

The Stack #

This is a one day build, I have both kids at home from school, so this is realistically only like 2-3 hours at most, so this has to be chosen based on familiarity.

Docker
k8s
Python
FastAPI
tailwind
httpx
OpenWeatherMap API
ipwho.is

This is the same stack (minue the apis) that I am using to build my startup fokais.com with. I am quite familiar with it and should be able to quickly make progress with it.

❯ tree
Permissions Size User   Date Modified Git Name
drwxr-xr-x     - waylon 16 Jan 12:21   -N  .
.rw-r--r--  3.8k waylon 16 Jan 12:24   -N ├──  deployment.yaml
.rw-r--r--   278 waylon 16 Jan 12:21   -N ├──  docker-compose.yml
.rw-r--r--   552 waylon 16 Jan 12:20   -N ├──  Dockerfile
.rw-r--r--   15k waylon 15 Jan 14:37   -N ├──  favicon.ico
.rw-r--r--  1.9k waylon 15 Jan 15:58   -N ├──  justfile
.rw-r--r--  1.1k waylon 15 Jan 11:39   -N ├──  LICENSE.txt
.rw-r--r--   51k waylon 15 Jan 14:38   -N ├──  package-lock.json
.rw-r--r--    69 waylon 15 Jan 14:38   -N ├──  package.json
drwxr-xr-x     - waylon 16 Jan 12:20   -N ├──  play_outside
.rw-r--r--   138 waylon 16 Jan 12:21   -N │  ├──  __about__.py
.rw-r--r--   115 waylon 15 Jan 11:39   -N │  ├──  __init__.py
.rw-r--r--  7.5k waylon 16 Jan 08:14   -N │  ├──  api.py
drwxr-xr-x     - waylon 15 Jan 21:30   -N │  ├──  cli
.rw-r--r--  3.5k waylon 15 Jan 21:30   -N │  │  └──  api.py
.rw-r--r--  2.8k waylon 16 Jan 12:20   -N │  ├──  config.py
.rw-r--r--  3.0k waylon 15 Jan 14:35   -N │  ├──  decorators.py
.rw-r--r--    51 waylon 15 Jan 14:50   -N │  └──  queries.py
.rw-r--r--  2.2k waylon 15 Jan 15:16   -N ├──  pyproject.toml
.rw-r--r--   506 waylon 15 Jan 11:39   -N ├──  README.md
drwxr-xr-x     - waylon 15 Jan 14:39   -N ├──  static
.rw-r--r--   21k waylon 16 Jan 08:10   -N │  ├──  app.css
.rw-r--r--   15k waylon 15 Jan 14:37   -N │  ├──  favicon.ico
.rw-r--r--   47k waylon 15 Jan 14:32   -N │  └──  htmx.org@1.9.8
drwxr-xr-x     - waylon 15 Jan 21:04   -N ├──  tailwind
.rw-r--r--  6.2k waylon 15 Jan 21:04   -N │  └──  app.css
.rw-r--r--   360 waylon 15 Jan 21:04   -N ├──  tailwind.config.js
drwxr-xr-x     - waylon 16 Jan 12:16   -N ├──  templates
.rw-r--r--  2.5k waylon 16 Jan 12:16   -N │  ├──  base.html
.rw-r--r--  2.2k waylon 16 Jan 12:11   -N │  ├──  card.html
.rw-r--r--   151 waylon 15 Jan 21:07   -N │  ├──  includestyles.html
.rw-r--r--   418 waylon 16 Jan 12:12   -N │  └──  index.html
drwxr-xr-x     - waylon 15 Jan 11:39   -N └──  tests
.rw-r--r--   115 waylon 15 Jan 11:39   -N    └──  __init__.py

htmx-btw">HTMX BTW #

I have been pairing up htmx with this stack quite a lot lately, and its fantastic, but honestly this idea just does not have a lot of endpoints, and I don’t think it needs it for a one day build, just toss everything into one page and call it good.

Getting Weather Data #

The first thing we need is the feels_like or Apartment Temperature. A quick google search lead me to https://openweathermap.org/ they have a very nice calculation for the feels like temerature already built in.

Now using openweathermap, we can get the feels like temperature, by latitude and longitude.

async def get_weather(lat_long):
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://api.openweathermap.org/data/2.5/weather?units=imperial&lat={lat_long['latitude']}&lon={lat_long['longitude']}&appid={config.open_weather_api_key}"
        )
        return response.json()

Where are you?? #

Since no one is going to know their current latitude and longitude we need a way to look this up for it to actually be useful. For this I leaned on https://ipwhois.io/

async def get_lat_long(ip_address):
    if ip_address is None:
        ip_address = "140.177.140.75"
    async with httpx.AsyncClient() as client:
        response = await client.get(f"https://ipwho.is/{ip_address}")
        return response.json()

Decision Tree #

For this I punted over to chatGPT to see what it came up with.

I am working on a web app to tell kids if its ok to play outside.  It is
targeted towards kids living in the  midwest united states.   I have the feels
like temperature already.  Set some values for the temperature

too cold,
no longer than 15 minutes,
coats required,
no longer than 15 minutes,
too hot

What it came up with wasn’t a terrible starting point, but its values essentially said kids cant play in the snow.

def determine_play_condition(feels_like_temperature):
    if feels_like_temperature < 20:
        return "It's too cold for extended play. Stay indoors and keep warm!"
    elif 20 <= feels_like_temperature <= 32:
        return "Perfect weather for snow play! Enjoy the winter wonderland!"
    elif 32 < feels_like_temperature <= 40:
        return "Coats and winter gear required for outdoor play. Stay cozy!"
    elif 40 < feels_like_temperature <= 50:
        return "You can play in the snow, but limit your time. It's getting warmer!"
    else:
        return "It's too warm for snow play. Find other fun activities indoors!"

# Example usage:
feels_like_temp = 25  # Replace with the actual feels like temperature
message = determine_play_condition(feels_like_temp)
print(message)

Some Massaging Later #

My wife sent me over this image from Tinkergarten, which is a chart made by the Iowa Department of Public Health. I used it as a guide to set some values, and added some conditions based on visibility and air quality index (aqi), which we have become all too familiar with over the past year with all the forest fires out Western US causing our lungs issues here in the Midwest.

(aqi) which we have become all too familiar with over the past year with all the forest fires out Western US

def determine_play_condition(weather, aqi=0):
    play_condition = PlayCondition()

    feels_like_temperature = weather["main"]["feels_like"]
    visibility = weather["visibility"]

    play_condition.message += hours_till_sunset(weather)

    if "after" in play_condition.message:
        play_condition.color = "bg-red-500"

    if visibility < 1000:
        play_condition.message += "It's too foggy. Find better activities inside!"
        play_condition.color = "bg-red-500"

    if aqi > 150:
        play_condition.message += "It's too polluted. Find better activities inside!"
        play_condition.color = "bg-red-500"
    elif aqi > 100:
        play_condition.message += "limit your time outside due to the poor air quality"
        play_condition.color = "bg-yellow-500"
    elif aqi > 50:
        play_condition.message += "Check the air quality outside at your discression."
        play_condition.color = "bg-yellow-500"
    else:
        play_condition.message += ""

    if feels_like_temperature < 10:
        play_condition.message += "It's too cold. Stay indoors and keep warm!"
        play_condition.color = "bg-red-500"
    elif feels_like_temperature < 30:
        play_condition.message += "You can play outside, but limit your time!"
        play_condition.color = "bg-yellow-500"
    elif feels_like_temperature < 40:
        play_condition.message += (
            "Coats and winter gear required for outdoor play. Stay cozy!"
        )
    elif feels_like_temperature < 50:
        play_condition.message += "Grab a warm jacket and enjoy your time outside!"
    elif feels_like_temperature < 60:
        play_condition.message += "Grab some long sleeves and enjoy your time outside!"
    elif feels_like_temperature > 90:
        play_condition.message += (
            "You can play outside, but limit your time in this heat!"
        )
        play_condition.color = "bg-yellow-500"
    elif feels_like_temperature > 109:
        play_condition.message += (
            "It's too hot for outdoor play. Find cooler activities indoors!"
        )
        play_condition.color = "bg-red-500"
    else:
        play_condition.message += "Enjoy your time outside!"
    return play_condition

Pulling the data together #

Since I will be needing all of the data together upon every request I put together one get_data function to return a dict of all of the data.

forecast

 I pulled the forecast endpoint from openweathermap as well, it looks like a

stripped down version of the regular weather endpoint, but every few hours over the course of the next 5 days.

async def get_data(request: Request):
    user_ip = request.headers.get("CF-Connecting-IP")
    lat_long = await get_lat_long(user_ip)
    weather = await get_weather(lat_long)
    forecast = await get_forecast(lat_long)
    air_quality = await get_air_quality(lat_long)
    weather["play_condition"] = determine_play_condition(
        weather,
        air_quality["list"][0]["main"]["aqi"],
    )

    forecast = [
        {"play_condition": determine_play_condition(x), **x}
        for x in forecast
        if datetime.fromtimestamp(x["dt"]).hour >= 6
        and datetime.fromtimestamp(x["dt"]).hour <= 21
    ]

    return {
        "request.client": request.client,
        "request.client.host": request.client.host,
        "user_ip": user_ip,
        "lat_long": lat_long,
        "weather": weather,
        "forecast": forecast,
        "air_quality": air_quality,
        "sunset": weather["sys"]["sunset"],
    }

FastAPI #

Fastapi here is a great framework, it uses pydantic to validate the data returned from the api, has a great dependency management system.

I am going to use none of that, all I need is one TemplateResponse using jinja. For good measure, I’ll toss in a /metadata route that returns the data.

from fastapi import FastAPI
app = FastAPI()


@app.get("/")
@no_cache
async def get_home(request: Request):
    data = await get_data(request)
    return config.templates.TemplateResponse("index.html", {"request": request, **data})

@app.get("/metadata")
async def root(
    request: Request,
):
    return await get_data(request)

No Cache Header #

I had some issues with cloudflare caching me and not letting me hit the api everytime. I’ve ran into this several times in the past, so I went to the cloudflare dashboard, manually busted the cache for the home route and popped a no_cache decorator on the get_home route.

def no_cache(func):
    not_cached_routes.append(f"{func.__module__}.{func.__name__}")

    @wraps(func)
    async def wrapper(*args, request: Request, **kwargs):
        # my_header will be now available in decorator
        if "request" in signature(func).parameters:
            kwargs["request"] = request

        if inspect.iscoroutinefunction(func):
            response = await func(*args, **kwargs)
        else:
            response = func(*args, **kwargs)

        response.headers["Cache-Control"] = "no-cache, no-store, must-revalidate"
        response.headers["Pragma"] = "no-cache"
        response.headers["Expires"] = "0"
        return response

    return wrapper

@app.get("/")
@no_cache
async def get_home(request: Request):
    data = await get_data(request)
    return config.templates.TemplateResponse("index.html", {"request": request, **data})

This solved my caching issues.

Templates #

I used jinja for templating its built right into FastAPI.

from fastapi.templating import Jinja2Templates

templates = Jinja2Templates(directory="templates")

This will get you by for quite awhile, and I probably could deal with templates working just like this for a one day build, but I have some nice feautures that I like from other projects, and at least one specific to just this project. Once they are in my config object, I use them like so.

Note

 The `request` parameter is a requirement for all templates.

@app.get("/")
@no_cache
async def get_home(request: Request):
    data = await get_data(request)
    return config.templates.TemplateResponse("index.html", {"request": request, **data})

Globals #

I need a nice way to convert the openweathermap timestamps to human readable values.

from datetime import datetime

templates.env.globals["datetime"] = datetime

Then in jinja I can format the weather.dt variable like so.

{{ datetime.fromtimestamp(weather.dt).strftime(format = '%A') }}

Putting it together #

I put a get_templates function in my config, and the config object is passed the results as config. Again I copy pasted a function https_url_for from my other project so that I can use url_for in my templates and it work on both localhost and production.

import os
from datetime import datetime
from datetime import timezone

from fastapi.templating import Jinja2Templates
from rich.console import Console

console = Console()


@pass_context
def https_url_for(context: dict, name: str, **path_params: Any) -> str:
    request = context["request"]
    http_url = request.url_for(name, **path_params)
    return str(http_url).replace("http", "https", 1)

def get_templates(config: BaseSettings) -> Jinja2Templates:
    templates = Jinja2Templates(directory="templates")
    templates.env.filters["quote_plus"] = lambda u: quote_plus(str(u))
    templates.env.filters["timestamp"] = lambda u: datetime.fromtimestamp(
        u, tz=timezone.utc
    ).strftime("%B %d, %Y")
    templates.env.globals["https_url_for"] = https_url_for
    templates.env.globals["config"] = config
    templates.env.globals["datetime"] = datetime
    templates.env.globals["len"] = len
    templates.env.globals["int"] = int
    console.print(f'Using environment: {os.environ.get("ENV")}')

    if os.environ.get("ENV") in ["dev", "qa", "prod"]:
        templates.env.globals["url_for"] = https_url_for
        console.print("Using HTTPS")
    else:
        console.print("Using HTTP")

    return templates

Styles #

I’ve been all in on tailwind lately. I was a long holdout. I used Sass back in 2016/2017 when I was first getting into webdev, then went many years just straight vanilla. Tailwind just has some really well thought out color and rythm that makes it easy. I have also really been appreciating the locality of behavior part of it. I can make components in jinja with everything they need.

I set up my tailwind.config.js like so. It includes the typography plugin and some extra box shadows that are centered.

/** @type {import('tailwindcss').Config} */
module.exports = {
  content: ["templates/**/*.html"],
  plugins: [require("@tailwindcss/typography")],
  theme: {
    extend: {
      boxShadow: {
        xlc: "0 0 60px 15px rgba(0, 0, 0, 0.3)",
        lgc: "0 0 20px 0px rgba(0, 0, 0, 0.3)",
      },

    },
  },
};

For my tailwind/app.css I set the background color of the page dark, text white, and make the scrollbar not default.

@tailwind base;
@tailwind components;
@tailwind utilities;

html {
  scroll-behavior: smooth;
  @apply bg-zinc-800 text-white autofill:bg-yellow-500;
}

::-webkit-scrollbar {
  @apply h-4 w-4;
}

::-webkit-scrollbar-track {
  @apply rounded-full bg-zinc-900;
}

body::-webkit-scrollbar-track {
  @apply rounded-full bg-pink-600;
}

::-webkit-scrollbar-thumb {
  @apply rounded-full bg-zinc-600 hover:bg-zinc-500;
}

body::-webkit-scrollbar-thumb {
  @apply rounded-full bg-cyan-500 hover:bg-cyan-400;
}

Compiling tailwind #

I run tailwind using npx after first installing the typography plugin. This will use my tailwind.config.js file, and the tailwind/app.css file as input, and output to static/app.css.

Note

 --watch will watch for changes, and automatically recompile as you make changes to any templates.

# install the typography plugin
npm i @tailwindcss/typography
# compile the css
npx tailwindcss --input tailwind/app.css --output static/app.css --watch

Deployment #

Docker #

CI #

K8s #

TEXT BASED #

play-outside

feeds

2024-04-02T20:01:44Z

I have several feeds that you can consume my content with. Many posts will cross between them, so if you would like to follow a certain stream of content in your rss reader thats what this is for. This one might feel like it has some extra noise with multiple posts per day, including starts, and instagram style posts, for a bit less look at the [[ main ]] feed.

TLDR

If you are looking for everything look here <https://waylonwalker.com/archive/rss.xml>

All #

This is a feed of everything that I have published on my site to date including Posts tagged: blog, [[ til ]], Thoughts, [[ stars ]], Posts tagged: shots, [[ pings ]].

Main #

Like the All feed, but drops the shots (instagram style posts) and stars (github stars).

til - daily(ish) #

think gist with a tweet

This is a feed of all of my TIL’s (Today I Learned). These are generally short single topic posts that I write up quickly. They are generally things that are fairly new to me, often things that I struggled to find support for on the internet, and felt that It needed yet one more article with my voice.

shots - daily(ish) #

think Instagram

This feed is a collection of images, sometimes I post a few a day, sometimes a few weeks will go by, it really just depends on what I have going on.

pings - daily(ish) #

think tweet

These are ephemeral thoughts that pop in my head and I post as they occur. Maybe inspired by something I see or hear, but typically no more context is provided. I hate how social media has become very much trigger engagement bait… This is adjacent to that, but I try to be less hard on the bait.

Thoughts - daily+ #

These are generally my thoughts on a web page or some sort of url, except a rare few don’t have a link. These are dual published off of my thoughts.waylonwalker.com site. It’s a fully dynamically rendered site 2000’s style. Posts are stored in a database and instantly available. Almost all of the posts were written in a small <textarea> field within a chrome extension that I built for it.

These posts are intended to in two ways. One, link building for the author. I hope that I give the people helping me out along the way just a little bit of a boost. Two, they serve as a permanant commented bookmark for me to search, and come back to later when I have forgotten where I have seen something.

stars #

I pull in all of my github stars into one feed pesos style.

Recent #

This is a shorter version of All, as it is filtered down to only the posts created within the last 30 days.

Blog - weekly/monthly #

These are my full big B Blog posts. They are what I started writing, typically take a bit more time to put together. I write fewer of them these days than I used to. I’d like to think that they are the more polished posts of the bunch.

Today #

These are only the posts that went live the day that the site last built. If my scheduled builds keep working as intended they are today’s posts.

Python #

These are all the posts related to the python programming language.

Linux #

These are all the posts related to linux.

vim #

These are all the posts related to the greatest way to edit text, vim.

Careful, you might not make it out

kedro #

These are all the posts related to the python data engineering framework kedro.

gratitude #

I used to do daily gratitude posts, I might get back into it.

Thoughts

2024-04-01T16:14:47Z

All thoughts posts cross posted to my site are prefixed with a thought balloon 💭.

The tech #

Since this blog is mostly a tech blog about software development, and my journey as I learn, lets talk tech.

python
fastapi
htmx
sqlite
docker
https://fly.io

fastapi #

The core of the site is a python web server running fastapi. Most of the endpoints return html via jinja templates to the browser and json to anything else. So you go to the list of posts at [https://thoughts.waylonwalker.com/posts/waylonwalker/?page_size=9999999999 in](https://thoughts.waylonwalker.com/posts/waylonwalker/?page_size=9999999999 in){.hoverlink} a web browser it will be a rendered feed, but from curl you will get json.

htmx #

The main page is an infite scrolling feed of the posts. All loaded in with htmx. Which just works so beautifully for this. I really like python its my jam, I can make endpoints that return things out of a database very effectively with it, but it takes me some time to do the js side, htmx just makes these common patterns available right in html.

sqlite #

The data storage is a sqlite database using the sqlmodel orm. This was my very first time using sqlmodel, which is an ORM built on top of pydantic. It works really well with fastapi. The only thing that it lacks is the same history, community, and stack overflow posts that sqlalchemy has.

docker #

Docker is my go to way for containerizing web applications. I know it really well and makes it easy to build and deploy on a number of platforms.

https://fly.io #

I have this running on https://fly.io. Its my first real production application running on https://fly.io, and its been great for me. It runs my container for me, gives me tools to scale, and ssh right into the running container. It’s still free for the time being, but goes up quick for my cheap taste if I were to scale it up much.

Id highly recomend it if you have a server that you want to get running on the public internet, and you don’t want to manage any of your own infrastructure.

kubernetes 6 months in

2024-03-26T13:20:24Z

I stumbled into kubernetes December 2023 when I was looking for a better way to self host applications. I was looking for something that didn’t require logging into a server and building and deploying like a cave man. I wanted a smoother experience than docker compose was giving me.

https://waylonwalker.com/looking-for-a-heroku-replacement/

This post turned into a list of tools that I have adopted into my k8s workflow, and plan to keep. enjoy.

Kompose #

Kompose is a great tool for gettting going and converting your docker-compose to kubernetes manifests or helm templates. It was a great tool for me to get started with, but I was afraid that it was hindering me learning more and just blindly using its output so I have tried to use it less and less. I’m now not solely leaning on it, but using it to get out quick POCs with low friction.

Kompose really helped me go 0 to 60 and get right into kubernetes with my existing docker compose files and very little change. I found it a great way to get my feet wet and learn.

Argocd #

https://argoproj.github.io/cd

Argocd is the first tool I am going to reach for in any kubernetes deployment that I work on. The number one reason is that it keeps track of all of everything internally, and if you remove something it cleans it up. Very quickly if you are not diligent you will end up with stuff laying around not sure if you need it, or where it came from.

For instance if I create a new deployment and don’t include the namespace, If I just change the yaml with the new namespace without taking it down first, that original one will be left behind and I will be stuck manually cleaning it up.

Argo Rollouts #

https://argoproj.github.io/rollouts

I’m looking into rollouts, and have played with it, but argocd really does most of what I need with low friction. I thought I needed rollouts to prevent downtime, but argo tends to do a good job of cutting over to a new pod only after its up and ready. This solves most of my problems, and I can pretty quickly roll back with low friction. If I needed to manage a high availabilty application Id look closer.

KubeSeal #

https://sealed-secrets.netlify.app/

KubeSeal is a fantastic low friction pair to argocd. The biggest benefit to kubeseal is that you can safely commit secrets straight to git. It uses a one way encryption method that is very secure and you can only decode the secrets if you gain access to the keys.

kubernetes kubeseal

poc is not product

2024-01-28T09:49:57Z

A poc is not a product. I started focais, not in a rush, but as something that I already had a POC for and thought it would be easy. I wanted to build tools to make creating blog posts like this one easier. I stared with shots a tool that takes screenshots of websites.

POC (proof of concept) #

For the poc, I made a single fastapi endpoint that takes a url and returns a screenshot of the page. It converts the url into a key that I can lookup to see if I have the shot, if I don’t I go get it. With the open source libraries out there, this is not too hard of a task.

Progress Thus Far #

/shot

But this wasn’t enough

All it does so far for this first tool is take screenshots of websites, and give you a hosted image.

Users #

To bring in users, I need to create a signup flow, with a database to store users, login, logout, and email recovery. I’ve never had to use an email service before that wasn’t already mandated by a company or an iternal smtp server. After some searching I chose resend for email.

Progress thus far #

/shot
email service for account recovery
database to store users
/login
/forgot-password
/recover-account
/logout
/signup
/access-token
/account

User Management #

Eventually I am going to need some user management for site admins (me), but for now I just need to see users and some of their attributes such as subscription level, number of shots, and such. If I’m ever contacted for support this will give me some first insight into what happened.

Subscription Levels #

For payment I chose stripe. I was able to setup subscription levels with config inside of stripe. I setup a pricing page with jinja based on the data right out of stripe, a way to create checkout sessions, a page to come back to once you have executed a checkout, and a way to cancel your subscription.

Progress thus far #

/shot
email service for account recovery
database to store users
/login
/forgot-password
/recover-account
/logout
/signup
/access-token
/account
stripe account
stripe api integration
/pricing
/create-checkout-session
/checkout-success
/cancel-subscription
/reactivate-subscription

Back to the Product #

Now that I have users, with subscriptions, and different subscription levels. A shot might not be the same per url. For instance a shot from a low tier subscription might have a watermark, while a higher tier will remove the watermark. Now I need a way to track shots per user, and keep them up to date with subscriptions.

Progress thus far #

/shot
email service for account recovery
database to store users
/login
/forgot-password
/recover-account
/logout
/signup
/access-token
/account
stripe account
stripe api integration
/pricing
/create-checkout-session
/checkout-success
/cancel-subscription
/reactivate-subscription
POST /shot
GET /shot/{id}

Keeping the internet working #

At the end of all of this once a shot is created I want to keep it working for users regarless of subscription. It really bothers me when part of the internet just stops working. I hope to keep running costs low and enough monthly subscribers to cover those costs. And if you decide that fokais is not a product giving you any more value your shots will still be here with us. At the moment the biggest cost is compute running headless chrome to take the shots anyways, continuing to host them over time is must less of an issue.

It’s a Lot, but not near enough #

I’m a couple of months in on building this for a few days at a time here an there. I am in no rush for it to start gaining revenue. I am learning a ton so far. Whether fokais becomes a big thing or not, I will be able to bring these skills to a new startup or to my professional career.

It already feels 1000% more complicated than it was just to get the thing to work for just me, but in the end it will be worth it.

It’s still not complete #

There’s feels like quite a bit to do before getting launched, testing out the checkout workflow is HUGE thing. I’ve never made a product for sale by myself before, so I am definitly feeling some pressure when taking money to have a product that works, and importantly not loosing the subscription or account.

thoughts on unit tests

2023-12-22T08:49:07Z

Your browser does not support the audio element.

Theo’s response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements “Everything must be unit tested, tests make code more maintainable.” In reality when we are not writing complex low level code unit tests are probably the wrong approach.

Most of us are assemblers #

So much of software engineering is assembling existing well tested code. Crud applications, UI, Data Pipelines, building on top of battle tested code.

Manufacturing Analogy - Unit Testing #

This kind of reminds me of Manufacturing. Individual components are QA tested with tests that look more like unit test. Parts like bearings, pistons, shafts, valves, they are all tested against sophisticated statistics of sample measurements. This is quite similar to unit testing.

You see measuring the individual sizes of these components does not guarantee the actual function of the component. Before the Henry Ford assembly line parts like this were all different sizes, and they were all custom matched to each other. The one industry that I can thing of that still exists today like this is gunsmithing. parts are made very close to size, but they need some lapping and final finishing for high quality function.

Integration Testing #

So how are pumps, engines, valve assemblies, and so on tested? They are integration testsed based on their function. They are not generally unit tested to measurements of thier size. A pump is tested on a test bench and will be required to output a spec flow, a valve will have air or fluid applied and tested for leaks. An engine is a more complicated case where it will be ran on a suite of cycles and should output a spec level of power, while emissions, temperature, fuel consumption and hundreds of other measurements stay within their specs.

As assemblers we should be thinking of integration testing #

There is 100% a place for unit testing in the software engineering industry, just like manufacturing, but I don’t think it makes sense everywhere. Library development and lower level algorithms look a lot more like parts that can be physically measured. But that’s probably not the code most of us are writing most of the time.

We are likely assembling a lot of code to build a product such as a data pipeline, or a web application for instance. For these cases we should be thinking about the final function of the product. Like Prime says in the video, think of the product as a black box, as if you have no idea how anything works in the box, but you know how it should function.

I’m starting to lean more into simple tools for this qa work on we applications that I build. They are easy to build and maintain, and do not get in the way when the direction of the project changes. If I rip out half my code it would be likely to see 3/4’s of my unit tests failing. Not a fun thing to maintain.

Uptime #

This is the simplest form of integration testing that I am pulling into fokais, and it’s nearly fully automatable without much thought on my behalf. I am building web applications in fastapi, these api’s have endpoints that should be reachable. Tools like Uptime Kuma make it trivial to pull in all of my endpoints and simply ask if they are reachable once per minute.

With uptime tests I can more confidently move through dev to prod. If dev has been running for a number of minutes and alarms are not going off, it’s probably functioning.

Functionality Testing #

I’ve used tools built upon selenium in the past, such as testproject.io. It is a pretty great experience, and makes it pretty easy to define, when a user logs into the dev app, and clicks the dashboard, can they see their name and account information.

These tests are likely to remain true for the life of the project. No matter what changes inside the black box, when a user goes to their account they should see their account information. It might be server rendered with jinja, fetch with htmx, or client side rendered with reactjs, it does not matter.

Load Testing #

The last thing I want to really put as a tool in my toolbelt is load testing. It’s pretty easy to degrade the experience of a webpage with a bad database query or api call. Load testing can ensure that your application can handle the load that you want it to in dev before you release to prod. I am looking into bringing in Locust.io for load testing fokais.com.

Logging and Monitoring #

Inevitably users will use your product in ways that were never intended. It will be nearly impossible to cover everything with any sort of test. This is where logging and monitoring come in. If you can set up error monitoring and alerts, then you can check production logs to see what the heck happend and potentially resolve the issue. Without monitoring, you will have wild bugs that you never thought could exist.

Unit Tests are not a Golden fix all #

No tests will never be perfect, they will not catch everything. The point that I was inspired to make after watching Theo’s video is that unit tests may not the right thing for your product, and then requiring coverage metrics on PR may be more of a detriment to the product than help.

Looking for a Heroku replacement, What I found was shocking!

2023-12-04T20:29:46Z

Your browser does not support the audio element.

I’ve long hosted my personal blog as a static site on waylonwalker.com. It’s all markdown, converted to html, and shipped as is. It’s been great, I’ve moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it’s just pointing ci to a different host after the site has built.

What about server side #

Now the part that I have struggled with is how to cheaply host a server rendered application that can just live on forever without me paying for it. This is a harder problem as it costs more to keep servers spinning, memory, and disk all ready for you to use at a moments notice.

Honestly #

I never really deployed anything that useful on heroku, but it seems like the klenex of the bunch that’s why they are in the title. I’ve moved between digital ocean and fly.io, and have had some great experiences with both. I just don’t want to build something that is going perpetually cost money, I’m cheap and dont want to feel the burden of paying for something that I might not be using all the time.

fly.io #

fly is absolutely amazing to get off the ground. Their cli is top notch. They have servers all over the place and have a great interface to get your application deployed close to your users. You just can’t run all that much on it before you end up off the free tier.

serverless #

These platforms so far have just not felt right for me, again, I have to pay for them, many of them I have to worry about how hard they would get hit, and often they are focused on javascript.

What I am looking for #

easy to deploy
cheap to host
free to use
generous free tier
no chance of big bills
reliable, not 9 9’s, but mostly up.
allows me to run a bunch of stuff that I lightly use
that buttery smooth cli that I get out of fly.io
runs containers

The last one is probably the most important. I don’t get tons of time to work on side projects, and they are mostly ideas I just want to get out there and play with, the idea may flop quickly, but I want that DX to be easy to go from nothing to publically deployed.

What about going back to self hosting?? #

I have a Gateway FX6860 built in 2011, and was gifted to me around 2016. She doesn’t have much power to her, but she does have more than free tier power. I have 8 dedicated cpu, 16GB RAM, 500GB boot disk, and a 4TB hard disk.

Cloudflare tunnels #

Cloudflare Tunnels allow me to route traffic to my own server sitting right next to me and have it publically accessible. They take care of ssl certs, they support wildcard routes, and configuration is pretty easy. I can map addresses to a port on the machine, wildcard to a reverse_proxy, and end with a 404.

I’m missing that buttery smooth cli #

At this point I’m hosting a few things that I use frequently, but I don’t have that buttery smooth deploy experience that I get from fly.io. I have to log into the machine, edit some files, and docker compose up every time I want to deploy.

I was certain there was a better way #

I knew there was something better out there that was not a complex pain in the ass to setup. I know small companies run their own infra with a small team. There has to be tools that don’t take an enterprise to manage.

What about kubernetes #

In my search I keep seeing kubernetes as the solution, just run k8s, k3s, k0s, minikube, or kind. But EVERYTHING I have heard about kubernetes is that its a pain in the ass to deploy, takes a team to manage, and if you don’t have a $1M problem to solve don’t bother cause k8s will create a $1M problem for you. For this reason alone I turned my nose up to k8’s and nearly didn’t even give it a chance. Everyone jokes about blogs and simple sites running on k8s, and how rediculous that must be.

If you don’t have a $1M problem to solve don’t bother cause k8s will create a $1M problem for you.

Let’s jump on k3s #

I’ve never ran kubernetes myself, but after seeing it so many times in my searches for a fly.io replacement, I decided to give it a shot. I chose k3s as it seems like a nice balance of easy to setup, maintain, and feature complete kubernetes service.

# install and start k3s
curl -sfL https://get.k3s.io | sh -
# check to see if your nodes are started
sudo kubectl get nodes

My main hiccup so far was the machine I am running on runs a fairly new ubuntu install with zfs on root, and it would not start the master node. Rather than figuring out how to make zfs play nice I just pointed k3s to a drive that is not zfs.

# manuallly
sudo k3s server -d /mnt/vault/.rancher/k3s
# without editing systemd service
sudo ln -s /mnt/vault/.rancher/k3s /var/lib/rancher/k3s

Next I needed to be able to completely manage my k3s cluster form my main machine while this one sits far away in a closet.

# from the server
sudo cp /etc/rancher/k3s/k3s.yaml ~/.config/kube

# from my local machine
scp falcon@falcon:~/.config/kube/k3s.yaml ~/.config/kube/falcon-k3s.yaml
sudo cp /etc/rancher/k3s/k3s.yaml /home/waylon/.config/kube
sudo chown -R waylon:waylon ~/.config/kube
export KUBECONFIG=~/.config/kube/k3s.yaml

kompose #

Since everything I was running prior was in docker compose, I found kompose.io to be a fantastic tool to help me start converting my docker deployments into kubernetes.

A Month Later #

I started this post a month ago, and I am still enjoying k3s. For clarity, I did have a bit of k8’s experience going in, but zero experience running it by myself. k3’s seems to have made it pretty straightforward so far. My worst issues have been with ingress. Docker registries were a bit of a pain due to their large blob sizes, and a service I wanted to try to self host (sshx) required grpc, which is not supported by cloudflare tunnels.

TLDR #

Don’t believe everyone’s pre-conceived notions about tech you have never tried. Most of these things come from the echo chamber that is twitter anyways. Create your own opinions by trying new things, learning for yourself, and forming your own opnions.

why-is-postgres-default

2023-09-30T21:26:36Z

Serious question.

No one ever got fired for choosing PostgreSQL #

But, why. It’s the most loved db, right? Right? Maybe it’s time to rethink it.

Don’t get me wrong, if I need a relational db as a service, PostgreSQL is going to be my first choice, but why do I need to run a separate application for it?

Tutorials use sqlite #

Why is that? Because there is nothing else to stand up. Nothing else to maintain. And you probably already have it installed on just about anything that has a battery.

SQLite runs in memory #

Don’t need, or maybe don’t want to persist state. Run it in memory. This is a nice feature for running tests.

Less exposure #

SQLite is a file on your filesystem. It’s not a web service. It’s not a cloud service. Not that postgres is insecure, but it is one more endpoint that you have to think about securing.

this means that is probably also cheaper 🤑

SQLite is easy to replicate #

Want to run your new feature with prod data? Pull a replica or backup. It’s a file, you can cp, scp, rsync it whatever you have available.

SQLite Does everything of what I need in an application db #

SQLite has a rock solid set of features that covers everything that I need, and when it’s not I am probably thinking about pulling data into my application code and running something custom anyways.

SQLite runs on the edge #

Most applications are read heavy and light on writes. Services like turso have recognized this and built a business around it. They give you a master db that all writes go to, and read replicas that can run on the edge with your application code. You have a lot of users in different regions you can run your application close to them without suffering from query latency or replication complexity.

SQLite probably does what you need it to #

For a large number of use cases SQLite is probably the best thing to get you off the ground quick, and likely all that you will need. Consider using it before defaulting to postgres.

You probably already know how to maintain it #

If you know a bit of cli commands and remember that your database is just a file, this will feel very intuitive. If not you can probably poke around a file system gui and still make most of it work.

## create a database
touch database.db
## back it up
cp database.db database.db.bak
## drop the database
rm database.db
## restore the backup
cp database.db.bak database.db
## drop the backup
rm database.db.bak
## create an offsite backup
aws s3 cp database.db s3://my-backup-bucket/
scp database.db username@to_host:/remote/directory/
rsync -a database.db username@to_host:/remote/directory/

Need High Concurrent writes #

You might consider postgres.

Need Remote connection string #

You might consider postgres.

A Case For Tailwindcss

2023-09-10T19:46:19Z

I was watching @theprimeagen recently and I think he sold me on using tailwindcss. The thing about tailwind is that it is not a big component library, it’s a set of css classes mapped to a few (usually one) style.

All css classes are shitty, so you might as well use someone else’s shitty css classes on all your projects rather than thinking you’re being smart with a new set of classes that you will hate in 6 months when you come back to the project. roughly quoted from memory of @theprimeagen

It’s tiny #

So unlike big component libraries like tailwind, it comes with a cli that that it uses to create the final css file. It is able to treeshake out all the tailwind classes that you are not using and only ship the ones that you are using.

It’s hard to clash #

Since the classes are so small and single purpose it’s hard to end up with something like .card in two places that mean different things causing you to duplicate most of that css anyways so that the whole design doesn’t break when you change one or the other.

the classes make sense #

The classes are really clear and easy to understand, they mentally map over directly to the css that they implement.

Things like margin are easy to grab once you get the grasp of it. m for margin, and a number for how much, and it’s really easy to bump just a bit more.

.m-6 {
  margin: 1.5rem;
}

.m-5 {
  margin: 1.25rem;
}

.m-4 {
  margin: 1rem;
}

.m-3 {
  margin: 0.75rem;
}

.m-2 {
  margin: 0.5rem;
}

.m-1 {
  margin: 0.25rem;
}

.m-0 {
  margin: 0px;
}

Font Sizes are well thought out #

Rather than having a bunch of random sizes, tailwind has a really well thought out design system that does not grow linearly, but with what makes sense. Not only the font size, but they have given you the appropriate line-height to go with it.

.text-2xl {
  font-size: 1.5rem;
  line-height: 2rem;
}

.text-xl {
  font-size: 1.25rem;
  line-height: 1.75rem;
}

.text-lg {
  font-size: 1.125rem;
  line-height: 1.75rem;
}

.text-sm {
  font-size: 0.875rem;
  line-height: 1.25rem;
}

.text-xs {
  font-size: 0.75rem;
  line-height: 1rem;
}

Colors #

This goes for colors too. When I want to do things like make a button look clickable, all I need to do is bump the color up or down by 100, I don’t need to know deep color theory to understand if it will be noticable, the team has designed it to work better than I can come up with.

.bg-stone-100 {
  --tw-bg-opacity: 1;
  background-color: rgb(245 245 244 / var(--tw-bg-opacity));
}

.bg-stone-200 {
  --tw-bg-opacity: 1;
  background-color: rgb(231 229 228 / var(--tw-bg-opacity));
}

.bg-stone-300 {
  --tw-bg-opacity: 1;
  background-color: rgb(214 211 209 / var(--tw-bg-opacity));
}

.bg-stone-400 {
  --tw-bg-opacity: 1;
  background-color: rgb(168 162 158 / var(--tw-bg-opacity));
}

.bg-stone-50 {
  --tw-bg-opacity: 1;
  background-color: rgb(250 250 249 / var(--tw-bg-opacity));
}

.bg-stone-500 {
  --tw-bg-opacity: 1;
  background-color: rgb(120 113 108 / var(--tw-bg-opacity));
}

.bg-stone-600 {
  --tw-bg-opacity: 1;
  background-color: rgb(87 83 78 / var(--tw-bg-opacity));
}

.bg-stone-700 {
  --tw-bg-opacity: 1;
  background-color: rgb(68 64 60 / var(--tw-bg-opacity));
}

.bg-stone-900 {
  --tw-bg-opacity: 1;
  background-color: rgb(28 25 23 / var(--tw-bg-opacity));
}

Am I going to start using tailwind? #

Right now I am working on some projects that I am going to use tailwind on. I really see myself liking it. I don’t like that I need to install it with npm, but I can get past that. Cli tools seem to work out pretty well, but anytime I have a project that installs a bagillion node modules they all come crashing down in 6 month or a year when the project has not been touched and I just need one small change.

I really like the idea of all these classes being consistent across all my projects and really being a short hop away from hand writing css vs a heavy component library. Most of all I like their well thought out design system that I definitly could not get right myself.

Make the easy things easy

2023-09-06T19:46:19Z

It’s so easy to get out of rhythm, get busy, and drop the ball on some things that you really want to do or should do. This blog is a good example. I took some time off for some family reasons, but have taken a long time to get back to it simply because I am out of rhythm. As I am trying to get back into the rhythm there is some tooling that I have set up for it that I completely forgot about that feel good to use again.

Repetitive Tasks #

Simple Repetitive Tasks that I have to do often can just feel soul crushing, and one main thing that got me interested in programming.

AI tools are becoming more and more useful at solving these problems. For instance code generation tools like co-pilot or codeium are really good at boilerplate and pattern repetition. Things that used to be a few vim macros is now just banging on tab.

I often look for setting up templates or some sort of snippet to replace a big chunk of boilerplate that I know I will need over and over.

timebox #

Don’t be afraid to give up on making a better workflow for something that just might not be worth it. Give yourself a session or two to show some progress, if you don’t get it, don’t be afraid to put it on pause, let it stew, or just forget about it. Often times I will come back later with new knowledge and knock out an thorn in my setup with new skill much easier the next time around.

Low Hanging Fruit #

Often things like having a quick template to get a blog post up and running are worth it. It might not take that long to setup a new post, set the date, title, tags, and create the file, but sometimes that is enough of a barrier to stop me from even making a post. Making a template and a script that I can call in a few keystrokes makes me much more likely to make a post.

Learning #

I am often reminded of this xkcd comic about automation, and how easy it is to overdo it. Spending far more time automating things than it would have to do the original task is a very easy pit to fall into. I’ve been guilty of it many times. I’m sure most people who have written any automation code is as well.

What this comit misses is learning. If you never take the time to learn any automation you will not be any good at it and it will always take longer to automate than to just do the thing. You have to tackle some things that you think are achievable with skills you think are transferable to other tasks. Scraping that one weird endpoint out of the Microsoft Graph API that is not in the cli is a prime example. Often there is a series of undocumented n+1 queries that you have to execute and each leave you feeling like you are almost at the data that you can see in the ui, but want in the terminal. Learning this api will leave you with nothing but potentially one task automated.

Learning how to leverage cli tools like curl, grep, xargs are great examples of skills that can solve many automation problems quickly, but the first time you will end up spending hours looking for the right thing before you realize what you need.

Underground Bases with Wyatt

2023-07-09T10:41:29Z

Playing minecraft with Wyatt today he started a server all on his own and had me join. All vanilla, only one rule, underground bases.

I spawned into the server and it was already night time. I gathered up some wood on my way down a tree, and was attacked by zombies before I could get any tools, so I ran up another tree and crafted a crafting table.

Now to follow the rules, it’s time to head underground to build my base.

I made a mistake and died, but look at this view from my respawn point.

I took a break

2023-05-27T19:45:12Z

Life comes in waves, and sometimes you need to set down some of your projects to focus on others. For the first part of 2023 I’ve really had a lot of family stuff to focus on, we also are pretty new homeowners and are still trying to get our new to us house cleaned up and modernized.

Side Projects #

You can see in my growing list of repos that I have poked around on quite a few side projects over the past few months. This has been quite relaxng to me, mostly things that I use to learn from, but also a lot that are tools and things I use that bring me joy.

Pydantic #

I haven’t wrote about it at all yet, but I have really been starting to lean into pydantic on all of these side projects. I have really been enjoying the type system. A good friend @pypeaday got me hooked and we have been throwing around this phrase that he learned from a math professor “Make it So”. The idea boils down to leveraging pydantic to make all the values you want to exist up front, or fail validation, then have no more checks scattered all over your code.

I’ve really been deleting a bunch of logic from places within my code and putting it right in the pydantic models. This gives me one place I will know to look for it rather than many places some logic might come up. Then when it’s time to do things, the logic is typically a bit simpler.

Markata #

Markata continues to be my biggest side project. Currently its it a big refactoring phase to get everything moved into pydantic. It’s been quite awhile since I’ve had a release, but there has been steady progress going into the pydantic branch.

AI #

Ai is really hot in 2023. I’ve been trying to keep my finger on the pulse and play with it. I tried to make a library to interface with openai lockhart. It was a cli where you can store prompt template. I also made a full text adventure game with a new library from prefect.io, called marvin. marvin star wars text adventure.

I took a break #

It’s defintitely good to take breaks from big projects, and for me its often things like family and work that need to come first and I set things aside. I am not saying that I am back to daily writing, but I am going to give it a go and see where it takes me. I’ve got a big backlog of ideas to get out.

Playing Star Wars Text Adventure with a 10 yr old

2023-05-15T12:08:00Z

The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star

❯ sw-adventure game run
[05/15/23 18:47:42] INFO     marvin.marvin: Using OpenAI model "gpt-3.5-turbo"                         logging.py:50
18:47:42.699 | INFO    | marvin.marvin - [default on default]Using OpenAI model "gpt-3.5-turbo"[/]
[18:47:42] Starting game                                                                                  game.py:30
generating your character
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez          │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He     │
│   health           │ 100     │ always dreamed of adventure and excitement. As soon as he was old enough, he      │
│   imperial credits │ 5000    │ joined the Imperial Navy and quickly rose through the ranks. He is now a skilled  │
│   fuel level       │ 100     │ pilot and loyal member of the Empire.                                             │
│                              │                                                                                   │
│       Imperial               │ A nimble and deadly starfighter used by the Imperial Navy                         │
│     Starfighter              │                                                                                   │
│   year built │ 5             │                                                                                   │
│   capacity   │ 1             │                                                                                   │
│   fuel level │ 100           │                                                                                   │
│                              │                                                                                   │
│     Operation Shadow Hand    │ Infiltrate and sabotage the Rebel base on Yavin IV, using any means necessary.    │
│   place  │ Yavin IV          │ Eliminate their leadership and secure the plans to the Death Star.                │
│   year   │ 5                 │                                                                                   │
│   risk   │ 50 hp             │                                                                                   │
│   reward │ 10000 credits     │                                                                                   │
│   leader │ Director Krennic  │                                                                                   │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: sneak into the deathstar, but accidently drop a fart

Your mission has been completed
Zorin Kreez attempted to sneak into the Death Star, but accidentally let out a loud fart, alerting nearby
Stormtroopers. He was quickly captured and taken into custody. The mission was a complete failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez            │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He   │
│   health           │ 100       │ always dreamed of adventure and excitement. As soon as he was old enough, he    │
│   imperial credits │ 5000      │ joined the Imperial Navy and quickly rose through the ranks. He is now a        │
│   fuel level       │ 100       │ skilled pilot and loyal member of the Empire.                                   │
│                                │                                                                                 │
│       Imperial                 │ A nimble and deadly starfighter used by the Imperial Navy                       │
│     Starfighter                │                                                                                 │
│   year built │ 5               │                                                                                 │
│   capacity   │ 1               │                                                                                 │
│   fuel level │ 100             │                                                                                 │
│                                │                                                                                 │
│      Operation Star Saber      │ Infiltrate the secret Rebel base on Coruscant and retrieve the stolen Imperial  │
│   place  │ Coruscant           │ data. Avoid detection and eliminate any Rebel agents if necessary.              │
│   year   │ 6                   │                                                                                 │
│   risk   │ 80 hp               │                                                                                 │
│   reward │ 12000 credits       │                                                                                 │
│   leader │ Admiral Rae Sloane  │                                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: release a deadly fart that will drop any stormtrooper in a 12 ft radius

Your mission has been completed
Zorin Kreez attempted to release a deadly fart to drop any stormtrooper in a 12 ft radius, but he ended up falling
victim to the toxic gas himself and passed out. The mission was a complete failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez        │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He       │
│   health           │ 100   │ always dreamed of adventure and excitement. As soon as he was old enough, he        │
│   imperial credits │ 5000  │ joined the Imperial Navy and quickly rose through the ranks. He is now a skilled    │
│   fuel level       │ 100   │ pilot and loyal member of the Empire.                                               │
│                            │                                                                                     │
│       Imperial             │ A nimble and deadly starfighter used by the Imperial Navy                           │
│     Starfighter            │                                                                                     │
│   year built │ 5           │                                                                                     │
│   capacity   │ 1           │                                                                                     │
│   fuel level │ 100         │                                                                                     │
│                            │                                                                                     │
│    Operation Foul Odor     │ Infiltrate the Death Star and disrupt the Rebel Alliance's plans by releasing a     │
│   place  │ Death Star      │ deadly fart that will incapacitate their forces. Avoid detection and eliminate any  │
│   year   │ 7               │ Rebel agents if necessary.                                                          │
│   risk   │ 90 hp           │                                                                                     │
│   reward │ 15000 credits   │                                                                                     │
│   leader │ Darth Vader     │                                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: spend 500 credits on beans

Your mission has been completed
Zorin Kreez spent 500 imperial credits on beans, hoping to prepare for the upcoming mission. Unfortunately, the
beans did not sit well with him, causing him to have severe digestive issues during the mission. The resulting
flatulence was so intense that it alerted nearby Stormtroopers, resulting in Zorin's capture. The mission was a
complete failure.
You earned 0 imperial credits
You spent 500 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez            │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He   │
│   health           │ 100       │ always dreamed of adventure and excitement. As soon as he was old enough, he    │
│   imperial credits │ 4500      │ joined the Imperial Navy and quickly rose through the ranks. He is now a        │
│   fuel level       │ 100       │ skilled pilot and loyal member of the Empire.                                   │
│                                │                                                                                 │
│       Imperial                 │ A nimble and deadly starfighter used by the Imperial Navy                       │
│     Starfighter                │                                                                                 │
│   year built │ 5               │                                                                                 │
│   capacity   │ 1               │                                                                                 │
│   fuel level │ 100             │                                                                                 │
│                                │                                                                                 │
│    Operation Wretched Stench   │ Infiltrate the Rebel base on Hoth and release a deadly stench that will         │
│   place  │ Rebel Base on Hoth  │ incapacitate their forces. Avoid detection and eliminate any Rebel agents if    │
│   year   │ 8                   │ necessary.                                                                      │
│   risk   │ 85 hp               │                                                                                 │
│   reward │ 12000 credits       │                                                                                 │
│   leader │ General Veers       │                                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: Quietly enter the Rebel Base and eliminate any Rebel agent with a silent but deadly wet fart

Your mission has been completed
Zorin Kreez attempted to eliminate a Rebel agent with a wet fart, but the fart was not silent and alerted nearby
Stormtroopers. The resulting chaos led to Zorin being captured and taken into custody. The mission was a complete
failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                │ Zorin Kreez was born on Tatooine and grew up in a small farming community.  │
│   health           │ 100           │ He always dreamed of adventure and excitement. As soon as he was old        │
│   imperial credits │ 4500          │ enough, he joined the Imperial Navy and quickly rose through the ranks. He  │
│   fuel level       │ 100           │ is now a skilled pilot and loyal member of the Empire.                      │
│                                    │                                                                             │
│       Imperial                     │ A nimble and deadly starfighter used by the Imperial Navy                   │
│     Starfighter                    │                                                                             │
│   year built │ 5                   │                                                                             │
│   capacity   │ 1                   │                                                                             │
│   fuel level │ 100                 │                                                                             │
│                                    │                                                                             │
│      Operation Stinky Sabotage     │ Infiltrate the Rebel base on Yavin IV and sabotage their weapons and        │
│   place  │ Rebel Base on Yavin IV  │ communication systems with a silent but deadly wet fart. Avoid detection    │
│   year   │ 9                       │ and eliminate any Rebel agents if necessary.                                │
│   risk   │ 90 hp                   │                                                                             │
│   reward │ 13000 credits           │                                                                             │
│   leader │ General Veers           │                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: spend 500 credits on fart spray

Your mission has been completed
Zorin Kreez spent 500 imperial credits on fart spray, hoping to prepare for the upcoming mission. However, the spray
turned out to be ineffective and Zorin ended up being captured by the Rebels. The mission was a complete failure.
You earned 0 imperial credits
You spent 500 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                │ Zorin Kreez was born on Tatooine and grew up in a small farming community.  │
│   health           │ 100           │ He always dreamed of adventure and excitement. As soon as he was old        │
│   imperial credits │ 4000          │ enough, he joined the Imperial Navy and quickly rose through the ranks. He  │
│   fuel level       │ 100           │ is now a skilled pilot and loyal member of the Empire.                      │
│                                    │                                                                             │
│       Imperial                     │ A nimble and deadly starfighter used by the Imperial Navy                   │
│     Starfighter                    │                                                                             │
│   year built │ 5                   │                                                                             │
│   capacity   │ 1                   │                                                                             │
│   fuel level │ 100                 │                                                                             │
│                                    │                                                                             │
│      Operation Stinky Sabotage     │ Infiltrate the Rebel base on Yavin IV and sabotage their weapons and        │
│   place  │ Rebel Base on Yavin IV  │ communication systems with a silent but deadly wet fart. Avoid detection    │
│   year   │ 9                       │ and eliminate any Rebel agents if necessary.                                │
│   risk   │ 90 hp                   │                                                                             │
│   reward │ 13000 credits           │                                                                             │
│   leader │ General Veers           │                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: spend 1000 on nuke

Your mission has been completed
Zorin Kreez attempted to purchase a nuke with 1000 imperial credits, but was quickly reprimanded by Imperial
authorities for attempting to use resources for personal gain. Zorin was demoted and his mission was a complete
failure.
You earned 0 imperial credits
You spent 1000 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                   │ Zorin Kreez was born on Tatooine and grew up in a small farming          │
│   health           │ 100              │ community. He always dreamed of adventure and excitement. As soon as he  │
│   imperial credits │ 3000             │ was old enough, he joined the Imperial Navy and quickly rose through     │
│   fuel level       │ 100              │ the ranks. He is now a skilled pilot and loyal member of the Empire.     │
│                                       │                                                                          │
│       Imperial                        │ A nimble and deadly starfighter used by the Imperial Navy                │
│     Starfighter                       │                                                                          │
│   year built │ 5                      │                                                                          │
│   capacity   │ 1                      │                                                                          │
│   fuel level │ 100                    │                                                                          │
│                                       │                                                                          │
│           Operation No Nukes          │ Zorin Kreez has been detained by Imperial authorities for attempting to  │
│   place  │ Imperial Detention Center  │ purchase a nuke with Imperial resources. His mission is to escape the    │
│   year   │ 10                         │ detention center and clear his name. Avoid detection and eliminate any   │
│   risk   │ 95 hp                      │ Rebel agents if necessary.                                               │
│   reward │ 0 credits                  │                                                                          │
│   leader │ Admiral Piett              │                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: Approach Admiral Piett with the fart nuke

Your mission has been completed
Zorin Kreez was quickly apprehended by Imperial authorities when he attempted to approach Admiral Piett with the
fart nuke. His mission was a complete failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│         Zorin Kreez                   │ Zorin Kreez was born on Tatooine and grew up in a small farming          │
│   health           │ 100              │ community. He always dreamed of adventure and excitement. As soon as he  │
│   imperial credits │ 3000             │ was old enough, he joined the Imperial Navy and quickly rose through     │
│   fuel level       │ 100              │ the ranks. He is now a skilled pilot and loyal member of the Empire.     │
│                                       │                                                                          │
│       Imperial                        │ A nimble and deadly starfighter used by the Imperial Navy                │
│     Starfighter                       │                                                                          │
│   year built │ 5                      │                                                                          │
│   capacity   │ 1                      │                                                                          │
│   fuel level │ 100                    │                                                                          │
│                                       │                                                                          │
│          Operation No Nukes 2         │ Zorin Kreez has escaped from Imperial detention and acquired a fart      │
│   place  │ Imperial Detention Center  │ nuke. His mission is to approach Admiral Piett with the nuke, eliminate  │
│   year   │ 10                         │ any Rebel agents if necessary, and clear his name.                       │
│   risk   │ 99 hp                      │                                                                          │
│   reward │ 0 credits                  │                                                                          │
│   leader │ Admiral Piett              │                                                                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
What do you do ❯: Jump in the Imperial Starfighter as quickly as possible, drop the fart nuke on the Imperial Detention Center just before jumping into hyperspace.

Your mission has been completed
Zorin Kreez attempted to carry out his mission plan, but was quickly intercepted by Imperial authorities who had
received a tip-off about his intentions. Zorin was arrested and taken into custody. The mission was a complete
failure.
You earned 0 imperial credits
You spent 0 imperial credits
You gained 0 health
You lost 0 health

Pydantic and singledispatch

2023-05-03T17:31:13Z

I was reading about pydantic-singledispatch from Giddeon’s blog and found it very intersting. I’m getting ready to implement pydantic on my static site generator markata, and I think there are so uses for this idea, so I want to try it out.

The Idea #

Let’s set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use functools.singledispatch and type hints to provide unique execution for each environment. We might want something like a path_prefix in prod for environments like GithubPages that deploy to /<name-of-repo> while keeping the root at / in dev.

Settings Model #

Here is our model for our settings. We will create a CommonSettings model that will be used by all environments. We will also create a DevSettings model that will be used in dev and ProdSettings that will be used in prod. We will use env as the discriminator so pydantic knows which model to use.

from typing import Literal, Union

import pydantic
from pydantic import Field
from rich import print
from typing_extensions import Annotated


class CommonSettings(pydantic.BaseSettings):
    """Common settings for all environments"""
    debug: bool = False
    secret_key: str = "secret"
    algorithm: str = "HS256"
    access_token_expire_minutes: int = 60


class DevSettings(CommonSettings):
    """Settings for dev"""
    env: Literal["dev"]


class ProdSettings(CommonSettings):
    """Settings for prod"""
    env: Literal["prod"]


class Settings(pydantic.BaseSettings):
    """Settings for all environments"""
    __root__: Annotated[Union[DevSettings, ProdSettings], Field(discriminator="env")]

    class Config:
        env_prefix = "APP_"



# Create our settings
settings = Settings(__root__={"env": "dev"}).__root__
# or
settings = Settings.parse_obj({"env": "dev"}).__root__
print(settings)

DevSettings(debug=False, secret_key='secret', algorithm='HS256', access_token_expire_minutes=60, env='dev')

Singledispatch #

Now let’s create our where_am_i function. We will use functools.singledispatch to provide a unique execution for each environment. It will leverage type hints to provide a unique execution for each environment.

from functools import singledispatch

@singledispatch
def where_am_i(obj):
   '''
   Where am I?
   '''

@where_am_i.register
def dev(obj: DevSettings):
   '''
   Where am I?
   '''
   print('I am in dev')

@where_am_i.register
def prod(obj: ProdSettings):
   '''
   Where am I?
   '''
   print('I am in prod')

Output #

Let’s call our eample function where_am_i with our settings and see the results.

where_am_i(settings)

results in

I am in dev

Let’s check prod

where_am_i(Settings.parse_obj({'env': 'prod'}).__root__)

results in

I am in prod

Environment Variables #

So far one down side to the TaggedUnion technique is that I am unable to pull env from environment variables. I’m sure there is a way around this with a different model design. Maybe following exactly what Giddeon did.

os.environ.clear()
os.environ['APP_ENV'] = 'prod'
where_am_i(Settings().__root__)

results in

ValidationError: 1 validation error for Settings
__root__
  field required (type=value_error.missing)
<darkmark.darkmark.DarkMark object at 0x7fcc58bec5d0>

FIN #

I’m really digging pydantic lately and excited to get it built into markata. Not 100% sure if I have a

minecraft documentary

2022-12-06T13:38:54Z

This is my first time journaling a Minecraft hardcore world, my son Wyatt is also documenting his journey in a survival world on wyattbubbylee.com.

Day 0 #

init

I logged into a brand new hardcore world. I was welcomed by a great Acacia biome spawn full of resources. I quickly cut my first tree, crafted an axe and set out to find my first sheep. I was able to find enough sheep for a bed, several cows and pigs.

I crafted a set of wooden tools, and farmed out a wheat farm till my wooden hoed died at the shore of a nearby stream. I found a small stone outcropping in the side of a hill and harvested nearly a full stack of cobblestone from my first wooden pick.

I ended the first day by sleeping in my bed safe from mobs.

Achievements #

bed
furnace
stone
wheat farm

Day 1 #

Thoughout the course of day one I collected wood and started the framework for my new house.

Day 2 #

The sun sets over the new frame of my house on Day 1

Day two continued the framework of the house, and a bit of sand collection for windows. I was a bit late to bed and saw four mobs spawn on the journeymap before I made it in.

Day 3 #

end of day 2

On day three I completed the roofline of the base, gathered some wood, and cleared out all the grass in the tree farm. The tree farm has a fresh planting of oak and acacia trees in it.

Day 4 #

Thoughout day 4 I kept farming trees and creating my first outter perimeter fence made of oak.

end of day 3

Day 5 #

start of of day 5

On day 5 I completed the perimeter fence and worked on my first animal pen.

End of Day 5

Day 6 #

I completed another animal pen.

a new animal pen from day 6

Day 7 #

On day 7 I left the game paused and went to some meetings, I came back to a death screen 😥. I am not going out this way, I opened to lan with cheats and reset my gamemode to survival.

Day 8 #

I started a storage silo by doing some terraforming and placing down the first chests.

Day 9 #

On Day 9 I started a challenge from Wyatt to create a tiny base. I chose to make a secret storage area in the storage silo.

Day 10 #

On day 10 Wyatt joined me, he built a dock and I found a zombie spawner.

We conquered the Zombie Spawner

Standing on Wyatt’s new dock

Day 11 #

Wyatt planted some beets, and made space for pumpkins.

Day 12 #

On day 12 we gathered up some farm animals.

Day 13-15 #

Wyatt and I went spelunking in the caves for iron.

Day 16 #

Wyatt started his floating base

Day 17 #

On Day 18 I created the crows nest on top of the storage silo.

Packages I Maintain

2022-08-30T20:02:38Z

Markata #

I open sourced the static site framework that I use to build my-blog among other side projects. It’s a plugins all the way down static site generator, that makes me happy to use.

{% gh_repo_list_topic “waylonwalker”, “markata” %}

Repos used to build this blog #

my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don’t have to worry about carrying around large images with my lightweight text files just to make some posts.

{% gh_repo_list_topic “waylonwalker”, “personal-website” %}

Kedro #

I am a heavy user of the kedro framework, and a big advocate for using some sort of DAG framework for your data pipelines. kedro is built all in python which makes it easy for a python dev like me to extend, run, maintain, and deploy.

{% gh_repo_list_topic “waylonwalker”, “kedro” %}

Neovim Plugins #

I use vim for all of my text editing needs. It brings me joy to make any part of it just a bit smoother. I have written a few plugins that help me achieve that silky smooth workflow.

{% gh_repo_list_topic “waylonwalker”, “neovim-plugin” %}

I was on Talk Python

2022-08-27T20:52:02Z

After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science.

I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on.

During the episode I tried my best to let Yetu and Ivan take the spotlight as the maintainer and chime in with my experience as a user of kedro.

Video Version #

https://youtu.be/WTcjvwkXoY0

Michael made the call available on youtube as well as the audio only podcast

The one reason I switched to arch

2022-07-23T12:05:55Z

The community, that’s it, end of post, roll the credits.

I’m a tinkerer #

I am a tinkerer, I am not going to run a stock desktop manager, mostly becuase that’s just not how my brain works. I need to tweak everything to fit my needs. Grantid I have not spent much time in many full fledged linux desktop environments. They are far more customizable than windows ever will be, I absolutely love that about them. Inevitibly I end up in a situation where I hit a wall, it just won’t do what I want it to do, or my lack of understanding what came wtih it holds me back.

minimal #

I love minimal installs. I love just building up my system from the bottom up with things that I like, I understand, and that I can script.

I’m a noob #

I spend a lot of my time in the terminal. I’d like to think I know how to use a linux command line for software development really well, but there are a lot of things that I still dont know all that well, mostly because I don’t need to.

aur">The AUR #

Arch’s community maintained repo

Now by far this is the thing that really kicked me over the edge and made arch feel like something that I needed to use. While on any Debian based distro it was inevitible that I would stumble accross a bug that was fixed in a newer version, not available in the standard repos, want new features that are not available in the standard repos, or want software that is not packaged into one of the standard repos.

This makes creating scripted installs/updates difficult. I have to pull releses from github, or some random website, or clone and build packages myself. It’s not too hard, but since I am the only one that maintains my personal install script it’s inevitible that something changes in the build process or release on something and breaks it over time.

PPA’s are not the same. Well maintained ppa’s are fine, but many of them are not. PPA’s are just deb files that someone hosts and their quality can be hit or miss depending on if the maintainer still uses it or not. While the AUR is not completely absent of stale packages it’s rolling release nature tends to keep them always up to date, and when it doesn’t the community helps out by flagging these packages in the AUR and upvoting maintained packages.

The Arch Wiki #

I have underutilized the arch wiki in the past, but dang it has some really good guides. For me (at my level) they edge on being to brief, generally I find wiki’s overly versose and hard to find what you are looking for. The arch wiki gives you what you need and no more. The articles and forum really have answered many of my questions and depened my understanding of linux.

I am learning #

While I am still a noob to a lot of this I am learning, and a combination of the minimal install with the arch wiki have really helped me learn a lot about lower level things like partitioning, grub, systemd, xorg. On most other systems these were just taken care of out of the box.

Arch is not hardcore #

I think there is an old sentimate that arch is for hardcore users only. Their typical install process really gatekeeps it this way, but there are other installers such as archinstall, which I used and now ships with the arch iso, that make it not much harder than any other distro.

If you are a tinkerer who likes to try different things, likes to set you machine up just the way you like it, and script it, arch really feels much easier to do this with. I’d even stand behind it being a distro for noob tinkerers.

Does distro matter?? #

Idk if it really matters all that much in the end, but the community behind the aur and arch wiki make setting up an up to date machine so much easier than on any lts based distro.

Using Different versions of python with pipx | pyenv

2022-05-28T20:38:42Z

I love using pipx for automatic virtual environment management of my globally installed python cli applications, but sometimes the application is not compatible with your globally installed pipx

Which version of python is `pipx` using?? #

This one took me a minute to figure out at first, please let me know if there is a better way. I am pretty certain that this is not the ideal way, but it works.

My first technique was to make a package that printed out sys.version.

# what version of python does the global pipx use?
pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

# what version of python does the local pipx use?
python -m pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

Let’s setup some other versions of python with pyenv #

If you don’t already have pyenv installed, you can follow their install instructions to get it.

pyenv install 3.8.13
pyenv install 3.10.5

I usually require a virtual environment #

I set the PIP_REQUIRE_VIRTUALENV environment variable to true to ensure that the virtual environment is activated when pip installing, this makes it so that I can’t accidentally use the global env, which is typically not what I want to do.

export PIP_REQUIRE_VIRTUALENV=true
# for windows users
set PIP_REQUIRE_VIRTUALENV=true

This goes right into my shell startup script ~/.bashrc or ~/.zshrc.

Exceptions happen #

This is my one exception. I’ve had better luck putting pipx right in the global python environment. Not the system python, but each python version that I install with pyenv.

export PIP_REQUIRE_VIRTUALENV=false
# for windows users
set PIP_REQUIRE_VIRTUALENV=false

Let’s install pipx #

First up is python 3.10.5

pyenv global 3.10.5
pip install pipx
pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0]

Next is python 3.8.13j

pyenv global 3.8.13
pip install pipx
pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers

3.8.13 (default, Aug  8 2022, 21:06:56)
[GCC 12.1.0]

Now once I close this shell I will always end up with PIP_REQUIRE_VIRTUALENV=true, since it’s in my shell startup script. So make sure that you reset it or kill this shell before doing any damage.

LIVE-REPLAY - Python dev | Markata todoui | 4/6/2022

2022-05-12T19:28:20Z

https://youtu.be/-42A5210HYo

Super fun steam Broadcasted live on Twitch – Watch live at https://www.twitch.tv/waylonwalker

We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow.

During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off.

dotfiles: https://github.com/WaylonWalker/devtainer
today’s project: https://github.com/WaylonWalker/markata-todoui
website: https://waylonwalker.com/

How I Quickly Capture Screenshots directly into My Blog

2022-04-30T14:05:49Z

When I am creating blog posts it’s often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content.

Making Screenshots #

When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog.

take screenshot
optimize
conversion
publish
create img tag

Created in 🐍Python #

I created this tool for myself in python because that is what I am most familiar with, but realistically most of what I am calling are shell scripts that I could do in just about any language.

Install my screenshot tool #

My screenshot tool is quite hacky and not configurable, but works wonderfully for me. If you like it and want to use it, make it configurable or fork it. For now it lives on GitHub and since it has a setup.py with entrypoints we can install it with pipx.

pipx install git+https://github.com/WaylonWalker/screenshots.waylonwalker.com

This is just a tool for me, it does not need to be in a package manager like pip.

calling screenshot #

Now that screenshot is installed we can call it and make a screenshot. I’ll take a screenshot of the frontmatter of this exact post.

I have this tool exposed as a command that can be ran in the command line by calling screenshot. I will rarely use it this way, but makes it easy to create a hotkey for later.

Success #

Once the screenshot is successful, I get a notify-send message popup telling me so.

xbindkeys #

Let’s make this workflow just a bit smoother. I want a desktop hotkey that I can press at any time without opening a split or making sure I’m in zsh or vim, I want it always there. For hotkeys like this I use the Desktop manager agnostic keybinding tool xbindkeys. I can add the screenshot command and the corresponding keybinding I want to my ~/.xbindkeysrc file and restart with xbindkeys -f ~/.xbindkeysrc.

"screenshot"
    Shift + Mod4 + alt + p

Now when I press Shift + Mod4 + alt + p I am presented with a little zenity box asking me what I want to name my screenshot, followed by flameshot.

xbindkeys allows me to bind this hotkey to my system so that no matter where I am it will work.

Name it #

The one question I ask myself when creating the hotkey is for a filename. On my ubuntu machine I do that with a simple gui application called zenity. It looks like this when I open it up.

Under the hood my screenshot tool is running the following command in a subprocess.

zenity --entry --text="filename"

Copier < 6.0.0b0 considered dangerous

2022-04-30T10:05:45Z

Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories.

!!UPDATE #

As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update.

This is a PSA #

I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I do so that they know the dangers of using copier inside an existing directory.

The issue #

The fix #

https://github.com/copier-org/copier/pull/273

As of the time of writing this version is still in beta, if you still want to use copier with existing directtories, I’d strongly encourage you to install the --pre release.

pipx install copier --pip-args='--pre'

confirm #

copier --version
# copier 6.0.0b0

My update commit #

https://github.com/WaylonWalker/devtainer/commit/a06e47462e2c6d26ad69fbdc2e7990a1a73ab4b7

PyOhio CFP's

2022-04-12T12:47:52Z

Here are some CFP’s that I used for PyOhio 2022.

https://pretalx.com/pyohio-2022/cfp

Idea to blog post in minutes - Shorter #

Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it’s plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle.

Lets build a whole blog site in 5 minutes.

Add Kedro to your Pandas Workflow - Short #

Sometimes python scripts/notebooks take a long time to run, let kedro automatically save your datasets so that you can maintain your production code with ease. Lets take a pipeline with an issue 30 minutes in and solve the issue in 5 mintues.

Smoother Python with automatic imports | pyflyby

2021-12-20T11:34:47Z

This is not a flaky works half the time kind of plugin, it’s a seriously smooth editing experience. I’ve just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython.

I can’t tell you how pumped I am for this, and how good its felt to use over the past few weeks. It’s glorious.

YouTube video #

Listen to me rant on how great pyflyby is

https://youtu.be/2QW5DJiEJH4

Give the video a watch, I did not have noise-cancelling on in obs. My apologies for the background hum and the mic stand bumps. I did my best to fix them up.

Installation #

How to install pyflyby for automatic python imports

pyflyby is hosted on pypi, so you can get it with pip. I have had no issues installing it on 3.8+ so far.

pip install pyflyby

Configuration setup with stow #

always stow your dotfiles

If you’re going to configure any of your tools the first thing you should do is set it up with stow, seriously don’t sleep on the stow. If you don’t have stow installed or choose not to use stow you can skip this part.

cd ~/dotfiles
mkdir ipython
touch ipython/.pyflyby
stow ipython

Seriously don’t sleep on the stow.

How to Configure pyflyby #

it’s just a file full of import statements

pyflyby is configured simply by putting all of your import statements that you want to automatically import into your ~/.pyflyby file. You can import pandas, from pandas import DataFrame, or even import pandas as pd, and all of these will work pretty much as expected.

# comments start with a #
# import your favorite libraries
import visidata as vd
import fsspec
import difflib
import s3fs
import seaborn as sns
import plotly

# also supports from imports
from rich.layout import Layout
from rich.live import Live

# duplicates are allowed
import plotly
import plotly

# duplicate names from different libraries are not allowed
import copy
from numpy import copy

Add all the things you would like to be imported automatically, just as you would import them. I went kinda crazy and added over 200 to mine based on packages that I use.

Commas are even supported #

yep all the import styles are supported

This following example will set up auto import for both DataFrame and Series, they will both work separately. I removed these from my config as I felt it was cleaner without, but it works with them.

from pandas import DataFrame, Series

Even imports with a comma will be treated separately.

Jupyter note #

Both work the same, use what your comfortable with

I only really mention ipython here, but the same all applies to Jupyter as well. I just really like ipython itself, c’mon its right there in the terminal integrating with the rest of your terminal experience so well.

Ipython setup #

Automatically import python libraries in ipython with pyflyby

The recommended way to set uppyflybyfrom the docs is to run the following magic command. This works well, but I want even less typing, I want pyflyby automatically installed and importing things without me even thinking about it.

%load_ext pyflyby

Ipython setup next level #

automatically import modules in python without %load_ext

I really want pyflyby to just work in every environment without me thinking much about it. I want it to load automatically, and even to attempt to install itself if it’s missing.

from IPython import get_ipython
import subprocess


ipython = get_ipython()

try:
    ipython.magic("load_ext pyflyby")
except ModuleNotFoundError:
    print("installing pyflyby")
    subprocess.Popen(
        ["pip", "install", "pyflyby"],
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    ).wait()
    ipython.magic("load_ext pyflyby")

Note: if installation fails you will still make it into ipython, there will just be a traceback to the failed command as you enter.

I’ve had zero issues with this, but if there ever comes a time when it does not work in certain environments for you. I’d strongly suggest you to add this to a separate profile.

Ipython-Config

Check out this article for a bit more in depth ipython configuration

ipython auto import examples #

pyflyby can import all the various import types just fine.

import something
from module import something
import something as alias

df = pd.read_csv("https://waylonwalker.com/cars.csv")
[PYFLYBY] import pandas as pd

Getting Help #

Want help on something that you have in your pyflyby config, just give it the ?, ??, or help and pyflyby will import it for you.

Popen?

Just Ask Ipython for help

Autocomplete #

This is next level python auto-import

pyflyby even goes as far as helping tab completion. If you try to tab complete Pop it will complete to Popen without even adding Popen to your local namespace. If you ask for something inside of a module i.e. requests.<tab>, then it will import requests.

# does not populate the namespace
Pop<tab>

# !!does populate the local namespace
requests.<tab>

What happens when a module is not installed #

ModuleNotFoundError

When you are in an environment where you do not have a module installed that is in your pyflyby config, it will throw a ModuleNotFoundError when it tries to import, and it will not import or try to install for you. You will have to change environments or install that module.

❯ pd?
[PYFLYBY] import pandas as pd
[PYFLYBY] Error attempting to 'import pandas as pd': ModuleNotFoundError: No module named 'pandas'
[PYFLYBY] Traceback (most recent call last):
[PYFLYBY]   File "/home/u_walkews/.local/lib/python3.8/site-packages/pyflyby/_autoimp.py", line 1610, in _try_import
[PYFLYBY]     exec_(stmt, scratch_namespace)
[PYFLYBY]   File "<string>", line 1, in <module>
[PYFLYBY] ModuleNotFoundError: No module named 'pandas'
Object `pd` not found.

❯ df = pd.read_csv("https://waylonwalker.com/cars.csv")
<ipython-input-3-69b040434562>:1 in <module>

NameError: name 'pd' is not defined

nvim pyflyby setup #

automatically importing python modules in vim, neovim, nvim

This is by far the best part of this article. It makes development so fluid. It’s not necessarily all about the speed. It really helps you move at the speed of your thoughts, without needing to worry about imports. Remembering where deeply nested modules are does not need to be a thing.

function! s:PyPreSave()
    Black
endfunction

function! s:PyPostSave()
    execute "silent !tidy-imports --black --quiet --replace-star-imports --action REPLACE " . bufname("%")
    execute "e"
endfunction

:command! PyPreSave :call s:PyPreSave()
:command! PyPostSave :call s:PyPostSave()

augroup waylonwalker
    autocmd!
    autocmd bufwritepre *.py execute 'PyPreSave'
    autocmd bufwritepost *.py execute 'PyPostSave'
    autocmd bufwritepost .tmux.conf execute ':!tmux source-file %'
    autocmd bufwritepost .tmux.local.conf execute ':!tmux source-file %'
    autocmd bufwritepost *.vim execute ':source %'
augroup end

refactoring #

This is where it really shines

This setup really shines when you are refactoring. You can freely move modules and classes around without worrying about bringing imports with them. Often when refactoring some modules from one file to another the most tedious part is editing the imports. Often you can’t even grab whole lines because there are several imports and some are needed in both places but not all. pyflyby handles all this like a champ.

Where to install for vim #

just make sure the tidy-imports command is available to vim

pyflyby goes into the environment that you have active at the time that you start neovim. Typically, this is the virtual environment that I am using for the project I am editing.

What gets imported/removed #

only give me what I actually use

Anything within the base config of pyflyby or your own config specified in ~/.pyflyby will get automatically imported if it is used within the file/console. If you are working in a file, and stop using a module, it will automatically get removed.

Anything that is used, and found in the config is added
Anything that is unused gets removed

Where does it put imports #

after the last import

pyflyby does not sort imports into paragraphs or by category. When it needs to add new imports. It will find the last paragraph of imports in your file, add the new one, and sort that paragraph alphabetically.

from collections import Counter

import requests

from plugins.custom_seo import post_render
# <-- pyflyby will put the import here

What about isort #

put those imports where they go

I did not like that I was getting pre-commit issues when using pyflyby, so I added isort to my chain of autocommands to automatically run isort and make my pre-commit happy.

function! s:PyPostSave()
    execute "silent !tidy-imports --black --quiet --replace-star-imports --action REPLACE " . bufname("%")
    execute "silent !isort " . bufname("%")
    execute "e"
endfunction

Let’s write some code

def get():
    """
    Get all the posts from waylonwalker.com.

    Yes theres an rss feed, you should be subscribed if your not already.

    Oh, and we don't need no stinkin error handing because it's always live
    """
    r = requests.get("https://waylonwalker.com/rss.xml")
    return r.content

Save it and pyflyby will inject requests into our file automatically, no need to type that out anymore.

import requests

def get():
    """
    Get all the posts from waylonwalker.com.

    Yes theres an rss feed, you should be subscribed if your not already.

    Oh, and we don't need no stinkin error handing because it's always live
    """
    r = requests.get("https://waylonwalker.com/rss.xml")
    return r.content

What about init / api’s #

careful to fill in the __all__ like you are supposed to

Files such as init.py often import things they do not need, this is simply there for a convenience of the library user and to make the api cleaner. These type of modules should implement a __all__ list of all the unused things that are imported according to pep8. Pyflyby will remove any unused modules unless they are in the __all__ list.

# snippet from kedro.extras.datasets.pandas

__all__ = [
    "CSVDataSet",
    "ExcelDataSet",
    "FeatherDataSet",
    "GBQTableDataSet",
    "ExcelDataSet",
    "AppendableExcelDataSet",
    "HDFDataSet",
    "JSONDataSet",
    "ParquetDataSet",
    "SQLQueryDataSet",
    "SQLTableDataSet",
]

py command #

one liners that need imports

pyflyby also comes with a cli command to run one liners. It’s pretty genius, I’m sure I will find a use or two for it, but so far it’s been more of a novelty for me.

py help pd
py help pd.DataFrame

py pd.read_csv 'https://waylonwalker.com/cars.csv'

Links #

You must use augroup with autocmd in vim | Here's how

2021-12-08T08:43:43Z

If you are running vim autocmd’s without a group, you’re killing your performance. Granted your probably not sourcing your vimscript files with autocmd’s too often, but every time you source that vimscript you are adding another command that needs to run redundantly.

https://youtu.be/2ITTn4Dl0lc

This is what I had #

Not silky smooth

For WAAY too long I have had something like this in my vimrc or init.vim. It formats my python for me on every save, works great except if I source my dotfiles more than once I start adding how many times black runs.

autocmd bufwritepre *.py execute 'Black'

Why is a bare autocmd bad #

let me demonstrate

Lets create a new file called format.vim and give it the :so %. Works great, it starts telling me that its formatting.

autocmd bufwritepre *.py :echo("formatting with black")

BUT as every time I give it the :so % it formats an extra time on every single save.

Setting up an augroup #

I’ve been told I need an augroup to prevent duplicates. So I did it, and nothing changes, it still ran as many times as it was sources on every save.

augroup black
    autocmd bufwritepre *.py :echo("formatting with black")
augroup end

Clearing out the augroup #

What you need to do is clear out all commands in the augroup with autocmd! right at the beginning of the group.

augroup black
    autocmd!
    autocmd bufwritepre *.py :echo("formatting with black")
augroup end

My Final silky smooth setup #

Now this is what I have in my dotfiles for a silky smooth setup that does not run automds like crazy as I am making changes to my dotfiles.

augroup waylonwalker
    autocmd!
    autocmd bufwritepre *.py execute 'PyPreSave'
    autocmd bufwritepost *.py execute 'PyPostSave'
    autocmd bufwritepost .tmux.conf execute ':!tmux source-file %' autocmd bufwritepost .tmux.local.conf execute ':!tmux source-file %'
    autocmd bufwritepost *.vim execute ':source %'
augroup end

vim-help
youtube video for this article

Code Review from the comfort of vim | Diffurcate

2021-12-04T11:34:47Z

I often review Pull requests from the browser as it just makes it so easy to see the diffs and navigate through them, but there comes a time when the diffs get really big and hard to follow. That’s when its time to bring in the comforts of vim.

https://youtu.be/5NKaZFavM0E

Plugins needed #

This all stems from the great plugin by AndrewRadev. It breaks a down into a project. So rather than poping into a pager from git diff, you can pipe to diffurcate and it will setup a project in a tmp directory for you and you can browse this project just like any other except it’s just a diff.

Plug 'AndrewRadev/diffurcate.vim'

My aliases #

First to quickly checkout PR’s from azure devops I have setup an alias to fuzzy select a pr and let the az command do the checkout.

alias azcheckout='az repos pr checkout --id $(az repos pr list --output table | tail -n -2 | fzf | cut -d " " -f1)'

Next I have a few aliases setup for checking diffs. The first one checks what is staged vs the current branch, the others check the current branch vs main or master.

alias diffstaged="git diff --staged | nvim - +Diffurcate '+Telescope find_files'"
alias diffmain="git diff main.. | nvim - +Diffurcate '+Telescope find_files'"
alias diffmaster="git diff master.. | nvim - +Diffurcate '+Telescope find_files'"

diffcommit() {
    git diff $1 | nvim - +Diffurcate '+Telescope find_files'
}

Links #

diffurcte.vim

Setup a yaml schema | yamlls for a silky smooth setup

2021-12-03T23:24:52Z

I’ve gone far too long without a good setup for editing yaml files, I am missing out on autocomplete and proper diagnostics. This ends today as I setup yaml-language-server in neovim.

https://youtu.be/xo4HrFoKF4c

The video for this one is part of a challenge-playlist I put out for myself to constantly improve my dotfiles for all of December.

init.vim #

I have my init.vim setup to only source other modules, if you want everything in a single config, feel free to do as you wish. I broke mine up earlier this year as I doubled into nvim and am not going back.

source ~/.config/nvim/plugins.vim
lua require'waylonwalker.cmp'
lua require'waylonwalker.lsp-config'

Plugin setup #

You will need the following plugins. I use plug, if you don’t you will have to convert the syntax over to the plugin manager you use.

neovim/nvim-lspconfig is for configuring the lsp. It comes with a bunch of sane defaults for most servers, so you pretty much just have to call setup on that server unless you want to change the defaults.

hrsh7th/nvim-cmp is what I use for autocomplete. If you are using something else you might need to set that up in a different way in order to get the autocomplete to work. You will still get the diagnostics with just lsp-config.

kabouzeid/nvim-lspinstall will aide in installing lsp’s if you want. I have chosen not to because I want to have my full setup scripted so when I setup any new machine I just run my ansible-playbook. This library is nice to just set things up quick and play with them.

" /home/u_walkews/.config/nvim/plugins.vim
Plug 'neovim/nvim-lspconfig'

" if you want to use nvim-cmp
Plug 'hrsh7th/nvim-cmp'
Plug 'hrsh7th/cmp-nvim-lsp'

" if you want to use lsp-install
Plug 'kabouzeid/nvim-lspinstall'

cmp config #

Make sure that you have nvim_lsp as a source in your cmp config. This is my config as of now, its likely to change in the future, set yours up how you like. hrsh7th has a really good readme if you want help configuring cmp.

Again if you don’t use cmp you can skip this step, cmp is for autocomplete. You can use a different plugin for autocomplete, or not use a plugin at all if that’s your thing.

--  ~/.config/nvim/lua/waylonwalker/lsp-config.lua
-- Setup nvim-cmp.
local cmp = require'cmp'

cmp.setup({
snippet = {
    expand = function(args)
    -- For `vsnip` user.
    vim.fn["vsnip#anonymous"](args.body)

    -- For `luasnip` user.
    -- require('luasnip').lsp_expand(args.body)

    -- For `ultisnips` user.
    -- vim.fn["UltiSnips#Anon"](args.body)
    end,
},
mapping = {
  ['<C-n>'] = cmp.mapping.select_next_item({ behavior = cmp.SelectBehavior.Insert }),
  ['<C-p>'] = cmp.mapping.select_prev_item({ behavior = cmp.SelectBehavior.Insert }),
  ['<Down>'] = cmp.mapping.select_next_item({ behavior = cmp.SelectBehavior.Select }),
  ['<Up>'] = cmp.mapping.select_prev_item({ behavior = cmp.SelectBehavior.Select }),
  ['<C-d>'] = cmp.mapping.scroll_docs(-4),
  ['<C-f>'] = cmp.mapping.scroll_docs(4),
  ['<C-Space>'] = cmp.mapping.complete(),
  ['<C-e>'] = cmp.mapping.close(),
  ['<CR>'] = cmp.mapping.confirm({
    behavior = cmp.ConfirmBehavior.Replace,
    select = true,
    })
},
sources = {
    { name = 'nvim_lsp' },
    { name = 'vsnip' },
    { name = 'path' },
    { name = 'buffer' },
    { name = 'calc' },
    { name = 'tmux' },
}
})

lsp config #

Next up is the heart of this post, the lsp-config.lua. This one is pretty straight forward, require lspconfig (which you need the plugin for), then set it up with cmp and the extra schemas. I’m sure there are yaml schemas for tons of things, I’ll probably add more in the future, but for now, this is what I have.

--  ~/.config/nvim/lua/waylonwalker/lsp-config.lua
require'lspconfig'.yamlls.setup{
    on_attach=on_attach,
    capabilities = require('cmp_nvim_lsp').update_capabilities(vim.lsp.protocol.make_client_capabilities()),
    settings = {
        yaml = {
            schemas = {
                ["https://raw.githubusercontent.com/kedro-org/kedro/develop/static/jsonschema/kedro-catalog-0.17.json"]= "conf/**/*catalog*",
                ["https://json.schemastore.org/github-workflow.json"] = "/.github/workflows/*"
            }
        }
    }
}

Follow the YouTube channel or the rss-feed to stay up to date.

Also Check out My python lsp setup #

https://waylonwalker.com/setup-pylsp/

Open files FAST from zsh | or bash if thats your thing

2021-12-02T11:34:47Z

https://youtu.be/PQw_is7rQSw

I am often in a set of tmux splits flying back and forth, accidentally close my editor, so when I come back to that split and hit my keybinds to edit files I enter them into zsh rather than into nvim like I intended. Today I am going to sand off that rough edge and get as similar behavior to nvim as I can with a couple of aliases.

Make sure you check out the YouTube video to see all of my improvements.

what’s an alias #

If you have never heard of an alias before it’s essentially a shortcut to a given command. You can pass additional flags to the underlying command and they will get passed in. Most of the time they are just shorter versions of commands that you run often or even like in this case a common muscle memory typo that occurs for you.

My new alias’s for fuzzy editing files from zsh #

Here are the new aliases that I came up with to smooth out my workflow. These give me a similar feel to how these keys work in neovim but from zsh.

# fuzzy select file to edit
alias p='nvim `fzf --preview="bat --color always {}"`'

# give me the same syntax as edit while in neovim
alias :e='nvim '

Follow the YouTube channel or the rss feed to stay up to date.

playlist for my dotfiles challenge

How linux users install a text editor

2021-11-30T23:18:24Z

In honor of the neovim 0.6.0 release, I decided to do a funny skit installing neovim, and fix up my install script in the process as part of my challenge to fix up my dotfiles. I ran into one snag where I was not updating the repo that I cloned. I moved it to the directory I now keep third-party git repos and set it to update with ansible.

https://youtu.be/64oKLphhBuo

The thing that took me the longest to realize was…. I had a path issue pointing me to an old install of the appimage over the fresh build, fixed that up and now we are on 0.7.0 nightly.

https://neovim.io/ https://github.com/neovim/neovim https://github.com/neovim/neovim/releases/tag/v0.6.0

30 days dotfile ricing

2021-11-29T11:34:47Z

https://youtu.be/Jq1Y48F_rOU

I am challenging myself to 30 days of dotfile ricing. I have been on linux desktop for a few months now and have a pretty good workflow going, I have the coarse edits done to my workflow, but it has some rough edges that need sanded down. It’s time to squash some of those little annoyances that still exist in my setup.

This is primarily going to be focused on productivity, but may have a few things to just look better. This will comprise heavily of aliases, zsh, and nvim config.

Follow the YouTube channel or the rss feed to stay up to date.

JUT | Read Notebooks in the Terminal

2021-11-20T10:38:00Z

Trying to read a .ipynb file without starting a jupyter server? jut has you covered.

https://youtu.be/t8AvImnwor0

watch the video version of this post on YouTube

install #

jut is packaged and available on pypi so installing is as easy as pip installing it.

pip install jut

! This is my first time including snippets of the video in the article like this, let me know what you think!

examples #

jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb
jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --head 3
jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --tail 2

what are all the commands available for jut? #

Take a look at the help of the jut cli to explore all the options that it offers.

jut --help

There is some good information on the projects readme as well.

without installing #

using pipx

Don’t want jut cluttering up your venv, or want to save yourself from making a new one, pipx can manage a separate virual environment for you. This is one of the biggest selling points for me.

pipx run jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --head 3

nbconvert #

jut is the lightweight option that I think will fit the bill often for me, but it just doesn’t always cut it. Mostly if there are images in the notebook or large output that is hard to read, its time to pull out the medium guns that sit between fulling running jupyter and jut.

pip install nbconvert

nbconvert does not have its own cli, instead it sits under the jupyter command.

html">converting to html #

Need to see images, go here!

wget https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb
jupyter nbconvert flame_temperature.ipynb --to html
python -m http.server

Note, nb convert does not work with a url, you will need to have the notebook locally.

what other options does nbconvert offer? #

nbconvert also offers a standard help flag that you can access by passing in the --help flag

jupyter nbconvert --help

converting to markdown #

nbconvert also supports converting to many other formats, one option that is quite interesting for use in the terminal is markdown. We could simply convert the notebook to markdown and cat it out.

jupyter nbconvert flame_temperature.ipynb --to maarkdown
cat flameflame_temperature.md

viewing markdown with glow #

Glow is a terminal markdown viewer that looks really good. These days I use bat as cat so I don’t get quite as much benefit from glow, but it still looks pretty good.

glow flameflame_temperature.md

viewing markdown as slides with lookatme #

lookatme

Lookatme is my slideshow tool of choice. Creating slides in markdown is such a fantasic user experience, It realy lets you go from outline to finished slide deck fluidly. Refactoring the whole thing is also so much easier than with gui tools. Once you have your idea fleshed out it does make the process of making slides in powerpoint much easier if thats what you need.

On to nbconvert, without even changing the notebook we can look at the notebook as a slideshow in the terminal. The only thing that it needs is some markdown headers to start new slides from.

lookatme flameflame_temperature.md

viewing markdown with rich #

Bringing this full circle, lets take a look at the converted markdown with rich. Here you will notice a surprising similarity to what we saw with jut.

pip install rich
python -m rich.markdown flame_temperature.md

Rich still cannot pull directly from a url or display markdown with out being installed and managed by yourself. Unlike how jut can leverage pipx to manage the installation sandbox for you.

Links #

jut - View notebooks in the terminal
nbconvert - convert notebooks to other formats
flame-temperature notebook - The sample notebook I used
glow - Terminal Markdown viewer
lookatme - Terminal Markdown slideshow tool
pipx - Run python cli’s without maintianing a virtual environment
rich - Beautiful python terminal formatter

Waylon Walker

2021-11-20T10:38:00Z

Hi, Hello, I’m Waylon #

Husband, dad of two, and hobbyist builder of things on the internet.

When I’m not wrangling data pipeline platforms or building web platforms, you’ll find me gaming with my kids, making art, or skating around the neighborhood. Reliving my mechanical engineering days with my 3d printer. Winding down at the end of the day binge-watching Big Bang Theory with my wife.

What I Do #

I’m a Senior Software Engineer who specializes in data pipelines and Python-based web platforms. I help teams turn messy data into reliable systems that actually work.

Why I Built This Site #

from scratch

I got tired of:

Build times that took forever
Node modules folders that became black holes
Bloated pages that took ages to load
SEO tools that felt like an afterthought

So I built my own platform from scratch using pluggy and diskcache. It’s under-funded, over-dreamed, barely documented, and I love it. This site is my sandbox for learning, teaching, and sharing ideas on my own terms.

Infrastructure #

Because apparently I don’t have enough hobbies, I also run this site from a Kubernetes cluster in my basement. Nothing says “I love DevOps” like maintaining your own bare-metal cluster just to host a static blog.

Explore #

Curious about the tech? Check out about this site for my philosophy on this space, or /colophon for the full stack breakdown.

Meta Pages #

Evergreen pages about me and this site:

kedro catalog create

2021-11-15T07:18:55Z

I use kedro catalog create to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key.

https://youtu.be/_22ELT4kja4

What is Kedro

👆 Unsure what kedro is? Check out this post.

Running Kedro Catalog Create #

The command to ensure there are catalog entries for every dataset in the passed in pipeline.

kedro catalog create --pipeline history_nodes

Create’s new yaml file, if needed
Fills in new dataset entries with the default dataset
Keeps existing datasets untouched
it will reformat your yaml file a bit
- default sorting will be applied
- empty newlines will be removed

CONF_ROOT #

Kedro will respect your CONF_ROOT settings when it creates a new catalog file, or looks for existing catalog files. You can change the location of your configuration files by editing your CONF_ROOT variable in your projects.

settings.py.

# settings.py
# default settings
CONF_ROOT = "conf"

# I like to package my configuration
CONF_ROOT = str(Path(__file__).parent / "conf")

I prefer to keep my configuration packaged inside of my project. This is partly due to how my team operates and deploys pipelines.

File Location #

The kedro catalog create command will look for a yaml file based on the name of the pipeline (CONF_ROOT/catalog/<pipeline-name>.yml). If it does not find one it will create one and make entries for each dataset in the pipeline. It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline. If you are going to use this command its important that you follow this pattern or copy what it generates into your own catalog file of choice.

⚠️ It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline.

MemoryDataSet’s #

When you run kedro catalog create you get MemoryDataSet, that’s it. As of 0.17.4 its hard coded into the library and not configurable.

range12:
  type: MemoryDataSet

Your free to use what you want though #

Let’s switch this dataset over to a pandas.CSVDataSet so that the file gets stored and we can pick up and read the file without re-running the whole pipeline.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv

Continue adding nodes #

As we work we will keep adding nodes to our kedro pipeline, in this case we added another node that created a dataset called range13.

kedro catalog create --pipeline history_nodes

After telling kedro to create new catalog entries for us we will see that it left our range12 entry alone and created range13 for us.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range13:
  type: MemoryDataSet

Formatting is not worthwhile #

If we decide this is too cramped for us we could add some space between datasets. The next time we run kedro catalog create empty lines will be removed.

range12:
  type: pandas.CSVDataSet

range13:
  type: MemoryDataSet

Continuing to work #

If we coninue adding new nodes, and tell kedro to create catalog entries again, all of our effort given to formatting will be lost. I wouldn’t worry about it unless you have an autoformatter that you can run on your yaml files. The productivity gains in an semi-automated catalog are worth it.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range121:
  type: MemoryDataSet
range13:
  type: MemoryDataSet

Sorting Order #

Notice the sorting order in the last entry, range121 comes before range13. This is all based on how pythons yaml.safe_dump works, kedro has set the default_flow_style to False. You can see where they write your file in the source code currently here

nvim conf 2021 | IDE's are slow | Waylon Walker

2021-10-29T00:00:00Z

https://youtu.be/E18m4KkJUnI

Slides 👇 #

welcome #

Other possible titles #

Using Vim as a Team Lead
I 💜 Tmux
Why I stopped using @code
Get there fast
How I vim

It’s ok #

Use a graphical IDE if it works for you.

Trick it out #

vim is so well integrated into the terminal, take advantage

It wasn’t working for me anymore #

dozens of instances #

As a team lead I bounce betweeen a dozen projects a per day

https://pbs.twimg.com/media/FAEmRjYUcAUk2eR?format=jpg&name=large

Move With Intent #

Running vim inside tmux lets me move swiftly between the exact project I need.

https://twitter.com/_WaylonWalker/status/1438849269407047686/photo/1 //: <> (__)

Hub and Spoke #

direct link to specific projects
fuzzy into all projects
fuzzy into open projects

How I navigate tmux in 2021#hub-and-spoke

Other Things That Make this Possible #

tmux
direnv

vim adjacent things

yes, vim is ugly, make it yours #

@rook

command! Q :q

@_waylonwalker

nnoremap <leader>6 <c-^>

lsp #

lua vim.lsp.buf.definition()

treesitter #

Plug 'nvim-treesitter/nvim-treesitter-textobjects'

telescope #

fuzzy matching like a boss

fzf is ok too

Check Messages #

Just Ask Ipython for help

2021-10-10T21:38:26Z

It happens to the best of us #

We can’t all remember every single function signature out there, it’s just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal.

https://youtu.be/TZrRAP-9UMk

Docstrings #

In any python repl you can access the docstring of a function by calling for help.

help(df.rolling)

In Ipython we can even get some syntax highlighting with the ?.

df.rolling?

Source Code #

Sometimes the docstrings are not good enough, and don’t give us the content we need, and we just need to look at the source. Without leaving your terminal there are two ways I often use to get to the source of a function I am trying to use.

import inspect
inspect.getsource(df.rolling)

The more common way I do it is with the ipython ??.

df.rolling??

Bonus rich.inspect #

You thought the syntax highlighting was good with ipython, check out what rich.inspect can do! As far as I know there is no way to get to source code, but the results are still fantastic.

pip install rich

Install rich

from rich import inspect
inspect(cars.rolling, help=True)

then inspect

Connect with me #

If you liked this one, check out the YouTube Channel, catch me live on twitch, or connect on twitter, I’d love to hear from you.

twitter: https://twitter.com/_WaylonWalker twitch: https://www.twitch.tv/waylonwalker github: https://github.com/waylonwalker/ twitch: https://www.twitch.tv/waylonwalker

Hacktoberfest 2021?

2021-10-10T09:43:52Z

Are you participating in Hacktoberfest 2021?

I got my start with open source with Hacktoberfest 2018. I remember being so nervous for those first pr’s. I have continued to participate every year, and it feels good to give back.

It made a big change on me #

Something that it really did for me was to lower the hurdle to contribute regularly. I am not a huge contributor to open source. I am not necessarily a regular maintainer of any large project, but I do often raise issues when I see things as a user that would make things easier or more clear.

After participating it does not feel like such a big deal to go through the motions of making my own fork, making a small change, and submitting it for review.

What can you do? #

As a user, you might be in a unique position to see a library from the perspective of other users rather than from the eyes of a maintainer. You can help bring the user perspective to your favorite library.

https://twitter.com/_WaylonWalker/status/1446440878864846909

My favorite error messages are those that I helped tweak just a bit to make it clear to the user what is happening and how to fix it. As much as we all hate getting errors, It does warm my heart when I see one that I helped make better.

Discuss #

Are you participating this year?

Show us some of your PR’s, and why they make a difference.

Let me know on twitter or in this dev.to discussion.

I made a neovim plugin

2021-10-05T08:04:44Z

I’ve slowly adding more and more lua functions into my neovim configuration, and recently I noticed a pattern for a class of functions that reach out to run shell commands that can be abstracted away.

https://youtu.be/8m5ipBuopPU

Telegraph.nvim #

Check out the project readme for the most up to date details on the plugin itself.

Motivation #

I want a simple way to make remaps into shell commands that can open new tmux windows, popups, or just run a command with context from the editor.

For example I want to make remaps to do things like open the current file in lookatme.

# vim :terminal
nnoremap <leader>s <cmd>Telegraph pipx run lookatme {filepath} --live-reload --style gruvbox-dark<cmd>

# tmux session
nnoremap <leader><leader>s <cmd>lua require'telegraph'.telegraph({cmd='pipx run lookatme {filepath} --live-reload --style gruvbox-dark', how='tmux'})<CR>

# tmux popup
nnoremap <leader><leader>S <cmd>lua require'telegraph'.telegraph({cmd='pipx run lookatme {filepath} --live-reload --style gruvbox-dark', how='tmux_popup'})<CR>

The main goal here is that remaps become one liners that can just be put directly in my init.vim without creating a whole new lua module for each shell command I want to bind.

how #

telegraph takes in a how argument to determine where to tun the command.j

term(default) runs command in the built in terminal
tmux runs command in a new tmux session and joins it.
tmux_popup runs command in a tmux popup window.
tmux_popup_session runs command in a tmux session and displays it in a popup
subprocess runs command in a subprocess

Format strings #

The current set of format strings that can be used with telegraph. Placing these in braces {} within your command will get rendered into something with context from the editor.

cword - the current word under the cursor
cWORD - the current BIG Word under the cursor
cline - the current line under the cursor
filepath - the filepath of the current file
filename - the filename of the current file
parent - the parent directory of the current file
current_session_name - name of the current tmux session
cwd - the current working directory

Give it a ⭐ #

Check out the repo and give it a star if its something that interests you.

Setting Parameters in kedro

2021-09-19T00:00:00Z

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files.

https://youtu.be/Jj5cQ5bqcjg

What is Kedro

👆 Unsure what kedro is? Check out this post.

parameters files #

You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a base and local parameters file.

conf/base/parameters.yml
conf/local/parameters.yml

base #

The base environment should contain all of the default values you want to run.

# /conf/base/parameters.yml
test_size: 0.2
random_state: 3
features:
  - engines
  - passenger_capacity
  - crew
  - d_check_complete
  - moon_clearance_complete
  - iata_approved
  - company_rating
  - review_scores_rating

NOTE base will always be loaded first.

accessing parameters #

Parameters can be accessed through context or through the catalog. Generally when you are working with nodes it will be loaded through the catalog.

Loding with the context.

context = session.load_context()
context.params

Loading with the catalog.

catalog.load('parameters')

Loading a specific key with the catalog.

catalog.load('params:test_size')

using parameters in nodes #

Here is an example from the complete spaceflights demo. The entire parameters dict is passed in, then the features key is accessed.

def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
    """Splits data into features and targets training and test sets.

    Args:
        data: Data containing features and target.
        parameters: Parameters defined in parameters.yml.
    Returns:
        Split data.
    """
    X = data[parameters["features"]]
    y = data["price"]
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=parameters["test_size"], random_state=parameters["random_state"]
    )
    return X_train, X_test, y_train, y_test

Comprehensive guide to creating kedro nodes

👆 Check out this complete guide to creating kedro nodes.

local #

The local parameters by default are in conf/local/parameters.yml. They will override the base parameters in a shallow fashion. If a top level key exists in local, it will override that entire key in your parameters.

# /conf/local/parameters.yml
env: local
features:
  - company_rating
  - review_scores_rating

NOTE If you have not explicitly set your environment, local will be the default environment selected to override base.

env #

You can also have other environments that override the base environment.

# /conf/new/parameters.yml
env: new

NOTE if you use an env local will not be applied

Activating this environment #

Following the configuration docs we can activate the environment by setting an environment variable in our shell or passing in –env to our kedro cli command.

Setting an environment variable.

export KEDRO_ENV=test

Passing in the env to a kedro cli command.

kedro run --env=test

Setting the Environment Variable in python.

import os
os.environ['KEDRO_ENV'] = 'new'

Links #

Writing your first kedro Nodes

2021-09-13T22:40:45Z

https://youtu.be/-gEwU-MrPuA

Before we jump in with anything crazy, let’s make some nodes with some vanilla data structures.

import node #

You will need to import node from kedro.pipeline to start creating nodes.

from kedro.pipeline import node

func #

The func is a callable that will take the inputs and create the outputs.

inputs / outputs #

Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is the catalog entry to use for that keyword.

our first node #

Sometimes in our pipelines our data is coming from an api where we already have python functions built to pull with. Thats ok, kedro supposrts that with inputs=None.

def create_range():
    return range(100)

make_range = node(
    func=create_range,
    inputs=None,
    outputs='range'
    )

second node #

Now we have some data to work from, lets use that as our input.

def square_range():
    return [i**2 for i in range]

square_range = node(
    func=square_range,
    inputs='range',
    outputs='range_squared'
    )

Multiple Inputs #

Kedro can take lists or dicts as either input or output when your function needs more than one input or output.

def concat(range, range_two):
    return [*range, *range_two]

concat_ranges = node(
    func=concat,
    inputs=['range', 'range_squared']
    outputs='concat'
    )

## inputs could also be defined as a dict
concat_ranges = node(
    func=concat,
    inputs={'range': 'range', 'range_two': 'range_squared'}
    outputs='concat'
    )

Links #

Running your Kedro Pipeline from the command line

2021-08-24T22:40:45Z

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli.

https://youtu.be/ZmccpLy-OEI

What is Kedro

👆 Unsure what kedro is? Check out this post.

Kedro run #

To run the whole darn project all we need to do is fire up a terminal, activate our environment, and tell kedro to run.

kedro run

Specific Pipelines #

Running a sub pipeline that we have created is as easy as telling kedro which one we want to run.

kedro run --pipeline dp

Single Nodes #

While developing a node or a small list of nodes in a larger pipeline its handy to be able to run them one at a time. Besides the use case of developing a single node I would not reccomend leaning very heavy on running single nodes, let the DAG do the work of figuring out which nodes to run for you.

kedro run --pipeline dp --node create_model_input_table_node
kedro run --pipeline dp -n create_model_input_table_node

Some DAG concepts #

We will cover more of the benefits that we get from the graph nature of the DAG in the future, but here is a quick peek at some things we can do.

kedro run --pipeline dp --to-outputs preprocessed_shuttles
kedro run --pipeline dp --from-inputs preprocessed_shuttles
kedro run --pipeline dp --to-nodes create_model_input_table_node

Multiple things #

You can stack up multiple kedro dag concepts into a single run command.

kedro run --pipeline dp --to-nodes create_model_input_table_node --to-nodes preprocess_shuttles_node

kedro Virtual Environment

2021-08-23T20:40:45Z

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment.

https://youtu.be/ZSxc5VVCBhM

conda
venv
pipenv

conda #

I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don’t have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

conda info --envs

stores environment in a root directory i.e. ~/miniconda3
conda can use its own way to manage environments environment.yml
the python interpreter is packaged with the environment

virtualenv #

Virtual env (venv) is another very respectable option that is built right into python, and requires no additional installs or using a different distribution of pytyhon.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

environments are typically stored in the project directory
does not package the interpreter

pipenv #

Pipenv is another virtual enviroment tool that comes with its own system for managing dependencies using a pipfile. It’s main benefit is that it creates a lockfile that will allow users to replicate the exact version of all their packages. The typical requirements.txt workflow can easily break as new version of dependecies are released between testing and deplpoyment.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

stores environment in a root directory i.e. ~/.local/share/virtualenvs/
pipenv can use its own way to manage environments pipfile
does not package the interpreter

Kedro Install

2021-08-21T20:40:45Z

Kedro comes with an install command to install and manage all of your projects dependencies.

https://youtu.be/IWimEs-hHQg

cd into your project directory and activate env #

You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment.

Kedro New

this post covers kedro new

kedro Virtual Environment

This post covers creating your virtual environment for kedro

install kedro #

Make sure you have kedro installed in your current environment, if you dont already have it.

pip install kedro==0.17.4

pip-tools #

Kedro uses the pip-tools package under the hood to pin dependencies in a very robust way to ensure that the project will continue to work on everyone’s machine day, including production, day in and day out. No matter what happens to the dependencies you have installed.

pip-compile #

The command that kedro uses from pip-tools is pip-compile. It will look at what you have in a requirements.in file, compile the dependencies down to exact versions, and create a requirements.txt that is fully pinned down, and updatable by re-running pip-compile.

requirements.in #

If kedro does not see a requirements.in file it will automatically move your requirements.txt to requirements.in and run pip-compile.

No requirements.in found. Copying contents from requirements.txt...

kedro install #

Lets go ahead and run kedro install on one of the projects we already create and environment for in a previous post, kedro-conda.

kedro install

The first time you run this on a new repo it is likely that you will run into this warning about creating a new requirements.in file.

No requirements.in found. Copying contents from requirements.txt...

kedro install flags #

Kedro does let you avoid pip-compile all together, by using the --no-build-reqs flag.

kedro install --no-build-reqs

It also lets you upgrade all of your dependencies with build-reqs. I would reccomend doing this on its own branch, and own pull request. If you are working on a team you want everyone to be on the same page when it comes to dependencies. If you are not, you surely do not want something to break with a new set of dependencies without a way of rolling back.

kedro install --build-reqs

Kedro Git Init

2021-08-20T22:40:45Z

Immediately after kedro new, before you start running kedro install or your first line of code the first thing you should always do after getting a new kedro template created is to git init.

https://youtu.be/IGba3ytf_6U

git-init">git init #

Its as simple as these three commands to get started.

git init
git add .
git commit -m init

I don’t care if this project is for learning, if it will never have a remote or not, use git.

Kedro New

2021-08-18T20:40:45Z

https://youtu.be/uqiv5LAiJe0

Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template.

What is Kedro

Unsure what kedro is, Check out yesterdays post on What is Kedro.

pipx #

I reccomend using pipx when running kedro new. pipx is designed for system level cli tools so that you do not need to maintain a virtual environment or worry about version conflicts, pipx manages the environment for you.

The kedro team does not reccomend pipx in their docs as they already feel like there is a bit of a tool overload for folks that may be less familiar with

pipx kedro new

I like using pipx as it gives you better control over using a specific version or always the latest version, unlike when you run what you have on your system depends on when you last installed or upgraded.

Kedro New #

The kedro team also has a set of starters, by passing in --starter you can start with a different template. Here is an example with the kedro spaceflights starter.

pipx run kedro new --starter spaceflights

=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
 [New Kedro Project]: Spaceflights Complete

Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
 [spaceflights-complete]:

Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter
or underscore.
 [spaceflights_complete]:

Change directory to the project generated in /home/u_walkews/git/spaceflights-complete

A best-practice setup includes initialising git and creating a virtual environment before running ``kedro install`` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

Other versions of kedro with pipx #

pipx not only ensures that you run the latest version, it can also run a very specific version.

pipx run --spec kedro==0.16.6 kedro new

kedro Virtual Environment

The next post in this series will help you create your virtual environment for your new kedro project.

What is Kedro

2021-08-17T12:48:00Z

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract DataSets that the user specifies through Catalog entries. These Catalog entries are loaded, ran through a function, and saved by Nodes. The order that these Nodes are executed are determined by the Pipeline, which is a DAG. It’s the runner’s job to manage the execution of the Nodes.

https://youtu.be/Wf4rnFsaFFU

What is Kedro

This is an updated version of my original what-is-kedro article

Hot Take #

If you are doing a series of operations to data with python, especially if you are using something as supported as pandas, you should be using a framework that gives you a pipeline as a DAG and abstracts io.

Orchestrators #

Like I said, kedro is unopinionated it does determine where or how your data should be ran. The kedro team does support the following Orchestrators with very little add on to the base template.

DataSets #

Did I say kedro is unopionated? Datasets are what allow kedro too be so flexible accross a number of different python objects. Any python object can be made into a kedro dataset. Kedro comes out of the box with many purpose built DataSets like storing pandas DataFrames to parquet, csv, or a sql table. If kedro does not come with support for the type of python objects you work with don’t worry, you can for the closest option they support and build your own. Or if you do not want to build your own, you can use a PickleDataSet for anything.

Catalog #

You will not often be creating your own datasets, most of what you need whould already be taken care of by the kedro framework. What you will need to do is to use the existing DataSets to build your data catalog.

Kedro takes care of all fo the file io for you, you simply need to use the catalog to tell kedro what type of DataSet to use and any extra information that DataSet needs. Much of the time this is simply a filepath.

Typically the catalog is specified in yaml format. If you are not familiar with yaml, I suggest learnxinyminutes.com/docs/yaml/ as a resource of examples.

test:
  type: pandas.CSVDataSet
  filepath: s3://your_bucket/test.csv #

Here is the most basic yaml catalog entry taken from the kedro docs

cars:
  type: pandas.CSVDataSet
  filepath: data/01_raw/company/cars.csv
    sep: ','
    load_args:
  save_args:
    index: False
    date_format: '%Y-%m-%d %H:%M'
    decimal: .

Here is a bit more complex example that takes in load_args and save_args docs

Nodes #

Nodes are a very core part of kedro to build the DAG. These nodes are what provides the definition of what catalog entries, get passed into which function, and output to another catalog entry.

import pandas as pd
import numpy as np

def clean_data(cars: pd.DataFrame,
               boats: pd.DataFrame) -> Dict[str, pd.DataFrame]:
    return dict(cars_df=cars.dropna(), boats_df=boats.dropna())

def halve_dataframe(data: pd.DataFrame) -> List[pd.DataFrame]:
    return np.array_split(data, 2)

nodes = [
    node(clean_data,
         inputs=['cars2017', 'boats2017'],
         outputs=dict(cars_df='clean_cars2017',
                      boats_df='clean_boats2017')),
    node(halve_dataframe,
         'clean_cars2017',
         ['train_cars2017', 'test_cars2017']),
    node(halve_dataframe,
         dict(data='clean_boats2017'),
         ['train_boats2017', 'test_boats2017'])
]

Here is an example of three nodes taken from their docs

Pipeline #

The kedro Pipeline, is a DAG (Directed Acyclic Graph). It is a graph object that flows in one direction. You can slice into the pipeline using a few built in graph method to_nodes, from_nodes, to_outputs, and from_inputs. You can chain up these method calls since each one returns a new Pipeline object. You can also ask a pipline for its edges with inputs, and outputs. You can also list every dataset along the way with all_inputs or all_outputs. Lastly you can convert it back into a list of nodes with nodes.

from kedro.pipeline import Pipeline, node

# using our nodes from last tim
Pipeline(nodes)

Runner #

The runner is the bridge between kedro and the orchestrators. The kedro team provides some basic runners for running pipelines locally, built right into the framework, but adding on new runners for different orchestrators is done through the use of adding in a new runner to your project.

Hooks #

Kedro allows you to hook into a number of lifecycle methods through the use of the pluggy framework. Yes the one that pytest is built on. There are a number of different lifecycle methods that allow us to hook in around where kedro is running such as before_pipeline_run or after_catalog_loaded.

Links #

How I Kedro

2021-08-16T22:34:45Z

https://youtu.be/bw5_FWDVRpU

Ubuntu #

I recently switched over to using Ubuntu, it works well pretty much out of the box for me. I am using gnome with a dark theme.

Gnome Terminal #

I am still using the built in default gnome terminal, it just works. It does all the things that I need it to do. It supports transparency renders my fonts and allows me to highlight things well.

One Dark Theme

dotfiles #

You can find my dotfiles on github. Feel free to read through and take anything that you find useful. I would encourage you not to steal them, but to integrate the parts that you want into your own dotfiles. dotfiles are a very personal thing. They are an extension of ones fingertips designed for how you think and type.

zsh #

I use zsh as my default shell. I like to use it as my interactive shell. It works, and does a bit better with things like tab completion out of the box.

starship #

I use the starship prompt for my shell. It works well out of the box. It looks good and includes all of the information that I would ever need.

tmux #

As a team lead I am in and out of many projects per day, tmux allows me to get in and out of these files with super speed. I was using a mix of vscode and tmux in until October 2020. At this point I got moved development machines and pushed myself to use only the terminal. I felt that vscode was just getting slower and slower, and I was getting less benefit from it. Especially now that the lsp is a part of nvim.

neovim #

I really like the raw speed and customizability of neovim. You can see all the customization, and plugins I have added in my dotfiles.

OneBuddy Theme
pylsp
kedro-lsp

ipython #

I really like ipython, it lets me edit code in my code editor, then import it or run it quickly. Ipython gives me the right level of tooling. I don’t need markdown mixed in my code, I put those notes into docstrings, a readme, or wiki. When I need to see plots I just store them as png or html and view them in my browser.

I do a bit of customization to my ipython session that you can find in my dotfiles repo. I use a custom prompt and use rich formatting and tracebacks if rich is installed.

custom prompt
rich traceback

Links #

tmux show-messages

2021-08-14T09:03:09Z

https://youtu.be/LLk94fKpGg4

As we push the limits of tmux further and further you are bound to end up in a situation where you are mashing down a hotkey and it’s just not doing what you want it to do, and you have no idea why.

show-messages is a tmux command that can be used to show what tmux is actually doing behind the scenes. This might highlight any hot key conflicts you might have in your ~/.tmux.conf.

man page for show-messages #

In case you wnat more information about show-messages, here is the man page.

show-messages [-JT] [-t target-client]
            (alias: showmsgs)

        Show server messages or information.  Messages are stored, up to a
        maximum of the limit set by the message-limit server option.  -J and -T
        show debugging information about jobs and terminals.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux ta

2021-08-13T09:03:09Z

https://youtu.be/nT0FA1RNZEs

Now your creating, jumping, and killing sessions like a boss. You are slicing through projects with ease, let me show you one more thing that can be the cream on top of this silky smooth setup we have been working towards.

Chris Toomey’s Tmux Course #

This script is simply my fork of Chris Toomey’s tat script straight out of his course. It helps us create or jump to project specific sessions with ease.

a directory of projects #

My version of the ta script will let you pass it a directory, and it will give you a fuzzy popup.

ta ~/git

setting up a keybinding #

bind C-g display-popup -E "ta ~/git"

default layout #

By default I have my projects open with a vertical split, vim is on top, with my file finder open and the lower split is set to just my terminal. This is what I do 90% of the time that I open a project anyways.

More projects #

I also have a directory setup that is a symlink-gallery of all of my projects, both private and public. This makes it easy to have one key that lets me see all of my projects.

rm -rf ~/projects
mkdir ~/projects
ln -sf ~/work/* ~/projects
ln -sf ~/git/* ~/projects

https://waylonwalker.com/symlink-gallery/

This post covers how I combine all my projects into a single directory.

https://waylonwalker.com/tmux-nav-2021/

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux display-message

2021-08-12T09:03:09Z

https://youtu.be/utfLA6L8o5s

You’ve got some long running tasks, and you’re trying to stay productive and knock tasks off that board, but you keep finding that your processes finish and you stay on other tasks for longer than you should. You were in the flow and just did not check back in on it. With display-message you can send yourself a notification when that long running task is complete.

from the man page #

Here is a snippet of display-message from the tmux man page. I rarely need to do anything other than just display message, but there are other flags for it.

display-message [-aINpv] [-c target-client] [-d delay] [-t target-pane] [message]
            (alias: display)

        Display a message.  If -p is given, the output is printed to stdout,
        otherwise it is displayed in the target-client status line for up to

        delay milliseconds.  If delay is not given, the message-time option is
        used; a delay of zero waits for a key press.  ‘N’ ignores key presses
        and closes only after the delay expires.  The format of message is
        described in the FORMATS section;

        information is taken from target-pane if -t is given, otherwise the
        active pane.

        -v prints verbose logging as the format is parsed and -a lists the
        format variables and their values.

        -I forwards any input read from stdin to the empty pane given by
        target-pane.

notifier #

With display-message we can do things like setup notifications that will work cross platform.

cmatrix -t 5 && tmux display-message done

Without setting the target-pane display-message defaults to the active pane. This is a very handy feature that allows us to switch windows, or sessions and still recieve the message.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

killing tmux

2021-08-11T09:03:09Z

https://youtu.be/QWPyYx54JbE

Now it’s time to switch gears, we are onto a different part of our day and there are just too many sessions running and we need to clean up shop.

kill-server #

One viable option is to nuke the whole dang thing. I actually do this more than you might think.

tmux kill-server

save and commit your work diligently before kill-server

kill-session #

A more reasonable option might be to kill a single session.

# kills the current session
tmux kill-session

# kills the session named scratch
tmux kill-session -t scratch

choose-tree #

Killing sessions one by one from the command line can be a bit tedious, and involve more keystrokes than necessary. Another option built right into tmux is choose-tree. By default choose-tree is bound to prefix+s, that’s pressing control+b then s. Once you are in choose-tree, you can navigate around with your configured navigation scheme, press x to kill a session, or pane or window then y to confirm. You can also batch delete by tagging items with t, and killing them all at once with X.

tmux choose-tree

check out this post for a bit more information on choose-tree

fuzzy matcher #

Here is my preferred way, using a fuzzy matcher. I recently improved this one by making it a popup and cleaning it up based on a repsonse, tmux-output-formatting by DJ Adams. I press prefix+k to bring up a kill-session menu.

bind k display-popup -E "\
    tmux list-sessions -F '#{?session_attached,,#{session_name}}' |\
    fzf --reverse -m --header=kill-session |\
    xargs -I {} tmux kill-session -t {}"

note this is setup to multiple sessions all at once.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux has-session

2021-08-09T09:03:09Z

https://youtu.be/XucVVgGmesM

I see you there, trying to script out your tmux layouts. Tryig to get each project setup just perfect with a script, but you keep stumbling over yourself with duplicate session error messages

The has-session tmux command is a handy tool to prevent this duplicate session error message when scripting your tmux layouts.

command line #

The command is pretty straight forward, you simply ask tmux if the session name you are looking for exists.

tmux has-session -t "waylonwalker_com"

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux list-keys

2021-08-08T09:03:09Z

https://youtu.be/Y1MYmL8ZolE

Tmux list keys can be a useful tool to help remind you of what kebindings you have setup. You can search for them and scroll just like in tmux copy-mode.

command line #

You can call list-keys from the command line but the interface is not very usable by itself. It might be nice to mix with grep or a pager in some circumstances.

tmux list-keys

tmux command line #

Running list-keys from within the tmux command line puts you into a much more pleasant copy-mode.

list-keys

default keybinging #

By default tmux comes with list-keys bound to prefix+?.

bind-key          ? list-keys

list-keys man page #

You can see the additional flags provided by tmux in the man page for list-keys.

list-keys [-1aN] [-P prefix-string -T key-table] [key]
            (alias: lsk)

        List key bindings.  There are two forms: the default lists keys as
        bind-key commands; -N lists only keys with attached notes and shows
        only the ke y and note for each key.

        With the default form, all key tables are listed by default.  -T lists only keys in key-table.

        With the -N form, only keys in the root and prefix key tables are
        listed by default; -T also lists only keys in key-table.  -P specifies
        a prefix to print before each key and -1 lists only the first matching
        key.  -a lists the command for keys that do not have a note rather than
        skipping them.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux status-bar

2021-08-07T09:03:09Z

https://youtu.be/mvgM8UH11_U

The tmux status bar can be a handy tool to remind yourself where you are within tmux. It can also include a bunch of system information like battery status, cpu, mem, whatever you can get from the command line. Honestly I like to keep it minimal, and actually keep it turned off most of the time. I find that it helps a little bit for others to follow along if I keep it on in certain circumstances.

show the status bar #

You can set a hotkey to show or hide the status bar.

bind s set-option -g status
bind C-s set-option -g status

setting the background transparent #

I really want a minimal status bar with very little bling, I want it to get out of the way an not draw too much attention, so step one is to set the background to transparent.

# default statusbar colors
#――――――――――――――――――――――――――――――――
set-option -g status-bg default
set-option -g status-fg colour240

setting default colors #

I want my status bar to somewhat match the rest of my theme, so I set the default foreground as magenta and the default background as transparent.

# default window title colors
#―――――――――――――――――――――――――――――――
set-window-option -g window-status-style fg=magenta
set-window-option -g window-status-style bg=default

my status bar #

Honestly I set this up quite awhile ago, and it does everything I need it to for now. It shows me the current session that I am in on the left and lists out the windows for the session in the middle.

set -g status-left-length 85
set -g status-left "working on#[fg=colour135] #S"
set -g window-status-current-format "#[fg=black,bold bg=default]│#[fg=white bg=cyan]#W#[fg=black,bold bg=default]│"
set -g window-status-current-format "#[fg=black,bold bg=default]│#[fg=colour135 bg=black]#W#[fg=black,bold bg=default]│"
set -g status-style bg=default
set -g status-right "#[fg=magenta] #[bg=gray] %b %d %Y %l:%M %p"
set -g status-right '#(gitmux "#{pane_current_path}")'
set -g status-justify centre

For more format options search for FORMATS in the tmux manpage.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux source-file

2021-08-06T09:03:09Z

https://youtu.be/dDq0depPrTs

So you have been tricking out that .tmux.conf, you’re looking for a silky smooth workflow that lets you fly through tmux with super speed, but every time you tweak out that .tmux.conf you have to restart your whole session. Not amymore,

Let’s add this to the bottom of our tmux.conf so that you can see everytime it gets sourced.

display-message "hello beautiful"

command #

We can run this command from your shell to re-source your changed .tmux.conf

tmux source-file ~/.tmux.conf

It also works from the tmux command line.

source-file ~/.tmux.conf

tmux hotkey #

It’s very common to set this up as a keybinding so that you can do it easily without needing to memorize the exact command.

bind -T prefix r source-file ~/.tmux.conf
bind -n M-r source-file ~/.tmux.conf

from vim #

This is my preferred way of re-sourcing my .tmux.conf. It sits quietly in the background, and I dont need to remember to do anything. If you are a vim user you can automate this process by creating a autocmd bufwritepost. This will shell out the tmux source-file everytime you save your .tmux.conf.

autocmd bufwritepost .tmux.conf execute ':!tmux source-file %'
autocmd bufwritepost .tmux.local.conf execute ':!tmux source-file %'

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux fzf session jumper

2021-08-05T09:03:09Z

https://youtu.be/DkJ9rb85LC0

Quickly getting between tmux splits is critical skill for productivity. You can get by with next or prev session for awhile, but if you have more than about three session you need something a bit more targeted.

Full Screen selector #

I have used this fzf one keybinding for quite awhile, honestly I did not make it up, and cannot remember where it came from. It will open up a session picker in a new full screen window.

bind C-j new-window -n "session-switcher" "\
    tmux list-sessions -F '#{?session_attached,,#{session_name}}' |\
    sed '/^$/d' |\
    fzf --reverse --header jump-to-session --preview 'tmux capture-pane -pt {}'  |\
    xargs tmux switch-client -t"

Like with many of my keybindings I have swapped this one out for a popup version. It just feels so smooth.

bind C-j display-popup -E "\
    tmux list-sessions -F '#{?session_attached,,#{session_name}}' |\
    sed '/^$/d' |\
    fzf --reverse --header jump-to-session --preview 'tmux capture-pane -pt {}'  |\
    xargs tmux switch-client -t"

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux next/prev session

2021-08-04T09:03:09Z

https://youtu.be/8kZnjHPYnKU

Now that we are splitting up work into their own sessions, lets talk about how to navigate between them without the command line. Navigating sessions is what kept me using a too many splits and windows workflow for far too long. It was rough, I was always tripping over panes and windows until I got too frustrated and just ran tmux kill-server to start on a new blank slate.

bind -n M-P switch-client -p
bind -n M-N switch-client -n

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux start application

2021-08-03T23:51:21Z

https://youtu.be/c-a2Bnv_NJ0

Scripting tmux to open up specific applications can be intimidating your first time. It can be tricky to get it to start in the right directory. If you are trying to assign applictaions to a keybinding it can be easy to mess up and have weird things happen every time your ~/.tmux.conf gets sourced.

Open htop in an above split #

I used this one for a number of years to take a quick peek into my systems performance while a memory intensive task was running.

bind -n M-t split-window htop \; swap-pane -U

🗒️ note that the swap-pane -U will make the htop split active immediately

With the new tmux popup windows I really like the flow of just peeking at htop in a popup and jumping back into what I was doing. It can have a more consistennt look, and not mess with the window layouts.

bind -n M-t popup -E -h 95% -w 95% -x 100% "htop"

Open an applicaiton in the current directory #

One thing that can be tricky is getting apps that need to be in a specific directory started in the directory that you want. Here are two examples I use to open vifm or gitui.

bind -n M-e split-window -c '#{pane_current_path}' vifm . .\; resize-pane -Z;
bind C-k split-window -c '#{pane_current_path}' 'gitui'\; resize-pane -Z;

🗒️ note that split-window takes in a -c flag before the application you want to run to specify the startup directory.

I’ve been converted over to using popups for these as well. I’ll admit that the workflow of creating a new full screen window may have been better, but this can be a bit less jarring, espessially if you have anyone following along with what you are doing.

bind -n M-e display-popup -d '#{pane_current_path}' -E vifm
bind C-k display-popup -d '#{pane_current_path}' -E 'gitui'

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux rename session

2021-08-03T23:51:21Z

https://youtu.be/WRLRiQDjVIA

So you have been working on your tmux workflow, you’ve dropped a too many window workflow for scoping work that belongs together into separate sessions, but you cannot remember what session your work is in. If your diligent you have named your window when you created it, but sometimes its intent has changed or your were just plain too lazy at the time for the extra characters needed to name it. Don’t worry we can still give that session a descriptive name.

Let’s rename some sessions in the terminal.

# rename the current session to me
tmux rename-session me

# rename the me session to scratch
tmux rename-session -t me scratch

There is a default keybinding that you can use <prefix>+$ to rename the current session in the tmux command line.

bind-key          $ command-prompt -I #S "rename-session '%%'"

I’ve also had this keybinding kicking around for years, but I rarely use it anymore. You will see why in an upcoming video.

bind -n M-W command-prompt "rename-session '%%'"

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux targeted session

2021-08-02T23:51:21Z

https://youtu.be/5KE7Il7SOEk

This is something that I made up but use every single day, this is what keeps much of what is on my blog or my teams private work wiki going. I have a few very important directories that I have assigned directly to a hotkey for fast session switching.

bind -n M-i new-session -A -s waylonwalker_com "cd ~/git/waylonwalker.com/ && nvim"
bind i popup -E -h 95% -w 95% -x 100% "tmux new-session -A -s waylonwalker_com 'cd ~/git/waylonwalker.com/ && nvim'"
bind -n M-I popup -E "tmux new-session -A -s waylonwalker_com 'cd ~/git/waylonwalker.com/ && nvim'"

tmux new-session

This one is building off of yeserday’s new-session post, make sure you check that one out as well.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux new-session

2021-08-01T23:51:21Z

https://youtu.be/LbQNdCAUogE

This one starts a new chapter in our series that is going to open up a whole new set of workflow productivity options, understanding how the new-session command is a critical command in our adventure into tmux glory. This is going to open the door for some seriously game changing hotkeys and scripting.

# create a new session
tmux new-session

# create a new session detached
tmux new-session -d


# create a new session and name it
tmux new-session -s me

# create a new named session and attach to it if one exists
tmux new-session -As me

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux detach

2021-08-01T23:51:21Z

https://youtu.be/A1qx3tNKDdA

tmux detach is a handy tmux command that will quit your current session while keeping it running. The full name of the comamnd is detach-client, detach is a shorthand.

default keybinding

bind-key          d detach-client

I have mine bound to mod+d where mod is alt.

bind -n M-d detach-client

https://waylonwalker.com/tmux-nav-2021/

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux attach

2021-07-31T23:51:21Z

https://youtu.be/JQ0yDCVu44E

attach is one of the most useful features of tmux. If you have no interest in tmux for pane and window management, you should use tmux for this. It can be a life saver if you ever get disconnected from the host machine or accidently close your terminal you can connect right back into the session you were just in using attach.

attach #

tmux attach

this command will simply attach back to tmux if you are ever disconnected

If you ever run long running tasks on a remote machine by sshing into this you should be doing it inside tmux, or something like tmux so that you do not loose your work.

attach to a specific session #

If you have multiple sessions running you can select the session that you want to attach to by passing -t <name-of-session>.

tmux attach -t scratch

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

Git in Depth Notes

2021-07-31T08:44:44Z

These are my notes from taking @nnja’s FEM course git-in-depth.

requirements #

git --version > than 2.0

git">creating a git #

echo "hello" | git hash-object --stdin

tmux ls

2021-07-30T23:51:21Z

https://youtu.be/LY41GLn_DGg

tmux ls will list the sessions that you have running within the tmux server if tmux is currently running. This is handy to combine with commands such as attach.

tmux ls

tmux attach

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux command line

2021-07-29T23:51:21Z

https://youtu.be/SNu-4IrkjAs

So far we have covered a lot of tmux commands and how they map to keybindings but these same commands can be executed at the command line.

From the command line #

Let’s make a popup that displays our git status for 5s or until we close it manually. We can run the following command at the command line, in a split.

tmux display-popup -E -d '#{pane_current_path}' 'git status && sleep 5'

From the tmux command line #

Or we can open the tmux command line and run it from tmux’s built in command line, which is very similar to bim EX mode. By default the tmux command line can be opened with prefix+[.

display-popup -E -d '#{pane_current_path}' 'git status && sleep 5'

🗒️ note that the tmux command is called by default when inside of tmux.

Make it a keybinding #

Finally we can make it a keybinding by adding a bind command ahead of our tmux command, then we can execute this in the tmux command line or add it to our ~/.tmux.conf.

bind s display-popup -E -d '#{pane_current_path}' 'git status && sleep 5'

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux copy-mode

2021-07-28T23:51:21Z

https://youtu.be/-ypY_-VmBKk

tmux copy-mode is a tmux mode that lets you scroll, search, copy, and jump your way through a pane. There are a ton of keybindings for copy-mode, the main ones you will need to know are / for searching down ? for searching up, n for next item, space for starting a selection, and enter to copy the selection. Arrow keys will be used for navigation unless you have specified vi mode, then it will be hjkl.

Default keybinding to get into copy mode is prefix+[.

bind-key          [ copy-mode

If you are a vim user you will likely want to use vi style keys, add this to your ~/.tmux.conf file to enable vi mode.

setw -g mode-keys vi

full list of copy-mode keybindings from the man page.

  Command                                      vi              emacs
           append-selection
           append-selection-and-cancel                  A
           back-to-indentation                          ^               M-m
           begin-selection                              Space           C-Space
           bottom-line                                  L
           cancel                                       q               Escape
           clear-selection                              Escape          C-g
           copy-end-of-line [<prefix>]                  D               C-k
           copy-line [<prefix>]
           copy-pipe [<command>] [<prefix>]
           copy-pipe-no-clear [<command>] [<prefix>]
           copy-pipe-and-cancel [<command>] [<prefix>]
           copy-selection [<prefix>]
           copy-selection-no-clear [<prefix>]
           copy-selection-and-cancel [<prefix>]         Enter           M-w
           cursor-down                                  j               Down
           cursor-down-and-cancel
           cursor-left                                  h               Left
           cursor-right                                 l               Right
           cursor-up                                    k               Up
           end-of-line                                  $               C-e
           goto-line <line>                             :               g
           halfpage-down                                C-d             M-Down
           halfpage-down-and-cancel
           halfpage-up                                  C-u             M-Up
           history-bottom                               G               M->
           history-top                                  g               M-<
           jump-again                                   ;               ;
           jump-backward <to>                           F               F
           jump-forward <to>                            f               f
           jump-reverse                                 ,               ,
           jump-to-backward <to>                        T
           jump-to-forward <to>                         t
           jump-to-mark                                 M-x             M-x
           middle-line                                  M               M-r
           next-matching-bracket                        %               M-C-f
           next-paragraph                               }               M-}
           next-space                                   W
           next-space-end                               E
           next-word                                    w
           next-word-end                                e               M-f
           other-end                                    o
           page-down                                    C-f             PageDown
           page-down-and-cancel
           page-up                                      C-b             PageUp
           pipe [<command>] [<prefix>]
           pipe-no-clear [<command>] [<prefix>]
           pipe-and-cancel [<command>] [<prefix>]
           previous-matching-bracket                                    M-C-b
           previous-paragraph                           {               M-{
           previous-space                               B
           previous-word                                b               M-b
           rectangle-on
           rectangle-off
           rectangle-toggle                             v               R
           refresh-from-pane                            r               r
           scroll-down                                  C-e             C-Down
           scroll-down-and-cancel
           scroll-up                                    C-y             C-Up
           search-again                                 n               n
           search-backward <for>                        ?
           search-backward-incremental <for>                            C-r
           search-backward-text <for>
           search-forward <for>                         /
           search-forward-incremental <for>                             C-s
           search-forward-text <for>
           search-reverse                               N               N
           select-line                                  V
           select-word
           set-mark                                     X               X
           start-of-line                                0               C-a
           stop-selection
           top-line                                     H               M-R

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux join-pane

2021-07-27T23:51:21Z

https://youtu.be/Vm5rRtcVXLw

Join-pane allows you to join panes that you have broken away from your window, or created in a different window to the window you want it in. As far as I know there is not a default keybinding for it.

Before you can join a pane you must first have a pane marked to join. Once you mark a pane, go back to the window you want to join it to and join-pane.

My keybindings, you must add this to your ~/.tmux.conf file to use them.

# Mark and swap panes
#――――――――――――――――――――――――――――――――――――――――――――
bind -n M-m select-pane -m # mark
bind -n M-M select-pane -M # unmark

bind -n M-< join-pane

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

Also check out the full YouTube tmux-playlist to see all of the videos in this series.

tmux break-pane

2021-07-26T23:51:21Z

https://youtu.be/ICL609F2xnc

Break-pane is a handy tmux command when your layout gets too cramped and you want to just move a split into its own window. Calling break-pane does exactly that, it creates a new-window for you and moves your currently selected split into that window

Default key binding for break-pane

bind-key          ! break-pane

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux zoom

2021-07-25T23:51:21Z

https://youtu.be/Rn6mOarCQ-Y

Zooming into the current split in tmux is a valuable tool to give yourself some screen real estate. These days I am almost always presenting, streaming, or pairing up with a co-worker over a video call. Since I am always sharing my screen I am generally zoomed in to a level that is just a bit uncomfortable, so anytime I make a split it is really uncomfortable, being able to zoom into the split I am focused on is a big help, and also help anyone watching follow where I am currently working.

Default key bindings for zooming the current split

bind-key          z resize-pane -Z

I have rebound this to match the default binding with mod+z rather so that I get that single keystroke experience.

bind -n M-z resize-pane -Z

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux new-window

2021-07-24T23:51:21Z

https://youtu.be/YRPZBv-iYyE

New window as it sounds makes new windows in tmux. Windows are kind of like tabs. They are another screen within your sessions that you can name and make new panes in.

Default key bindings for creating and navigating windows in tmux.

bind-key          c new-window
bind-key          p previous-window
bind-key          n next-window

As always I have rebound these keys because I generally prefer a single keystroke over the prefix plus keybinding approach that tmux gives by default.

#――windows――――――――――――――――――――――――――――――――――――――
bind -n M-c new-window -c '#{pane_current_path}'
bind -n M-p previous-window
bind -n M-n next-window

When I started using tmux I did almost everything in one giant session with many panes and windows. It became a nightmare to manage and quickly get between two sets work efficiently. This year I leaned in on sessions quite heavily. Checkout this 👇 post to see that workflow in depth.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux slect-pane

2021-07-23T23:51:21Z

https://youtu.be/CPZJZjN9YTY

These are my MOST often used keybindings that I use in tmux. They allow me to jump between splits with ease with a vim style layout. I can hold mod and jump between panes with a familiar arrow key.

bind -n M-h select-pane -L
bind -n M-l select-pane -R
bind -n M-k select-pane -U
bind -n M-j select-pane -D

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux rotate-window

2021-07-22T23:51:21Z

https://youtu.be/06z5qf81ofo

Rotate window is the main way that I navigated tmux before I learned select-pane. It allows you to change your focused pane, or rotate the position of the panes easily.

Default keybindings

bind-key        C-o rotate-window
bind-key          o select-pane -t :.+

My keybindings look just a bit different than the default ones, I do not like needing to hit prefix for every command, especially for repeated commands. I set a similar keybinding to the default one that uses mod instead of prefix.

bind -n M-o select-pane -t :.+
bind -n M-O rotate-window

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux select-layout

2021-07-21T23:51:21Z

https://youtu.be/F0mHnwTrNNc

When you get many splits going in tmux sometimes its time for a new layout. There are four layout strategies that I use, main-vertical, main-horizontal, even-vertical, even-horizontal. Almost always I am useing the main ones with mod plus a or mod plus shift a keybindings.

# Select Layouts
#―――――――――――――――――
bind -n M-a select-layout main-vertical
bind -n M-A select-layout main-horizontal
bind -n M-E select-layout even-vertical
bind -n M-V select-layout even-horizontal

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux resize-panes

2021-07-20T23:51:21Z

https://youtu.be/hpFYE2LU7xc

Resizing panes in tmux can be quite difficult in default tmux, I use a set of keybingings to help resize panes in the rare occasions that I do need just a bit more space. I set the keybinding to the same as my split navigation bindings but shifted. They are very vim like (h,j,k,l).

# resize panes
#―――――――――――――――――――――――――――――
bind -n M-H resize-pane -L 2
bind -n M-L resize-pane -R 2
bind -n M-K resize-pane -U 2
bind -n M-J resize-pane -D 2

Most often when I need to resize panes I just grab the edge of the pane with my mouse. Yes the mouse, its not that often that I actually need to change the size of a pane.

# Enable mouse control (clickable windows, panes, resizable panes)
set -g mouse on

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux choose-tree

2021-07-19T23:51:21Z

https://youtu.be/79Y-kqAiMpw

Choose tree is a powerful tmux utility that provides a graphical interface to preview all sessions, windows, and panes, move between them kill them, move them and much more.

The default keybinding

bind-key -T prefix s choose-tree -s

my preferred keybinding to open sessions and windows collapsed and Zoomed in.

bind-key j choose-tree -swZ

From the man page.

choose-tree [-GNrswZ] [-F format] [-f filter] [-K key-format] [-O sort-order] [-t target-pane] [template]
        Put a pane into tree mode, where a session, window or pane may be chosen interactively from a tree.  Each session, window or pane is shown on
        one line.  A shortcut key is shown on the left in brackets allowing for immediate choice, or the tree may be navigated and an item chosen or
        otherwise manipulated using the keys below.  -s starts with sessions collapsed and -w with windows collapsed.  -Z zooms the pane.  The follow‐
        ing keys may be used in tree mode:

            Key    Function
            Enter  Choose selected item
            Up     Select previous item
            Down   Select next item
            +      Expand selected item
            -      Collapse selected item
            M-+    Expand all items
            M--    Collapse all items
            x      Kill selected item
            X      Kill tagged items
            <      Scroll list of previews left
            >      Scroll list of previews right
            C-s    Search by name
            m      Set the marked pane
            M      Clear the marked pane
            n      Repeat last search
            t      Toggle if item is tagged
            T      Tag no items
            C-t    Tag all items
            :      Run a command for each tagged item
            f      Enter a format to filter items
            H      Jump to the starting pane
            O      Change sort field
            r      Reverse sort order
            v      Toggle preview
            q      Exit mode

        After a session, window or pane is chosen, ‘%%’ is replaced by the target in template and the result executed as a command.  If template is
        not given, "switch-client -t '%%'" is used.

        -O specifies the initial sort field: one of ‘index’, ‘name’, or ‘time’.  -r reverses the sort order.  -f specifies an initial filter: the fil‐
        ter is a format - if it evaluates to zero, the item in the list is not shown, otherwise it is shown.  If a filter would lead to an empty list,
        it is ignored.  -F specifies the format for each item in the tree and -K a format for each shortcut key; both are evaluated once for each
        line.  -N starts without the preview.  -G includes all sessions in any session groups in the tree rather than only the first.  This command
        works only if at least one client is attached.

https://waylonwalker.com/tmux-nav-2021/

for more information on how I navigate tmux, check out this full post

tmux prefix

2021-07-18T23:51:21Z

https://youtu.be/BMkpbfhbkKM

The prefix key is an essential part of tmux, by default all of tmux’s key-bindings sit behind a prefix. This prefix is very similar to vim’s leader key. It is common for folks to change the default C-b (control b) to C-a or if they are a vim user something to match their vim leader key.

set -g prefix C-Space
bind Space send-prefix

A few of the essential default key-bindings.

%      vertical split
"      horizontal split
d      detach

up     select up one pane
down   select down one pane
right  select right one pane
left   select left one pane

t      clock
o      swap panes
c      create window
n      next window
p      previous window

A more complete list of key-bindings can be found in this gist https://gist.github.com/mzmonsour/8791835.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux splitting panes

2021-07-17T23:51:21Z

https://youtu.be/kzgyiHap1nQ

splitting panes is a core feature of tmux. It allows us to split the terminal vertically or horizontally into new panes.

bind -n M-s split-window -c '#{pane_current_path}'
bind -n M-v split-window -h -c '#{pane_current_path}'
bind -n M-X kill-pane

🗒️ note that ‘#{pane_current_path}‘will keep the split in the same directory as it’s parent, without this it will default to your home directory.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux last session

2021-07-16T23:51:21Z

https://youtu.be/RB87EEnnMnU

An ultimate productivity key-binding in tmux is one to switch to the last session. I use this to quickly get between sessions really quick. Often I am working and need to lookup a quick note, or copy something into my notes, then get back to where I was quickly.

bind -n M-b switch-client -l

I think of this hub and spoke model, and use last-session to quickly drive it.

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux floating popups

2021-07-15T23:51:21Z

https://youtu.be/2ZqFDsJywt8

Tmux popups are actually floating windows that you can drag around the screen. They always open in the middle (by default) when you open them, no matter where you leave them.

Here are a couple of keybindings I use to open up popup windows.

bind C-g display-popup -E "ta ~/git"
bind -n M-g display-popup -E "tmux new-session -A -s scratch"

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

tmux popups

2021-07-14T23:51:21Z

https://youtu.be/2I8fB28zfB4

Tmux-popups are a great feature that is relatively new to tmux, many repos such as the standard ubuntu repos do not have it. Popups came in 3.2a, if your package manager does not have it, you can follow the tmux’s install instructions to build from source.

# open a popup scratchpad
bind -n M-g display-popup -E "tmux new-session -A -s scratch"

How I navigate tmux in 2021

for more information on how I navigate tmux, check out this full post

I use popups quite a bit in my workflow to ssh into another machine for a short period, or make a new project with a template.

Incremental Versioned Datasets in Kedro

2021-07-08T11:12:25Z

Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work.

What is Kedro

👆 Unsure what kedro is? Check out this post.

How does our dataset change over time?? #

This was a question presented to me at work. We had some plots being produces as the output of our pipeline and the user wanted the ability to compare results over time. Luckily this was asked early in the project so we were able to proactively setup versioning on the right datasets.

To enable this all we needed to do now was to add versioned: true and we will be able to compare results over time. Yes kedro makes it that easy to setup.

set up a project #

Set up a new project just as usual. note I like using pipx for global cli packages. You can pick a specific version of kedro or opt for the latest while simply globally installing kedro and running kedro new is purely dependent on the last time you chose to update kedro.

pip install pipx
pipx run kedro new

cd versioned-partitioned-kedro-example
conda create -n versioned-partitioned-kedro-example python=3.8  -y
conda activate versioned-partitioned-kedro-example

pip install kedro
kedro install

git init
git add .
git commit -m "init project from pipx run kedro new"

I called my project versioned-partitioned-kedro-example. You can call your project whatever you like. If you try to use some special characters where they don’t belong, kedro will catch you. Under the hood, kedro is using a library called cookiecutter

⚠️ Please do not skip out on using a virtual environment. You may use whichever virtual environment tool you prefer, but please do not skip out. Wrecking a running project for learning is not fun.

update dependencies #

I popped open my dependencies, added kedro[pandas] and find-kedro. Since those are extra packages our example will require.

aiohttp
black==21.5b1
find-kedro
flake8>=3.7.9, <4.0
ipython
isort~=5.0
jupyter_client>=5.1, <7.0
jupyterlab~=3.0
jupyter~=1.0
kedro-telemetry~=0.1.0
kedro==0.17.4
kedro[pandas]
nbstripout~=0.4
pytest-cov~=2.5
pytest-mock>=1.7.1, <2.0
pytest~=6.2
requests
wheel>=0.35, <0.37

note I created find-kedro, and I like using it to create my pipeline object. Think of how pytest automatically picks up everything named test, find-kedro does the same thing for kedro. It picks up everything with node or pipeline in the name and creates pipelines out of it.

Install new dependencies #

After adding our additional dependencies to the requirements.in, we can tell kedro to install everything and compile the dependencies. Behind the scenes --build-reqs uses a library called pip-compile to create a requirements.txt file with hard pinned dependencies, which is ideal for creating reproducible projects. You and your future colleagues may not thank you for this, but they sure as heck won’t be cursing your name when they can’t get the project to run.

kedro install --build-reqs

git add .
git commit -m "added additional dependencies"

create a node #

For this example, we need a node to do much. This node will pass the cars.csv from a URL to a parquet file. I am going to use a lambda to build my identity function inline.

# pipelines/cars_nodes.py

from kedro.pipeline import node

nodes = []

nodes.append(
        node(
            func=lambda x:x,
            inputs='raw_cars',
            outputs='int_cars',
            name='create_int_cars',
            )
        )

🗒️ notefind-kedro' will automatically pick up these nodes for us after we set up our pipeline_registry.py`.

git add .
git commit -m "add create_int_cars node"

implement find-kedro #

Next, we need to tell kedro where our nodes are. This is where find-kedro comes in. Once we point to the directory where our modules of nodes/pipelines are, it creates the pipelines dictionary for us automatically. It will even separate each module into a pipeline and stitch them all into one default pipeline.

# pipeline_registry.py

"""Project pipelines."""
from typing import Dict
from pathlib import Path

from kedro.pipeline import Pipeline

from find_kedro import find_kedro


def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    Returns:
        A mapping from a pipeline name to a "Pipeline "object.
    """
    pipeline_dir = Path(__file__).parent / 'pipelines'
    return find_kedro(directory= pipeline_dir)

🗒️ This is very similar to the default ` pipeline_registry’except the last two lines.

git add .
git commit -m "implement find-kedro"

create a baseline catalog #

Once we have a pipeline setup, the kedro cli can automatically fill in missing catalog entries with MemoryDataSet’s. Thus, using the cli helps consistently scaffold the catalog and ensure we don’t end up with a typo in our dataset name.

kedro catalog create --pipeline cars_nodes

Kedro will kick out the following catalog file to base/catalog/cars_nodes.yml for us to get started with.

raw_cars:
  type: MemoryDataSet
int_cars:
  type: MemoryDataSet

🔥 use the kedro cli to fill in any missing datasets from the automatically catalog.

make a versioned dataset #

Kedro has scaffolded MemoryDataSet ’s for us. We will convert them to the appropriate dataset type and turn on versioning for our int layer, which is the first point we save in our environment.

raw_cars:
  type: pandas.CSVDataSet
  filepath: https://waylonwalker.com/cars.csv
int_cars:
  type: pandas.ParquetDataSet
  filepath: data/int_cars.parquet
  versioned: true

Commit your changes to the catalog.

git add .
git commit -m "create catalog"

run the pipeline #

Once we have the nodes and catalog setup, we can run the pipeline a few times to get some versioned data. Each time we run, it will save a new version inside the int_cars.parquet directory.

kedro run
kedro run
kedro run
kedro run
kedro run

🗒️ we put our data in the data directory. By default, this directory is included in the .gitignore and will not be picked up by git.

inspect the data #

Listing the files in data/int_cars.parquet shows that I now have five different datasets available. I can load old ones, but by default, kedro will load the latest one.

ls data/int_cars.parquet

2021-07-05T15.24.53.164Z
2021-07-05T15.29.56.144Z
2021-07-05T15.30.23.101Z
2021-07-05T15.30.26.555Z
2021-07-05T15.31.12.688Z

🗒️ kedro sets the version at the timestamp that the session starts. All datasets created within the same run will have the same version.

stack on an incremental dataset #

This is where things get interesting. Kedro comes with an incremental dataset that will load all of the files from a particular directory into a dictionary where the keys are the filename of the dataset. To load up all datasets into this dictionary all we need to do is add a new catalog entry that is a type: PartitionedDataSet, with a path pointing to the same place as the original, and a dataset type the same as the original.

int_cars_partitioned:
  type: PartitionedDataSet
  dataset: pandas.ParquetDataSet
  path: data/int_cars.parquet

catalog list #

Listing the catalog entries confirms that we have successfully added our new PartitionedDataSet.

In [17]: context.catalog.list()
Out[17]:
['raw_cars',
 'int_cars',
 'int_cars_partitioned',
 'parameters']

loading an incremental dataset #

Now we can easily load the datasets from every run we just did into a single dictionary, simply by running context.catalog.load('int_cars_incremental').

In [19]: context.catalog.load('int_cars_incremental')
2021-07-05 11:32:40,534 - kedro.io.data_catalog - INFO - Loading data from `int_cars_incremental` (IncrementalDataSet)...
Out[19]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.30.23.101Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.30.26.555Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.31.12.688Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2}

👆 notice that incremental datasets are all loaded for you, its a dict of filepath:dataset

stack on a partitioned dataset #

Let’s take a look at a similar type of dataset called PartitionedDataSet. We can add it to the catalog in a very similar way to how we added the IncrementalDataSet.

int_cars_incremental:
  type: IncrementalDataSet
  dataset: pandas.ParquetDataSet
  path: data/int_cars.parquet

loading a partitioned dataset #

Note that we get a dict with the same keys as before, but this time the values are a load function rather than loaded data. Partitioned datasets can be helpful if you are operating on datasets that take up more memory than you have available. In our case of coupling this with versioned datasets, its likely to grow quite large, so PartitionedDataSet ’s are likely a better option for this use.

In [18]: context.catalog.load('int_cars_partitioned')
2021-07-05 11:31:11,253 - kedro.io.data_catalog - INFO - Loading data from `int_cars_partitioned` (PartitionedDataSet)...
Out[18]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570820>>,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570850>>,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570910>>,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb15709a0>>}

incremental vs. partitioned #

IncrementalDataSet ’s and PartitionedDataSet ’s are very similar as they give you access to a whole directory of data that uses the same underlying dataset loader. The significant difference is whether you want your data pre-loaded or if you want to load and dispose of it as you iterate over it.

incremental loads the data
partitioned give a load function

creating nodes with partitioned datasets #

Let’s create a node with this PartitionedDataSet to collect stats on our dataset over time. This node does a dict comprehension to get the length of each version that we pulled.

def timeseries_partitioned(cars: Dict):
    return {k:len(car()) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_partitioned,
            inputs='int_cars_partitioned',
            outputs='int_cars_timeseries_partitioned',
            name='create_int_cars_timeseries_partitioned',
            )
        )

🗒️ note that inside of the dict comprehension car is a load function that we need to call.

creating nodes with incremental datasets #

Doing the same node with our IncrementalDataSet looks very similar, except this time car is loaded data inside of the dict comprehension, not a function that we need to call.

def timeseries_incremental(cars: Dict):
    return {k:len(car) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_incremental,
            inputs='int_cars_incremental',
            outputs='int_cars_timeseries_incremental',
            name='create_int_cars_timeseries_incremental',
            )
        )

More catalog entries #

After adding those nodes, we can add the catalog entries again with the command line. This will not overwrite any of the datasets we just created. It will only add to it.

kedro catalog create --pipeline cars_nodes

int_cars_timeseries_partitioned:
  type: MemoryDataSet
int_cars_timeseries_incremental:
  type: MemoryDataSet

int_cars_timeseries_partitioned:
  type: pickle.PickleDataSet
  filepath: data/int_cars_timeseries_partitioned.parquet
int_cars_timeseries_incremental:
  type: pickle.PickleDataSet
  filepath: data/int_cars_timeseries_incremental.parquet

Loading the new datasets #

Loading the two dtasets that we just created show that we have the ended up with the same result using both incremental and partitioned datasets. This result is a dictionary of filepaths mapped to the size of the dataset. Since the default filepaths are timestamps we could start doing some time series analysis to see how our dataset is changing over time.

In [32]: context.catalog.load('int_cars_timeseries_incremental')
2021-07-05 12:00:55,014 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_incremental` (PickleDataSet)...
Out[32]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
 '2021-07-05T16.43.43.088Z/int_cars.parquet': 32}

In [33]: context.catalog.load('int_cars_timeseries_partitioned')
2021-07-05 12:01:03,223 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_partitioned` (PickleDataSet)...
Out[33]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
 '2021-07-05T16.43.43.088Z/int_cars.parquet': 32,
 '2021-07-05T16.50.46.686Z/int_cars.parquet': 32}

Kedro - My Data Is Not A Table

☝️ I have a full article on creating datasets that are not tabular datasets using pickle.

cross posted to dev #

If you like this article give it some ❤️🦄🏷️ on dev https://dev.to/waylonwalker/incremental-versioned-datasets-in-kedro-3ajn

This post was primarily built live on https://twitch.tv/waylonwalker, give me a follow and join in the live show if that is something that interests you.

I Started Streaming on Twitch

2021-06-12T10:11:00Z

I recently started streaming on twitch.tv/waylonwalker and it’s been a blast so far.

python
kedro
Data Science
Data Engineering
webdev
digital gardening

Kedro Spaceflights #

It all started with kedro/issues/606, Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me.

Inspiration #

My introduction to twitch came from twitch.tv/theprimeagen. I watched him on YouTube, and then decided to drop into a stream. It was so fun to watch him live that I started following others in the science and tech category.

twitch.tv/teej_dv Brilliant neovim core dev, I learn a bunch about nvim every time I watch.
twitch.tv/cmgriffing Super Chill and engaging chat.
twitch.tv/cassidoo Fantastic discussion/chat.
twitch.tv/anthonywritescode Building the python ecosystem.
twitch.tv/kierisi Exploring data in r-studio, taking xgboost to Sliced.

Streaming What I Do #

After taking a jump in with the spaceflights tutorial I realized that it really was not that hard to do, and I can stream things that I am doing anyways. I will be streaming while I build open source kedro plugins, pipelines, and while I work on my website.

YouTube #

I started storing all the replay, and some clips on YouTube. If that is more of your thing be sure to check it out. To be completely honest the editing/ clips will likely come at a minimum, as they take me a lot more time to edit. I would greatly appreciate anyone who subscribes to help me get to the 100 subscriber baseline to get my custom url.

Come join me while I try to figure out what I am doing.

Deploying a kedro dataset to netlify

Here is a small clip from my latest

Kedro Spaceflights - part 2 | Stream replay June 7, 2021

2021-06-07T16:15:04Z

This was my seconf time ever streaming on twitch.tv/waylonwalker, and I completely botched my mic 2x.

https://youtu.be/_7MwgKu-844

Links #

Notes to get started #

pipx run kedro new
cd project
python -m venv .venv
source .venv/bin/activate
pip install kedro
kedro install

🌱 Digital Gardening | gif to Mp4 | Stream replay June 4, 2021

2021-06-06T16:15:04Z

https://youtu.be/I4VenHqIEng

Doing some Digital Gardening on stream

Ahrefs Errors
ahrefs large images
Automatic gif to mp4

gif to mp4 #

After this stream all gifs on my site are converted to mp4/webm if they exist.

![tmux-navigation-2021](https://images.waylonwalker.com/tmux-navigation-2021.gif)

Kedro Spaceflights - part 1 | Stream replay June 4, 2021

2021-06-04T16:15:04Z

This was my first time ever streaming on twitch.tv/waylonwalker. I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go.

https://youtu.be/Y07UBr9Ccjs

Kedro Spaceflights #

Notes #

pipx run kedro new
cd project
python -m venv .venv
source .venv/bin/activate
pip install kedro
kedro install

Using Kedro In Scripts

2021-05-23T11:00:44Z

With the latest releases of kedro 0.17.x, it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects.

New to Kedro #

What is Kedro

If your just learning about kedro check out this post walking through it

No More Rabbit Hole of Errors #

as of 0.17.2

I’ve tried to do this in kedro 0.16.x, and it turned into a rabbit hole of errors. First kedro needed a conf directory, if you tried to fake one in it would then ask for logging setup. These errors just kept coming to the point it wasnt worth doing and I might as well use a proper template for real projects and stick to simple function calls for things that are not a kedro project.

Kedro in a script #

To get kedro running, you will need a pipeline, catalog, and runner at a minimum. Those who have used kedro before the pipeline will look very similar to what you are familiar with, but the catalog will not be loaded from yaml and you will need to bring your own runner.

from kedro.pipeline import Pipeline, node
from kedro.io import DataCatalog
from kedro.runner.sequential_runner import SequentialRunner


# additional datasets you want to use
from kedro.extras.datasets.pandas.csv_dataset import CSVDataSet
from kedro.extras.datasets.pandas.parquet_dataset import ParquetDataSet

# the sequential runner is the simplest. It runs one node at a time.
runner = SequentialRunner()

# this is a super simple example pipeline
pipeline = Pipeline(
    [
        node(lambda: range(100), None, "range"),
        node(lambda x: [i ** 2 for i in x], "range", "range**2"),
        node(lambda x: [i for i in x if i > 5000], "range**2", "range>5k"),
        node(lambda x: x[:5], "range>5k", "range>5k-head"),
        node(lambda x: sum(x) / len(x), "range>5k", "range>5k-mean"),
    ]
)

# to get up and running, you can use an empty catalog
catalog = DataCatalog()

runner.run(pipeline, catalog)

👆 Above is the minimal setup to get a kedro pipeline running

more practically #

More often, your kedro pipelines are going to use a function rather than a lambda, and pandas DataFrames.

def clean_columns(df: pd.DataFrame):
    df.columns = [col.lower().strip() for col in df.columns]

pipeline = Pipeline(
    [
        node(clean_columns, "raw_data", "clean_columns", name="create_clean_columns"),
    ]
)

catalog = DataCatalog(
    {
        "raw_data": ParquetDataSet(filepath=f"data/raw_data.parquet")
        "clean_columns": ParquetDataSet(filepath=f"data/clean_columns.parquet")
    }
)

One single node pipeline to get you started

Semi-automatic catalog #

For some reason, when I tried to use the DataCatalogWithDefault it did not pick up my datasets right. I suspect this has something to do with not setting up a proper session, so this is what I did in a pinch to get that catalog goodness for my DataFrames without setting up each one manually.

catalog = DataCatalog(
    {
        name: ParquetDataSet(filepath=f"data/{name}.parquet")
        for name in pipeline.all_outputs()
    }
)

⚠ If all of your datasets are pandas dataframes

For the example above that does not use DataFrames, I would pickle all of my outputs to enable re-loading them later.

catalog = DataCatalog(
    {
        name: PickleDataSet(filepath=f"data/{name}.pkl")
        for name in pipeline.all_outputs()
    }
)

🔥 for use with non-pandas datasets

Logging #

Once you explicitly add datasets, kedro will start logging when it’s loading, running, or saving each node. Things will begin to look a bit more familiar to anyone who has used kedro before.

ww3 ↪main ©kedro-in-scripts v3.8.8 ipython
❯ runner.run(pipeline, catalog)
2021-04-18 09:30:58,099 - kedro.pipeline.node - INFO - Running node: <lambda>(None) -> [range]
2021-04-18 09:30:58,100 - kedro.io.data_catalog - INFO - Saving data to `range` (PickleDataSet)...
2021-04-18 09:30:58,104 - kedro.runner.sequential_runner - INFO - Completed 1 out of 5 tasks
2021-04-18 09:30:58,105 - kedro.io.data_catalog - INFO - Loading data from `range` (PickleDataSet)...
2021-04-18 09:30:58,105 - kedro.pipeline.node - INFO - Running node: <lambda>([range]) -> [range**2]
2021-04-18 09:30:58,105 - kedro.io.data_catalog - INFO - Saving data to `range**2` (PickleDataSet)...
2021-04-18 09:30:58,111 - kedro.runner.sequential_runner - INFO - Completed 2 out of 5 tasks
2021-04-18 09:30:58,111 - kedro.io.data_catalog - INFO - Loading data from `range**2` (PickleDataSet)...
2021-04-18 09:30:58,112 - kedro.pipeline.node - INFO - Running node: <lambda>([range**2]) -> [range>5k]
2021-04-18 09:30:58,112 - kedro.io.data_catalog - INFO - Saving data to `range>5k` (PickleDataSet)...
2021-04-18 09:30:58,115 - kedro.runner.sequential_runner - INFO - Completed 3 out of 5 tasks
2021-04-18 09:30:58,115 - kedro.io.data_catalog - INFO - Loading data from `range>5k` (PickleDataSet)...
2021-04-18 09:30:58,115 - kedro.pipeline.node - INFO - Running node: <lambda>([range>5k]) -> [range>5k-mean]
2021-04-18 09:30:58,115 - kedro.io.data_catalog - INFO - Saving data to `range>5k-mean` (PickleDataSet)...
2021-04-18 09:30:58,118 - kedro.runner.sequential_runner - INFO - Completed 4 out of 5 tasks
2021-04-18 09:30:58,119 - kedro.io.data_catalog - INFO - Loading data from `range>5k` (PickleDataSet)...
2021-04-18 09:30:58,119 - kedro.pipeline.node - INFO - Running node: <lambda>([range>5k]) -> [range>5k-head]
2021-04-18 09:30:58,119 - kedro.io.data_catalog - INFO - Saving data to `range>5k-head` (PickleDataSet)...
2021-04-18 09:30:58,122 - kedro.runner.sequential_runner - INFO - Completed 5 out of 5 tasks
2021-04-18 09:30:58,122 - kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully.

Kedro Viz #

I was not able to quickly get kedro viz up and running for my use case. If you really wanted to you could start modifying their format_pipelines_data function in server.py. Or you could render a new template and put your pipeline there for viz purposes.

It’s possible, but might as well stick to the template

cli #

For something that I would be using this on, I am probably not going to put much effort into the cli as it is not likely something that we will have a team of developers interacting with constantly. I would just put together the minimum necessary to run my application how I need.

if __name__ == "__main__":
    import sys

    if '--skip-raw' in sys.argv:
        runner.run(pipeline.from_inputs('range**2'), catalog)
    else:
        runner.run(pipeline, catalog)

Keeping it simple

If I want to go down the route of having a full cli built out I am probably going to use the full kedro template, or something very similar.

It’s a bit Rough #

While I might use this in production somewhere, it will be inside of some other not kedro application. I will still be using something quite similar to their template for my pipelining projects. It misses some excellent things that bring me to kedro like hooks, plugins, credentials, catalog, logging config, cli, and viz.

Silence Kedro Logs

2021-05-20T13:13:38Z

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output.

Silence a Python log #

First, how does one silence a python log? Python loggers can be retrieved by the logging module’s getLogger function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways.

python logging levels #

Level	Numeric value
CRITICAL	50
ERROR	40
WARNING	30
INFO	20
DEBUG	10
NOTSET	0

Get or Create a logger #

Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in.

logger = logging.getLogger('kedro')

🔥 If a logger doesn’t exist under the passed in name, it will create one for you.

Set Level #

Once we get the logger, we need to silence it by setting the log level. Typically it’s not appropriate to completely turn off loggers as you would still want information in the case of a complete failure. If you are building a cli such as one that prints out the pipelines to the console, you may not want to see logs that happen during regular operation as this would make it more challenging to integrate with other shell applications.

logger.setLevel(logging.ERROR)

⚠ Be sure to leave some logging left. After the point of error, you are not going to get a clean output anyways. So let the user see what happened.

It is possible to set the log level before kedro even registers the logger, if there is no logger currently setup under getLogger, it will create one.

Silent all kedro loggers #

As of kedro==0.17.3 this function covers every logger issued by kedro. I generated this list of known_kedro_loggers by looking through their codebase and filling in a few others I found by running it.

def silent_loggers() -> None:
    """All logs need to be silent in order for a clean kedro diff output."""
    known_kedro_loggers = [
        "ProfileTimeTransformer",
        "hooks_handler",
        "kedro.__init__",
        "kedro",
        "kedro.config",
        "kedro.config.config",
        "kedro.extras.decorators.memory_profiler",
        "kedro.framework.cli",
        "kedro.framework.session.session",
        "kedro.framework.session.store",
        "kedro.framework.session",
        "kedro.io.cached_dataset",
        "kedro.io.data_catalog",
        "kedro.io",
        "kedro.journal",
        "kedro.pipeline",
        "kedro.pipeline.decorators",
        "kedro.pipeline.node",
        "kedro.pipeline.pipeline",
        "kedro.runner",
        "kedro.runner.runner",
        "kedro.versioning.journal",
        "py4",
    ]
    for logger in [
        *known_kedro_loggers,
        *list(logging.root.manager.loggerDict.keys()),  # type: ignore
    ]:
        logging.getLogger(logger).setLevel(logging.ERROR)

This function comes right from a plugin I am currently working on kedro-diff. Check it out, give it a star, and watch it for release.

What is Kedro

Not familiar with kedro, check out this article to see what it’s all about.

Master the log #

Python logs can seem super confusing at first, understanding how to get a logger and set its level are the first steps to mastering it.

Python Diskcahe is locked

2021-05-14T18:38:45Z

Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale.

Minimum reproducible error #

Minimum reporducible error is one of my superpowers in development. I do this very often to sus out what is really happening. My day to day work is processing data with python, I keep a number of very small data sets handy to break and fix. This helps separate complexities of the project and the problem.

Let’s break it #

Markata has a lot going on. It’s a plugins all the way down static site generator built in python. Trying to find the root cause through the layers of plugin and cli modules can be a pain, but in this case building a very simple minimum reporducible error was much easier.

from pathlib import Path
from diskcache import FanoutCache
import time

CACHE_DIR = Path(".") / ".markata.cache"

if __name__ == "__main__":

    cahe = FanoutCache(CACHE_DIR, statistics=True)
    item = cache.get("me")
    print(item)
    time.sleep(20)

📝 time.sleep(20) is here to simulate doing some other work, while the cache object is still open.

Running locked_diskcache.py #

Here I have my editor showing the file on the top split and have ran it in the two lower splits. Notice that the first (left split) immediately prints out the result, while the second one (right split) does not print out the result until the first is completely finished.

Sorry, your browser doesn't support embedded videos.

Using a context manager #

This time lets put the cache in a context manager so that it automatically closes after it gets the item.

from pathlib import Path
from diskcache import FanoutCache
import time

CACHE_DIR = Path(".") / ".markata.cache"

if __name__ == "__main__":

    with FanoutCache(CACHE_DIR, statistics=True) as cache:
        item = cache.get("me")
    print(item)
    time.sleep(20)

Running unlocked_diskcahce.py #

Notice when we run this time when I run both splits, they are able to immediatly print out their result and get to work on that sleep statement. This time the right right split only takes ~20s rather than ~40s since it no longer needs to wait for the left one to unlock the cache.

Sorry, your browser doesn't support embedded videos.

I’ve learned #

Keep your diskcache open for as little as needed, especially if you plan to have it open on multiple processes. Hopefully this problem solving session helps someone else with their discache problems or find a better way to problem solve with minimum viable errors.

Useful Links #

How I navigate tmux in 2021

2021-05-11T09:14:07Z

In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into logical groups.

Update #

Since making this post, I have made ~20 other posts in short form that all have a YouTube video to go along with them you can find them all on my tmux-playlist.

Chris Toomey’s Tmux Course #

I took Chris’s tmux course in December 2020 and it was fantastic. Even as a seasoned tmux user, I learned quite a bit. Before the course, I was proficient in navigating within each of my tmux sessions but rarely started more than one session. A few months later, I have adopted a lot of what I learned from Chris and made it my own.

I am now keeping projects to their own session and can move between them fluidly with just a few keystrokes. For high-traffic projects, I have them bound to a single keystroke for instant switching. This change has been a game-changer from the mess of windows I used to have and the nightmare it was to find work I was doing and end up duplicating project work in two separate windows.

📝 NOTE: Some of my config comes straight from the course, and some of it has been extended to my liking.

Let’s take a quick look at how I am navigating through tmux on a day-to-day basis.

Sorry, your browser doesn't support embedded videos.

👆 Overview of how I navigate tmux

tmux ls #

Throughout this article, I have several recordings showing how I use manage sessions with my keybindings. I will often run a tmux ls command to highlight running sessions at various points to help guide the viewer.

ta #

my attach/session switch script

At the heart of my tmux navigation is a highly customized version of Chris’s tat script that I renamed ta. Many folks add this to their bashrc alias ta=tmux attach. Simply calling ta will do the same thing as shown below. If you’re in a tmux session, it does nothing, and if you’re not in one, it will attach you to the first one.

get the full script from GitHub.

Sorry, your browser doesn't support embedded videos.

👆 attaching to a session by default

In my ~/.bashrc or ~/.zshrc I add the ta command to keep myself in a tmux session at all times. Whenever I open my terminal, I am automatically dropped into a tmux session, but if I am opening a split while in tmux it’s smart enough to know not to nest tmux sessions.

ta

Another article can dive into my ta command. This one is more about the methodology, workflow, and keybindings to get me there. It’s available in my script.

but there’s more #

gettin fuzzy

Give it a directory, and a fzy dropdown will let you choose a subdirectory to start your session in, and name the session after that directory.

ta ~/git

🔥 Bonus, use direnv to automatically set settings, echo your git status, activate your environment or whatever else you need.

Sorry, your browser doesn't support embedded videos.

👆 give it a directory, it will ask for input to which project and start a new named session in that directory.

Note that starting from outside currently does not start in a split layout like it does when starting from within tmux. I am still playing with this, but generally, I want my terminal session to be plain when I first start my terminal. I usually am starting work after the first default session.

🤔 I still use both fzy and fzf. It probably doesn’t make sense to use both, but I am currently giving fzy a try.

prefix+w #

tmux choose-tree

By default, tmux comes with a tmux choose-tree command bound to prefix+w, which opens in full screen. The upper section of the screen will show every window opened. While selected, you can show the splits in each window by hitting l, or fold it with h. You can search for a session name by hitting /.

Sorry, your browser doesn't support embedded videos.

# ~/.tmux.conf

# expanded to show all splits
bind s choose-tree
# simpler window to show only sessions
bind S choose-session

Keybindings in choose-tree/choose-session #

The default keybindings of the tmux choose-tree and choose-session that I use are listed below. J/K are very intuitive, but I just learned about h,l,/. When I do use one of these, the / (search) can be super helpful to find sessions/windows faster.

action	key
fold	h
unfold	l
up	k
down	j
search	/

prefix+c-w prefix+c-g #

open a project

I have set up to make it easy to open my non-work projects (in my ~/git directory) and my work projects (in my ~/work directory). I bound prefix+c-g and prefix+c-w to open a new session in their respective directories. I like mapping common prefix commands with control to keep my pinky mashed on that control key.

# ~/.tmux.conf

bind C-w new-window -n "work-session-picker" "ta ~/work"
bind C-g new-window -n "git-session-picker" "ta ~/git"

Sorry, your browser doesn't support embedded videos.

prefix+c-j #

jump to session

Now that I have ta rocking with a good create or attach setup, I am rarely toggling through a list of running sessions, but I am doing it with prefix+c-j when I do it. Keeping my finger on control and pressing <space>+j. This keybinding uses fzf to fuzzy match to an existing session and attach.

bind C-j new-window -n "session-switcher" "tmux list-sessions | sed -E 's/:.*$//' | grep -v \"^$(tmux display-message -p '#S')\$\" | fzf --reverse | xargs tmux switch-client -t"

Sorry, your browser doesn't support embedded videos.

M-N M-P #

next/prev

Next and Previous sessions. This is super handy when working with under 3 sessions to be able to cycle through sessions holding shift+alt and pressing n or p.

Sorry, your browser doesn't support embedded videos.

tkill #

time to clean up

It’s easy to get a long crufty list of sessions running throughout the day. Typically this is not too bad on system resources compared to running vscode in every working project, but it does make it more challenging to manage and wade through the sessions list. I use a handy shell alias that’s been in my zshrc for quite some time.

alias tkill="for s in \$(tmux list-sessions | awk '{print \$1}' | rg ':' -r '' | fzy); do tmux kill-session -t \$s; done;"

I don’t have this one set up with a nice hotkey, but it works for my fingers. I often pop open a lower split(M-s), run tkill, and close (M-x).

Sorry, your browser doesn't support embedded videos.

Last Session #

back

While M-n and M-p work well with a small, focused number of sessions, I often end up with too many sessions open, and it’s not efficient to remember a double M-N followed by a triple M-P to get back and forth. Most often, I want to get between two sessions quickly, no matter what the order is.

bind -n M-B switch-client -l
bind -n M-b switch-client -l

Sorry, your browser doesn't support embedded videos.

Once I get two sessions back to back, I can switch between them with insane speed and precision.

More Precision #

one keystroke

The final layer of precision is for my most current project. I need to get to these with a single keystroke. These are bound to a set of keybindings that were readily available, just above the home row.

bind C-t new-session -A -s todo "cd ~/work/todo && nvim -O backlog.md doing.md done.md"
bind -n M-i new-session -A -s ww3 "cd ~/git/ww3/ && nvim"
bind -n M-o new-session -A -s images_waylonwalker_com "cd ~/git/images.waylonwalker.com/ && nvim"

Sorry, your browser doesn't support embedded videos.

These few directories are always at my fingertips, encouraging me to keep better notes

And yes, I did steal this last one from Harpoon-man By The Way.

Hub and Spoke #

M-i M-b

I have really been digging this hub and spoke workflow where I am rocking away on a project hit M-I, take some notes then hit M-b to get back to where I was.

Model of my current workflow

Example #

Here is an example of how I use the hub and spoke model to get to notes on my blog and back to my project quickly.

Sorry, your browser doesn't support embedded videos.

Example workflow #

open tmux session with ta
prefix+c-g start work in a project using a fuzzy matcher
M-t over to my todo list
M-b back to my project
M-i to my blog to look up notes/make notes
M-b back to my project
prefix+c-g start work in another project using a fuzzy matcher
M-t over to my todo list
M-b back to my project
prefix+c-j fuzzy back to the first project
M-b back to the second project

Please let me know your thoughts. @waylonwalker, this one took me a bit longer to put together with all of the animated gif’s, but I think it helps visually show how I navigate tmux every day.

If you liked it, give it a share and tag me on twitter. I don’t often ask but this article took a bit more to put together than my normal post.

Trim unused git branches

2021-05-07T09:47:24Z

Trim branches no longer on origin #

git remote prune origin --dry-run
git remote prune origin

Find branches already merged #

git checkout main
# list remote branches that have already been merged into main
git branch -r --merged
# list local branches that have already been merged into main
git branch --merged

What is if name == "main_", and how do I use it.

2021-05-07T09:47:24Z

When a python module is called it is assigned the __name__ of __main__ otherwise if it’s imported it will be assigned the __name__ of the module.

Concrete example #

Let’s create a module to play with __name__ a bit. We will call this module nodes.py. It is a module that we may want to run by it’self or import and use in other modules.

#!python
# nodes.py

if __name__ == "nodes":
    import sys
    import __main__

    print(f"you have imported me {__name__} from {sys.modules['__main__'].__file__}")

if __name__ == "__main__":
    print("you are running me as main")

I have set this module up to execute one of two if statements based on whether the module it’self is being ran or if the module is being imported.

Note it is not common to have a if __name__ == "nodes": block, this is just for demnonstration purposes.

running python nodes.py #

Running a python script with the command python <filename.py> will execute your script top to bottom.

python nodes.py

This will print out you are running me as main

Sorry, your browser doesn't support embedded videos.

How to Install miniconda on linux (from the command line only)

If you don’t already have python installed try using miniconda or replit.com

running ./nodes.py #

You can also simply execute the script from bash if you first set the module to be executable.

chmod +x nodes.py
./nodes.py

Sorry, your browser doesn't support embedded videos.

Note once you have set the file to be executable, it will remain executable chmod +x nodes.py is only needed one time, even if you edit the file.

pipeline.py #

Let’s create a second module pipeline.py and import the first module nodes and see what happens.

#!python
# pipeline.py
import nodes

Just like nodes we can run pipeline either way if it’s executable

python pipeline.py
# must run chmod +x pipeline.py first.
./pipeline.py

Either way it will print out you have imported me nodes from ./pipeline.py

Sorry, your browser doesn't support embedded videos.

REPL #

If we were to import nodes from the repl we would see an error in this case, due to the fact that there is no __main__ file since it’s a repl session.

Use Cases #

The main use case for if __name__ == "__main__": is flexibility. Simply importing a module should not execute any code, print anything to the screen, change your filesystem, or generally have any side effects in most cases. It is something that most python users would not expect. We can use this block to make it such that the module can be both imported and executed.

rich #

The rich library uses it to make examples of each module print to the screen if it’s executed. I personally think this is a fantastic idea.

Sorry, your browser doesn't support embedded videos.

etl #

In my world of data analysis we often setup a script of functions that will behave as an etl pipeline of sorts. Since we may want to reuse some of these functions in other scripts it’s common to hide the actual execution of these functions in a if __name__ == "__main__": block so that we don’t start making changes to the data simply by importing the module.

cli #

Most cli applications will leverage if __name__ == "__main__": to run something when called as a script instead of being imported. This allows us dt do things such as testing much easier.

Check out the example on the first page of the click framework’s docs

Recap #

if __name__ == "__main__": is not so cryptic or scary, it’s just looking to see if this module was called as a script or imported from somewhere else, and executing some different behavior based on how it was called.

if __name__ == "__main__":
    print("you are running me as main")

example from rich.live
click framework’s docs
try it yourself in your browser with replit.com
StackOverflow: What does if name == “main”: do?

Create a Virtual File Gallery with Symlinks

2021-05-05T08:23:33Z

Creating a directory that is a union of several directories can be achieved with a few symlinks at the command line.

Creating a Virtual File Gallery #

Here is how I am creating a virtual directory of all my projects that is a combination of both work and not-work projects. I am creating symlinks for every directory under ~/work and ~/git.

rm -rf ~/projects
mkdir ~/projects
ln -sf ~/work/* ~/projects
ln -sf ~/git/* ~/projects

⚠ Notice that first I am recreating the directory each time. This will ensure that any project that is deleted from their actual directory is removed from the virtual gallery.

Updating the gallery #

Since links are always kept up to date without any extra work, all the data is still in the same place it started. But as new directories are added to any project directory they will not be automatically added to the virtual gallery.

cron
bashrc/zshrc

If you’re concerned about system resources, you can add it to a cron job to run at a regular schedule that makes sense to you. For me, I just popped those 4 lines right in my ~/.zshrc. It’s a bit overkill, maybe bloat, but it runs in an impercieveable amount of time.

Automatically CD to the real directory #

When you cd into a ~/projects/my-proj directory, your $PWD will still be ~/projects/my-proj. I did not want this for my use case. I wanted to follow the symlink to the real directory. I found two options that worked for me.

alias cd='cd -P'
set -o physical

Add either of these to your .bashrc/.zshrc to follow symlinks to the actual directory.

How to Install micromamba on linux (from the comamnd line only)

2021-04-21T13:32:27Z

I really like using conda (miniconda) as my python virtual environment manager of choice. It’s simple and it includes its own python interpreter using the version that I specify at creation.

Mamba #

from their readme

Mamba is a reimplementation of the conda package manager in C++.

parallel downloading of repository data and package files using multi-threading
libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE
core parts of mamba are implemented in C++ for maximum efficiency

At the same time, mamba utilize the same command line parser, package installation and deinstallation code and transaction verification routines as conda to stay as compatible as possible.

Installing Micromamba #

Similar to miniconda micromamba can be installed with a few lines of bash

wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
./bin/micromamba shell init -s bash -p ~/micromamba
source ~/.bashrc

Creating Environments with Micromamba #

Creating new environments with micromamba is pretty similar to using conda.

micromamba create -n mamba-new python=3.9 -y -c conda-forge

-c is required #

I was unable to figure out how to configure channels to micromamba, so I needed to add -c conda-forge to my commands.

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

WARNING No 'channels' specified
Encountered problems while solving:
  - nothing provides requested python 3.9**

ERROR   Could not solve for environment specs

⚠ micromamba thows this error when -c conda-forge is missing from the create command.

Speed #

micromamba is built for speed. I tried it out in a replit.com session, while it felt quite snappy creating a new environment was still within a few seconds of conda on subsequent environment creations. Their marketing says it should be faster, but for what I use conda for I didn’t see it.

pip #

I used conda install years ago while on windows machines struggling to compile c-extensions and install certain troublesome packages, but I haven’t used a conda install in years, pip works just fine for my use.

Useful Links #

GitHub: https://github.com/mamba-org/mamba
Gitter: https://gitter.im/QuantStack/Lobby
Documentation: https://mamba.readthedocs.io/en/latest/

Vim Wsl Clipboard

2021-04-17T00:00:00Z

I’ve long used neovim from within windows wsl, and for far too long, I went without a proper way to get text out of it and into windows.

wsl has access to cmd applications #

wsl can access clip.exe. You can do some cool things with it, such as cat a file into the clipboard, sending output from a command to the clipboard, or set an autocmd group in vim to send yank to the windows clipboard.

using clip.exe #

Let’s say you want to send a teammate the tail of a log file over chat. You can tail the file into clip.exe.

tail -n 1 info.log | clip.exe

pipe streams of text into clip.exe

make it a bit more natural #

I recently made mine feel a bit more natural by aliasing it to clip.

alias clip=clip.exe

pop this in your ~/.bashrc or ~/.zshrc

yanking to windows clipboard from vim #

I use neovim as my daily text editor and its a pain to share code with a teammate over chat, stack overflow, into a gist, or whatever you need. The following snippet has been quite useful and flawless for me.

if system('uname -r') =~ "Microsoft"
    augroup Yank
        autocmd!
        autocmd TextYankPost * :call system('/mnt/c/windows/system32/clip.exe ',@")
        augroup END
endif

add this to your ~/.vimrc or your ~/.config/nvim/init.vim

Wsl2 #

Based on some feedback from l-sannin the ‘uname -r’ command now returns uname -r command returns '5.10.16.3-microsoft-standard-WSL2' So you will need an all lowercase microsoft.

if system('uname -r') =~ "microsoft"
  augroup Yank
  autocmd!
  autocmd TextYankPost * :call system('/mnt/c/windows/system32/clip.exe ',@")
  augroup END
endif

kedro replit

2021-04-15T12:37:29Z

I am trying to see what an embeded replit

Pytest capsys

2021-04-05T08:46:08Z

Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it.

capsys #

capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys

using capsys #

Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.

def test_print(capsys):
    print('hello')
    captured = capsys.readouterr()
    assert 'hello' in captured.out
    print('world')
    captured = capsys.readouterr()
    assert 'world' in captured.out

Building Rich a Dev Server

2021-03-27T11:23:26Z

Draft Post

I’ve really been digging @willmcgugan’s rich library for creating TUI like interfaces in python. I’ve only recently started to take full advantage of it.

Dev Server #

I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to.

automatic port number
auto-restart
display ( port, pid, uptime )

finding the port #

I am very novice at best when it comes to sockets, the following function came from searching StackOverflow for how to tell if a port is in use. I recursively check if a port is being used, if it is I increment by one until I find an unused port to return.

def find_port(port=8000):
    """Find a port not in ues starting at given port"""
    import socket

    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        if s.connect_ex(("localhost", port)) == 0:
            return find_port(port=port + 1)
        else:
            return port

The Dev Server #

I am going super basic here and just running python -m http.server <port>. It works for what I need it for, it hosts my files for the browser to display, and if I try a route without an index.html it gives me a decent file list.

import subprocess

proc = subprocess.Popen(["python", "-m", "http.server", str(find_port)],)

Optionally if you wanted a live-reload you could opt into live-reload from pypi.

The above snippet will start my dev server on the first open port starting at 8000 and give me a subprocess.Popen object. From there I can see a bit of information about the process.

# returns the process id
proc.pid
# returns none if proc is still running
proc.poll()

Rich #

a quick aside

rich will assist in creating a beautiful terminal interface with minimal effort. Here we are going to build a reuable component to later use inside of a rich layout. When using rich.print or the live display rich will execute a __rich__ method on our objects.

class Min:
    def __rich__(self) -> Panel:
        return Panel("hello world")


def make_min_layout():
    layout = Layout()
    layout.split(Layout(name="upper"), Layout(name="lower"))
    layout["upper"].update(Min())
    layout["lower"].update(Min())

    return layout

__repr__ - custom method
Panel - box around a renderable
Layout - split and nest renderables

There are many components to rich, but the basics I am using so far here are making my own components with a __repr__ method, Panel, and Layout. Panel is an object that will by default take up as much space as it can and draw a rounded border around itself. Layout is an object that accepts other rich renderables, can be split and nested.

Final Result #

Here is a image of the final result running. Here I have the server running on the top split and kill the running server several times. You will see a quick flash of server died followed by the sever back up and running on a new pid.

Sorry, your browser doesn't support embedded videos.


class Server:
    def __init__(self, auto_restart=True):

        self.port = find_port()
        self.start_server()
        self.auto_restart = auto_restart

    def start_server(self):
        import subprocess

        self.proc = subprocess.Popen(
            ["python", "-m", "http.server", str(self.port)],
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )

    def __rich__(self) -> Panel:
        if not self.proc.poll():
            return Panel(
                f"[green]serving on port: [gold1]{self.port} [green]using pid: [gold1]{self.proc.pid}[/]"
            )
        else:
            if self.auto_restart:
                self.start_server()

            return Panel(f"[red]server died")

Future State #

Future state this is going to be integrated into the main layout for my personal website SSG markata.

Site Down During Build

2021-03-22T00:00:00Z

Recently I noticed a new netlify site of mine was down while I was checking to see if new content was live. Later found out this was consistent after each and every push the site would go gown as soon as I hit push, and would not come back until the build finished.

Is this normal? #

Do other Netlify sites go down during build???

Short Answer NO. All of my google fu lead me to believe I was alone and none of my other sites do this.

Digging into my build #

My deploy script ends with the following. After resetting keys and watching it build half a dozen times I determined that everything was working as normal here.

- name: Deploy to Netlify
uses: nwtgck/actions-netlify@v1.1.12
with:
    publish-dir: "./markout"
    production-branch: markout
    production-deploy: true
    deploy-message: "Deploy markout from GitHub Actions"
env:
    NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
    NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}

Opening the Nelify Console #

After poking at the netlify console for hours I realized that the issue was that netlify was still auto-deploying from a no longer existing directory and would cause 404’s for every page. During build, then my build from GitHub Actions would deploy with the netlify cli.

Netlify really likes to put a lot of warnings up when you are not deploying from them. I tured off automatic deploys, swore to the netlify gods this is what I wanted. Pushed a new deploy and 🎉 THE SITE DID NOT GO DOWN.

jdiv class=‘center-img’>

Kedro pipeline_registry.py

2021-03-20T00:00:00Z

With the realease of kedro==0.17.2 came a new module in the project template pipeline_registry.py. Here are some notes that I learned while playing with this new module.

migrating to `pipeline_registry.py` #

create a src/<package-name>/pipeline_registry.py file create a
register_pipelines function in pipeline_registry.py that mirrors the
register_pipelines method from your hooks.py module do not bring the
hook_impl decorator remove register_pipelines method on your ProjectHooks
class

You should now have something that looks like this in your src/<package-name>/pipeline_registry.py.

"""Project pipelines."""
from typing import Dict

from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    Returns: A mapping from a pipeline name to a ``Pipeline`` object.
    """
    return {"__default__": Pipeline([])}

pipeline_registry only works in kedro>=0.17.2

Conflict Resolution #

What happens If I register pipelines in both places

I was not able to find any official documentation on how conflict resolution worked so I stepped into a project and added to both my hooks.py and pipeline_registry.py file. I noticed that it would pick up pipelines from both modules, but pipelines from hooks.py always take precedence. The entire duplicate pipeline will be over written by the one from hooks.py.

kedro automatically merges pipelines from both hooks.py takes precedence

Ready to update #

In my experience there were no issues upgrading from 0.17.1 to 0.17.2. I would reccomend only having one register_pipelines so decide to migrate to the new pipeline_registry.py or keep it in your hooks.py, but both is only going to lead to confusion.

https://twitter.com/datajoely/status/1375193511264456705

Stand With Your Team

2021-01-22T00:00:00Z

People who are quick to toss team members under a bus are not well trusted or highly thought of and it will lead to some toxic team dynamics.

Building Steam #

While collaborating on any project there are going to be decisions made that aren’t necessarily your favorite, during a summer internship my mentor made some decisions that I was not on board with, but I accepted his wisdom and moved forward with little push back.

Full Spead Ahead #

During a review, leadership showed interest in the option that I wanted to go towards. I was quick to jump up and say I told you so right then and there and pitch reasons why my idea was so much better.

The Aftermath #

I walked out so proud that leadership recognized my idea, but I will never forget the look on my teammate’s face and what he told me after the meeting.

“Wow quick to throw me under the bus in front of everyone 😢”. I

~ my teammate

Realizing what Happened #

It was at this moment that I threw my mentor, a good friend in front of a bus that was at full speed. I will never forget this moment and the importance of being a good trustworthy team player.

Speak your Mind, be heard, shut up #

This day I learned that there is a time to speak your mind. While that bus is sitting idle it would have hurt no one.

There were times to hear out ideas, and there were times that we need to accept the decisions the team made. The team that you were a part of and had a chance to steer.

There are times where it’s appropriate to sit down and shut up.

Your Story 👇 #

Do you have a story of how not standing with your team caused you pain later on. I would love to hear and learn from your experience.

Blogging For Me

2021-01-21T00:00:00Z

I create this blog with one person in mind, me.

There are others like me #

This is not completely selfish, as there are likely many others out there that think similarly to me. Everyone comes from different backgrounds and varying levels of experience. In no way do you need to be an expert to create content others will benefit from.

Accurate #

I am as accurate as possible. I don’t know everything, and If I waited for that to happen I would never post, or write at such a high level no one else (including me) would ever want to read.

Generally I know that I am directionally accurate for most of the article. In fact there is often a part of the article that I don’t feel the best about. It may have been some code that could have been better. I try to point these things out. Often when I am and point out where I know there is improvement to be had I get the most welcoming comments, “that was just fine how you did that”, or “I tried that myself and saw a better abstraction.

Create the articles I want to read #

Generally the articles I want to read are directly relatable to things I am currently working on and vary over time. Over the last year its been leaning a bit more towards automation with bash, python, kedro tools.

On Brand #

I try to stay on brand to who I am. I remember writing my first articles. I was trying to be whitty, and include funny memes. Thats not me. I am not a commedian, and generally memes go right over my head. I can remember back in High School, I was the one that didn’t understand a lot of pop culture and was kinda treated as if I was always living under a rock. I knew it was true and I owned it, so no one ever got too mean about it.

Honest, never fake #

I will not try to pretend that I know everything. You may read my blog on occasion and think I know a lot, also possible that you think “wow what an idiot”. I welcome you to share your thoughts and techniques in a positive way. I often cringe at some of my old posts and think about how differently I would approach something not that long after posting. I honest, never fake know what I need to know and don’t pretend to know anything outside of my wheelhouse.

Minimal Kedro Pipeline

2021-01-20T00:00:00Z

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in setup.py and 27 in mini_kedro_pipeline.py.

📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner.

Minimal Kedro Pipeline #

I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along.

Installation #

pip install git+https://github.com/WaylonWalker/mini-kedro-pipeline

Caveats #

This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your hooks.py or run.py modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file.

This is a sharable pipeline that can be used across many different projects.

Usage #

# hooks.py

import mini_kedro_project as mkp

class ProjectHooks:
    @hook_impl
    def register_pipelines(self) -> Dict[str, Pipeline]:
        """Register the project's pipeline.

        Returns:
            A mapping from a pipeline name to a ``Pipeline`` object.

        """

        return {"__default__": Pipeline([]), "mkp": mkp.pipeline}

Implemantation #

This builds on another post that I made about creating the minimal python package. I am not sure if it should be called a package, it’s a module, but what do you call it after you build it and host it on pypi?

[[ minimal-python-package ]]

Directory structure #

.
├── .gitignore
├── README.md
├── setup.py
└── my_pipeline.py

setup.py #

This is a very minimal setup.py. This is enough to get you started with a package that you can share across your team. In practice, there is a bit more that you might want to include as your project grows.

from setuptools import setup

setup(
    name="MiniKedroPipeline",
    version="0.1.0",
    py_modules=["mini_kedro_pipeline"],
    install_requires=["kedro"],
)

mini_kedro_pipeline.py #

The mini kedro pipeline looks like any set of nodes in your project. Many projects will separate nodes and functions, I prefer to keep them close together. The default recommendation is also to have a create_pipelines function that returns the pipeline.

This pattern creates a singleton, if you were to reference the same pipeline in multiple places within the same running interpreter and modify the one you would run into issues. I don’t foresee myself running into this issue, but maybe as more features become available I will change my mind.

"""
An example of a minimal kedro pipeline project
"""
from kedro.pipeline import Pipeline, node

__version__ = "0.1.0"
__author__ = "Waylon S. Walker"

nodes = []


def create_data():
    "creates a dictionary of sample data"
    return {"beans": range(10)}


nodes.append(node(create_data, None, "raw_data", name="create_raw_data"))


def mult_data(data):
    "multiplies each record of each item by 100"
    return {item: [i * 100 for i in data[item]] for item in data}


nodes.append(node(mult_data, "raw_data", "mult_data", name="create_mult_data"))

pipeline = Pipeline(nodes)

Kedro in scripts #

Using Kedro In Scripts

If you enjoyed this one check out this companion article where I build a fully runnable kedro project in a single script.

Go forth and share your pipelines across projects. Let me know, do you share pipelines or catalogs across projects?

Markdown Cli

2021-01-20T00:00:00Z

This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown.

Listing things #

posts
tags
draft posts

data #

frontmatter
filepath
content
template
html

render content #

Markdown.Markdown
support extentsions

frontmatter cleaning. #

provide ways to hook in or clean up the frontmatter

Markata.Markata methods #

load
render
save

Markata.Post methods #

load
render
save

Markata plugins #

before_load
before_post_load
after_load
after_post_load
before_save
before_post_save
after_save
after_post_save

Markata plugins #

cleanse_frontmatter
html_feed
json_feed
rss_feed
save_posts

CLI #

$ markata list tags

python
data

$ markata

[
  {
    "title": "post title",
    "description": "this is a post",
    "filepath": "path_to.md",
    "content": "the content of the post",
    "html": "<p>the content of the post</p>"
    },
    ...
]

My Content Strategy For 2021

2021-01-19T00:00:00Z

I am making another push in 2021 to get my content out in the world and meeting users where they are. See how I plan to execute.

Platforms #

waylonwalker.com
Twitter
DEV
hashnode
Medium
LinkedIn
Anchor

Markdown #

My content is written in markdown, all markdown. I find that markdown does a really great job at getting out of the way and letting ideas flow onto the page. I am never fussing with fonts and formatting while physically writing posts. Not that I don’t spend way more time than I need to tweak these things on my own personal site where everything gets posted.

Articles #

Much of what I create is inside of short articles that get posted to my personal site waylonwalker.com. These will get cross-posted to DEV, hashnode, Medium.

I have made cross-posting a bit easier for myself by posting the markdown for each article next to the post on my personal site. Add .md to any post and there is the source.

Should I be giving my articles to Medium?? Personally, I am not a fan of the bait and switch that they did with adding a paywall years after launch. From what I can tell authors are not getting paid much from it, users are discouraged when they are blocked, and they really only promote articles that are behind the paywall.

In the end, I am a fan of meeting users where they are, if they like Medium I don’t want them to miss my content.

More Catalytic #

soft skills

My articles are very code-heavy, I do want to transition over to some more evergreen type of posts that are long-lasting. I want to create some more content around how to be a good developer, data scientist, engineer, whatever you call yourself.

Personally, this is some of the content I enjoy consuming, but for some reason is not something I tend to create.

Slides #

I am really enjoying the slide output of my articles on LinkedIn. It makes for something that is unique and easy to digest. I have a pandoc script that takes in the markdown from my posts and kicks out a set of slides, so it’s quick and easy to do. Not every post works well this way by many have so far.

Threads #

With less success (shares, views, likes) I have been converting articles over to Twitter threads as they are released, again not every article works well this way, but many do.

https://twitter.com/_WaylonWalker/status/1349716127887347717

Simplicity #

just write markdown

In the end, everything needs to be simple and build off of the original markdown post. Automation should be in a place where it makes sense. I am no longer manually creating cover art or article descriptions, those are scripted out of the article itself.

I have come cli tooling in place to simplify my process. Expect more of that to come.

Audio #

I have set up a podcast where I read out my posts, but the time that I have available to read them is very constraining to the process. It may be something that I figure out how to work out or something that I drop. It’s something that I think would be really nice, but still trying to figure out how to do it.

https://waylonwalker.com/audio-for-blog/

I personally really like this as I often find a cool blog article that I want to read, but it’s time to cook dinner or drive somewhere. Articles with audio make it super convenient to consume while doing a task requiring eye contact.

TLDR #

Meet Users Where They are.

Quickly Edit Posts

2021-01-18T00:00:00Z

Recently I automated starting new posts with a python script. Today I want to work on the next part that is editing those posts quickly.

Automating my Post Starter

Check out this post about setting up my posts with python 🐍

Enter Bash #

For the process of editing a post I just need to open the file in vim quickly. I dont need much in the way of parsing and setting up the frontmatter. I think this is a simple job for a bash script and fzf.

change to the root of my blog
fuzzy find the post
open it with vim
change back to the directory I was in

bash function #

For this I am going to go with a bash function. This is partly due to being able to track where I was and get back. Also the line with nvim will run fzf everytime you source your ~/.alias file which is not what we want.

Lets setup the boilerplate. Its going to create a function called ep "edit post", track our current directory, create a sub function _ep. Then call that function and cd back to where we were no matter if _ep fails or succeeds.

boilerplate

ep () {
    _dir=$(pwd)
    _ep () {
        # open file here
    }
    _ep && cd $_dir || cd $_dir
}

Creating Reusable Bash Scripts

check out this post for more information about writing reusable bash scripts.

FZF #

Let’s focus in on that _ep function here that is going to do the bulk of the work to edit the post.

cd to where I want to edit from

cd ~/git/waylonwalkerv2/

Next I need to find all markdown pages within my posts directory. There is probably a better way to filter with the find command directly, but I am not worried about perf here and I knew how to do it without google.

find all markdown

find ~/git/waylonwalkerv2/src/pages/ | grep .md$

Now that we can list all potential posts, sending the selected post back to neovim is as easy as piping those files into fzf inside of a command substitution that is called with neovim.

putting together the edit command

$EDITOR $(find ~/git/waylonwalkerv2/src/pages/ | grep .md$ | fzf)

Final Script #

final ep function

ep () {
    _dir=$(pwd)
    _ep () {
        cd ~/git/waylonwalkerv2/
        $EDITOR $(find ~/git/waylonwalkerv2/src/pages/ | grep .md$ | fzf)
    }
    _ep && cd $_dir || cd $_dir
}

Gitui is a blazing fast terminal git interface

2021-01-17T00:00:00Z

Gitui is a terminal-based git user interface (TUI) that will change the way that you work with git. I have been a long-time user of the git cli, and it’s been hard to beat, mostly because there is nothing that keeps my fingers on the keyboard quite like it, except gitui which comes with some great ways to very quickly walk through a git project.

installation #

Go to their [releases]https://github.com/extrawurst/gitui/releases) page, download the latest build, and pop it on your PATH. I have the following stuffed away in some install scripts to get the latest version.

install latest release

GITUI_VERSION=$(curl --silent https://github.com/extrawurst/gitui/releases/latest | tr -d '"' | sed 's/^.*tag\///g' | sed 's/>.*$//g' | sed 's/^v//')
wget https://github.com/extrawurst/gitui/releases/download/v${GITUI_VERSION}/gitui-linux-musl.tar.gz -O- -q | sudo tar -zxf - -C /usr/bin/

run `gitui` #

It opens blazing fast.

gitui

Quick Commits #

Sometimes I edit a number of files and want to commit them one at a time, this is painful in the git cli and my main use case for gitui. gitui shows unstaged changes at the top, staged changes on the bottom, and a diff on the right.

Navigate with hjkl #

By default, gitui uses arrow keys, but simply copying vim_style_key_config.ron to your config directory will get you vim-like keybindings.

workflow #

Generally, I pop open gitui, use j/k to get to the file I want to commit, glance at the diff to the right, press enter to stage the file, sc to switch focus to the saged files and commit, write my commit message hit enter and done.

w/s: to toggle focus between working and staged changes
j/k: to scroll each section
h/l: switch between left and right side
enter: toggle file from working or staging
c: start a commit message
p: push
: quit

Other Panes #

I am in the Status [1] pane 90% of the time, but it also has three other panes for Log [2], Stashing [3], and Stashes [4]. I do not really use the stashes panes, but the Log [2] pane is quite useful to quickly go through the last set of commits and see the diff for each of them.

What UI do you use for git #

Let me know what ui you use for git, do you stick to the cli, use a gui, or use a similar TUI interface?

Kedro - My Data Is Not A Table

2021-01-14T00:00:00Z

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.

DataFrames are the heart of most pipelines #

These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types.

What is Kedro

unfamiliar with kedro, check out this post

Sometimes datasets are not tables #

There are times when our data doesn’t fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python object to disk. Beware that pickle files coming from an unknown source can run malicous code and are considered unsafe. For the most part though when you read and write your own pickle files they are a good tool to consider.

See more about pickle from python.org.

Cataloging Pickle #

I may have a dictionary that describes some cars.

{
  'truck-012-abc': {
    'type': 'truck'
    'sales': [12, 2, 3, 4, 8]
    'weight': 9024,
    'accesories': ['leather', 'audio-1']
}

In the catalog we will simply set the type as pickle.PickleDataSet and give it a filepath.

cars:
  filepath: data/cars.pkl
  type: pickle.PickleDataSet

This filepath does not have to be on the local filesystem it can be on the cloud thanks to how kedro utilizes fsspec for each of its datasets.

Loading the dataset #

The benefit of cataloging this dataset compared to leaving it as a MemoryDataSet is that you can easily load this data back into memory for further development or debugging without running any of the pipeline.

catalog.load('cars')

Quickly Change Conda Env With Fzf

2021-01-11T00:00:00Z

Changing conda environments is a bit verbose, I use a function with fzf that both lists environments and selects the one I want in one go.

Conda #

I have used conda as a virtual environment tool for years now. I started using conda for its simplicity to install packages on windows, but now that has gotten so much better and it’s been years since I have run a conda install command. I’m sure that I could use a different environment manager, but it works for me and makes sense.

What environment manager do you use for python?

Conda environments are stored in a central location such as ~/miniconda3/envs/ and not with the project. They contain both the python interpreter and packages for that env.

Conda create #

Conda environments are created with the conda create command. At this point, you will need to name your env and select the python version.

conda create -n my_env python=3.8

After running this command you will have a directory ~/miniconda3/envs/my_env with a base python install. It will not be active yet.

List environments #

Before activating an environment I often want to list the environments that I have installed which are often upwards of 70, so it’s hard to remember them all.

conda info --envs

After running this command you will see something like the following.

# conda environments:
#
base                     /home/waylon/miniconda3
my_env                   /home/waylon/my_env

Activating an environment #

Activating a conda environment will do some magic to your current shells $PATH variable to ensure that the environment that you select is preferred over the base environment.

conda activate my_env

Ready to work #

Now you can install packages for your project in an isolated environment safe from wrecking another project or being wrecked by another project.

pip install -r requirements.txt

Using fzf #

a bit less verbose

fzf is an amazing tool for the terminal that is a generic fuzzy matcher. It is super performant and can handle insane amounts of text and is brilliant at figuring out what you mean with just a few characters. We can use it here to list out all of our conda environments and select the one we want to activate with just a few keystrokes.

Selecting the environment #

Piping our list of environments directly into fzf gives us a fuzzy selection where we can type a few characters and it will return the row we were looking for.

conda info --envs | fzf

This returns us something like this which also includes the path where it is located.

my_env                 /home/walkews/miniconda3/envs/my_env

getting just the environment name #

To get just the name without the path I pipe the output into awk. There are many ways to do this in bash, this is the way that worked for me at the time I made this function.

conda info --envs | fzf | awk '{print $1}'

Time to activate #

Functions that use fzf can be a bit odd, running them in a subshell with the $() syntax generally makes it super simple to utilize the output. No matter how many times I have tried without running it in a subshell it’s always buggy without it.

conda activate "$(conda info --envs | fzf | awk '{print $1}')"

This will now run conda activate on the environment that we select with fzf.

Make it a function #

We don’t want to type that out every time we want to activate an environment. I keep a function called a in my ~/.bashrc and ~/.zshrc so that I can activate an environment with a single character. Yes, I switch environments often enough to justify the valuable namespace of a single character.

a () {
  conda activate "$(conda info --envs | fzf | awk '{print $1}')"
}

Creating Reusable Bash Scripts

for more information on writing reusable bash scripts check out one of my favorite articles

I am always on the lookout for cool new use cases for fzf, if you have one please share it with me.

Vim Replace Visual Star

2021-01-10T00:00:00Z

Replacing text based on whats in the current search register is a quite handy tool that I use often. I believe I picked this tip up from Nick Janetakis, check out his YouTube channel for some amazing vim tips.

https://www.youtube.com/watch?v=fP_ckZ30gbs

If there is one thing that I Like most about vim it’s the ability to hack on it and make it work well for you.

Replacing text in vim #

Vim can often be a bit verbose, but that’s ok because we can hack on it, and make our own shortcuts and keybindings. For instance, finding and replacing text requires using a command at the vim command-line :. Replacing foo with bar looks like this :%s/foo/bar/g, the final g means all of the foos, not just the first one on the line.

making it better #

I have a keybinding in my init.vim that will allow me to search for a pattern with the usual / character, page through them as normal with n and N, but when I press <C-R> it will populate the replace command for me so that all I need to do is type out the new text.

nnoremap <c-r> :%s/<C-R>///g<Left><Left>

Note on the `<C-R>/` #

In command mode : vim allows you to paste any text from any register into the current command. The <C-R>/ will paste the text from the current search register into the command.

<C-R> in command mode can paste text from any register, you can see what registers are in use with the :reg command. There are a lot of them and many get populated automatically as you yank text or create macros.

Save Vim Macro

Also see how to use to save macros to key bindings easily

Minimal Python Package

2021-01-10T00:00:00Z

What does it take to create an installable python package that can be hosted on pypi?

What is the minimal python package #

setup.py
my_module.py

This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file.

Directory structure #


.
├── setup.py
└── my_pipeline.py

setup.py #

from setuptools import setup

setup(
    name="",
    version="0.1.0",
    py_modules=["my_pipeline", ],
    install_requires=["kedro"],
)

name #

The name of the package can contain any letters, numbers, “_”, or “-”. Even if it’s for internal/personal consumption only I usually check for discrepancy with pypi so that you don’t run into conflicts.

Note that pypi treats “-” and “_” as the same thing, beware of name clashes

version #

This is the version number of your package. Most packages follow semver. At a high level its three numbers separated by a . that follow the format major.minor.patch. It’s a common courtesy to only break APIs on major changes, new releases on minor, and fixes on patch. This can become much more blurry in practice so checkout semver.org.

py_modules #

Typically most packages use the packages argument combined with find_packages, but for this minimal package, we are only creating one .py file.

Using packages instead #

from setuptools import setup, find_packages

setup(
    name="",
    version="0.1.0",
    packages=find_packages(),
    install_requires=["kedro"],
)

install_requires #

These are your external dependencies that come from pypi. They go in this list but are often pulled in from a file called requirements.txt. Other developers may look for this file and want to do a pip install -r requirements.txt.

Clean? #

One thing to be careful of here is that everything sits at the top level API, when you users import your module and hit tab they are going to see a lot of stuff unless you hide all of your internal functions behind an _.

Minimal #

Can you create a python package with less than two files and less than 8 lines? Should you? I really like a minimal point to get started from for quick and simple prototypes. You can always pull a more complicated cookiecutter template later if the project is successful.

If Tmux

2021-01-09T00:00:00Z

I do much of my work from tmux, I love it so much that I want to setup some functionality that puts me in tmux even if I didn’t ask for it.

Bash Function #

Bash function to check if the shell is in a tmux session.

in_tmux () {
  if [ -n "$TMUX" ]; then
    return 0
  else
    return 1
  fi
  }

Using the bash function #

I often open up vim to do some quite edits, but before I know it I have several splits open and I need access to another shell utility, but I forgot to start in tmux. This function makes sure tht I start in tmux everytime.

Using if_tmux to ensure vim is opened in tmux.

vim () {
  in_tmux \
    && nvim \
    || bash -c "\
    tmux new-session -d;\
    tmux send-keys nvim Space +GFiles C-m;\
    tmux -2 attach-session -d;
    "
  }

I am not quite sure if this is proper use of the && and ||, let me know if you have a better way to do one thing if in_tmux returns true and another if it returns faslse.

Save Vim Macro

2021-01-04T00:00:00Z

If you are like me, you have created a macro or two that is pure glory, and you forget how you made it after a day or so, or you immediately want to store it away as a custom keybinding. As with most things with vim, it’s easy to do once you understand it.

Creating a Macro #

One of the earliest things we all learn to do in vim is to create macros, custom sets of functionality stored in a register that can be replayed later.

To create a macro, get into normal mode, then type q followed by a letter that you want to store the macro under.

qq

Note: a common throw-away macro register is q because it’s easy to hit qq from normal mode to start recording.

Replaying a Macro #

Macros can be replayed using @ followed by the letter that you stored the macro under.

@q

Registers #

Registers are nothing more than a single character key mapping to a value of some text. As you yank, delete, or create macros in vim, it automatically stores text into these registers.

When you hit p paste it’s simply pasting in the default register. You can also paste in any other register by hitting "qp where q is the register that you want to paste in.

Listing Registers #

To see what you have stored in each register, use the :reg command. This is a powerful tool that I have underutilized for a long time. It is really great to see what you have in each register.

:reg

making a macro into a shortcut #

a little <c-r> magic

Thgve magical shortcut that makes it easy is that control + r <C-R> followed by a register will paste that register wherever you currently are, including the command mode.

:nnoremap {binding} <C-R>{register}

Editing a Macro #

relieve some of that Macro Pressure

Now that we understand that macros are simply strings of text placed into a register, it becomes a bit more intuitive to edit them after being created.

First, paste the contents of the register into your current working buffer.

<C-R>q

Then edit the macro and add it back to that buffer and delete it.

"qdd

If your macro had multiple lines in it, you might need to.

"qdj
"qd2j

Make it recursive #

One use case of editing a macro may be making it recursive after trying it out a few times. Macros can become recursive by simply calling themselves after running.

Paste in the macro.

<C-R>q

Go to the end of the line and add @q to get called again.

A @q

Replace the q register with the updated macro.

"qd

Note: don’t use this in a shortcut as the macro may change. If you want to call the keybinding again, you will have to use noremap instead of nnoremap, but be careful as recursive remaps can be dangerous.

Recap #

" record a macro
q{register}

" play a macro
@{register}

" list registers
:reg

" map a macro to a keyboard shortcut
:nnoremap {binding} <C-R>{register}

" edit a macro
<C-R>{register}
"{register}dd

" make a macro recursive
<C-R>{register}A@q<esc>"{register}dd

Live Substitution In Neovim

2021-01-04T00:00:00Z

Replacing text in vim can be quite frustrating especially since it doesn’t have live feedback to what is changing. Today I was watching Josh Branchaud’s Vim-Unalphabet series on Youtuve and realized that his vim was doing this and I had to have it.

https://twitter.com/_WaylonWalker/status/1346081617199198210

How to do it #

I had to do a bit of searching and found a great post from vimcasts that shows exactly how to get the live search and replace highlighting using inccomand

:h inccommand #

'inccommand' 'icm' string (default "")
   global

 "nosplit": Shows the effects of a command incrementally, as you type.
 "split"  : Also shows partial off-screen results in a preview window.

 Works for |:substitute|, |:smagic|, |:snomagic|. |hl-Substitute|

 If the preview is too slow (exceeds 'redrawtime') then 'inccommand' is
 automatically disabled until |Command-line-mode| is done.

Add this to your config #

I believe that this is a neovim only feature, add it into your ~/.config/nvim/init.vim file. You can see it in my dotfiles as well.

set inccommand=nosplit

See it in Action #

The Video that inspired this #

Check out Josh Branchaud’s great series on the Vim-Unalphabet.

https://www.youtube.com/watch?v=5jMiYtXz2QA

Newsboat

2021-01-02T00:00:00Z

Web browsers are a black hole of productivity. I try to use them as little as possible when it is time to focus. I try to use help, ?, or ?? with ipython, or –help at the command line as much as possible. What about that time I am trying to see what my online friends are posting on their sites? I used to used google reader quite heavily before that was taken down.

Newsboat #

I am going to give a terminal rss reader a try for a bit and see how that goes for me. I have really struggled to get into an rss reader since google reader died.

installation #

I installed with the reccomended snap for Ubuntu.

sudo snap install newsboat

Adding feeds #

super simple

Running help for newsboat directed me towards their config files at the bottom.

❯ newsboat --help
newsboat r2.22
usage: /snap/newsboat/3849/usr/local/bin/newsboat [-i <file>|-e] [-u <urlfile>] [-c <cachefile>] [-x <command> ...] [-h]
    -e, --export-to-opml        export OPML feed to stdout
    -r, --refresh-on-start      refresh feeds on start
    -i, --import-from-opml=<file>   import OPML file
    -u, --url-file=<urlfile>    read RSS feed URLs from <urlfile>
    -c, --cache-file=<cachefile>    use <cachefile> as cache file
    -C, --config-file=<configfile>  read configuration from <configfile>
    -X, --vacuum            compact the cache
    -x, --execute=<command>...  execute list of commands
    -q, --quiet         quiet startup
    -v, --version           get version information
    -l, --log-level=<loglevel>  write a log with a certain loglevel (valid values: 1 to 6)
    -d, --log-file=<logfile>    use <logfile> as output log file
    -E, --export-to-file=<file> export list of read articles to <file>
    -I, --import-from-file=<file>   import list of read articles from <file>
    -h, --help          this help
        --cleanup           remove unreferenced items from cache

Files:
    - configuration:  /home/nic/snap/newsboat/3849/.newsboat/config
    - feed URLs:      /home/nic/snap/newsboat/3849/.newsboat/urls
    - cache:          /home/nic/snap/newsboat/3849/.newsboat/cache.db

Support at #newsboat at https://freenode.net or on our mailing list https://groups.google.com/g/newsboat
For more information, check out https://newsboat.org/

I just need to edit its urls file.

nvim ~/snap/newsboat/3849/.newsboat/urls

The Urls file is just a list of urls to rss feeds. Adding mine in allowed me to see all of my posts.

https://waylonwalker.com/rss.xml

Config #

I took most of my config from a blog post that I found by Evan Travers. It set some sane defaults to the reading width and vim keys.

# https://evantravers.com/articles/2020/04/15/reworking-my-rss-reading/
# Hide feeds where all the items are read.
show-read-feeds no

# Make the text width readable
text-width 50

# Use multiple threads to download all the news faster.
reload-threads 11

# browser ~/bin/newsboat-browser.sh
# browser "/usr/bin/brave-browser %u"

# unbind keys
unbind-key ENTER
unbind-key j
unbind-key k
unbind-key J
unbind-key K
unbind-key ^D
unbind-key ^U
unbind-key o
unbind-key g
unbind-key G

# bind keys - vim style
bind-key j down
bind-key k up
bind-key l open
bind-key h quit
bind-key ^D pagedown
bind-key ^U pageup
bind-key b toggle-source-view
bind-key U toggle-show-read-feeds
bind-key u show-urls
bind-key g home
bind-key G end
bind-key b open-in-browser-and-mark-read
bind-key B open-in-browser
bind-key i sort
bind-key I rev-sort

GUI Browser #

No matter how many different guides I tried I keedp getting error code 127 when trying to open-in-browser. Please let me know if you know how to fix this. For now I am just going to roll with it.

Here’s how it looks #

browsing a feed in newsboat

reading an article in newsboat

Large Refactor At The Command Line

2020-12-30T00:00:00Z

As projects grow patterns that worked early on break and we need to change things to make the project easier to work with, and more welcoming to new developers.

git">git #

Before you start mucking up a project with wild commands at the terminal check that you have a super clean git status. We may make some mistakes and need a way to undo 100’s files and git makes it really easy if you start with a clean history.

git status

If we are ready to begin work we should see a response like this.

On branch main
nothing to commit, working tree clean

It would also be wise to do this inside of a branch. The minute you try to do something wild in your working branch someone will walk by and ask you to do a five-minute task, but your deep in refactoring and haven’t left yourself a clean way back.

git branch my-big-refactor

grepr #

Time for the meat of this refactor replacing text across our project. I often will pop this bash function into my terminal session and tweak it as needed. This function is called grepr for grep then replace. It will recursively search for a given pattern inside your working directory, then use sed to replace that pattern with another.

grepr() {grep -iRl "$1" | xargs sed -i "s/$1/$2/g"}

If your pattern contains / characters such as for URLs you can swap the /’s in the sed command for |’s.

grepr() {grep -iRl "$1" | xargs sed -i "s|$1|$2|g"}

You can find this function and more of my bash notes.

📝 Bash Notes

Example #

I recently flattened this blog so blogs are under the top-level rather than under /blog and I used this technique to swap internal links to the new format.

grepr() {grep -iRl "$1" | xargs sed -i "s|$1|$2|g"}


grepr "https://waylonwalker.com/blog/" "https://waylonwalker.com/"

git diff #

After running the replace command the first thing I want to see is everything that changed. Looking at git diff will highlight exactly what changed since our last commit.

git diff

Work in small steps #

If you’re happy with the results commit them now. It’s best to do these commands that have a large effect on the entire project in small steps.

git add .
git commit -m "moved routes from /blog to /"

Working in small steps gives us an easy way to undo steps that may have been a mistake before it’s too late.

Master No More

I used the technique from this post to switch master to main on my blog.

git reset #

How I do Mass Undo

be careful work from a branch, make sure you started clean

Let’s say I wanted to change every occurrence of one variable name to another. Lets try to replace replace pandas.CSVDataSet with pandas.ParquetDataSet.

grepr() {grep -iRl "$1" | xargs sed -i "s|$1|$2|g"}


grepr "pandas.CSVDataSet" "pandas.ParquetDataSet"

Upon inspection of the git diff we notice that there was an unintentional change to the docs/standard-storage.md file. To revert the entire change we can run.

note These resets are irreversible. Make sure that you started with a clean git status and you are confident that you didn’t have any work on your machine, not in the remote repo.

match the remote and wipe out any changes

git reset --hard origin/main

match our last commit

git reset --hard HEAD

agr #

I have an alternative version that I occasionally use as well that utilizes the silver searcher ag. It does a great job at following your .gitignore rules with no fuss, and can filter down to file extensions simply with flags like --md

agr() {ag -l "$1" | xargs sed -i "s/$1/$2/g"}

git clean #

how I remove untracked files

Sometimes our refactoring requires moving files around. If we want to undo steps like this git will not clean up untracked files.

mv conf/base/sales-catalog.yml conf/base/sales/catalg.yml

clean up untracked files

git clean -f

clean up untracked directories

git clean -d

clean up ignored files

git clean -x

-x can be a bit dangerous, be careful with it. You can lose significant time by wiping out a node_modules, venv, or credentials.

git checkout #

How I undo single files

If our command was mostly successful, but just a few extra files were touched I will manually revert them with git checkout <filename>

git checkout conf/base/supply-catalog.yml

git checkout – #

How I undo an entire directory

Sometimes we need to undo an entire directory. This command will undo changes to all of the tracked files in the repo.

git checkout -- /src/pages/blog

gitui #

I really love using gitui as a handy terminal interface to browse logs, diffs, and commit a few files at a time. It starts up crazy fast and is very intuitive to navigate through diffs of changes like this one file at a time if the git diff gets too overwhelming.

https://github.com/extrawurst/gitui/

Ipython-Config

2020-12-20T00:00:00Z

I use my ipython terminal daily. It’s my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration.

install ipython #

Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython.

pip install ipython

You are using a virtual environment right? Virtual environments like venv or conda can save you a ton of pain down the road.

profile_default #

When you install ipython you start out with no config at all. Runnign ipython profile create will start a new profile called profile_default that contains all of the default configuration.

ipython profile create

This command will create a directory ~/.ipython/profile_default

multiple configurations #

You can run multiple configurations by naming them with ipython profile create [profile_name] This command will create a directory ~/.ipython/[profile_name]

ipython profile create my_profile
ipython --profile=my-profile

startup #

Inside the profile there will be a startup directory ~/.ipython/profile_default/startup. Ipython will execute each of the files in this directory on startup. This is particularly handy to create custom prompts, search, or import packages automatically for certian profiles.

Custom Ipython Prompt

This post creates a custom ipython prompt by creating a ~/.ipython/profile_default/startup/prompt.py file.

ipython_config.py #

There are tons of options that are in the ipython_config.py file. My favorite is to automatically enable my favorite magic command autoreload.

Autoreload in Ipython

c.InteractiveShellApp.extensions = ['autoreload'
c.InteractiveShellApp.exec_lines = []'%autoreload 2']
c.InteractiveShellApp.exec_lines.append('print("Warning: disable autoreload in ipython_config.py to improve performance.")')

Want automatic imports?? #

Smoother Python with automatic imports | pyflyby

This article covers how I setup automatic imports in ipython

Custom Ipython Prompt

2020-12-20T00:00:00Z

I’ve grown tired of the standard ipython prompt as it doesn’t do much to give me any useful information. The default one gives out a line number that only seems to add anxiety as I am working on a simple problem and see that number grow to several hundred. I start to question my ability 🤦‍♂️.

Configuration #

If you already have an ipython config you can move on otherwise check out this post on creating an ipython config.

Ipython-Config

The Dream Prompt #

I want something similar to the starship prompt I am using in the shell. I want to be able to quickly see my python version, environment name, and git branch.

python version
active environment
git branch

This is my zsh prompt I am using for inspiration

Basic Prompt #

This is mostly boilerplate that I found from various google searches, but this gets me a basic green chevron as my prompt.

from IPython.terminal.prompts import Prompts, Token

class MyPrompt(Prompts):
    def in_prompt_tokens(self, cli=None):
        return [ ( Token.Prompt, "❯ ",), ]

    def out_prompt_tokens(self, cli=None):
        return []

ip = get_ipython()
ip.prompts = MyPrompt(ip)

The rest of this post will build off of this boilerplate and add to the in_prompt_tokens method of MyPrompt

Colors #

I mostly set the colors of my prompt throughout this post by guessing and trying different attributes under the Token.

Red If Error #

I found that the Prompts subclass has many of the same methods as the ipython object, so I would often use that for inspection. Looking through the ipython class I found a boolean under shell.last_execution_succeeded that would give me if the last execution was successful or not. I did an inline if statemetn to set the color to a Token.Generic.Error if this was false.

def in_prompt_tokens(self, cli=None):
    return [
        (
            Token.Prompt
            if self.shell.last_execution_succeeded
            else Token.Generic.Error,
            "❯ ",
        ),
    ]

Python Version #

Next up to list out the python version that is running. I grabbed the version from platform.python_version, this seemed to get me the most concise version that I was looking for to match the starship prompt.

update imports

from platform import python_version

update prompt

def in_prompt_tokens(self, cli=None):
    return [
        (
            (Token.Name.Class, "v" + python_version()),
            (Token, " "),
            Token.Prompt
            if self.shell.last_execution_succeeded
            else Token.Generic.Error,
            "❯ ",
        ),
    ]

Python environment #

Since I use conda for my environments I chose to pull the name of the environment from the CONDA_DEFAULT_ENV environment variable that is set by conda when you change your environment.

update imports

from platform import python_version
import os

update prompt

def in_prompt_tokens(self, cli=None):
    return [
        (
            (Token.Prompt, "©"),
            (Token.Prompt, os.environ["CONDA_DEFAULT_ENV"]),
            (Token, " "),
            (Token.Name.Class, "v" + python_version()),
            (Token, " "),
            Token.Prompt
            if self.shell.last_execution_succeeded
            else Token.Generic.Error,
            "❯ ",
        ),
    ]

Git Branch #

Git branch was a bit tricky. There might be a better way to get it, but I was sticking with things I knew, the git cli and python. I did need to do a bit of googling to figure out that git has a --show-current option.

getting the current git branch

def get_branch():
    try:
        return (
            subprocess.check_output(
                "git branch --show-current", shell=True, stderr=subprocess.DEVNULL
            )
            .decode("utf-8")
            .replace("\n", "")
        )
    except BaseException:
        return ""

NOTE If this is run form a non-git directory you will quickly find git errors after every command as this function tries to ask for the git branch. Sending stderr to devnull will avoid this inconvenience.

add git branch to prompt

def in_prompt_tokens(self, cli=None):
    return [
        (
            (Token.Generic.Subheading, "↪"),
            (Token.Generic.Subheading, get_branch()),
            (Token, " "),
            (Token.Prompt, "©"),
            (Token.Prompt, os.environ["CONDA_DEFAULT_ENV"]),
            (Token, " "),
            (Token.Name.Class, "v" + python_version()),
            (Token, " "),
            Token.Prompt
            if self.shell.last_execution_succeeded
            else Token.Generic.Error,
            "❯ ",
        ),
    ]

Add current directory name #

I am a big fan of pathlib so that is what I will use to get the path. If I planned on using python <3.6 I would probably use something else, but this is what I know and I can’t think of the last time I used <3.6> for anything.

update imports

from pathlib import Path

add git branch to prompt

def in_prompt_tokens(self, cli=None):
    return [
        (
            (Token, ""),
            (Token.OutPrompt, Path().absolute().stem),
            (Token, ""),
            (Token.Generic.Subheading, "↪"),
            (Token.Generic.Subheading, get_branch()),
            (Token, " "),
            (Token.Prompt, "©"),
            (Token.Prompt, os.environ["CONDA_DEFAULT_ENV"]),
            (Token, " "),
            (Token.Name.Class, "v" + python_version()),
            (Token, " "),
            Token.Prompt
            if self.shell.last_execution_succeeded
            else Token.Generic.Error,
            "❯ ",
        ),
    ]

Final Script #

That’s it for my prompt at the moment. I have been using it for about a week. It seems to have everything I need so far, and skips on things I don’t need.

Enjoy the full script.

my final prompt

from IPython.terminal.prompts import Prompts, Token
from pathlib import Path
import os
from platform import python_version
import subprocess

def get_branch():
    try:
        return (
            subprocess.check_output(
                "git branch --show-current", shell=True, stderr=subprocess.DEVNULL
            )
            .decode("utf-8")
            .replace("\n", "")
        )
    except BaseException:
        return ""


class MyPrompt(Prompts):
    def in_prompt_tokens(self, cli=None):
        return [
            (Token, ""),
            (Token.OutPrompt, Path().absolute().stem),
            (Token, " "),
            (Token.Generic.Subheading, "↪"),
            (Token.Generic.Subheading, get_branch()),
            (Token, " "),
            (Token.Prompt, "©"),
            (Token.Prompt, os.environ["CONDA_DEFAULT_ENV"]),
            (Token, " "),
            (Token.Name.Class, "v" + python_version()),
            (Token, " "),
            (Token.Name.Entity, "ipython"),
            (Token, "\n"),
            (
                Token.Prompt
                if self.shell.last_execution_succeeded
                else Token.Generic.Error,
                "❯ ",
            ),
        ]

    def out_prompt_tokens(self, cli=None):
        return []


ip = get_ipython()
ip.prompts = MyPrompt(ip)

Want automatic imports?? #

Smoother Python with automatic imports | pyflyby

This article covers how I setup automatic imports in ipython