-
uv s3 sync with dotenv
I often want to run an s3 sync in an isolated environment, I don't want to set any environment variables, I don't want anything secret in my history, and I don't want to change my dotenv into something that exports variables, I just want s3 sync to work. is the tool that I've been using for this, and this uv one liner lets it run fully isolated from the project. one liner multi-line same thing formatted for readability There are probably 10 ways to skin this cat, but this is what I did, if yo
-
First 3d Printed Threads
Working on an upcoming project that requires some threaded screws. Trying to keep a low budget on this one with as much to come off of the printer as I can. It might become a slant3d portals product if it works out. I always like making test prints for stuff like this especially to see how the feel is off of the printer that is going to print the final product and take much longer. First try was a success. b485b759-719a-4aa0-aa8d-f98e0a5e1ac3-1080p.mp4 What worked I started out looking up stan
-
starlette head request
Starlette has a head request that works right along side your get requests. This morning I fiddled around with custom routes for and , but had to manually set some things about the file, and was still missing in the end. Turns out as a developer you can [[ just ]] add a route to your routes and starlette will strip the content for you, while preserving all of those good headers that fastapi created automatically for you. Here is an example of the response with curl.
-
python extras are for shipping
Python has two ways of adding optional dependencies to your projects pyproject.toml file and . dependency-groups for development Dependency grooups are used when working on the project, they do not ship with the project, users cannot select to install them with the project. These are for things like running tests, linting, or docs. You might want to run these in ci, or keep your dev machines tight. For the most part you can probably keep these in . Depending on your team, fluency, and to
-
Shots Get An Upgrade To the Main Feed
Today I updated my jinja templates and feeds to include shots posts in the main feed. These posts will prominently show the image in the feed with a small post for each one. screenshot-2025-09-21T01-19-39-934Z.png Shots are instagram style imge posts inspired by https://justin.searls.co/shots/.
-
zmk two hand hold down
I don't know about you, but I don't re-flash my keyboard enough to ever remember where I put the bootloader. Sometimes its the last thing I think about in a refactor and I end up cornering myself into a place where I cant get into that layer anymore. I've started putting hardware switches on my newer builds, but some older builds don't have a hardware one, so it requires disassembly and jumping the microcontroller. Even when I have one though, I gotta flip my board over and its annoying some
-
vanilla html hover text
I needed to display some hover text in a web app that I am using tailwind and jinja on. It has no js, and no build other than the tailwind. I want this to remain simple . Turns out that you can use a span with a title attribute to get hover text in HTML.
-
Why Make a Website in 2025
Inspired by Jim and Dave It's Fun If it's not something that you enjoy, you might as well move on there are far better ways to spend your time in 2025. Only the weirdos read this shit anymore, the masses have long moved on to curated social media feeds, and on to chatting with llms. I enjoy spending some time in the digital garden every once in awhile tweaking templates, creating [[ markata ]] plugins to do interesting things. I enjoy expressing my thoughts, or at least shouting them into th
-
uv script management
I've been leaning on lazy-self-installing-python-scripts more and more, but I did not realize how much tooling that uv gives you to help manage your scripts. The result is a script that looks like this, its executable as what looks like regular command in your shell.
-
Keycap Cut Away
I was curious how/if my custom keycap design was hitting my switches. So I set out to find out what the fitup inside of this assembly looks like, but not theoretically, a fully sliced view into their fit up in the flesh. The set up To setup for this cut, I flooded the edge of a 2x4 with hot glue, and inserted the cap such that the step was tangent with the edge. This way I could use the edge as a guide to cut one side off and leave the stem in tact. I took a handsaw to it and filed it smooth.
-
I'm Out On Agents
Its the year 2025 and we are only a few years into having 6 months to live before ai takes our jobs, and the big push right now is agents, managing agents. I will fully concede to I'm not doing it right, or a future state gets better than where we are right now, but right now they kinda suck. !!! Note Transparency Chat Chat is what really kicked off ai uses and goes back as old as computers, but it always sucked. Then chatgpt rocked the world with the biggest launch day in history and showed u
-
modd
Today I gave modd a try, and it seems like a good file watcher executor. I tried using libnotify to send desktop notifications, but all I got was modd, I might not have notifications setup right on the awesomewm machine. config goes in I installed it using installer from jpillora, pulling pre-built binaries right out of the github repo. Then you can install it, and on file change it will run the commands you configured.
-
qr codes in python
I have a couple of use cases for simple qr codes in python coming up. One is for blog posts, the other is for auth into a new server application logged to a terminal. I tried the library and it does not look as nice to me and I found to be quite nice. result Here is the final svg result. Here is what it looks like in the terminal. screenshot-2025-08-05T13-53-17-368Z.png If you want fancier qrcodes check out https://mydigitalharbor.com/
-
pygments htmlformatter
I've been a long user of pygments, it's been the thing that injects with funny little class names like and into the code blocks of my website. I've even gone as far as implementing a plugin for md-it , but I had no idea how to re-style it. I long ago got a theme that looked good enough from somewhere and just used it, maybe I pulled something from their docs site and forgot. Today I learned you can list all the themes easily from the library itself, and render out new css. And now you g
-
command palettes are overrated
Command palettes are slow, and overrated, you should treat yourself better. You probably installed VSC*** out of the box and your co-workers see you using the mouse and reprimanded you as they should. Mouse usage is not OK if you are a software dev, you should have the cheap ass free mouse that came with your cousins dell machine five years ago and only use if for emergencies. If you want to be fast you cannot do that by moving cursors to imprecise locations and clicking with your hand. You a
-
just fucking use kubernetes
You want to run containers? JUST FUCKING USE KUBERNETES. !!! tip "Obvious satire" If you don't like harsh language this is not the post for you. Obviously ripping off motherfuckingwebsite . !!! warning "ThIs is AI SLoP" If you don't like if you can fuck off to the next post, I'm having fun here, but satire is not my strong suit and needed some help. "But it’s complicated!" Shut up. Close twitter and fucking do something. Life is complicated. You know what else is complicated? Email. DNS. L
-
markata parallel render
-
principal-engineer-at-meta
Jake Bolam principal engineer at Meta, has some of the best career advice for those looking to become principal or just be better at their craft. This video was such a banger I had to bring it in as a full post, and not just a thought. It was a random YouTube auto play, something that I probably wouldn't have clicked on given title an thumbnail, but turned out to be very impactful. Jake is such a smart guy with a lot of great insights, and I can tell he thinks really quick on his feet, he jus
-
csv
-
copier vcs quirks
Copier has a few quirks with vcs that I just discovered by trying to test out some changes. I may have some config that I have long forgotten about somewhere deep in my dotfiles, I don't think so, but id love to be wrong and corrected, please reach out. What Doesn't Work I tried throwing everything at this template to make it work. I tried a bunch of flags that did not work. I tried making commits to the local repo to get rid of the dirty warning. I really wanted to test new changes locally wi
-
rich theme
The rich console is themeable, I've been a long time user of rich and had no Idea. You can define your own theme keywords and use them just like you use normal rich keywords in square brackets like .
-
kubernetes node labels
If you need to target a specific k8s node in the cluster, you can use labels. You want to treat your nodes as much like cattle as you can, but sometimes budgets get in the way. You might be like me and just run any free hardware you can get in your cluster, or you might have some large storage or gpu needs that you can't afford to put on every node in the cluster. To use the label in a pod set to the label that you applied.
-
javascript b64
I'm currently [[replacing-google-search-apps-with-self-hosted-web-apps]] and decided to create a simple b64 encoder/decoder, just start typing to enter text, escape to deselect, then e/d to encode/decode. I'm trying to make these apps super simple, self hosted out of minio, static html, and javascript. It's been fun to get back to some simple interactive web development like this. No build just a website that does something. No broken builds, no containers to deploy, just push to minio. Here i
-
web wakelock
I'm trying to replace my usage of google inline search apps with real apps, today I used a stopwatch to time some things out at work by opening stopwatch. This was something I just wanted running in a tab on another screen, it was not timing running code or anything, I was using it as a reminder to check browser caches every 5 minutes or so for some testing. So tonight I whipped up a stopwatch , clock and timer , all of which are using the wakelock API to keep the screen on while the app is r
-
m9a devlog 1
It's sad to see textualize.io close the doors, but textual is still alive and maintained as a n open source project. I tried to use it very early, and struggled, this was before docs and tutorials really existed, before a lot of the widgets and components existed. Then as we all do I got busy and moved on to other things in life and did not have the capacity to build TUIs. I like tuis I like tuis, I like staying in the terminal. I use lf daily to move files around when I want something more
-
grafana dashboards as k8s configmap
I'm trying to learn proper logs, monitoring, otel, and grafana. Today I imported a bunch of pre-made k8s dashboards and made a few of my own for specific apps, and it made me want to know how I can turn my own custom dashboards into infrastructure as code. Turns out grafana makes it pretty easy to do this, if you have the grafana dashboard sidecar running. It will pick up any ConfigMap with the grafana_dashboard label and import it. Go to Dashboards -> Pick a Dashboard -> Export -> JSON. imag
-
fix feed descriptions
Today I fixed a bug in markata that has been occurring for a few months where the description for posts come out as None if coming from cache, the issue was a pretty simple check and pull properly from cache. This fixes all the descriptions in feeds and metadata on the post. Better description While in there we went ahead and improved our get_description to more accurately return plain text without escaped characters, remove cutoff words, and add an elipsis if the description cuts off the text.
-
ruff single line import
I've been using ruff to lint my python code for quite awhile now, I was pretty early to jump on it after release. Some of my projects have had a nice force-single-line setting and some have not. I dug into the docs and it was not clear what I needed to make it work. Turns out I was missing I sort in the select list.
-
Markata list posts by year
I was looking back at my [[analytics]] page today and wondered what were my posts about back at the beginning. My blog is managed by [[markata]] so I looked at a few ways you could pull those posts up. Turns out it's pretty simple to do, use the markata map with a filter. !!! Note the filter is python eval that should evaluate to a boolean, all of the attributes of the post are available to filter on. Result You could use the command as well right within your shell and the same map and filte
-
portal-platformer-devlog-1
Here is the current state of my platformer yet to really be named, I want to make something in between hollow knight and portal. Starting I made one once in make code arcace on a pybadge. It was quick and dirty, but fun to work on. It had the basic of blocks that I could move, blocks i could put a portal onto, and a goal for each level. Some levels you can just walk through and some levels required you to really think about where to place the portal. History So this version of the game is a l
-
backup distrobox image
Today I'm upgrading my distrobox, but don't want to end up in a situation where I can't get anything done becauase I work out of my distrobox.
-
Changing k8s Storage Class - Migration Job
I'm setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I'm all in on gitops so option 1 was not an option. So changing storageclass it is. Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute. Migration Job This migration job will create a new pvc with the
-
pre-commit exclude
I run tailwind for my personal blog, whenever I update it, pre-commit goes in and fixes end of file. I'm sick of these things fighting each other, since it is a generated app it is going to et ignored from pre-commit from now on.
-
newlines in css before
I'm building in a [[ fragmentions ]] implementation into my blog, I wanted to add some text before the fragment to indidate that it was the highlighted fragment that someone may have intended to share with you. To get a newline in a I need to use and . Here is what it looks like on my not yet live implementation of fragmentions. screenshot-2025-02-15T15-43-06-372Z.png
-
fragmention
This post is still WIP. ..... https://indieweb.org/fragmention##Challenges I've been digging through David Bushell's blog over the past day, he has some really good ideas about blogging and webdev. One really interesting post I came accross is url-fragment-text-directives . I've long had id's linked on my headings, though sometimes broken, or now showing the link, I've done my best to include them. Fragmentions extend this to allow any text to be linkable like this. fragmentioner ui: https://
-
testing nvim installs
Testing fresh nvim installs can be a pain, and hard to di without borking your known good install. I've been using to run a test nvim in a sandbox that wont bork my main install. This usually runs for me in under a minute, can be down under 15s if I remove some of the TreeSitter installs at the end. This beats a full docker build of my full devtainer to test out nvim packaging woes. I've started to use this as a recipe to run before deploying a new version of my dotfiles. So far its pairi
-
markdown split panel
Today I was playing with markdown split panels. I want to be able to compare and constrast occasionually, today the inspiration hit to do this using admonitions. screenshot-2025-02-04T02-28-46-750Z.png {.more-cinematic} {.rounded-xl } !!! Note Mobile Users 🔄 The Markdown This is what I am going for, one admonition that is easy to remember, that nests inside of itself , and I can put as much markdown on the inside that I want. Here is the result of that markdown. !!! vsplit I Have two opinions {
-
tailwind aspect
I've been back to putting some images on my blog lately and thinking about making them a bit thinner through the use of aspect ratio for simplicity. I'm leaning pretty heavy on tailwindcss these days due to some weird quirks of markdown-it-attrs I cannot have slashes in classes from markdown so I made a class to achieve this. Example screenshot-2025-01-31T14-50-00-094Z.png {.cinematic}
-
markdown it attrs with slashes dont work
Attrs does not like '/' characters in its classes, so to use some tailwind classes with custom values we must make new classes in our tailwind input css. Given the following markdown with attrs added to the image and to the paragraph block. We get the following output with only the middle one working correctly. screenshot-2025-01-31T14-50-00-094Z.png {.aspect-[2.39/1]} screenshot-2025-01-31T14-50-00-094Z.png {.cinematic} {.cinematic} screenshot-2025-01-31T14-50-00-094Z.png !!! Note The inline ve
-
markata search
A side effect of [[ markata-didyoumean ]] is that we are able to implement some rudimentary search based on slug, title, tags, and description. screenshot-2025-01-31T14-50-00-094Z.png {.more-cinematic} I made a video of the early work on using this side effect to search for content on markata.dev. Replay markata-search-1.mp4 This was first implemented to solve 404 pages giving users a way to try to find the page that they tried to access. But it turned out to be a decent way to search through t
-
Markata DidYouMean
Coming in Markata 0.9.1 is far better documentation. i.e. Documentation that actually exists for everything. As part of poking around I realized that I often go to look up the docs for a plugin and forget that the path is , sometimes I might try or or or I might even forget the plugin name exactly and try something like and get a 404. So I added a plugin to markata that takes care of this. screenshot-2025-01-31T14-53-31-264Z.png {.more-cinematic} I made a quick recording of this earl
-
pesos
Pesos is the act of Publish Elsewhere Syncicate to Own Site. It is an indieweb concept that I recently started applying to my own site. here does it skip again !!! note See https://indieweb.org/PESOS for more information, they have a ton of information about the indieweb In short it is the concept of pulling data from other sites that you use and republishing it to your own site. This gives a single source of information for you, and protection against sites and apis changing or rug pulling.
-
hover z-index and positioning
I broke my [[ sick-wikilink-hover ]] recently in a refactor, today I did some diving in to figure out what happened. Before As you can see in the screenshot below, the link is in a list of links, and when the hover image pops up it sits behind all of the other text. The z-index of the list-item is supposed to be raised above the others on hover. image Manually setting z-index to 20 in the inspector I noticed this message from devtools, "The position: static property prevents z-index from havin
-
I fixed lists in my blog
A small improvement to my blog today. I was checking out my [[ now ]] post and noticed that lists have left padding but no discs or numbers. I popped open my added these lines and rebuilt. And rebuild, which I actually do with [[ casey-just ]], but showing the command here for clarity. WHY the applies {.bg-blue-900 .border-r-8 .border-black .rounded-xl .max-w-xl .p-6 .font-bold} Dude, you know that @apply is an anti-pattern in tailwind right? {.bg-green-900 .border-l-8 .border-black .rounded-x
-
emoji in headless chrome in docker
I recently noticed that my og images were missing emoji. They were taken using headless chrome in a container. I fixed it by adding an emoji font in the containerfile / dockerfile. Before Here's what they were looking like with broken emoji fonts. image After And now with the fixed emoji font. image I put thought bubbles on my thoughts posts and stars on my github stars posts
-
one-shot-s3-cleanup
So I made a mistake in an app I am working on and ended up creating thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails... etc. I was able to delete them all, but I wanted to make a one-shot script to do it. screenshot-2025-01-31T21-20-56-934Z.png One Shot App I got the idea of the one shot app from Simon Willison and replicated his setup in a chatgpt project screenshot-2025-01-12T18-16-57-443Z.png Initial Prompt s3_cleanup Here is the result of the s3_cleanup script. It ende
-
nvim-manager
I recently built a cli application as a nearly-one-shot-app called nvim-manager . It manages your nvim dotfiles install. screenshot-2025-01-31T21-21-40-707Z.png {.more-cinematic} Why {.chat-left} Don't we have stow? {.chat-right} Ya, thats not enough. {.chat-left} Why not?? {.chat-right} Inevitably shit goes sideways and I break my vim install. How is nvim manager any better nvim-manager allows you to install pinned versions of your dotfiles, your friends dotfiles, and distros in ~/.config. Thi
-
postiz-file-upload
Today I learned that the docs in postiz are a bit behind, (fantastic docs btw, they are to the point, and cover almost all of what you need). The docs state that you need to include an R2 bucket to handle uploads. This issue shows that more work has been done, one of which is local storage. The compose file they use in the quick start has the required env variables to set this up. looking into my running instance I can see my images there.
-
/colophon
Colophon a page that describes how the site is made, with what tools, supporting what technologies Author Waylon Walker's Profile Picture All posts on this site are written by Waylon Walker , the typical content has changed and evolved over time. I go back and make a few corrections, but for the most part things stay pretty much as they were published originally. see more in [[ about ]] tech This site is a static site build with my own static site generator [[ markata ]], [[ thoughts ]] or as
-
slow nfs performance
I'm running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I'm already learning why this is not reccommended. screenshot-2025-01-31T21-18-43-543Z.png Slow I've been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don't care if they take a bit longer to keep my master node freed up for more critical work. I just sta
-
thoughts 0.0.4
This is such an improvement to the backend of my website it warrants a blog post of celebration. For far too long I've been dealing with a tiny ass edit form on thoughts. I tend to not edit them, and try to get them right in one go. This is kinda the point of a thought, its a quick post meant to be the size of a tweet, but sometimes I'm leaving thoughts on a video or long post and want to make sure I have a good save point, but I just keep the thing in draft and hope I don't loose if for far
-
Links rely on color to be distiniquishable
Today i got hit by this accessibility issue on my site. Low contrast links are not distiniquishable. I had not seen this error title before it was new to me, maybe I have bad memory or maybe it's new to me. screenshot-2024-12-18T02-25-53-014Z.png I ended up dropping the background color of the site down a notch as I didn't really care for the semi-dark brown anyways. I'm liking the near black much better now. screenshot-2024-12-18T02-45-53-807Z.png Now I got that 100 A11y score in lighthouse
-
price an stl print on slant3d
I've been playing with 3d printing some items through the slant3d api. I've been pricing out different prints by running a slice request through their api. make a project I've been using uv for project management. It's been working well for quick projects like this while making it reproducible, I'm still all in on hatch for libraries. Get an api key You will need an api key from the slant api, which currently requires a google account and a credit card to create. slicing an stl with teh slant a
-
k3s config after first install
After first setting up a new k3s instance your kubeconfig file will be located in /etc/rancher/k3s/k3s.yaml. You cans use it from here by setting $KUBECONFIG to that file. Or you can copy it to If you have installed k3s on a remote server and need the config on your local machine then you will need to modify the server address to reflect the remote server. !!! warning only do this if you don't already have a ~/.kube/config file, otherwise copy it to a new file and set your $KUBECONFIG env varia
-
Obsidian-new-file
Wikilinks are a core concept within obsidian to link to documents by [[slug]] wrapped in double square brackets. These are commonly used within wiki site generators. Obsidian gives you a keybinding to go to that file, but if it does not exist it will create the file for you in the root of the project. It's a nice way to quickly make new documents.
-
Slug
A slug is the part of the url that comes after the domain. Commonly matches the file name of a markdown file many blogging systems. These are typically human readable, unique identifiers for pages within the site.
-
Obsidian-go-to-definition
It was not obvious to me, but if you have a wikilink such as [[trying-obsidian]], you can jump to the file in obsidian, just like you can with lsp go to definition, the keybinding is alt + enter.
-
Obsidian Using Templater Like Copier
I've long used copier to create all of my posts for my blog, and it works really well for my workflow. I think of a title, call a template, and give it a title. out of the box obsidian did not seem to work this way. It seems like it wants me to right click a file tree and make a new file using the tree, this is not my jam. Here is what I came up with to replace my til template. is a return value, and it gets placed directly into the place it is in the file creates a tab-index point so I can
-
Obsidian Image Converter
I'm giving obsidian a go as an editor for my blog and one of the main things I want to fix in my workflow is the ability to quickly drop in images. on first look through the community plugins I found Image Converter. I set it up to convert to webp and drop them in a git submodule. I may make it something other than a git repo in the future, but I've learned that adding images to my blog repo quickly makes it heavy and hard to clone on other machines. obsidian-image-converter-20240731211310793
-
k8s kustomize diff
I've started leaning in on kubernetes kustomize to customize my manifests per deployment per environment. Today I learned that it comes with a diff command. You can enable color diffs by using an external diff provider like colordiff. You might need to install colordiff if you don't already have it. Now I can try out kustomize changes and see the change with kustomize diff.
-
kubectl dash k
Kubernetes ships with a feature called kustomize that allows you to customize your manifests in a declarative way. It's a bit like helm, but easier to use. I…
-
diskcache as debounce
I've been using fastapi more and more lately and one feature I just started using is background tasks [[ thoughts-333 ]]. !!! seealso basic diskcache example [[ python-diskcache ]] One Background Task per db entry I am using it for longer running tasks and I don't want to give users the ability to spam these long running tasks with many duplicates running at the same time. And each fastapi worker will be running in a different process so I cannot keep track of work in memory, I have to do it in
-
Install sealed-secreats via manifest
Yesterday I realized that I have overlooked the default installation method of the sealed secrets controller for [[ kubernetes-kubeseal ]] this whole time an jumped straight to the helm section. I spun up a quick [[ kind-cluster ]] and had it up quickly. I can't say this is any better or worse than helm as I have never needed to customize the install. According to the docs you can customize it with [[ kustomize ]] or helm.
-
kind cluster
kind {.hoverlink} is a very useful tool to quickly standup and teardown kubernetes clusters. I use it to run clusters locally. Generally they are short lived clusters for trying, testing, and learning about kubernetes. Kind is Kubernetes in Docker, its very fast to get a new cluster up and running. Other than checking a box in docker desktop it is the easiest way currently to get a cluster up and running. I've used docker desktop for k8s before I really developed on k8s and it was buggy at t
-
tailwind post template
Today I am playing around with tailwind, flexing the css muscle and learning how to build new and different layouts with it. I created a new post template that mimics a terminal look in css where I could inject the post title, description, and other frontmatter elements. I think this is a pretty cool layout, I could make a carbon.now.sh {.hoverlink} clone or more realistically I could make it into a template for blog pages and this could become og images. {.mx-auto .block .w-fit .mb-0} Check out
-
Latest Page in Markata
I just implemented a latest blog post link in Markata by asking for the first post slug from the blog feed. The implementation uses the jinja_md plugin to render jinja against the markdown and a tag to redirect. Setting up the feed Feeds are setup in configuration. They provide a handy way to create an html feed, rss feed, and quickly reference a filtered set of posts like this. For more information on markata check out the full [[ markata ]] post.
-
markata
This post is a work in progress. Markata is the static site generator that I created to build my website [[ about-this-site ]]. I built it for me and I enjoy using it. I know everying it can do and I can extend it to do more easily. I have set it up for some friends to also use it and am proud that it helps them publish their content. It's a meme to create your own static site generator to make your website. Yes its funny, I don't recommend it if your not ready for the level of work that com
-
sick wikilink hover
Today I set up some sick wikilink hover effects using tailwind see [[ a-case-for-tailwindcss ]]. When you hover over them they show an image preview of the link that you are going to. I cant find where I have seen this but it comes from some docs sites. I'll finish this article later, just excited to see it up.
-
control minio token access
To allow access only to the , you can pass add the Resource field to the User Policy when you create a new token.
-
How to List Sqlite tables
You can inspect sqlite tables with the sqlite shell. note that you get into the shell with I also learned that , and are helper functions that query the table on the database. Here is an output from my redka database. The sqlite_master table contains all the sqlite objects type, name, tbl_name, rootpage, and sql to create them.
-
Redka Runs on SQLite
With the liscense changes to redis there are several new forks out there. One that I am particularly interested in is redka . We now have redis running on port 6379 that we can connect to with a redis client. And we have a sqlite database that we can inspect. We can look at the values in the vstring table.
-
copier trust
I recently had to update my copier-gallery command to trust my own templates because some of them have shell scripts that run afterwards. Be warned that this could be a dangerous feature to run on random templates you get off the internet, but these are all mine, so if I wreck it its my own fault. All the the copier copy api can be found with help.
-
ipython f2
Today I accidentally ran f2 in ipython to discover that it opens your $EDITOR! I use this feature quite often in zsh, it is bound to for me, and since I have my environment variable set to it opens nvim when I hit . Today I discovered that Ipython has this bound to . If you know how to set it to let me know I've tried, a lot. better yet add to your .zshrc
-
python inline snapshot
is a new tool that I am trying out for python testing. It takes snapshots of your outputs and places them inline with the test. Here is the most basic starter. Now when I run my tests will fail because my assert has no value, but if I run it will fill out my snapshot values and the file will then look like this. It also works with pydantic models.
-
sqlmodel indexes
I've really been enjoying using sqlmodel for my projects that need a database. One thing that I definitely lacked on for too long was indexing my database. I hit a point with one database where it was taking 7s for pretty simple paginated queries to return 10 records. For every field that you will be querying on, you can create an index, by setting it equal to example courtesy of the docs !!! Note primary keys are indexed by default. The docs cover this pretty well, and in quite depth - Optim
-
sqlite vacuum
Today I learned how to VACUUM a sqlite database and cut its size in about half. It's a database that I have had running for quite awhile and has some decent traffic on it. Why is it important to do a VACUUM? In short its becuase the file system gets fragmented with as data is updated. On delete the files are removed from the database and marked as available for reuse in the filesystem, but the space is not reclaimed. To VACUUM a database, run the following sql command. You can do it right form
-
Tailwind Animations
I learned that tailwind animations are pretty easy to add only needing a few classes. For some reason though my brain broke, thinking that I could dynamically change the number and you can't cause there are only so many pre compiled classes without using an arbitrary value with brackets. Here are the classes that I used to transition my colors very slowly. And the entire square element.
-
Composing Typer clis
Typer makes it easy to compose your cli applications, like you might with a web router if you are more familiar with that. This allows you to build smaller applications that compose into a larger application. You will see similar patterns in the wild, namely the cli which always has the pattern. Lets setup the cli app itself first. You can put it in . Creating an app that will become a command is the same as creating a regular app in Typer. We need to create a callback that will become o
-
tailwind arbitrary values
I learned not to fear the arbitrary size feature of tailwind. While building out reader.waylonwalker.com I kept getting content flowing off the screen, and struggling to keep it on the screen. I really felt that I should be able to do this with vanilla tailwind, but after some encouragement from Twitter I decided to lean on arbitrary values and it worked. Don't fear the arbitrary values. Learn more about using-arbitrary-values from their docs docs
-
fix npm global install needs sudo
Each time I go to set up npm I am frustrated by the errors saying that I don't have permission to , and it's frustrating. And I forget what I need to do to tell npm to install packages in a directory I own, and my shell to look there so that I can use the executables. For the fix to remain persistent you need to put these two lines in your shell profile like or .
-
darkmode scrollbars
If you are designing a website in dark mode the scrollbars can be finicky to match the theme. Here is a pretty sane default that looks nice without being obnoxiously contrast to the rest of the site. Want a rounded scrollbar thumb? add these styles. This makes a very nice looking default darkmode scrollbar.
-
One Day Build - Play Outside
Inspired by Adam Savage and his One Day builds on youtube. I often build things, and want to make them generally useful for others and over configure out of the gate. This project is purely for me inspired by a need I have. play-outside !How-To This post will not directly show how to make a weather app, but document the process that I went through to make mine. It will show the tools that I used to make it, and the final result. The Situation It often goes in our house ask dad while he is bus
-
Udating Cloudflare Pages using the Wrangler cli
Before deploying to cloudflare pages with wrangler you need a cloudflare api token. You can get one at dash.cloudflare.com/profile/api-tokens . cloudflare-pages-api-token.png Install Wrangler Next install wrangler using npm. Create a Project Before you deploy to cloudflare pages you need to create a project. You might already have one, or you might want to create one in the webui, but you have the option to create it at the command line with wrangler. Deploy Now you can deploy your static appl
-
scheduling cron jobs in kubernetes
For my reader app I am using cronjobs to schedule my a new build and upload to cloudflare pages every hour. In this example I have built a docker image and pushed it to dockerhub. It uses a secret to access cloudflare, and the entrypoint itself does the build and upload.
-
jinja macros
I am working on a page for htmx-patterns and I ran into a situation with lots of duplication. Especially when i am using tailwind I run into situations where the duplication can get tedious to maintiain. The solution I found is macros. Now I can use the same code for all of my links, and call the macro to use it.
-
jinja loop variable and htmx
jinja has a loop variable that is very handy to use with htmx. Whether you want to implement a click to load more or an infinite scroll this loop variable is very handy. Now for every chunk of contacts that we load we will trigger the infinite scroll by loading more once the last one has intersected the screen.
-
fastapi jinja url_for with query params
Out of the box Starlette does not support url_for with query params. When trying to use url_for with query params it throws the following error. In my searching for this I found starlette issue #560 quite helpful, but not complete, as it did not work for me. !!! Note "https" If you want url_for to work in production you need some way to convert http to https. Here is how I make it work, for local development I then for each environment that I am running on a server I include it in the list
-
setting up a kind cluster with argocd installed
Kind (Kubernetes in Docker) is a tool that makes it easy to create and tear down local clusters quickly. I like to use it to test out new workflows. Argocd is a continuous delivery tool that makes it easy to setup gitops workflows in kubernetes. Here is how you can setup a new kind cluster and install argocd into it using helm, the kubernetes package manager. If you want to add repos and apps to your cluster you can use the argo cli to do that, but first you will need forward the argocd port an
-
kubernetes kubeseal
In my homelab kubernetes cluster I am using kubeseal to encrypt secrets. I have been using it successfully for a few months now wtih great success. It allows me to commit all of my secrets manifests to git with out risk of leaking secrets. You see kubeseal encrypts your secrets with a private key only stored in your cluster, so only the cluster itself can decrypt them using the kubeseal controller. kubeseal-post.png KubeSeal https://sealed-secrets.netlify.app/ {.hoverlink} installation Installa
-
kubernetes 6 months in
I stumbled into kubernetes December 2023 when I was looking for a better way to self host applications. I was looking for something that didn't require logging into a server and building and deploying like a cave man. I wanted a smoother experience than docker compose was giving me. https://waylonwalker.com/looking-for-a-heroku-replacement/ {.hoverlink} This post turned into a list of tools that I have adopted into my k8s workflow, and plan to keep. enjoy. Kompose Kompose is a great tool for g
-
thoughts on unit tests
thoughts on unit tests cover image Theo's response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements "Everything must be unit tested, tests make code more maintainable." In reality when we are not writing complex low level code unit tests are probably the wrong approach. thought 192, a thought about theo's reaction to prime's unit testing Most of us are assemblers So much of software engineerin
-
Hiding Form input During htmx Request
I am working on fokais.com's signup page, and I want to hide the form input during an htmx request. I was seeing some issues where I was able to prevent spamming the submit button, but was still able to get one extra hit on it. It also felt like nothing was happening while sending the email to the user for verification. Now I get the form to disappear and a spinner to show during the request. HTML Let's start off with the form. It uses htmx to submit a post request to the route. Note that there
-
Stripe Cancellations in FastAPI and SQLModel
Today I am working on fokais.com, trying to get to a point where I can launch by workig through stripe integrations. This is my first time using stripe, so there has been quite a bit to learn, and I am probably building in more than I need to before launching, but I am learning, and not in a rush to launch. I am building the fokais backent in python primarilyt with fastapi and sqlmodel on sqlite. My billing integration is going to be all Stripe. Stripe Subscription Cancellations Docs Here is a l
-
Looking for a Heroku replacement, What I found was...
I've long hosted my personal blog as a static site on waylonwalker.com. It's all markdown, converted to html, and shipped as is. It's been great, I've moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it's just pointing ci to a different host after the site has built. What about server side Now the part that I have struggled with is how to cheaply host a server rendered application that can just l
-
Still Loving Tailwind
I've been using tailwind for a few months now and I can still say I'm loving it. I've been using it to create some rapid prototypes that may or may not ever become something, a document that is likely to go to print (a resume), and some quick dashboards. I started using Tailwind a few month back A few months back in september of 2023 I made a case for tailwindcss . And have been using it on quite a few projects since. values are well thought out it's really easy to use classes that make sense t
-
how to host static content with fastapi
I wanted to host some static files through fastapi. Typical use cases for this might be some static web content like html/css/js. It could also be images or some data that doesn't need dynamically rendered. From the Docs The docs cover how to host static files, and give this solution that is built into fastapi. https://fastapi.tiangolo.com/tutorial/static-files/ {.hoverlink} Authenticated Static Files Thanks to #858 . OscartGiles posted this solution to add authentication to static files. I tr
-
Set up minio bucket entrypoint
I recently se tup minio object storage in my homelab for litestream sqlite backups. The litestream quickstart made it easy to get everything up and running on localhost, but I hit a wall when dns was involved to pull it from a different machine. Here is what I got to work First I had to configure the Key ID and Secret Access Key generated in the minio ui. Then set the the s3 signature_version to s3v4. Now when I have minio running on https://my-minio-endpoint.com I can use the aws cli to acces
-
why-is-postgres-default
Serious question. No one ever got fired for choosing PostgreSQL But, why. It's the most loved db, right? Right? Maybe it's time to rethink it. Don't get me wrong, if I need a relational db as a service, PostgreSQL is going to be my first choice, but why do I need to run a separate application for it? Tutorials use sqlite Why is that? Because there is nothing else to stand up. Nothing else to maintain. And you probably already have it installed on just about anything that has a battery. SQLite ru
-
Setup Tailwind for Jinja
I've recently given tailwindcss a second chance and am really liking it. Here is how I set it up for my python based projects. https://waylonwalker.com/a-case-for-tailwindcss {.hoverlink} Installation is used to install the cli that you will need to configure and compile tailwindcss. Setup You will need to create a tailwind.config.js file, to get this you can use the cli. Using tailwind with jinja templates To set up tailwind to work with jinja templates you will need to point the tailwind conf
-
A Case For Tailwindcss
I was watching @theprimeagen recently and I think he sold me on using tailwindcss. The thing about tailwind is that it is not a big component library, it's a set of css classes mapped to a few (usually one) style. All css classes are shitty, so you might as well use someone else's shitty css classes on all your projects rather than thinking you're being smart with a new set of classes that you will hate in 6 months when you come back to the project. roughly quoted from memory of @theprimeagen I
-
Python scandir ignores hidden directories
The next version of markata will be around a full second faster at building it's docs, that's a 30% bump in performance at the current state. This performance will come when virtual environments are stored in the same directory as the source code. "One lone jedi stands in Glowing chains of interconnected network of technological cubes, in the middle of a futuristic cyberpunk dubai city, in the art style of dan mumford and marc simonetti, atmospheric lighting, intricate, volumetric lighting, beau
-
Playing Star Wars Text Adventure with a 10 yr old
The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star
-
Pydantic and singledispatch
I was reading about pydantic-singledispatch from Giddeon's blog and found it very intersting. I'm getting ready to implement pydantic on my static site generator markata , and I think there are so uses for this idea, so I want to try it out. The Idea Let's set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use and type hints to provide unique execution for each environment. We might want something like
-
pipx textual devtools
I really like having global cli command installed with pipx. Since textual (the css release) is out I want to be able to pop into textual devtools easily from anywhere. "rusting tape machine robot, cinematic lighting, detailed, cell shaded, 4 k, warm colours, concept art, by wlop, ilya kuvshinov, artgerm, krenz cushart, greg rutkowski, pixiv. cinematic dramatic atmosphere, sharp focus, volumetric lighting, cinematic lighting, studio quality" -s50 -W832 -H416 -C12.0 -Ak_lms -S2404332231 Pipx In
-
textual app devtools
I am working through the textual tutorial, and I want to put it in a proper cli that I can pip install and run the command without . This is a fine pattern, but I also want this to work when I don't have a file to run. "An astronaut working in a lab, hacking on a computer terminal, htop is running, shallow depth of field beakers, test tubes, volumetric lighting, pink lighting, by victo ngai, killian eng vibrant colours, dynamic lighting, digital art" -s50 -W768 -H448 -C7.5 -Ak_lms -S3617210203
-
localstack status permission
I ran into an issue where I was unable to ask localstack for its status. I would run the command and it would tell me that it didn't have permission to read files from my own home directory. Let's fix it The issue I would run this to ask for the status. And get this error What happened It dawned on me that the first time I ran localstack was straight docker, not the python cli. When docker runs it typically runs as root unless the Dockerfile sets up a user and group for it. "cell shaded, long,
-
Markata Supports Jinja Plugins 0.5.0.dev2
Markata now allows you to create jinja extensions that will be loaded right in with nothing more than a . From the Changelog The entry for 0.5.0.dev2 from markata's changelog Created entrypoint hook allowing for users to extend marka with jinja exensions #60 0.5.0.dev2 "cybernetic soldier working on a rusting tape machine robot, cinematic lighting, detailed, cell shaded, 4 k, warm colours, concept art, by wlop, ilya kuvshinov, artgerm, krenz cushart, greg rutkowski, pixiv. cinematic dramatic a
-
Getting Started with Django REST framework
In my adventure to learn django, I want to be able to setup REST api's to feed into dynamic front end sites. Potentially sites running react under the hood. cell shaded full body shot of a cybernetic blue soldier with glowing eyes working ina lab, llustration, post grunge, pink glowing volumetric lighting, engulfed in smoke and fog, concept art by josan gonzales and wlop, by james jean, Victo ngai, David Rubín, Mike Mignola, Laurie Greasley, highly detailed, sharp focus,alien,Trending on Artstat
-
Markata now uses hatch
Markata now uses hatch as its build backend, and version bumping tool. , and are completely gone. "An astronaut working in a lab, there is a series of eggs ready to hatch baby snakes on the table, experiments running, beakers, test tubes, cyberpunk trending on artstation, neon lighting, volumetric lighting, pink lighting" -s50 -W800 -H450 -C7.5 -Ak_lms -S4048189038 {.more-cinematic} 0.5.0 is big Markata 0.5.0 is now out, and it's huge. Even though it's the backend of this blog I don't actuall
-
django create superuser
My next step into django made me realize that I do not have access to the admin panel, turns out that I need to create a cuper user first. "cybernetic soldier working on a rusting tape machine robot, cinematic lighting, detailed, cell shaded, 4 k, warm colours, concept art, by wlop, ilya kuvshinov, artgerm, krenz cushart, greg rutkowski, pixiv. cinematic dramatic atmosphere, sharp focus, volumetric lighting, cinematic lighting, studio quality" -s50 -W832 -H416 -C12.0 -Ak_lms -S3309980874 Run Mig
-
django disallowed host
I am continuing my journey into django, but today I am not at my workstation. I am ssh'd in remotely from a chromebook. I am fully outside of my network, so I can't access it by localhost, or it's ip. I do have cloudflared tunnel installed and dns setup to a . Settings I found this in and yolo, it worked first try. I am in from my remote location, and even have auth taken care of thanks to cloudflare. I am really hoping to learn how to setup my own auth with django as this is one of the t
-
Trying out django
I have no experience in django, and in my exploration to become a better python developer I am dipping my toe into one of the most polished and widely used web frameworks Django to so that I can better understand it and become a better python developer. If you found this at all helpful make sure you check out the django tutorial "An atronaut working in a lab, there is a long snake working along side, shallow depth of field beakers, test tubes, volumetric lighting, pink lighting, by victo ngai,
-
how to escape variables in jinja templates
While updating my site to use Markata's new configurable head I ran into some escaping issues. Things like single quotes would cause jinja to fail as it was closing quotes that it shouldnt have. Nuclear core being help up by glowing neon wires, cyberpunk synthwave, intricate abstract. delicate artwork. by tooth wu, wlop, beeple, dan mumford. pink volumetric lighting, octane render, trending on artstation, greg rutkowski very coherent symmetrical artwork. cinematic, hyper realism, high detail, o
-
I've made my ipython config too complicated, let's...
When I am developing python code I often have a repl open alongside of it running snippets ofcode as I go. Ipython is my repl of choice, and I hace tricked it out the best I can and I really like it. The problem I recently discovered is that I have way overcomplicated it. What Have I done?? So in the past the way I have setup a few extensions for myself is to add something like this to my directory. It sets up some things like rich highlighting or in this example automatic imports. I even
-
markata extend in configuration
Astronauts stunting some stylish color explosion {.more-cinematic} A long needed feature of markata has been the ability to really configure out templates with configuration rather. It's been long that you needed that if you really want to change the style, meta tags, or anything in the head you needed to write a plugin or eject out of the template and use your own. Adding some Head Now you can add some extra style to your site with the existing built-in template. You can have more than one Hea
-
Create a new Python Project with the Hatch Cli
I'm really getting into using hatch as my go to build system, and I am really liking it so far. I am slowly finding new things that just work really well. is one of those things that I didn't realize I needed until I had it. Hatch new cover image creating new versions created by myself with stable diffusion hatch-new-cli.webp Note! I am running all of these commands with pipx. I like to use pipx for all of my system level cli applications. To emphasis this point in the article I am going to
-
Versioning Python Projects with Hatch
Hatch has an amazing versioning cli for python packages that just works. It takes very little config to get going and you can start bumping versions without worry. Hatch version cover image {.more-cinematic} creating new versions created by myself with stable diffusion project layout For trying out the cli let's make a simple project with the terrible name . pyproject.toml The main hero of this post is the . This is what defines all of our PEP 517 style project setup. statically versioni
-
From Markdown to Blog with Markata | lightning tal...
Markata is a great python framework that allows you to go from markdown to a full website very quickly. You can get up and running with nothing more than Markdown. It is also built on a full plugin architecture, so if there is extra functionality that you want to add, you can create a plugin to make it behave like you want. Full transparancy... I built markata. The talk The talk is live on YouTube. Make sure you check out the other videos from the conference. There were quite a few quality t
-
I spoke at Python Web Conf 2022
I spoke at python webconf in March 2022 about how I deploy this blog on a continuous basis. Building this blog has brought me a lot of benefits. I have a set of custom curated notes to help describe a problem and how to solve it to me. At theis point it's not uncommon to google an Issue I am having and finding my own blog with exactly the solution I need at the top. I also bump into people from time to time that recognize me from the blog, its a nice conversation starter, and street cred. The
-
Packages I Maintain
Markata I open sourced the static site framework that I use to build my-blog among other side projects. It's a plugins all the way down static site generator, that makes me happy to use. {% gh_repo_list_topic "waylonwalker", "markata" %} Repos used to build this blog my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don't have to worry about carrying around large images with my lightweight text files just to make some posts. {%
-
I turned on 2FA for all of my pypi packages
I got all the pypi packages that I own behind 2 factor authentication. 💪 Recently this really made it's rounds in the python news since pypi was requiring critical package maintainers to have 2FA on and even offering them hardware tokens to help them turn this on. I feel like this caused a bit of confusion as turning on 2FA does not mean that you need to do anything different to deploy a package, and it DOES NOT require a hardware token. You can continue using your favorite 2FA app. You might
-
I was on Talk Python
After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science. I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on. During the episode I tried m
-
use vim to remove duplicates
I just love how some features of vim are so discoverable and memorable once you really start to grasp it. Sorting and uniqing your files or ranges is one of those examples for me. I recently used this to dedupe my autogenerated links section for rich-syntax-range-style . More often I am using it to sort and uniqify objects like arrays and lists. Here is what the markdown looks like. Then typing yields a uniqly sorted list of links.
-
Highlighting text ranges with Rich | python
Today I've been playing with py-tree-sitter a bit and I wanted to highlight match ranges, but was unable to figure out how to do it with rich , so I reached out to @textualizeio for help. https://twitter.com/_WaylonWalker/status/1562469770766589952 While waiting for that reply let's show how we got this far. imports Lets import all the classes that we need from rich and setup a console to print to. some code Now we need some code to highlight. I am going to rip my from another post . prin
-
markata 0.3.0 is 15-20% faster
image from Dall-e {.more-cinematic} a sprinter edging out his opponent by Dall-e It's about time to release Markata 0.3.0. I've had 8 pre-releases since the last release, but more importantly it has about 3 months of updates. Many of which are just cleaning up bad practices that were showing up as hot spots on my reports Markata started off partly as a python developer frustrated with using nodejs for everything, and a desire to learn how to make frameworks in pluggy. Little did I know how f
-
get the version of any python package | python api
A common meta thing that I need in python is to find the version of a package. Most of the time I reach for , but that does not always work. but not all projects have a In searching the internet for an answer nearly every one of them pointed me to . This works for most projects, but is simply a convention, its not required. Not all projects implement a , but most do. I've never seen it lie to me, but there is nothing stopping someone from shipping mismatched versions. If you maintain a pro
-
IPython q to exit
So many terminal applications bind to exit, even the python debugger, its muscle memory for me. But to exit ipython I have to type out . This is fine, but since q is muscle memory for me I get this error a few times per day. After digging way too deep into how IPython implements its I realized there was a very simple solution here. automatically imports all the scripts you put in your profile directory, all I needed was to create with the following. It was that simple. This is not a gam
-
Copier Slugify | python templating | using cookiec...
It's no secret that I love automation, and lately my templating framework of choice has been copier. One hiccup I recently ran into was having spaces in my templated directory names. This makes it harder to run commands against as you need to escape them, and if they end up in a url you end up with ugly all over. Cookiecutter has the solution Yes the solution comes from a competing templating framework. I install copier with pipx, so I need to inject cookiecutter in to my copier environment
-
Textual has devtools
Textual has devtools in the upcoming css branch, and its pretty awesome! It's still early Textual is still very early and not really ready for prime time, but it's quite amazing how easy some things such as creating keybindings is. The docs are coming, but missing right now so if you want to use textual be ready for reading source code and examples. On to the devtools As @willmcgugan shows in this tweet it's pretty easy to setup, it requires having two terminals open, or using tmux, and curren
-
Using Different versions of python with pipx | pye...
"cell shaded, long, full body, shot of a cybernetic blue soldier with glowing pink eyes, llustration, post grunge, cinebatic dramatic atmosphere, sharp focus, pink glowing volumetric lighting, concept art by josan gonzales and wlop, by james jean, Victo ngai, David Rubín, Mike Mignola, Laurie Greasley, highly detailed, sharp focus,alien,Trending on Artstation, HQ, deviantart, art by artgem" -s50 -W832 -H416 -C7.5 -Ak_lms -S70567464 I love using pipx for automatic virtual environment management o
-
LIVE-REPLAY - Python dev | Markata todoui | 4/6/20...
https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow. During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off. dotfiles: https://github.com/WaylonWalker/devtainer today's project:
-
How I Quickly Capture Screenshots directly into My...
When I am creating blog posts it's often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content. Making Screenshots When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog. take screenshot opti
-
Copier < 6.0.0b0 considered dangerous
Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories. !!UPDATE As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update. This is a PSA I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I
-
PyOhio CFP's
Here are some CFP's that I used for PyOhio 2022. https://pretalx.com/pyohio-2022/cfp {.hoverlink} Idea to blog post in minutes - Shorter Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it's plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle. Lets build a whole blog site in 5 minutes. Add Kedro to your Pandas Workfl
-
python functools total ordering
functools.total_ordering makes adding all of six of the rich comparison operators to your custom classes much easier, and more likely that you remember all of them. From the Docs: The class must define one of __lt__(), __le__(), __gt__(), or __ge__ In addition, the class should supply an __eq__() method. one of these lt () le () gt () ge () and required to have this one eq () Total Ordering Docs Here is an example using the Enum I was working on the other day.
-
Smoother Python with automatic imports | pyflyby
This is not a flaky works half the time kind of plugin, it's a seriously smooth editing experience. I've just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython. I can't tell you how pumped I am for this, and how good its felt to use over the past few weeks. It's glorious. YouTube video Listen to me rant on how great pyflyby is https://youtu.be/2QW5DJiEJH4 {.youtube-embed} Give the
-
kedro catalog create
I use to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key. https://youtu.be/_22ELT4kja4 {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Running Kedro Catalog Create The command to ensure there are catalog entries for every dataset in the pass
-
Just Ask Ipython for help
It happens to the best of us We can't all remember every single function signature out there, it's just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal. https://youtu.be/TZrRAP-9UMk Docstrings In any python repl you can access the docstring of a function by calling for . In Ipython we can even get some syntax highlighting with the . Source Code Sometim
-
Setting Parameters in kedro
Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files. https://youtu.be/Jj5cQ5bqcjg {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. parameters files You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a and para
-
Writing your first kedro Nodes
https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla data structures. import node You will need to import node from kedro.pipeline to start creating nodes. func The is a callable that will take the and create the . inputs / outputs Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is
-
Running your Kedro Pipeline from the command line
Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli. https://youtu.be/ZmccpLy-OEI {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Kedro run To run the whole darn proj
-
kedro Virtual Environment
Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment. https://youtu.be/ZSxc5VVCBhM {.youtube-embed} conda venv pipenv conda I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don't have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool. st
-
Kedro Install
Kedro comes with an command to install and manage all of your projects dependencies. https://youtu.be/IWimEs-hHQg cd into your project directory and activate env You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment. [[ kedro-new ]] this post covers kedro new [[ kedro-environment ]] This post covers creating your virtual environment for kedro install kedro Make sure you have kedro installed in your cur
-
Kedro Git Init
Immediately after , before you start running or your first line of code the first thing you should always do after getting a new kedro template created is to . https://youtu.be/IGba3ytf_6U git init Its as simple as these three commands to get started. I don't care if this project is for learning, if it will never have a remote or not, use git.
-
Kedro New
https://youtu.be/uqiv5LAiJe0 {.youtube-embed} Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template. [[ what-is-kedro ]] Unsure what kedro is, Check out yesterdays post on What is Kedro. pipx I reccomend using when running kedro new. is designed for system level cli tools so that you do not
-
What is Kedro
Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract that the user specifies through entries. These entries are loaded, ran through a function, and saved by . The order that these are executed are determined by the , which is a DAG . It's the 's job to manage the execution of the . https://youtu.be/Wf4rnFsaFFU {.youtube-embed}
-
Incremental Versioned Datasets in Kedro
Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work. [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. How
-
I Started Streaming on Twitch
I recently started streaming on twitch.tv/waylonwalker and it's been a blast so far. python kedro Data Science Data Engineering webdev digital gardening Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. kedro-issue-606 Inspiration My introduction to twitch c
-
Kedro Spaceflights - part 2 | Stream replay June 7...
This was my seconf time ever streaming on twitch.tv/waylonwalker , and I completely botched my mic 2x. https://youtu.be/_7MwgKu-844 Links Spaceflights Tutorial my spaceflights repo Notes to get started
-
🌱 Digital Gardening | gif to Mp4 | Stream replay J...
https://youtu.be/I4VenHqIEng Doing some Digital Gardening on stream Ahrefs Errors ahrefs large images Automatic gif to mp4 gif to mp4 After this stream all gifs on my site are converted to mp4/webm if they exist. tmux-navigation-2021
-
Kedro Spaceflights - part 1 | Stream replay June 4...
This was my first time ever streaming on twitch.tv/waylonwalker . I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go. https://youtu.be/Y07UBr9Ccjs Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time co
-
Using Kedro In Scripts
With the latest releases of kedro , it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro [[ what-is-kedro ]] If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors as of 0.17.2 I've tried to do this in kedro and it turned into a rabbit hole of erro
-
Silence Kedro Logs
Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log First, how does one silence a python log? Python loggers can be retrieved by the module's function. Then their log level can be changed. Much of kedro's chattiness comes from INFO level logs. I don't want to hear about anything for my current use case unless i
-
Python Diskcahe is locked
Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error Minimum reporducible error is one of my superpowers in development. I do this very often to sus ou
-
What is if __name__ == "__main___", and how do I u...
When a python module is called it is assigned the of otherwise if it's imported it will be assigned the of the module. Concrete example Let's create a module to play with a bit. We will call this module . It is a module that we may want to run by it'self or import and use in other modules. I have set this module up to execute one of two if statements based on whether the module it'self is being ran or if the module is being imported. Note it is not common to have a block, this is just
-
How to Install micromamba on linux (from the comam...
I really like using conda ( ) as my python virtual environment manager of choice. It's simple and it includes its own python interpreter using the version that I specify at creation. Mamba from their readme Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE core parts
-
kedro replit
I am trying to see what an embeded replit
-
Pytest capsys
Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys using capsys Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.
-
Building Rich a Dev Server
Draft Post I've really been digging @willmcgugan's rich library for creating TUI like interfaces in python. I've only recently started to take full advantage of it. Dev Server I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to. automatic port number auto-restart display ( port, pid, uptime ) finding the port I
-
Site Down During Build
Recently I noticed a new netlify site of mine was down while I was checking to see if new content was live. Later found out this was consistent after each and every push the site would go gown as soon as I hit push, and would not come back until the build finished. Is this normal? Do other Netlify sites go down during build??? Short Answer NO. All of my google fu lead me to believe I was alone and none of my other sites do this. Digging into my build My deploy script ends with the following. Aft
-
Kedro pipeline_registry.py
With the realease of came a new module in the project template . Here are some notes that I learned while playing with this new module. migrating to create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your . pipeline_registry only works in Conflict Resolution What happens If I register pipelines in both places I w
-
Minimal Kedro Pipeline
How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in and 27 in . 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline I have everything for this post hosted in this gihub repo , you can fork it, clone it, or just follow along. Installation Caveats This repo represents the minimal amount of structure to build a ked
-
Markdown Cli
This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things posts tags draft posts data frontmatter filepath content template html render content Markdown.Markdown support extentsions frontmatter cleaning. provide ways to hook in or clean up the frontmatter Markata.Markata methods load render save Markata.Post methods load render save Markata plugins befor
-
Kedro - My Data Is Not A Table
In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. DataFrames are the heart of most pipelines These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types. [[ what-is-kedro ]] unfamiliar with kedro, check out this post Sometimes datasets are not t
-
Quickly Change Conda Env With Fzf
Changing conda environments is a bit verbose, I use a function with fzf that both lists environments and selects the one I want in one go. Conda I have used conda as a virtual environment tool for years now. I started using conda for its simplicity to install packages on windows, but now that has gotten so much better and it's been years since I have run a command. I'm sure that I could use a different environment manager, but it works for me and makes sense. What environment manager do you u
-
Minimal Python Package
What does it take to create an installable python package that can be hosted on pypi? What is the minimal python package setup.py my_module.py This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file. Directory structure setup.py name The name of the package can contain any letters, numbers, "_", or "-". Even if it's for internal/personal consumption only I usually check for discrepancy with p
-
Ipython-Config
I use my ipython terminal daily. It's my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration. install ipython Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython. You are using a virtual environment right? Virtual environments like venv or conda ca
-
Custom Ipython Prompt
I've grown tired of the standard ipython prompt as it doesn't do much to give me any useful information. The default one gives out a line number that only seems to add anxiety as I am working on a simple problem and see that number grow to several hundred. I start to question my ability 🤦♂️. Configuration If you already have an ipython config you can move on otherwise check out this post on creating an ipython config. [[ ipython-config ]] The Dream Prompt I want something similar to the stars
-
Automating my Post Starter
One thing we all dread is mundane work of getting started, and all the hoops it takes to get going. This year I want to post more often and I am taking some steps towards making it easier for myself to just get started. When I start a new post I need to cd into my blog directory, start neovim in a markdown file with a clever name, copy some frontmatter boilerplate, update the post date, add tags, a description, and a cover. Todo List for starting a post frontmatter template Title slug tags date
-
Windowing Python Lists
In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools more-itertools` are full of cases of this. This post is a walkthrough of me solving a problem with rather than reaching for a for loop, or pandas. I am working on a one-line-link expander for my blog. I ended up doing it, just by modifying the markdown with python. I first split the post into line
-
Adding Audio to my blog posts
This is episode 1 of the Waylon Walker Audio experience, posts from waylonwalker.com {.hoverlink} in audio form. So I have had this idea for awhile to add audio to my blog posts. The idea partly comes from the aws blog, if you have ever been on their blog you will have noticed that they have a voiced by amazon polly section. What to Expect Honestly I don't know this is all new to me and I dont have much to go off of. For now its a test that may or may not work out. I will say that the time th
-
gatsby-remark-embedder
Inspired by discourse's link expansion I am rolling out expansions for one line links on the blog waylonwalker . I was able to find a gatsby plugin gatsby-remark-embedder that expands one line links for social cards for popular platforms like twitter and YouTube through a repose from Kyle Mathews to my tweet. https://twitter.com/kylemathews/status/1329817928666005504 Use Cases This covers a couple of use cases I have with very little effort. Twitter YouTube install This was super quick and sim
-
Expand One Line Links
I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML into the markdown. While this works, it's more manual/difficult for me does not look the best, and does not read well as Goals for new card The new card should be fully automated to expand with title, description, and cover image. Bonus if I am able to attach a comment behind it. fully automated card expa
-
reasons-to-kedro
There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines [[ what-is-kedro ]] Starter Template Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. [[ create-new-kedro-project ]] read more about how
-
What's New in Kedro 0.16.6
Kedro 0.16.6 is out! Let's take a look through the release notes Deployment Docs This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Argo Prefect Kubeflow Batch SageMaker Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies. Argo, Prefect, and Kubeflow have an interesting technique where
-
Designing a "Router" for kedro
nodes_global I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called . It worked fairly well but did have some issues around being set as a global variable. But... One thing in particular that it did not lend itself well to was being able to create a packagable pipeline that I could pip install and append into an
-
Reclaim memory usage in Jupyter
Today I ran into an issue where we had a one-off script that just needed to work, but it was just chewing threw memory like nothing. It started with a colleague asking me How do I clear the memory in a Jupyter notebook, these are the steps we took to debug the issue and free up some memory in their notebook. How do I clear the memory in a Jupyter notebook? Pre check the status of memory There are a number of ways that you can check the amount of memory on your system. The easiest is not necessa
-
Strip Trailing Whitespace from Git projects
A common linting error thrown by various linters is for trailing whitespace. I most often use flake8. I generally have [pre-commit]( https://waylonwalker.com/pre-commit-is-awesome hooks setup to strip this, but sometimes I run into situations where I jump into a project without it, and my editor lights up with errors. A simple fix is to run this one-liner. One-Liner to strip whitespace bash read more about how [[pre-commit-is-awesome]]
-
Three things to Automate with Python using Pandas
Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an alternative out there that can save us time, automate boring things, and allow us to open up our minds for the part that we add value, Thinking about the data. Focus on Value Add Operations Lets face it, stitching together spreadsheets is zero value add by
-
How to Install miniconda on linux (from the comman...
miniconda is a python distribution from continuum. It's a slimmed-down version of their very popular anaconda distribution. It comes with its own environment manager and has eased the install process for many that do not have a way to compile c-extensions. It made it much easier to install the data science stack on windows a few years ago. These days windows are much better than it was back then at compiling c-extensions. I still like its environment manager, which installs to a global directory
-
What's New in Kedro 0.16.4
If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks. This one comes a bit surprising as it was just casually mentioned in #435 auto enabled plugins mentioned in issue 435 Think pytest As mentioned in #435 this is the model that pytest uses. Not all plugins automatically start doing things right out of the box but require a CLI argument. simplicity It feels a bit crazy that simply installing a package will change the way that your
-
Integration testing with Python, TestProject.io, a...
!!! Caution None of the testproject.io urls resolve anymore in JAN 2025, I removed all of the broken links. As I continue to build out waylonwalker.com I sometimes run into some errors that are not caught because I do not have good testing implemented. I want to explore some integration testing options using GitHub's actions. Running integration tests will not prevent bugs from happening completely, but it will allow me to quickly spot them and rollback. 🤔 What to test first? The very first t
-
🐍 Practice Python Online
When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these you can try out something quick. Sometimes I see snippets from blogs or tweets and I need to try the out for myself to really understand. When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to sh
-
Kedro Catalog
I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. Process metadata will be attached to the dataset object under a attribute metadata will be updated metadata will be empty until a pipeline is ran with the hook on optionally a function to add metadata will be added metadata will be stored in a file next to the meta Problems This Hook Should solve what datasets have a columns with in the name what datasets were updated after last tuesday which pipe
-
How python tools configure
mypy Mypy's config parser seems to be one of the most complex. This is likely in part to it having the largest backwards compatability of all projects that I looked at. mypy/config_parser flake8 options/config.py black black portray only uses pyproject.toml portray/config.py interrogate only uses pyproject.toml
-
Why use a cms
When first learning to code its very common to hard code everything right into the code. This happens with most folks in just about any language. Whether its HTML or markdown for front end content, or even hardcoding parameters in our backend languages like python, or node.js. 🤷♀️ What's wrong with hard coding everything? Hard coding everything right into your code makes it really hard for non-technical collaborators to join. It makes it nearly impossible to hand websites off to clients without
-
🐍 Parsing RSS feeds with Python
I am looking into a way to replace my google reader experience that I had back in 2013 before google took it from us. I am starting by learning how to parse feeds with python, and without much previous knowledge, it proved to be much easier than anticipated thanks to the library. This is how I used python to parse rss and setup my own custom feed. Install Install the feedparser library. Get the content The feed object The feed is a feedparser.FeedParserDict. For all intents and purposes this
-
How I Built My GitHub Profile
I ran a discussion on dev that collected quite a list of examples in the comment section. So many great calls to action, animations, memes, and weird tricks. dev.to whats-on-your-github-profile My current profile Waylon Walkers GitHub profile social icons Upload all of your icons to the repo in a directory such as or , then link them with a attribute like below. I used html for mine, not sure if you can set the in markdown. note I did add a bit of (non-breaking-whitespace) between m
-
SLIDES - understanding python \*args and \*\*kwarg...
Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog @_waylonwalker . python args and kwargs
-
Gracefully adopt kedro, the catalog
Why use kedro catalog? While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all. Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different fo
-
How to find things in your kedro catalog
kedro 0.16.2 just dropped last week with a long-awaited feature... catalog search ! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful. The Catalog The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place
-
How Kedro handles your inputs
Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing. *args/**args review Check out this post for a review of how work in python. [[ python-args-kwargs ]] python args and kwargs article by @_waylonwalker All Kedro inputs are catalog Entries When kedro runs your pipeline it uses the catalog to imperatively load your data, mea
-
Refactoring your blog urls
I just did a quick refactoring of my JAMStack blog urls. Some didn't fit with my style, some had that I wanted to switch to , and others were ridiculously long. I've been using forestry as my CMS, I write many of my posts there, and sometimes it picks some crazy file names (based on my titles). It was time to refactor. [[ refactor-in-cli ]] When refactorings similar to this get really big I often need to do some project wide find an replace, I usually do this right from the command line. 🖊
-
understanding python \*args and \*\*kwargs
Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function a
-
pre-commit is awesome
I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it's easy. It will automatically run checks for you. In some cases, it will even automatically fix them for you. Out of the box, it will do things like automatically trim extra whitespace, fix file endings, and ensure file sizes are not too large for git. I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it
-
Kedro Static Viz 0.3.0 is out with Hooks Support
kedro-static-viz is out with support for the newly released hooks feature. This means that you can have automatically deploy a full gatsby site keeping your visualization always up to date. Even though it is a static site there is no functionality lost. The only thing that's missing is the flask server. With kedro-static-viz you can deploy your visualization to a number of static hosting providers such as GitHub pages free of charge with wicked fast performance ⚡ It's Fast Even though
-
How to get Dev Comments from an article Url
I want to incorporate some of the wonderful comments, \U0001F495, \U0001F984, and \U0001F516's that I have been getting on dev.to on my website. I have dabbled once or twice with no avail this time I am taking notes on my journey, so follow along and let's get there together. By the end of this post, I will have a way to get comments from posts on the client-side thanks to the wonderfully open dev.to API. I want to incorporate some of the wonderful comments, 💕, 🦄, and 🔖's that I have been gett
-
Adding google fonts to a gatsbyjs site
stack overflow link
-
Create Custom Kedro Dataset
Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key. But what is happening behind the scenes Under the hood there is an that each connector inherits from. It sets up a lot of the behind the scenes structure for us so
-
Interrogate is a pretty awesome, brand new, cli fo...
As usual while listening to python bytes 181 I heard of a tool that I had to try out right away! This thing is 🔥 hot off the press folks, we're talking the first release only 3 weeks ago. Its something that the python community needed years ago, and it belongs in your CI today . I had tried several tools that tried to do docstring coverage in the past but they were a bit cumbersome and were quickly forgotten about. Not interrogate, its dead simple! Nothing I have tried has come close to being
-
creating the kedro-preflight hook
Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and (nouns). With a , or (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability. kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and
-
📝 Kedro Preflight Notes
This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources. Must Haves check that inputs exist or are of a type to skip (sql) Good to haves check that all input and output databases are accessible with good credentials check for s3 bucket access check for spark install Implementation run params
-
Explicit vs Implicit Returns in Javascript
Often when reading through javascript examples you will find some arrow functions use parentheses while others use braces . This key difference is that parentheses will implicitly return the last statement while braces require an explicit return statement. It is important to understand the difference between them because it is likely that you will find code examples of both and trying to edit code written differently than you're used to may have unintended consequences. Arrow functions Arro
-
python-deepwatch
Is it possible to deep watch a single python function for changes? Shallow Watch keeping track of a python functions hash is quite simple. There is a method attached to every python function. Calling it will return a hash of the function. If the function changes the hash will change. Using hashlib provides a consistent hash. Now we have a consistent way to hash function code. Deep hashing Find dependencies setup a function in a module with a dependency Create Generic module importer by filepath
-
TIL: Bind arguments to dynamically generated lambd...
This past week I had a really weird bug in my kedro pipeline. For some reason data running through my pipeline was coming out completely made no sense, but if I manually request raw data outside of the pipeline it matched expectations. NOTE While this story is about a kedro pipeline, it can be applied anywhere closures are put into an iterable. Debugger to the rescue After a few days of looking at it off and on, I pinpointed that it was all the way down in the raw layer. Right as data is com
-
Four Github Actions for Python
If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simple cli I have opted to use the cli for most of these, that way I know exactly what is happening and have more control over it if I need. If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simpl
-
Variables names don't need their type
So often I see a variables inside of its name and it hurts me a little inside. Tell me I'm right or prove me wrong below. Examples Pandas are probably the worst offender that I see Sometimes vanilla structures too! Edge Cases? It's so common when you need to get inside a data structure in a special way that itsn't provided by the library.... I am not exactly sure of a good way around it. Containers are plural Always name your containers plural, so that naming while iterating is simple. Befo
-
Create New Kedro Project
This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro . You can learn more about this Hello World Example in the docs 🧹 Install Kedro 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization Create a Virtual Environment I use conda to control my virtual environments and will create a new environment called with the following command. note the latest compatible vers
-
What is YOUR Advice for New Data Scientists
Learn the business Learn Git Your code does not need to be amazing Keep Learning Learn Git You dont have to start out as a git wizard with the cleanest possible commit history. At first dont let yourself get too wrapped up in it, the most important part is that you make commits. You will find needs for more advanced stuff later. Get comfortable with this, then learn how to , , , etc... Your code does not need to be amazing Get the job done. Keep it in small bite size pieces. Make readable
-
Do You Hoist
I am working through Wes Bos's beginnerjavascript.com/ I just hit module 18 on hoisting. It's something that I always knew was there, Its not something I typically see used or use myself. Do you Hoist? Do you have any use cases that you use hoising? Why? It seems like a really cool feature in any language that uses it, but I dont really notice it in use. What is Hoising There are many articles that cover this in far more depth, but its the idea that variable declarations and functions are de
-
What is Kedro
[[ what-is-kedro ]] This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate and . This article will cover the 10K view of kedro, future articles will dive deper into each one. kedro is an open-source data pipeline framework. It pr
-
Custom Scrollbar Design
Getting a custom scrollbar on your site makes it stand out a bit compared to the very plain stock one that are on most sites. This is how I set mine up on my gatsby site. Inspired by Wes Bos's new uses.tech I wanted a custom scrollbar on my personal site. I had tried to do it in the past, but gave up after it was not working. Looking at the Source Since uses.tech is open source I jumped on github, searched for scroll and found this layout.js . Copy it to my own component My first step was
-
2020 waylonwalker.com rebrand
Moving into 2020 I have been really leaning on using purple as my theme color everywhere more and more. Its time for an update to my personal site, not just because it feels plain, not just because the cover art I am using for dev.to doesn't fit my current card layout, but because I feel inspired and I want to. Starting point  This is what we are working with. It has been my card design for
-
I just added react-headroom to my site
It was so easy to get a professional looking navbar with just 3 lines of code. This package seriously is so usable on mobile it is ridiculous. I found this package from day-4 of the 100 days of gatsby challenge. It is by the wonderful man who brought us gatsbyjs Kyle Mathews, so you know its gotta be good. install react-headroom installation is easy Import Headroom There was no instructions for es6 style imports that are common with gatsbyjs sites like mine, but it was intuitive to figure out.
-
Serverless things to investigate
Social Automation Zappier Free only one step is free can take rss to social Automate.io Form submission https://getform.io Free to get started https://formkeep.com/?#pricing Free 7 day trial $4.99 cheapest after
-
I finally fixed my Styled-Components in gatsby.js
I finally fixed my Styled-Components in gatsby.js. I am starting a redesign of my website. I have started cross posting to dev.to more regularly. With that I have been making more detailed cover images at the recommended . These images get cut off on my own site, which is a bit ridiculous to have my own content not look right on my own site. But before we start a heavy redesign I have a small issue that has plagued the site for at least a year! I have a small issue that has plagued the si
-
Should I switch to Zeit Now
Netlify I have happily had my personal site waylonwalker.com hosted on netlify for nearly 2 years now. In fact I have hosted about a dozen different toy projects to play with on there, 4 of which have gone far enough to get a custom domain name. They are fast to deploy and consistently do so on every to main. Zeit I have recently started playing with zeit again. I really like their cli tool, its dead simple and makes sense. I tried the netlify one early on and dont think I really gave it
-
Personal URL shortener with Netlify Redirects
I love using URL shorteners to easily share links without hitting character limits, but they loose their meaning. Services like bit.ly will save my links for me so that I can find them, but I would rather them to be easy to remember. https://bit.ly/2ruLwQz does not roll of the tongue so well. 301 🤸♀️ I recently discovered a really cool feature of netlify that I have always looked past, . It is so simple cool and powerful, every netlify site should do this! But how 🤷♀️ simply add a file to
-
simple click
cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Example Ideas Things a cli can do to enhance your library. 🆚 print version 🕶 print readme 📝 print changelog 📃 print config ✏ change config 👩🎓 run a tutorial 🏗 scaffold a project with cookiecutter 🖱 Click Click is the most popular python cli tool framework for python. There are others, some old, some new c
-
SqlAlchemy Models
Make a connection Make a session Make a Base Class Make your First Model Make your own Base Class to inherit From Use the Custom Base Class
-
Building Cli apps in Python
Packages Click Inputs Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option typically aliased with a shorthand ('-v', '--verbose') **From the Docs To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. Argument positional required no help text supplied by click Yaspin Yaspin Gif Cl
-
Kedro
See all of my kedro related posts in [[ kedro-feed ]]. #kedrotips I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips . 🗣 Heads up Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful mo
-
📝 Packages to Investigate Notes
jmespath Tabnine Bulwark |-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example
-
Debugging Python
Using pdb
-
Just Use Pathlib
Pathlib is an amazing cross-platform path tool. Import Create path object Current Directory Users Home Directory module directory Others Let's create a path relative to our current module. Check if files exist Make Directories rename files List files Glob Files recursively Write
-
Filtering Pandas
query Good for method chaining, i.e. adding more methods or filters without assigning a new variable. masking general purpose, this is probably the most common method you see in training/examples isin capable of including multiple strings to include contains Good For partial matches MASKS anything that we put inside of square brackets can be set as a variable then passed in. Operators & - and ~ - not | - or AVAILABLE and NAME AVAILABLE or NAME AVAILABLE and not NAME
-
Quick Progress Bars in python using TQDM
tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. for more gifs like these follow me
-
Clean up Your Data Science with Named Tuples
If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing , , and data. You may need to calculate total revenue, inventory
-
Background Tasks in Python for Data Science
This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth R
-
📝 Bash Notes
Bash is super powerful. File System Full Show Remaining Space on Drives show largest files in current directory Move files then symlink them Fuzzy One Liners edit in vim cat a file bash execute git add git reset Kill a process Finding things Files fd-find is amazing for finding files, it even respects your file 😲. Install with . ++Vanilla Bonus Content ** show matching text ** ++Vanilla Bonus ** show file names only ** ++Vanilla Bonus Recursively Replace text ++Vanilla Bon
-
Autoreload in Ipython
I have used for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library co
-
Forestry.io
Testing out forestry.io Sorry Netlify CMS I still ♥️ your product dont be forestry is simple I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really struggle configuring it. Forestry is so simple to setup. My favorite part is that I can code up my gatsby.js site, storing all editable text in markdown, and come back later and add the CMS based on existing documents. Configuration is Simple Forestry.io has this amazing feature to create cre
-
D3 Day 5
select rowid, key, raw, store_time, expire_time, access_time, access_count, tag, size, mode, filename from Cache where key='e3cd37c97980ab2c'; 'e3cd37c97980ab2c' 'e3cd37c97980ab2c' select rowid, key, raw, store_time, expire_time, access_time, access_count, tag, size, mode, filename from Cache where key='c14050404a107d5e'; Learn D3 in 5 days For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere! These are just stepping stones into real custom
-
D3 Day 4
Learn D3 in 5 days For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere! These are just stepping stones into real custom visualizations that cannot be done in any other tool today. I still cannot explain how excited I am to say "I created that in d3!!!" Todays Result Today I will be learning about d3 scales, and adding them to the bar chart that we created yesterday. Follow along as I try to create something interesting. today's_result R
-
D3 Day 3
Learn D3 in 5 days I recently subscribed to Ben Clinkinbeard's learn D3.js in 5 days, and am currently on day 3. I read through the first 2 days, and felt fairly comfortable with selecting elements, so I did not follow along on the first two days. I probably should have, but there are only so many hours in the day. Why Learn D3 D3 is the ubiquitous dynamic visualization library for building custom interactive visualizations on the web. It is a bit low level, and more verbose than many other libr
-
FlexBox
Flexbox-zombies I recently finished up the flexbox-zombies course to learn more about flexbox, and to become proficient with it. I can truly say that this course has changed the way that I create layouts. Flexbox is very intuitive now. What this course does really well at is explaining the concepts and hitting you with a ton of examples that you can work through really quickly. flexbox-zombies A clip from the final round against Dave Basic Setup Flexbox requires a wrapper container to work I