A poc is not a product. I started focais, not in a rush, but as something that I already had a POC for and thought it would be easy. I wanted to build tools to make creating blog posts like this one easier. I stared with a tool that takes screenshots of websites. POC (proof of concept) For the poc, I made a single fastapi endpoint that takes a url and returns a screenshot of the page. It converts the url into a key that I can lookup to see if I have the shot, if I don’t I go get it. With the open source libraries out there, this is not too hard of a task. Progress Thus Far /shot But this wasn’t enough All it does so far for this first tool is take screenshots of websites, and give you a hosted image. Users To bring in users, I need to create a signup flow, with a database to store users, login, logout, and email recovery. I’ve never had to use an email service before that wasn’t already mandated by a company or an iternal smtp server. After some searching I chose resend for email. Prog…
Theo’s response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements “Everything must be unit tested, tests make code more maintainable.” In reality when we are not writing complex low level code unit tests are probably the wrong approach. We Finally Agree On Unit Tests - YouTube!https://www.youtube.com/watch?v=MbU-PKukdMw Dec 22, 2023 Most of us are assemblers So much of software engineering is assembling existing well tested code. Crud applications, UI, Data Pipelines, building on top of battle tested code. Manufacturing Analogy - Unit Testing This kind of reminds me of Manufacturing. Individual components are QA tested with tests that look more like unit test. Parts like bearings, pistons, shafts, valves, they are all tested against sophisticated statistics of sample measurements. This is quite similar to unit testing. You see measuring the individual sizes of these components does not…
Your browser does not support the audio element. I’ve long hosted my personal blog as a static site on waylonwalker.com. It’s all markdown, converted to html, and shipped as is. It’s been great, I’ve moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it’s just pointing ci to a different host after the site has built. What about server side Now the part that I have struggled with is how to cheaply host a server rendered application that can just live on forever without me paying for it. This is a harder problem as it costs more to keep servers spinning, memory, and disk all ready for you to use at a moments notice. Honestly I never really deployed anything that useful on heroku, but it seems like the klenex of the bunch that’s why they are in the title. I’ve moved between digital ocean and fly.io, and have had some great experiences with both. I just don’t want to build something that is g…
Serious question. No one ever got fired for choosing PostgreSQL But, why. It’s the most loved db, right? Right? Maybe it’s time to rethink it. Don’t get me wrong, if I need a relational db as a service, PostgreSQL is going to be my first choice, but why do I need to run a separate application for it? Tutorials use sqlite Why is that? Because there is nothing else to stand up. Nothing else to maintain. And you probably already have it installed on just about anything that has a battery. SQLite runs in memory Don’t need, or maybe don’t want to persist state. Run it in memory. This is a nice feature for running tests. Less exposure SQLite is a file on your filesystem. It’s not a web service. It’s not a cloud service. Not that postgres is insecure, but it is one more endpoint that you have to think about securing. this means that is probably also cheaper 🤑 SQLite is easy to replicate Want to run your new feature with prod data? Pull a replica or backup. It’s a file, you can cp, scp, rsync…
I was watching @theprimeagen recently and I think he sold me on using tailwindcss. The thing about tailwind is that it is not a big component library, it’s a set of css classes mapped to a few (usually one) style. All css classes are shitty, so you might as well use someone else’s shitty css classes on all your projects rather than thinking you’re being smart with a new set of classes that you will hate in 6 months when you come back to the project. roughly quoted from memory of @theprimeagen It’s tiny So unlike big component libraries like tailwind, it comes with a cli that that it uses to create the final css file. It is able to treeshake out all the tailwind classes that you are not using and only ship the ones that you are using. It’s hard to clash Since the classes are so small and single purpose it’s hard to end up with something like in two places that mean different things causing you to duplicate most of that css anyways so that the whole design doesn’t break when you change o…
It’s so easy to get out of rhythm, get busy, and drop the ball on some things that you really want to do or should do. This blog is a good example. I took some time off for some family reasons, but have taken a long time to get back to it simply because I am out of rhythm. As I am trying to get back into the rhythm there is some tooling that I have set up for it that I completely forgot about that feel good to use again. Repetitive Tasks Simple Repetitive Tasks that I have to do often can just feel soul crushing, and one main thing that got me interested in programming. AI tools are becoming more and more useful at solving these problems. For instance code generation tools like co-pilot or codeium are really good at boilerplate and pattern repetition. Things that used to be a few vim macros is now just banging on tab. I often look for setting up templates or some sort of snippet to replace a big chunk of boilerplate that I know I will need over and over. timebox Don’t be afraid to give…
Playing minecraft with Wyatt today he started a server all on his own and had me join. All vanilla, only one rule, underground bases. I spawned into the server and it was already night time. I gathered up some wood on my way down a tree, and was attacked by zombies before I could get any tools, so I ran up another tree and crafted a crafting table. Now to follow the rules, it’s time to head underground to build my base. I made a mistake and died, but look at this view from my respawn point.
Life comes in waves, and sometimes you need to set down some of your projects to focus on others. For the first part of 2023 I’ve really had a lot of family stuff to focus on, we also are pretty new homeowners and are still trying to get our new to us house cleaned up and modernized. Side Projects You can see in my growing list of repos that I have poked around on quite a few side projects over the past few months. This has been quite relaxng to me, mostly things that I use to learn from, but also a lot that are tools and things I use that bring me joy. Pydantic I haven’t wrote about it at all yet, but I have really been starting to lean into pydantic on all of these side projects. I have really been enjoying the type system. A good friend @pypeaday got me hooked and we have been throwing around this phrase that he learned from a math professor “Make it So”. The idea boils down to leveraging pydantic to make all the values you want to exist up front, or fail validation, then have no mo…
The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star
I was reading about pydantic-singledispatch from Giddeon’s blog and found it very intersting. I’m getting ready to implement pydantic on my static site generator markata, and I think there are so uses for this idea, so I want to try it out. The Idea Let’s set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use and type hints to provide unique execution for each environment. We might want something like a path_prefix in prod for environments like GithubPages that deploy to while keeping the root at in dev. Settings Model Here is our model for our settings. We will create a CommonSettings model that will be used by all environments. We will also create a model that will be used in dev and that will be used in prod. We will use as the discriminator so pydantic knows which model to use. Singledispatch Now let’s create our function. We will use to provide a unique execution for each environment. It w…
This is my first time journaling a Minecraft hardcore world, my son Wyatt is also documenting his journey in a survival world on wyattbubbylee.com. Day 0 init I logged into a brand new hardcore world. I was welcomed by a great Acacia biome spawn full of resources. I quickly cut my first tree, crafted an axe and set out to find my first sheep. I was able to find enough sheep for a bed, several cows and pigs. I crafted a set of wooden tools, and farmed out a wheat farm till my wooden hoed died at the shore of a nearby stream. I found a small stone outcropping in the side of a hill and harvested nearly a full stack of cobblestone from my first wooden pick. I ended the first day by sleeping in my bed safe from mobs. Achievements bed furnace stone wheat farm Day 1 Thoughout the course of day one I collected wood and started the framework for my new house. Day 2 Day two continued the framework of the house, and a bit of sand collection for windows. I was a bit late to bed and saw four mobs s…
Markata I open sourced the static site framework that I use to build my-blog among other side projects. It’s a plugins all the way down static site generator, that makes me happy to use. {% gh_repo_list_topic “waylonwalker”, “markata” %} Repos used to build this blog my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don’t have to worry about carrying around large images with my lightweight text files just to make some posts. {% gh_repo_list_topic “waylonwalker”, “personal-website” %} Kedro I am a heavy user of the kedro framework, and a big advocate for using some sort of DAG framework for your data pipelines. kedro is built all in python which makes it easy for a python dev like me to extend, run, maintain, and deploy. {% gh_repo_list_topic “waylonwalker”, “kedro” %} Neovim Plugins I use vim for all of my text editing needs. It brings me joy to make any part of it just a bit smoother. I have written a few plugins that…
After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science. I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on. During the episode I tried my best to let Yetu and Ivan take the spotlight as the maintainer and chime in with my experience as a user of kedro. Video Version https://youtu.be/WTcjvwkXoY0 Michael made the call available on youtube as well as the audio only podcast
The community, that’s it, end of post, roll the credits. I’m a tinkerer I am a tinkerer, I am not going to run a stock desktop manager, mostly becuase that’s just not how my brain works. I need to tweak everything to fit my needs. Grantid I have not spent much time in many full fledged linux desktop environments. They are far more customizable than windows ever will be, I absolutely love that about them. Inevitibly I end up in a situation where I hit a wall, it just won’t do what I want it to do, or my lack of understanding what came wtih it holds me back. minimal I love minimal installs. I love just building up my system from the bottom up with things that I like, I understand, and that I can script. I’m a noob I spend a lot of my time in the terminal. I’d like to think I know how to use a linux command line for software development really well, but there are a lot of things that I still dont know all that well, mostly because I don’t need to. The AUR Arch’s community maintained repo…
https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch – Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow. During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off. dotfiles: https://github.com/WaylonWalker/devtainer today’s project: https://github.com/WaylonWalker/markata-todoui website: https://waylonwalker.com/
When I am creating blog posts it’s often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content. Making Screenshots When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog. take screenshot optimize conversion publish create img tag Created in 🐍Python I created this tool for myself in python because that is what I am most familiar with, but realistically most of what I am calling are shell scripts that I could do in just about any language. Install my screenshot tool My screenshot tool is quite hacky and not configurable, but works wonderfully for me. If you like it and want to use it, make it configurable or fork it. For now it lives on GitHub and since it has a setup.py with entrypoi…
Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories.!!UPDATE As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update. This is a PSA I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I do so that they know the dangers of using copier inside an existing directory. The issue The fix https://github.com/copier-org/copier/pull/273 As of the time of writing this version is still in beta, if you still want to use copier with existing directtories, I’d strongly encourage you to install the release. confirm My update commit https://github.com/WaylonWalker/devtainer/commit/a06e47462e2c6d26ad69fbdc2e7990a1a73ab4b7
Here are some CFP’s that I used for PyOhio 2022. https://pretalx.com/pyohio-2022/cfp Idea to blog post in minutes - Shorter Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it’s plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle. Lets build a whole blog site in 5 minutes. Add Kedro to your Pandas Workflow - Short Sometimes python scripts/notebooks take a long time to run, let kedro automatically save your datasets so that you can maintain your production code with ease. Lets take a pipeline with an issue 30 minutes in and solve the issue in 5 mintues.
functools.total_ordering makes adding all of six of the rich comparison
operators to your custom classes much easier, and more likely that you
remember all of them.
From the Docs: The class must define one of __lt__(), __le__(),
__gt__(), or __ge__ In addition, the class should supply an
__eq__() method.
one of these
and required to have this one
Total Ordering Docs
Here is an example using the Enum I was working on the other day.
from enum import Enum, auto
from functools import total_ordering
@total_ordering
class LifeCycle(Enum):
configure = auto()
glob = auto()
load = auto()
pre_render = auto()
render = auto()
post_render = auto()
save = auto()
def __lt__(self, other):
try:
return self.value < other.value
except AttributeError:
return self.value < other
def __eq__(self, other):
try:
return self.value == other.value
except AttributeError:
return self.value == other
This is not a flaky works half the time kind of plugin, it’s a seriously smooth editing experience. I’ve just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython. I can’t tell you how pumped I am for this, and how good its felt to use over the past few weeks. It’s glorious. YouTube video Listen to me rant on how great pyflyby is https://youtu.be/2QW5DJiEJH4 Give the video a watch, I did not have noise-cancelling on in obs. My apologies for the background hum and the mic stand bumps. I did my best to fix them up. Installation How to install pyflyby for automatic python imports pyflyby is hosted on pypi, so you can get it with pip. I have had no issues installing it on 3.8+ so far. Configuration setup with stow always stow your dotfiles If you’re going to configure any of your tools the first thing you should do is set it up with stow, seriously don’t sleep on the stow. If you…
Calendar data could not load. Try again.