Post Archive

csv

m9a devlog 1

It's sad to see textualize.io close the doors, but textual is still alive and maintained as a n open source project. I tried to use it very early, and struggled, this was before docs and tutorials really existed, before a lot of the widgets and components existed. Then as we all do I got busy and moved on to other things in life and did not have the capacity to build TUIs. I like tuis I like tuis, I like staying in the terminal. I use lf daily to move files around when I want something more

fix feed descriptions

Today I fixed a bug in markata that has been occurring for a few months where the description for posts come out as None if coming from cache, the issue was a pretty simple check and pull properly from cache. This fixes all the descriptions in feeds and metadata on the post. Better description While in there we went ahead and improved our get_description to more accurately return plain text without escaped characters, remove cutoff words, and add an elipsis if the description cuts off the text.

portal-platformer-devlog-1

Here is the current state of my platformer yet to really be named, I want to make something in between hollow knight and portal. Starting I made one once in make code arcace on a pybadge. It was quick and dirty, but fun to work on. It had the basic of blocks that I could move, blocks i could put a portal onto, and a goal for each level. Some levels you can just walk through and some levels required you to really think about where to place the portal. History So this version of the game is a l

markata search

A side effect of [[ markata-didyoumean ]] is that we are able to implement some rudimentary search based on slug, title, tags, and description. screenshot-2025-01-31T14-50-00-094Z.png {.more-cinematic} I made a video of the early work on using this side effect to search for content on markata.dev. Replay markata-search-1.mp4 This was first implemented to solve 404 pages giving users a way to try to find the page that they tried to access. But it turned out to be a decent way to search through t

Markata DidYouMean

Coming in Markata 0.9.1 is far better documentation. i.e. Documentation that actually exists for everything. As part of poking around I realized that I often go to look up the docs for a plugin and forget that the path is , sometimes I might try or or or I might even forget the plugin name exactly and try something like and get a 404. So I added a plugin to markata that takes care of this. screenshot-2025-01-31T14-53-31-264Z.png {.more-cinematic} I made a quick recording of this earl

I fixed lists in my blog

A small improvement to my blog today. I was checking out my [[ now ]] post and noticed that lists have left padding but no discs or numbers. I popped open my added these lines and rebuilt. And rebuild, which I actually do with [[ casey-just ]], but showing the command here for clarity. WHY the applies {.bg-blue-900 .border-r-8 .border-black .rounded-xl .max-w-xl .p-6 .font-bold} Dude, you know that @apply is an anti-pattern in tailwind right? {.bg-green-900 .border-l-8 .border-black .rounded-x

one-shot-s3-cleanup

So I made a mistake in an app I am working on and ended up creating thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails... etc. I was able to delete them all, but I wanted to make a one-shot script to do it. screenshot-2025-01-31T21-20-56-934Z.png One Shot App I got the idea of the one shot app from Simon Willison and replicated his setup in a chatgpt project screenshot-2025-01-12T18-16-57-443Z.png Initial Prompt s3_cleanup Here is the result of the s3_cleanup script. It ende

nvim-manager

I recently built a cli application as a nearly-one-shot-app called nvim-manager . It manages your nvim dotfiles install. screenshot-2025-01-31T21-21-40-707Z.png {.more-cinematic} Why {.chat-left} Don't we have stow? {.chat-right} Ya, thats not enough. {.chat-left} Why not?? {.chat-right} Inevitably shit goes sideways and I break my vim install. How is nvim manager any better nvim-manager allows you to install pinned versions of your dotfiles, your friends dotfiles, and distros in ~/.config. Thi

thoughts 0.0.4

This is such an improvement to the backend of my website it warrants a blog post of celebration. For far too long I've been dealing with a tiny ass edit form on thoughts. I tend to not edit them, and try to get them right in one go. This is kinda the point of a thought, its a quick post meant to be the size of a tweet, but sometimes I'm leaving thoughts on a video or long post and want to make sure I have a good save point, but I just keep the thing in draft and hope I don't loose if for far

markata

This post is a work in progress. Markata is the static site generator that I created to build my website [[ about-this-site ]]. I built it for me and I enjoy using it. I know everying it can do and I can extend it to do more easily. I have set it up for some friends to also use it and am proud that it helps them publish their content. It's a meme to create your own static site generator to make your website. Yes its funny, I don't recommend it if your not ready for the level of work that com

thinking about static sites in 2024

actions build k8s build fastapi sqlite

sick wikilink hover

Today I set up some sick wikilink hover effects using tailwind see [[ a-case-for-tailwindcss ]]. When you hover over them they show an image preview of the link that you are going to. I cant find where I have seen this but it comes from some docs sites. I'll finish this article later, just excited to see it up.

One Day Build - Play Outside

Inspired by Adam Savage and his One Day builds on youtube. I often build things, and want to make them generally useful for others and over configure out of the gate. This project is purely for me inspired by a need I have. play-outside !How-To This post will not directly show how to make a weather app, but document the process that I went through to make mine. It will show the tools that I used to make it, and the final result. The Situation It often goes in our house ask dad while he is bus

thoughts on unit tests

thoughts on unit tests cover image Theo's response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements "Everything must be unit tested, tests make code more maintainable." In reality when we are not writing complex low level code unit tests are probably the wrong approach. thought 192, a thought about theo's reaction to prime's unit testing Most of us are assemblers So much of software engineerin

Looking for a Heroku replacement, What I found was...

I've long hosted my personal blog as a static site on waylonwalker.com. It's all markdown, converted to html, and shipped as is. It's been great, I've moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it's just pointing ci to a different host after the site has built. What about server side Now the part that I have struggled with is how to cheaply host a server rendered application that can just l

Hosted Platform Brainstorm

shots shots templates gradient backgrounds premium feature upload your own background ogs scrape og info, generate og image og's build on shots, they are just hosted template endpoints that get a screenshots potentially auto approve og's from rss drops drop a file to share limited size, limited upload MB's / month archives deep permalink a website https://pypi.org/project/pywebcopy/ og-links ai-covers give me an url, and a selector, I will read your article and give you an ai generated cover ima

Pycon 2023

Keynote Speaker - James Powell I don't want to be an expert python developer. https://www.youtube.com/watch?v=iKzOBWOHGFE {.hoverlink} keynote-speaker---james-powell usage of keyword only arguments to prevent pain for users of libraries In an alternate timeline the maintainer of newton could have chose to use keyword only arguments to prevent pain for users of libraries, or poor api design due to fear of changing api on users.

Playing Star Wars Text Adventure with a 10 yr old

The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star

Pydantic and singledispatch

I was reading about pydantic-singledispatch from Giddeon's blog and found it very intersting. I'm getting ready to implement pydantic on my static site generator markata , and I think there are so uses for this idea, so I want to try it out. The Idea Let's set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use and type hints to provide unique execution for each environment. We might want something like

Packages I Maintain

Markata I open sourced the static site framework that I use to build my-blog among other side projects. It's a plugins all the way down static site generator, that makes me happy to use. {% gh_repo_list_topic "waylonwalker", "markata" %} Repos used to build this blog my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don't have to worry about carrying around large images with my lightweight text files just to make some posts. {%

I was on Talk Python

After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science. I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on. During the episode I tried m

Running My Blog on 3.11-dev

3.10.5: 109.441 3.11-dev: 108.856

Using Different versions of python with pipx | pye...

"cell shaded, long, full body, shot of a cybernetic blue soldier with glowing pink eyes, llustration, post grunge, cinebatic dramatic atmosphere, sharp focus, pink glowing volumetric lighting, concept art by josan gonzales and wlop, by james jean, Victo ngai, David Rubín, Mike Mignola, Laurie Greasley, highly detailed, sharp focus,alien,Trending on Artstation, HQ, deviantart, art by artgem" -s50 -W832 -H416 -C7.5 -Ak_lms -S70567464 I love using pipx for automatic virtual environment management o

LIVE-REPLAY - Python dev | Markata todoui | 4/6/20...

https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow. During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off. dotfiles: https://github.com/WaylonWalker/devtainer today's project:

How I Quickly Capture Screenshots directly into My...

When I am creating blog posts it's often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content. Making Screenshots When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog. take screenshot opti

Copier < 6.0.0b0 considered dangerous

Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories. !!UPDATE As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update. This is a PSA I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I

PyOhio CFP's

Here are some CFP's that I used for PyOhio 2022. https://pretalx.com/pyohio-2022/cfp {.hoverlink} Idea to blog post in minutes - Shorter Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it's plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle. Lets build a whole blog site in 5 minutes. Add Kedro to your Pandas Workfl

pipx examples

count lines of code

Copier Templates

%%include til/copier_endops %%include til/copier-template-variables %%include til/copier-answers

Smoother Python with automatic imports | pyflyby

This is not a flaky works half the time kind of plugin, it's a seriously smooth editing experience. I've just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython. I can't tell you how pumped I am for this, and how good its felt to use over the past few weeks. It's glorious. YouTube video Listen to me rant on how great pyflyby is https://youtu.be/2QW5DJiEJH4 {.youtube-embed} Give the

Kedro Course

find all nodes with raw in the name use parameters make and use a logger use find-kedro in spaceflights slide in a new node vizualize your pipeline find slow nodes move the configuration directory build kedro into docker and run it pyinstrument pdb make a new cli command make a hook load catalog entries slice a pipeline by tag by name from inputs to outputs

kedro catalog create

I use to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key. https://youtu.be/_22ELT4kja4 {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Running Kedro Catalog Create The command to ensure there are catalog entries for every dataset in the pass

Using Nix to manage my Python Interpreter

I recently started playing with nix. goals automatically select correct python version per project activating one doesn't bleed into the other Installing nix controlling nix-env searching for packages https://search.nixos.org/ shell

Just Ask Ipython for help

It happens to the best of us We can't all remember every single function signature out there, it's just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal. https://youtu.be/TZrRAP-9UMk Docstrings In any python repl you can access the docstring of a function by calling for . In Ipython we can even get some syntax highlighting with the . Source Code Sometim

Setting Parameters in kedro

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files. https://youtu.be/Jj5cQ5bqcjg {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. parameters files You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a and para

Writing your first kedro Nodes

https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla data structures. import node You will need to import node from kedro.pipeline to start creating nodes. func The is a callable that will take the and create the . inputs / outputs Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is

Running your Kedro Pipeline from the command line

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli. https://youtu.be/ZmccpLy-OEI {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Kedro run To run the whole darn proj

kedro Virtual Environment

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment. https://youtu.be/ZSxc5VVCBhM {.youtube-embed} conda venv pipenv conda I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don't have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool. st

Kedro Pipeline Create

Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself. https://youtu.be/HtyIKqlEoNw creating a new pipeline The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your , to be covered later, you will need to add it yourself. results The directory structure that it creates looks like this.

Kedro Install

Kedro comes with an command to install and manage all of your projects dependencies. https://youtu.be/IWimEs-hHQg cd into your project directory and activate env You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment. [[ kedro-new ]] this post covers kedro new [[ kedro-environment ]] This post covers creating your virtual environment for kedro install kedro Make sure you have kedro installed in your cur

Kedro Git Init

Immediately after , before you start running or your first line of code the first thing you should always do after getting a new kedro template created is to . https://youtu.be/IGba3ytf_6U git init Its as simple as these three commands to get started. I don't care if this project is for learning, if it will never have a remote or not, use git.

Kedro New

https://youtu.be/uqiv5LAiJe0 {.youtube-embed} Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template. [[ what-is-kedro ]] Unsure what kedro is, Check out yesterdays post on What is Kedro. pipx I reccomend using when running kedro new. is designed for system level cli tools so that you do not

What is Kedro

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract that the user specifies through entries. These entries are loaded, ran through a function, and saved by . The order that these are executed are determined by the , which is a DAG . It's the 's job to manage the execution of the . https://youtu.be/Wf4rnFsaFFU {.youtube-embed}

Incremental Versioned Datasets in Kedro

Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work. [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. How

Manage many git repos with ease

mu-repo

I Started Streaming on Twitch

I recently started streaming on twitch.tv/waylonwalker and it's been a blast so far. python kedro Data Science Data Engineering webdev digital gardening Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. kedro-issue-606 Inspiration My introduction to twitch c

Upcoming Stream

!!! Caution I'm no longer streaming As much as I would really love to make streaming work, its really hard for my family situation to make large blocks of time work for me. https://stackoverflow.com/questions/16720541/python-string-replace-regular-expression {.hoverlink} I am starting to stream 3 days per week, before I start work in the morning. These streams will likely be me just talking through things I am already doing. Making DAGs do 🔮Magical Things | Open Source 🐍Python | kedro plugins |

Kedro Spaceflights - part 2 | Stream replay June 7...

This was my seconf time ever streaming on twitch.tv/waylonwalker , and I completely botched my mic 2x. https://youtu.be/_7MwgKu-844 Links Spaceflights Tutorial my spaceflights repo Notes to get started

🌱 Digital Gardening | gif to Mp4 | Stream replay J...

https://youtu.be/I4VenHqIEng Doing some Digital Gardening on stream Ahrefs Errors ahrefs large images Automatic gif to mp4 gif to mp4 After this stream all gifs on my site are converted to mp4/webm if they exist. tmux-navigation-2021

Kedro Spaceflights - part 1 | Stream replay June 4...

This was my first time ever streaming on twitch.tv/waylonwalker . I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go. https://youtu.be/Y07UBr9Ccjs Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time co

Comprehensive guide to creating kedro nodes

The Kedro node is an essential part of the pipeline. It defines what catalog entries get passed in, what function gets ran, and the catalog entry to save the results under. does this link work? https://waylonwalker.com/what-is-kedro/ {.hoverlink} 👆 Unsure what kedro is? Check out this post. The node function The node function is the most common and reccomended way to define kedro nodes. It is a function that constructs and returns objects for you. Creating your first kedro node function The

Using Kedro In Scripts

With the latest releases of kedro , it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro [[ what-is-kedro ]] If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors as of 0.17.2 I've tried to do this in kedro and it turned into a rabbit hole of erro

Creating pypi-list with kedro

I had an idea come to me via twitter. Short one word name packages are becoming hard to find on pypi. Short one word readable package names that are not a play on words are easy to remember, easy to spell correctly, and quick to type out. Simple index I started with the simple index. Pypi provides a single page listing to every single package hosted on pypi via the simple-index

Silence Kedro Logs

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log First, how does one silence a python log? Python loggers can be retrieved by the module's function. Then their log level can be changed. Much of kedro's chattiness comes from INFO level logs. I don't want to hear about anything for my current use case unless i

python lsp setup

Setting up python with the native nvim>0.5 lsp was mr lsp-config https://github.com/neovim/nvim-lspconfig pyls#190 https://github.com/palantir/python-language-server/issues/190 mypy Getting mypy working with lsp was tricky for me. I had some issues trying to run mypy in ci and pyright in my editor and I really wanted them to match.

Python Diskcahe is locked

Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error Minimum reporducible error is one of my superpowers in development. I do this very often to sus ou

Vim Fugitive

Add current file and commit with diff in a split :on[ly] C-W o :on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default. C-I C-O cycle through the jumplist This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize and , I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific c

What is if __name__ == "__main___", and how do I u...

When a python module is called it is assigned the of otherwise if it's imported it will be assigned the of the module. Concrete example Let's create a module to play with a bit. We will call this module . It is a module that we may want to run by it'self or import and use in other modules. I have set this module up to execute one of two if statements based on whether the module it'self is being ran or if the module is being imported. Note it is not common to have a block, this is just

Custom Kedro Logger

DRAFT -

How to Install micromamba on linux (from the comam...

I really like using conda ( ) as my python virtual environment manager of choice. It's simple and it includes its own python interpreter using the version that I specify at creation. Mamba from their readme Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE core parts

kedro replit

I am trying to see what an embeded replit

Zev Averbach Interview

Zev Averbach, Frustrated spreadsheet jockey to software developer at 36 Q: Tell me about your journey as a spreadsheet jockey into Data Engineering? A: First of all, it's hilarious that I accidentally found your questions for this interview by Googling myself. 😊 I've always been a frustrated software user, and that frustration led me to be a "power user" (keyboard shortcuts etc) of my most used applications, as well as a "visual coder" using desktop automation like Alfred and Keyboard Maestro (M

Pytest capsys

Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys using capsys Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.

Building Rich a Dev Server

Draft Post I've really been digging @willmcgugan's rich library for creating TUI like interfaces in python. I've only recently started to take full advantage of it. Dev Server I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to. automatic port number auto-restart display ( port, pid, uptime ) finding the port I

Automatic Conda Environments

I have automated my process to create virtual environments in my python projects, here is how I did it. I've really been digging my new tmux session management setup. Now I have leveled it up by adding direnv to my workflow. It will execute a shell script whenever I cd into a directory. One thing I wanted to add to this was, automatic activation of python environments whenever I cd into a directory, or create a new environment if one does not exist. https://waylonwalker.com/tmux-nav-2021/ {.hove

fix crlf for entire git repo

Final Result

How I Review Pipeline Code

I have started doing more regular PR's on my teams Kedro pipelines. I generally take a two phase approach to the review in order to give the reviewee both quick and detailed feedback. [[ what-is-kedro ]] initial scan (Phase1) passing ci Variable Names Antipatterns No commented out code Docsttrings generally make sense Phase1 is typically a quick scan over the PR right within the PR window in my browser. Passing CI flake8 black isort interrogate pytest build The very first thing that needs to h

Kedro pipeline_registry.py

With the realease of came a new module in the project template . Here are some notes that I learned while playing with this new module. migrating to create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your . pipeline_registry only works in Conflict Resolution What happens If I register pipelines in both places I w

🐍 Pluggable Architecture with Python

pytest has open sourced their amazing plugin framework , it allows library authors to give their users a way to modify the libaries behavior without needing to submit a change that may not make sense to the entire library. Previous Experience My experience so far as a plugin user, and plugin author has been great. Building and using plugins are incredibly intuitive. I wanted to dive a bit deeper and see how they are implemented inside of a library and its a bit of a mind bend the first time yo

Create Og Image Covers Using Python

There are so many tutorials out there for creating OG images with or , but I have yet to see many using python. PIL Template Text Overlays

⚙ How Python Tools Are Configured

There are various ways to configure python tools, config files, code, or environment variables. Let's look at a few projects that allow users to configure them through the use of config files and how they do it. Motivation This will not include how they are implemented, I've looked at a few and its not simple. This will focus on where config is placed and the order in which duplicates are resolved. The motivation of this article is to serve as a bit of a reference guide for those who may want

Minimal Kedro Pipeline

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in and 27 in . 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline I have everything for this post hosted in this gihub repo , you can fork it, clone it, or just follow along. Installation Caveats This repo represents the minimal amount of structure to build a ked

Markdown Cli

This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things posts tags draft posts data frontmatter filepath content template html render content Markdown.Markdown support extentsions frontmatter cleaning. provide ways to hook in or clean up the frontmatter Markata.Markata methods load render save Markata.Post methods load render save Markata plugins befor

Blog Data With Python

Generating an api for a blog is much simpler than one might expect with python. Markdown Frontmatter Fill in the blanks fix missing data Fast

Kedro - My Data Is Not A Table

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. DataFrames are the heart of most pipelines These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types. [[ what-is-kedro ]] unfamiliar with kedro, check out this post Sometimes datasets are not t

Quickly Change Conda Env With Fzf

Changing conda environments is a bit verbose, I use a function with fzf that both lists environments and selects the one I want in one go. Conda I have used conda as a virtual environment tool for years now. I started using conda for its simplicity to install packages on windows, but now that has gotten so much better and it's been years since I have run a command. I'm sure that I could use a different environment manager, but it works for me and makes sense. What environment manager do you u

Minimal Python Package

What does it take to create an installable python package that can be hosted on pypi? What is the minimal python package setup.py my_module.py This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file. Directory structure setup.py name The name of the package can contain any letters, numbers, "_", or "-". Even if it's for internal/personal consumption only I usually check for discrepancy with p

Ipython-Config

I use my ipython terminal daily. It's my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration. install ipython Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython. You are using a virtual environment right? Virtual environments like venv or conda ca

Custom Ipython Prompt

I've grown tired of the standard ipython prompt as it doesn't do much to give me any useful information. The default one gives out a line number that only seems to add anxiety as I am working on a simple problem and see that number grow to several hundred. I start to question my ability 🤦‍♂️. Configuration If you already have an ipython config you can move on otherwise check out this post on creating an ipython config. [[ ipython-config ]] The Dream Prompt I want something similar to the stars

Ipython Ninjitsu

?docstring ??sourcecode %run %debug %autoreload %history autoformat %reset !shell commands ?docstring Stop going to google everytime your stuck and stay in your workflow. The ipython is a superhero for productivity and staying on task. Note This does jump you into a pager, a j,k or up, down to navigate, q to quit. ??sourcecode Docstring not enough for you use case. I often run into cases where the docstring is not clear enough and I need to see the implementation for myself to see what a f

Automating my Post Starter

One thing we all dread is mundane work of getting started, and all the hoops it takes to get going. This year I want to post more often and I am taking some steps towards making it easier for myself to just get started. When I start a new post I need to cd into my blog directory, start neovim in a markdown file with a clever name, copy some frontmatter boilerplate, update the post date, add tags, a description, and a cover. Todo List for starting a post frontmatter template Title slug tags date

Windowing Python Lists

In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools more-itertools` are full of cases of this. This post is a walkthrough of me solving a problem with rather than reaching for a for loop, or pandas. I am working on a one-line-link expander for my blog. I ended up doing it, just by modifying the markdown with python. I first split the post into line

Testing Data Pipelines

Lint/Format/Doc black flake8 interrogate mypy Pipeline Assertions pipeline constructs pipeline as expected nodes pipeline has minimum nodes test minimum tags test alternate tags Catalog Assertions test catalog follows naming structure Node Tests test function does the correct operations on test data Great Expectations

reasons-to-kedro

There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines [[ what-is-kedro ]] Starter Template Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. [[ create-new-kedro-project ]] read more about how

What's New in Kedro 0.16.6

Kedro 0.16.6 is out! Let's take a look through the release notes Deployment Docs This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Argo Prefect Kubeflow Batch SageMaker Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies. Argo, Prefect, and Kubeflow have an interesting technique where

Designing a "Router" for kedro

nodes_global I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called . It worked fairly well but did have some issues around being set as a global variable. But... One thing in particular that it did not lend itself well to was being able to create a packagable pipeline that I could pip install and append into an

Reclaim memory usage in Jupyter

Today I ran into an issue where we had a one-off script that just needed to work, but it was just chewing threw memory like nothing. It started with a colleague asking me How do I clear the memory in a Jupyter notebook, these are the steps we took to debug the issue and free up some memory in their notebook. How do I clear the memory in a Jupyter notebook? Pre check the status of memory There are a number of ways that you can check the amount of memory on your system. The easiest is not necessa

Strip Trailing Whitespace from Git projects

A common linting error thrown by various linters is for trailing whitespace. I most often use flake8. I generally have [pre-commit]( https://waylonwalker.com/pre-commit-is-awesome hooks setup to strip this, but sometimes I run into situations where I jump into a project without it, and my editor lights up with errors. A simple fix is to run this one-liner. One-Liner to strip whitespace bash read more about how [[pre-commit-is-awesome]]

Three things to Automate with Python using Pandas

Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an alternative out there that can save us time, automate boring things, and allow us to open up our minds for the part that we add value, Thinking about the data. Focus on Value Add Operations Lets face it, stitching together spreadsheets is zero value add by

How to Install miniconda on linux (from the comman...

miniconda is a python distribution from continuum. It's a slimmed-down version of their very popular anaconda distribution. It comes with its own environment manager and has eased the install process for many that do not have a way to compile c-extensions. It made it much easier to install the data science stack on windows a few years ago. These days windows are much better than it was back then at compiling c-extensions. I still like its environment manager, which installs to a global directory

What's New in Kedro 0.16.4

If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks. This one comes a bit surprising as it was just casually mentioned in #435 auto enabled plugins mentioned in issue 435 Think pytest As mentioned in #435 this is the model that pytest uses. Not all plugins automatically start doing things right out of the box but require a CLI argument. simplicity It feels a bit crazy that simply installing a package will change the way that your

Integration testing with Python, TestProject.io, a...

!!! Caution None of the testproject.io urls resolve anymore in JAN 2025, I removed all of the broken links. As I continue to build out waylonwalker.com I sometimes run into some errors that are not caught because I do not have good testing implemented. I want to explore some integration testing options using GitHub's actions. Running integration tests will not prevent bugs from happening completely, but it will allow me to quickly spot them and rollback. 🤔 What to test first? The very first t

🐍 Practice Python Online

When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these you can try out something quick. Sometimes I see snippets from blogs or tweets and I need to try the out for myself to really understand. When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to sh

Kedro Catalog

I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. Process metadata will be attached to the dataset object under a attribute metadata will be updated metadata will be empty until a pipeline is ran with the hook on optionally a function to add metadata will be added metadata will be stored in a file next to the meta Problems This Hook Should solve what datasets have a columns with in the name what datasets were updated after last tuesday which pipe

How python tools configure

mypy Mypy's config parser seems to be one of the most complex. This is likely in part to it having the largest backwards compatability of all projects that I looked at. mypy/config_parser flake8 options/config.py black black portray only uses pyproject.toml portray/config.py interrogate only uses pyproject.toml

🐍 Parsing RSS feeds with Python

I am looking into a way to replace my google reader experience that I had back in 2013 before google took it from us. I am starting by learning how to parse feeds with python, and without much previous knowledge, it proved to be much easier than anticipated thanks to the library. This is how I used python to parse rss and setup my own custom feed. Install Install the feedparser library. Get the content The feed object The feed is a feedparser.FeedParserDict. For all intents and purposes this

SLIDES - understanding python \*args and \*\*kwarg...

Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog @_waylonwalker . python args and kwargs

Gracefully adopt kedro, the catalog

Why use kedro catalog? While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all. Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different fo

How to find things in your kedro catalog

kedro 0.16.2 just dropped last week with a long-awaited feature... catalog search ! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful. The Catalog The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place

How Kedro handles your inputs

Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing. *args/**args review Check out this post for a review of how work in python. [[ python-args-kwargs ]] python args and kwargs article by @_waylonwalker All Kedro inputs are catalog Entries When kedro runs your pipeline it uses the catalog to imperatively load your data, mea

understanding python \*args and \*\*kwargs

Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function a

pre-commit is awesome

I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it's easy. It will automatically run checks for you. In some cases, it will even automatically fix them for you. Out of the box, it will do things like automatically trim extra whitespace, fix file endings, and ensure file sizes are not too large for git. I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it

Kedro Static Viz 0.3.0 is out with Hooks Support

kedro-static-viz is out with support for the newly released hooks feature. This means that you can have automatically deploy a full gatsby site keeping your visualization always up to date. Even though it is a static site there is no functionality lost. The only thing that's missing is the flask server. With kedro-static-viz you can deploy your visualization to a number of static hosting providers such as GitHub pages free of charge with wicked fast performance ⚡ It's Fast Even though

Create Custom Kedro Dataset

Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key. But what is happening behind the scenes Under the hood there is an that each connector inherits from. It sets up a lot of the behind the scenes structure for us so

Interrogate is a pretty awesome, brand new, cli fo...

As usual while listening to python bytes 181 I heard of a tool that I had to try out right away! This thing is 🔥 hot off the press folks, we're talking the first release only 3 weeks ago. Its something that the python community needed years ago, and it belongs in your CI today . I had tried several tools that tried to do docstring coverage in the past but they were a bit cumbersome and were quickly forgotten about. Not interrogate, its dead simple! Nothing I have tried has come close to being

creating the kedro-preflight hook

Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and (nouns). With a , or (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability. kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and

📝 Kedro Preflight Notes

This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources. Must Haves check that inputs exist or are of a type to skip (sql) Good to haves check that all input and output databases are accessible with good credentials check for s3 bucket access check for spark install Implementation run params

python-deepwatch

Is it possible to deep watch a single python function for changes? Shallow Watch keeping track of a python functions hash is quite simple. There is a method attached to every python function. Calling it will return a hash of the function. If the function changes the hash will change. Using hashlib provides a consistent hash. Now we have a consistent way to hash function code. Deep hashing Find dependencies setup a function in a module with a dependency Create Generic module importer by filepath

TIL: Bind arguments to dynamically generated lambd...

This past week I had a really weird bug in my kedro pipeline. For some reason data running through my pipeline was coming out completely made no sense, but if I manually request raw data outside of the pipeline it matched expectations. NOTE While this story is about a kedro pipeline, it can be applied anywhere closures are put into an iterable. Debugger to the rescue After a few days of looking at it off and on, I pinpointed that it was all the way down in the raw layer. Right as data is com

Four Github Actions for Python

If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simple cli I have opted to use the cli for most of these, that way I know exactly what is happening and have more control over it if I need. If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simpl

Variables names don't need their type

So often I see a variables inside of its name and it hurts me a little inside. Tell me I'm right or prove me wrong below. Examples Pandas are probably the worst offender that I see Sometimes vanilla structures too! Edge Cases? It's so common when you need to get inside a data structure in a special way that itsn't provided by the library.... I am not exactly sure of a good way around it. Containers are plural Always name your containers plural, so that naming while iterating is simple. Befo

Create New Kedro Project

This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro . You can learn more about this Hello World Example in the docs 🧹 Install Kedro 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization Create a Virtual Environment I use conda to control my virtual environments and will create a new environment called with the following command. note the latest compatible vers

What is Kedro

[[ what-is-kedro ]] This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate and . This article will cover the 10K view of kedro, future articles will dive deper into each one. kedro is an open-source data pipeline framework. It pr

Long variable names are good

🏷️ Long variable names are a good thing. Self documenting code is more important than poorly documented code. Simply adding a few characters to your variable names can go a long ways. Containers are plural Aliases are welcome Scope is important

simple click 2

simple click

cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Example Ideas Things a cli can do to enhance your library. 🆚 print version 🕶 print readme 📝 print changelog 📃 print config ✏ change config 👩‍🎓 run a tutorial 🏗 scaffold a project with cookiecutter 🖱 Click Click is the most popular python cli tool framework for python. There are others, some old, some new c

SqlAlchemy Models

Make a connection Make a session Make a Base Class Make your First Model Make your own Base Class to inherit From Use the Custom Base Class

Building Cli apps in Python

Packages Click Inputs Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option typically aliased with a shorthand ('-v', '--verbose') **From the Docs To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. Argument positional required no help text supplied by click Yaspin Yaspin Gif Cl

Kedro

See all of my kedro related posts in [[ kedro-feed ]]. #kedrotips I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips . 🗣 Heads up Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful mo

📝 Packages to Investigate Notes

jmespath Tabnine Bulwark |-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example

Debugging Python

Using pdb

Just Use Pathlib

Pathlib is an amazing cross-platform path tool. Import Create path object Current Directory Users Home Directory module directory Others Let's create a path relative to our current module. Check if files exist Make Directories rename files List files Glob Files recursively Write

Filtering Pandas

query Good for method chaining, i.e. adding more methods or filters without assigning a new variable. masking general purpose, this is probably the most common method you see in training/examples isin capable of including multiple strings to include contains Good For partial matches MASKS anything that we put inside of square brackets can be set as a variable then passed in. Operators & - and ~ - not | - or AVAILABLE and NAME AVAILABLE or NAME AVAILABLE and not NAME

Making good documentation in python

Tools Sphinx Portray I just started using portray and it is amazingly simple to use! Methodology

Pyspark

I have been using pyspark since March 2019, here are my thoughts.

Quick Progress Bars in python using TQDM

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. for more gifs like these follow me

Clean up Your Data Science with Named Tuples

If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing , , and data. You may need to calculate total revenue, inventory

Background Tasks in Python for Data Science

This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth R

📝 Bash Notes

Bash is super powerful. File System Full Show Remaining Space on Drives show largest files in current directory Move files then symlink them Fuzzy One Liners edit in vim cat a file bash execute git add git reset Kill a process Finding things Files fd-find is amazing for finding files, it even respects your file 😲. Install with . ++Vanilla Bonus Content ** show matching text ** ++Vanilla Bonus ** show file names only ** ++Vanilla Bonus Recursively Replace text ++Vanilla Bon

Autoreload in Ipython

I have used for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library co

Python Tips

Dictionaries Unpacking **kwargs func(**input) locals().update(d)

Generating Readme Tables From Pandas

Generating Readme Tables From Pandas I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, and . Lets say I have a Pandas DataFrame in memory as already. All I would need to do to convert the first 5 rows to markdown and copy it to the clipboard is the following. This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without opening the entire file.

Pycon 2018 Roundup

These are my notes from pycon 2018 videos. I love the python community and especially the conference talks. This year I am going to take some notes from my favorite talks and post them here. This is an Incomplete working post. Jake VanderPlas - Performance Python: Seven Strategies for Optimizing Your Numerical Code Always profile before making any optimizations. Vectorize with Numpy Looping in python can be slow Use specialized data structures. scipy.spacial pandas xarray scipy.sparse sp

My favorite pandas pattern

My favorite pandas pattern I work with a lot of transactional timeseries data that includes categories. I often want to create timeseries plots with each category as its own line. This is the method that I use almost data to achieve this result. Typically the data that am working with changes very slowly and trends happen over years not days or weeks. Plotting daily/weekly data tends to be noisy and hides the trend. I use this pattern because it works well with my data and is easy to explain to

background tasks in python

I have tried most of the different methods in the past and found that copying and pasting the threadpoolexecutor example or the processpoolexecutor example from the standard library documentation to be the most reliable. Since this is often something that I stuff in the back of a utility module of a library it is not something that I write often enough to be familiar with, which makes it both hard to write and hard to read and debug. If you are looking for a good overview of the difference

Pycon 2017 Roundup

Pycon 2017 Roundup Good afternoon fellow Data Geeks. Last week Pycon released 141 videos of greatness. Here are my top picks from the event. #3 Kelsey Hightower - Keynote - Pycon 2017 https://www.youtube.com/watch?v=u_iAXzy3xBA&t=1795s #2 Al Sweigart Yes, It's Time to Learn Regular Expressions PyCon 2017 https://www.youtube.com/watch?v=abrcJ9MpF60 #1 Trey Hunner Readability Counts PyCon 2017 https://www.youtube.com/watch?v=knMg6G9_XCg What's on Tap This afternoon we have a cup of from one of