All Posts

I'm Out On Agents

Its the year 2025 and we are only a few years into having 6 months to live before ai takes our jobs, and the big push right now is agents, managing agents. I will fully concede to I'm not doing it right, or a future state gets better than where we are right now, but right now they kinda suck. !!! Note Transparency Chat Chat is what really kicked off ai uses and goes back as old as computers, but it always sucked. Then chatgpt rocked the world with the biggest launch day in history and showed u

command palettes are overrated

Command palettes are slow, and overrated, you should treat yourself better. You probably installed VSC*** out of the box and your co-workers see you using the mouse and reprimanded you as they should. Mouse usage is not OK if you are a software dev, you should have the cheap ass free mouse that came with your cousins dell machine five years ago and only use if for emergencies. If you want to be fast you cannot do that by moving cursors to imprecise locations and clicking with your hand. You a

just fucking use kubernetes

You want to run containers? JUST FUCKING USE KUBERNETES. !!! tip "Obvious satire" If you don't like harsh language this is not the post for you. Obviously ripping off motherfuckingwebsite . !!! warning "ThIs is AI SLoP" If you don't like if you can fuck off to the next post, I'm having fun here, but satire is not my strong suit and needed some help. "But it’s complicated!" Shut up. Close twitter and fucking do something. Life is complicated. You know what else is complicated? Email. DNS. L

principal-engineer-at-meta

Jake Bolam principal engineer at Meta, has some of the best career advice for those looking to become principal or just be better at their craft. This video was such a banger I had to bring it in as a full post, and not just a thought. It was a random YouTube auto play, something that I probably wouldn't have clicked on given title an thumbnail, but turned out to be very impactful. Jake is such a smart guy with a lot of great insights, and I can tell he thinks really quick on his feet, he jus

csv

m9a devlog 1

It's sad to see textualize.io close the doors, but textual is still alive and maintained as a n open source project. I tried to use it very early, and struggled, this was before docs and tutorials really existed, before a lot of the widgets and components existed. Then as we all do I got busy and moved on to other things in life and did not have the capacity to build TUIs. I like tuis I like tuis, I like staying in the terminal. I use lf daily to move files around when I want something more

fix feed descriptions

Today I fixed a bug in markata that has been occurring for a few months where the description for posts come out as None if coming from cache, the issue was a pretty simple check and pull properly from cache. This fixes all the descriptions in feeds and metadata on the post. Better description While in there we went ahead and improved our get_description to more accurately return plain text without escaped characters, remove cutoff words, and add an elipsis if the description cuts off the text.

portal-platformer-devlog-1

Here is the current state of my platformer yet to really be named, I want to make something in between hollow knight and portal. Starting I made one once in make code arcace on a pybadge. It was quick and dirty, but fun to work on. It had the basic of blocks that I could move, blocks i could put a portal onto, and a goal for each level. Some levels you can just walk through and some levels required you to really think about where to place the portal. History So this version of the game is a l

backup distrobox image

Today I'm upgrading my distrobox, but don't want to end up in a situation where I can't get anything done becauase I work out of my distrobox.

Changing k8s Storage Class - Migration Job

I'm setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I'm all in on gitops so option 1 was not an option. So changing storageclass it is. Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute. Migration Job This migration job will create a new pvc with the

fragmention

This post is still WIP. ..... https://indieweb.org/fragmention##Challenges I've been digging through David Bushell's blog over the past day, he has some really good ideas about blogging and webdev. One really interesting post I came accross is url-fragment-text-directives . I've long had id's linked on my headings, though sometimes broken, or now showing the link, I've done my best to include them. Fragmentions extend this to allow any text to be linkable like this. fragmentioner ui: https://

markdown split panel

Today I was playing with markdown split panels. I want to be able to compare and constrast occasionually, today the inspiration hit to do this using admonitions. screenshot-2025-02-04T02-28-46-750Z.png {.more-cinematic} {.rounded-xl } !!! Note Mobile Users 🔄 The Markdown This is what I am going for, one admonition that is easy to remember, that nests inside of itself , and I can put as much markdown on the inside that I want. Here is the result of that markdown. !!! vsplit I Have two opinions {

markata search

A side effect of [[ markata-didyoumean ]] is that we are able to implement some rudimentary search based on slug, title, tags, and description. screenshot-2025-01-31T14-50-00-094Z.png {.more-cinematic} I made a video of the early work on using this side effect to search for content on markata.dev. Replay markata-search-1.mp4 This was first implemented to solve 404 pages giving users a way to try to find the page that they tried to access. But it turned out to be a decent way to search through t

Markata DidYouMean

Coming in Markata 0.9.1 is far better documentation. i.e. Documentation that actually exists for everything. As part of poking around I realized that I often go to look up the docs for a plugin and forget that the path is , sometimes I might try or or or I might even forget the plugin name exactly and try something like and get a 404. So I added a plugin to markata that takes care of this. screenshot-2025-01-31T14-53-31-264Z.png {.more-cinematic} I made a quick recording of this earl

pesos

Pesos is the act of Publish Elsewhere Syncicate to Own Site. It is an indieweb concept that I recently started applying to my own site. here does it skip again !!! note See https://indieweb.org/PESOS for more information, they have a ton of information about the indieweb In short it is the concept of pulling data from other sites that you use and republishing it to your own site. This gives a single source of information for you, and protection against sites and apis changing or rug pulling.

hover z-index and positioning

I broke my [[ sick-wikilink-hover ]] recently in a refactor, today I did some diving in to figure out what happened. Before As you can see in the screenshot below, the link is in a list of links, and when the hover image pops up it sits behind all of the other text. The z-index of the list-item is supposed to be raised above the others on hover. image Manually setting z-index to 20 in the inspector I noticed this message from devtools, "The position: static property prevents z-index from havin

I fixed lists in my blog

A small improvement to my blog today. I was checking out my [[ now ]] post and noticed that lists have left padding but no discs or numbers. I popped open my added these lines and rebuilt. And rebuild, which I actually do with [[ casey-just ]], but showing the command here for clarity. WHY the applies {.bg-blue-900 .border-r-8 .border-black .rounded-xl .max-w-xl .p-6 .font-bold} Dude, you know that @apply is an anti-pattern in tailwind right? {.bg-green-900 .border-l-8 .border-black .rounded-x

one-shot-s3-cleanup

So I made a mistake in an app I am working on and ended up creating thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails... etc. I was able to delete them all, but I wanted to make a one-shot script to do it. screenshot-2025-01-31T21-20-56-934Z.png One Shot App I got the idea of the one shot app from Simon Willison and replicated his setup in a chatgpt project screenshot-2025-01-12T18-16-57-443Z.png Initial Prompt s3_cleanup Here is the result of the s3_cleanup script. It ende

nvim-manager

I recently built a cli application as a nearly-one-shot-app called nvim-manager . It manages your nvim dotfiles install. screenshot-2025-01-31T21-21-40-707Z.png {.more-cinematic} Why {.chat-left} Don't we have stow? {.chat-right} Ya, thats not enough. {.chat-left} Why not?? {.chat-right} Inevitably shit goes sideways and I break my vim install. How is nvim manager any better nvim-manager allows you to install pinned versions of your dotfiles, your friends dotfiles, and distros in ~/.config. Thi

/colophon

Colophon a page that describes how the site is made, with what tools, supporting what technologies Author Waylon Walker's Profile Picture All posts on this site are written by Waylon Walker , the typical content has changed and evolved over time. I go back and make a few corrections, but for the most part things stay pretty much as they were published originally. see more in [[ about ]] tech This site is a static site build with my own static site generator [[ markata ]], [[ thoughts ]] or as

slow nfs performance

I'm running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I'm already learning why this is not reccommended. screenshot-2025-01-31T21-18-43-543Z.png Slow I've been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don't care if they take a bit longer to keep my master node freed up for more critical work. I just sta

thoughts 0.0.4

This is such an improvement to the backend of my website it warrants a blog post of celebration. For far too long I've been dealing with a tiny ass edit form on thoughts. I tend to not edit them, and try to get them right in one go. This is kinda the point of a thought, its a quick post meant to be the size of a tweet, but sometimes I'm leaving thoughts on a video or long post and want to make sure I have a good save point, but I just keep the thing in draft and hope I don't loose if for far

kind cluster

kind {.hoverlink} is a very useful tool to quickly standup and teardown kubernetes clusters. I use it to run clusters locally. Generally they are short lived clusters for trying, testing, and learning about kubernetes. Kind is Kubernetes in Docker, its very fast to get a new cluster up and running. Other than checking a box in docker desktop it is the easiest way currently to get a cluster up and running. I've used docker desktop for k8s before I really developed on k8s and it was buggy at t

markata

This post is a work in progress. Markata is the static site generator that I created to build my website [[ about-this-site ]]. I built it for me and I enjoy using it. I know everying it can do and I can extend it to do more easily. I have set it up for some friends to also use it and am proud that it helps them publish their content. It's a meme to create your own static site generator to make your website. Yes its funny, I don't recommend it if your not ready for the level of work that com

sick wikilink hover

Today I set up some sick wikilink hover effects using tailwind see [[ a-case-for-tailwindcss ]]. When you hover over them they show an image preview of the link that you are going to. I cant find where I have seen this but it comes from some docs sites. I'll finish this article later, just excited to see it up.

One Day Build - Play Outside

Inspired by Adam Savage and his One Day builds on youtube. I often build things, and want to make them generally useful for others and over configure out of the gate. This project is purely for me inspired by a need I have. play-outside !How-To This post will not directly show how to make a weather app, but document the process that I went through to make mine. It will show the tools that I used to make it, and the final result. The Situation It often goes in our house ask dad while he is bus

kubernetes 6 months in

I stumbled into kubernetes December 2023 when I was looking for a better way to self host applications. I was looking for something that didn't require logging into a server and building and deploying like a cave man. I wanted a smoother experience than docker compose was giving me. https://waylonwalker.com/looking-for-a-heroku-replacement/ {.hoverlink} This post turned into a list of tools that I have adopted into my k8s workflow, and plan to keep. enjoy. Kompose Kompose is a great tool for g

thoughts on unit tests

thoughts on unit tests cover image Theo's response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements "Everything must be unit tested, tests make code more maintainable." In reality when we are not writing complex low level code unit tests are probably the wrong approach. thought 192, a thought about theo's reaction to prime's unit testing Most of us are assemblers So much of software engineerin

Looking for a Heroku replacement, What I found was...

I've long hosted my personal blog as a static site on waylonwalker.com. It's all markdown, converted to html, and shipped as is. It's been great, I've moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it's just pointing ci to a different host after the site has built. What about server side Now the part that I have struggled with is how to cheaply host a server rendered application that can just l

why-is-postgres-default

Serious question. No one ever got fired for choosing PostgreSQL But, why. It's the most loved db, right? Right? Maybe it's time to rethink it. Don't get me wrong, if I need a relational db as a service, PostgreSQL is going to be my first choice, but why do I need to run a separate application for it? Tutorials use sqlite Why is that? Because there is nothing else to stand up. Nothing else to maintain. And you probably already have it installed on just about anything that has a battery. SQLite ru

A Case For Tailwindcss

I was watching @theprimeagen recently and I think he sold me on using tailwindcss. The thing about tailwind is that it is not a big component library, it's a set of css classes mapped to a few (usually one) style. All css classes are shitty, so you might as well use someone else's shitty css classes on all your projects rather than thinking you're being smart with a new set of classes that you will hate in 6 months when you come back to the project. roughly quoted from memory of @theprimeagen I

Playing Star Wars Text Adventure with a 10 yr old

The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star

Pydantic and singledispatch

I was reading about pydantic-singledispatch from Giddeon's blog and found it very intersting. I'm getting ready to implement pydantic on my static site generator markata , and I think there are so uses for this idea, so I want to try it out. The Idea Let's set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use and type hints to provide unique execution for each environment. We might want something like

Packages I Maintain

Markata I open sourced the static site framework that I use to build my-blog among other side projects. It's a plugins all the way down static site generator, that makes me happy to use. {% gh_repo_list_topic "waylonwalker", "markata" %} Repos used to build this blog my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don't have to worry about carrying around large images with my lightweight text files just to make some posts. {%

I was on Talk Python

After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science. I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on. During the episode I tried m

Using Different versions of python with pipx | pye...

"cell shaded, long, full body, shot of a cybernetic blue soldier with glowing pink eyes, llustration, post grunge, cinebatic dramatic atmosphere, sharp focus, pink glowing volumetric lighting, concept art by josan gonzales and wlop, by james jean, Victo ngai, David Rubín, Mike Mignola, Laurie Greasley, highly detailed, sharp focus,alien,Trending on Artstation, HQ, deviantart, art by artgem" -s50 -W832 -H416 -C7.5 -Ak_lms -S70567464 I love using pipx for automatic virtual environment management o

LIVE-REPLAY - Python dev | Markata todoui | 4/6/20...

https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow. During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off. dotfiles: https://github.com/WaylonWalker/devtainer today's project:

How I Quickly Capture Screenshots directly into My...

When I am creating blog posts it's often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content. Making Screenshots When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog. take screenshot opti

Copier < 6.0.0b0 considered dangerous

Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories. !!UPDATE As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update. This is a PSA I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I

PyOhio CFP's

Here are some CFP's that I used for PyOhio 2022. https://pretalx.com/pyohio-2022/cfp {.hoverlink} Idea to blog post in minutes - Shorter Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it's plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle. Lets build a whole blog site in 5 minutes. Add Kedro to your Pandas Workfl

Smoother Python with automatic imports | pyflyby

This is not a flaky works half the time kind of plugin, it's a seriously smooth editing experience. I've just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython. I can't tell you how pumped I am for this, and how good its felt to use over the past few weeks. It's glorious. YouTube video Listen to me rant on how great pyflyby is https://youtu.be/2QW5DJiEJH4 {.youtube-embed} Give the

kedro catalog create

I use to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key. https://youtu.be/_22ELT4kja4 {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Running Kedro Catalog Create The command to ensure there are catalog entries for every dataset in the pass

Just Ask Ipython for help

It happens to the best of us We can't all remember every single function signature out there, it's just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal. https://youtu.be/TZrRAP-9UMk Docstrings In any python repl you can access the docstring of a function by calling for . In Ipython we can even get some syntax highlighting with the . Source Code Sometim

Setting Parameters in kedro

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files. https://youtu.be/Jj5cQ5bqcjg {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. parameters files You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a and para

Writing your first kedro Nodes

https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla data structures. import node You will need to import node from kedro.pipeline to start creating nodes. func The is a callable that will take the and create the . inputs / outputs Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is

Running your Kedro Pipeline from the command line

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli. https://youtu.be/ZmccpLy-OEI {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Kedro run To run the whole darn proj

kedro Virtual Environment

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment. https://youtu.be/ZSxc5VVCBhM {.youtube-embed} conda venv pipenv conda I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don't have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool. st

Kedro Install

Kedro comes with an command to install and manage all of your projects dependencies. https://youtu.be/IWimEs-hHQg cd into your project directory and activate env You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment. [[ kedro-new ]] this post covers kedro new [[ kedro-environment ]] This post covers creating your virtual environment for kedro install kedro Make sure you have kedro installed in your cur

Kedro Git Init

Immediately after , before you start running or your first line of code the first thing you should always do after getting a new kedro template created is to . https://youtu.be/IGba3ytf_6U git init Its as simple as these three commands to get started. I don't care if this project is for learning, if it will never have a remote or not, use git.

Kedro New

https://youtu.be/uqiv5LAiJe0 {.youtube-embed} Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template. [[ what-is-kedro ]] Unsure what kedro is, Check out yesterdays post on What is Kedro. pipx I reccomend using when running kedro new. is designed for system level cli tools so that you do not

What is Kedro

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract that the user specifies through entries. These entries are loaded, ran through a function, and saved by . The order that these are executed are determined by the , which is a DAG . It's the 's job to manage the execution of the . https://youtu.be/Wf4rnFsaFFU {.youtube-embed}

Incremental Versioned Datasets in Kedro

Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work. [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. How

I Started Streaming on Twitch

I recently started streaming on twitch.tv/waylonwalker and it's been a blast so far. python kedro Data Science Data Engineering webdev digital gardening Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. kedro-issue-606 Inspiration My introduction to twitch c

Kedro Spaceflights - part 2 | Stream replay June 7...

This was my seconf time ever streaming on twitch.tv/waylonwalker , and I completely botched my mic 2x. https://youtu.be/_7MwgKu-844 Links Spaceflights Tutorial my spaceflights repo Notes to get started

🌱 Digital Gardening | gif to Mp4 | Stream replay J...

https://youtu.be/I4VenHqIEng Doing some Digital Gardening on stream Ahrefs Errors ahrefs large images Automatic gif to mp4 gif to mp4 After this stream all gifs on my site are converted to mp4/webm if they exist. tmux-navigation-2021

Kedro Spaceflights - part 1 | Stream replay June 4...

This was my first time ever streaming on twitch.tv/waylonwalker . I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go. https://youtu.be/Y07UBr9Ccjs Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time co

Using Kedro In Scripts

With the latest releases of kedro , it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro [[ what-is-kedro ]] If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors as of 0.17.2 I've tried to do this in kedro and it turned into a rabbit hole of erro

Silence Kedro Logs

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log First, how does one silence a python log? Python loggers can be retrieved by the module's function. Then their log level can be changed. Much of kedro's chattiness comes from INFO level logs. I don't want to hear about anything for my current use case unless i

Python Diskcahe is locked

Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error Minimum reporducible error is one of my superpowers in development. I do this very often to sus ou

What is if __name__ == "__main___", and how do I u...

When a python module is called it is assigned the of otherwise if it's imported it will be assigned the of the module. Concrete example Let's create a module to play with a bit. We will call this module . It is a module that we may want to run by it'self or import and use in other modules. I have set this module up to execute one of two if statements based on whether the module it'self is being ran or if the module is being imported. Note it is not common to have a block, this is just

How to Install micromamba on linux (from the comam...

I really like using conda ( ) as my python virtual environment manager of choice. It's simple and it includes its own python interpreter using the version that I specify at creation. Mamba from their readme Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE core parts

kedro replit

I am trying to see what an embeded replit

Pytest capsys

Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys using capsys Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.

Building Rich a Dev Server

Draft Post I've really been digging @willmcgugan's rich library for creating TUI like interfaces in python. I've only recently started to take full advantage of it. Dev Server I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to. automatic port number auto-restart display ( port, pid, uptime ) finding the port I

Site Down During Build

Recently I noticed a new netlify site of mine was down while I was checking to see if new content was live. Later found out this was consistent after each and every push the site would go gown as soon as I hit push, and would not come back until the build finished. Is this normal? Do other Netlify sites go down during build??? Short Answer NO. All of my google fu lead me to believe I was alone and none of my other sites do this. Digging into my build My deploy script ends with the following. Aft

Kedro pipeline_registry.py

With the realease of came a new module in the project template . Here are some notes that I learned while playing with this new module. migrating to create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your . pipeline_registry only works in Conflict Resolution What happens If I register pipelines in both places I w

Minimal Kedro Pipeline

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in and 27 in . 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline I have everything for this post hosted in this gihub repo , you can fork it, clone it, or just follow along. Installation Caveats This repo represents the minimal amount of structure to build a ked

Markdown Cli

This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things posts tags draft posts data frontmatter filepath content template html render content Markdown.Markdown support extentsions frontmatter cleaning. provide ways to hook in or clean up the frontmatter Markata.Markata methods load render save Markata.Post methods load render save Markata plugins befor

Kedro - My Data Is Not A Table

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. DataFrames are the heart of most pipelines These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types. [[ what-is-kedro ]] unfamiliar with kedro, check out this post Sometimes datasets are not t

Quickly Change Conda Env With Fzf

Changing conda environments is a bit verbose, I use a function with fzf that both lists environments and selects the one I want in one go. Conda I have used conda as a virtual environment tool for years now. I started using conda for its simplicity to install packages on windows, but now that has gotten so much better and it's been years since I have run a command. I'm sure that I could use a different environment manager, but it works for me and makes sense. What environment manager do you u

Minimal Python Package

What does it take to create an installable python package that can be hosted on pypi? What is the minimal python package setup.py my_module.py This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file. Directory structure setup.py name The name of the package can contain any letters, numbers, "_", or "-". Even if it's for internal/personal consumption only I usually check for discrepancy with p

Ipython-Config

I use my ipython terminal daily. It's my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration. install ipython Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython. You are using a virtual environment right? Virtual environments like venv or conda ca

Custom Ipython Prompt

I've grown tired of the standard ipython prompt as it doesn't do much to give me any useful information. The default one gives out a line number that only seems to add anxiety as I am working on a simple problem and see that number grow to several hundred. I start to question my ability 🤦‍♂️. Configuration If you already have an ipython config you can move on otherwise check out this post on creating an ipython config. [[ ipython-config ]] The Dream Prompt I want something similar to the stars

Automating my Post Starter

One thing we all dread is mundane work of getting started, and all the hoops it takes to get going. This year I want to post more often and I am taking some steps towards making it easier for myself to just get started. When I start a new post I need to cd into my blog directory, start neovim in a markdown file with a clever name, copy some frontmatter boilerplate, update the post date, add tags, a description, and a cover. Todo List for starting a post frontmatter template Title slug tags date

Windowing Python Lists

In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools more-itertools` are full of cases of this. This post is a walkthrough of me solving a problem with rather than reaching for a for loop, or pandas. I am working on a one-line-link expander for my blog. I ended up doing it, just by modifying the markdown with python. I first split the post into line

Adding Audio to my blog posts

This is episode 1 of the Waylon Walker Audio experience, posts from waylonwalker.com {.hoverlink} in audio form. So I have had this idea for awhile to add audio to my blog posts. The idea partly comes from the aws blog, if you have ever been on their blog you will have noticed that they have a voiced by amazon polly section. What to Expect Honestly I don't know this is all new to me and I dont have much to go off of. For now its a test that may or may not work out. I will say that the time th

gatsby-remark-embedder

Inspired by discourse's link expansion I am rolling out expansions for one line links on the blog waylonwalker . I was able to find a gatsby plugin gatsby-remark-embedder that expands one line links for social cards for popular platforms like twitter and YouTube through a repose from Kyle Mathews to my tweet. https://twitter.com/kylemathews/status/1329817928666005504 Use Cases This covers a couple of use cases I have with very little effort. Twitter YouTube install This was super quick and sim

Expand One Line Links

I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML into the markdown. While this works, it's more manual/difficult for me does not look the best, and does not read well as Goals for new card The new card should be fully automated to expand with title, description, and cover image. Bonus if I am able to attach a comment behind it. fully automated card expa

reasons-to-kedro

There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines [[ what-is-kedro ]] Starter Template Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. [[ create-new-kedro-project ]] read more about how

What's New in Kedro 0.16.6

Kedro 0.16.6 is out! Let's take a look through the release notes Deployment Docs This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Argo Prefect Kubeflow Batch SageMaker Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies. Argo, Prefect, and Kubeflow have an interesting technique where

Designing a "Router" for kedro

nodes_global I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called . It worked fairly well but did have some issues around being set as a global variable. But... One thing in particular that it did not lend itself well to was being able to create a packagable pipeline that I could pip install and append into an

Reclaim memory usage in Jupyter

Today I ran into an issue where we had a one-off script that just needed to work, but it was just chewing threw memory like nothing. It started with a colleague asking me How do I clear the memory in a Jupyter notebook, these are the steps we took to debug the issue and free up some memory in their notebook. How do I clear the memory in a Jupyter notebook? Pre check the status of memory There are a number of ways that you can check the amount of memory on your system. The easiest is not necessa

Strip Trailing Whitespace from Git projects

A common linting error thrown by various linters is for trailing whitespace. I most often use flake8. I generally have [pre-commit]( https://waylonwalker.com/pre-commit-is-awesome hooks setup to strip this, but sometimes I run into situations where I jump into a project without it, and my editor lights up with errors. A simple fix is to run this one-liner. One-Liner to strip whitespace bash read more about how [[pre-commit-is-awesome]]

Three things to Automate with Python using Pandas

Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an alternative out there that can save us time, automate boring things, and allow us to open up our minds for the part that we add value, Thinking about the data. Focus on Value Add Operations Lets face it, stitching together spreadsheets is zero value add by

How to Install miniconda on linux (from the comman...

miniconda is a python distribution from continuum. It's a slimmed-down version of their very popular anaconda distribution. It comes with its own environment manager and has eased the install process for many that do not have a way to compile c-extensions. It made it much easier to install the data science stack on windows a few years ago. These days windows are much better than it was back then at compiling c-extensions. I still like its environment manager, which installs to a global directory

What's New in Kedro 0.16.4

If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks. This one comes a bit surprising as it was just casually mentioned in #435 auto enabled plugins mentioned in issue 435 Think pytest As mentioned in #435 this is the model that pytest uses. Not all plugins automatically start doing things right out of the box but require a CLI argument. simplicity It feels a bit crazy that simply installing a package will change the way that your

Integration testing with Python, TestProject.io, a...

!!! Caution None of the testproject.io urls resolve anymore in JAN 2025, I removed all of the broken links. As I continue to build out waylonwalker.com I sometimes run into some errors that are not caught because I do not have good testing implemented. I want to explore some integration testing options using GitHub's actions. Running integration tests will not prevent bugs from happening completely, but it will allow me to quickly spot them and rollback. 🤔 What to test first? The very first t

🐍 Practice Python Online

When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these you can try out something quick. Sometimes I see snippets from blogs or tweets and I need to try the out for myself to really understand. When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to sh

Kedro Catalog

I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. Process metadata will be attached to the dataset object under a attribute metadata will be updated metadata will be empty until a pipeline is ran with the hook on optionally a function to add metadata will be added metadata will be stored in a file next to the meta Problems This Hook Should solve what datasets have a columns with in the name what datasets were updated after last tuesday which pipe

How python tools configure

mypy Mypy's config parser seems to be one of the most complex. This is likely in part to it having the largest backwards compatability of all projects that I looked at. mypy/config_parser flake8 options/config.py black black portray only uses pyproject.toml portray/config.py interrogate only uses pyproject.toml

Why use a cms

When first learning to code its very common to hard code everything right into the code. This happens with most folks in just about any language. Whether its HTML or markdown for front end content, or even hardcoding parameters in our backend languages like python, or node.js. 🤷‍♀️ What's wrong with hard coding everything? Hard coding everything right into your code makes it really hard for non-technical collaborators to join. It makes it nearly impossible to hand websites off to clients without

🐍 Parsing RSS feeds with Python

I am looking into a way to replace my google reader experience that I had back in 2013 before google took it from us. I am starting by learning how to parse feeds with python, and without much previous knowledge, it proved to be much easier than anticipated thanks to the library. This is how I used python to parse rss and setup my own custom feed. Install Install the feedparser library. Get the content The feed object The feed is a feedparser.FeedParserDict. For all intents and purposes this

How I Built My GitHub Profile

I ran a discussion on dev that collected quite a list of examples in the comment section. So many great calls to action, animations, memes, and weird tricks. dev.to whats-on-your-github-profile My current profile Waylon Walkers GitHub profile social icons Upload all of your icons to the repo in a directory such as or , then link them with a attribute like below. I used html for mine, not sure if you can set the in markdown. note I did add a bit of (non-breaking-whitespace) between m

SLIDES - understanding python \*args and \*\*kwarg...

Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog @_waylonwalker . python args and kwargs

Gracefully adopt kedro, the catalog

Why use kedro catalog? While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all. Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different fo

How to find things in your kedro catalog

kedro 0.16.2 just dropped last week with a long-awaited feature... catalog search ! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful. The Catalog The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place

How Kedro handles your inputs

Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing. *args/**args review Check out this post for a review of how work in python. [[ python-args-kwargs ]] python args and kwargs article by @_waylonwalker All Kedro inputs are catalog Entries When kedro runs your pipeline it uses the catalog to imperatively load your data, mea

Refactoring your blog urls

I just did a quick refactoring of my JAMStack blog urls. Some didn't fit with my style, some had that I wanted to switch to , and others were ridiculously long. I've been using forestry as my CMS, I write many of my posts there, and sometimes it picks some crazy file names (based on my titles). It was time to refactor. [[ refactor-in-cli ]] When refactorings similar to this get really big I often need to do some project wide find an replace, I usually do this right from the command line. 🖊

understanding python \*args and \*\*kwargs

Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function a

pre-commit is awesome

I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it's easy. It will automatically run checks for you. In some cases, it will even automatically fix them for you. Out of the box, it will do things like automatically trim extra whitespace, fix file endings, and ensure file sizes are not too large for git. I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it

Kedro Static Viz 0.3.0 is out with Hooks Support

kedro-static-viz is out with support for the newly released hooks feature. This means that you can have automatically deploy a full gatsby site keeping your visualization always up to date. Even though it is a static site there is no functionality lost. The only thing that's missing is the flask server. With kedro-static-viz you can deploy your visualization to a number of static hosting providers such as GitHub pages free of charge with wicked fast performance ⚡ It's Fast Even though

How to get Dev Comments from an article Url

I want to incorporate some of the wonderful comments, \U0001F495, \U0001F984, and \U0001F516's that I have been getting on dev.to on my website. I have dabbled once or twice with no avail this time I am taking notes on my journey, so follow along and let's get there together. By the end of this post, I will have a way to get comments from posts on the client-side thanks to the wonderfully open dev.to API. I want to incorporate some of the wonderful comments, 💕, 🦄, and 🔖's that I have been gett

Adding google fonts to a gatsbyjs site

stack overflow link

Create Custom Kedro Dataset

Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key. But what is happening behind the scenes Under the hood there is an that each connector inherits from. It sets up a lot of the behind the scenes structure for us so

Interrogate is a pretty awesome, brand new, cli fo...

As usual while listening to python bytes 181 I heard of a tool that I had to try out right away! This thing is 🔥 hot off the press folks, we're talking the first release only 3 weeks ago. Its something that the python community needed years ago, and it belongs in your CI today . I had tried several tools that tried to do docstring coverage in the past but they were a bit cumbersome and were quickly forgotten about. Not interrogate, its dead simple! Nothing I have tried has come close to being

creating the kedro-preflight hook

Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and (nouns). With a , or (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability. kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and

📝 Kedro Preflight Notes

This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources. Must Haves check that inputs exist or are of a type to skip (sql) Good to haves check that all input and output databases are accessible with good credentials check for s3 bucket access check for spark install Implementation run params

Explicit vs Implicit Returns in Javascript

Often when reading through javascript examples you will find some arrow functions use parentheses while others use braces . This key difference is that parentheses will implicitly return the last statement while braces require an explicit return statement. It is important to understand the difference between them because it is likely that you will find code examples of both and trying to edit code written differently than you're used to may have unintended consequences. Arrow functions Arro

python-deepwatch

Is it possible to deep watch a single python function for changes? Shallow Watch keeping track of a python functions hash is quite simple. There is a method attached to every python function. Calling it will return a hash of the function. If the function changes the hash will change. Using hashlib provides a consistent hash. Now we have a consistent way to hash function code. Deep hashing Find dependencies setup a function in a module with a dependency Create Generic module importer by filepath

TIL: Bind arguments to dynamically generated lambd...

This past week I had a really weird bug in my kedro pipeline. For some reason data running through my pipeline was coming out completely made no sense, but if I manually request raw data outside of the pipeline it matched expectations. NOTE While this story is about a kedro pipeline, it can be applied anywhere closures are put into an iterable. Debugger to the rescue After a few days of looking at it off and on, I pinpointed that it was all the way down in the raw layer. Right as data is com

Four Github Actions for Python

If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simple cli I have opted to use the cli for most of these, that way I know exactly what is happening and have more control over it if I need. If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simpl

Variables names don't need their type

So often I see a variables inside of its name and it hurts me a little inside. Tell me I'm right or prove me wrong below. Examples Pandas are probably the worst offender that I see Sometimes vanilla structures too! Edge Cases? It's so common when you need to get inside a data structure in a special way that itsn't provided by the library.... I am not exactly sure of a good way around it. Containers are plural Always name your containers plural, so that naming while iterating is simple. Befo

Create New Kedro Project

This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro . You can learn more about this Hello World Example in the docs 🧹 Install Kedro 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization Create a Virtual Environment I use conda to control my virtual environments and will create a new environment called with the following command. note the latest compatible vers

What is YOUR Advice for New Data Scientists

Learn the business Learn Git Your code does not need to be amazing Keep Learning Learn Git You dont have to start out as a git wizard with the cleanest possible commit history. At first dont let yourself get too wrapped up in it, the most important part is that you make commits. You will find needs for more advanced stuff later. Get comfortable with this, then learn how to , , , etc... Your code does not need to be amazing Get the job done. Keep it in small bite size pieces. Make readable

Do You Hoist

I am working through Wes Bos's beginnerjavascript.com/ I just hit module 18 on hoisting. It's something that I always knew was there, Its not something I typically see used or use myself. Do you Hoist? Do you have any use cases that you use hoising? Why? It seems like a really cool feature in any language that uses it, but I dont really notice it in use. What is Hoising There are many articles that cover this in far more depth, but its the idea that variable declarations and functions are de

What is Kedro

[[ what-is-kedro ]] This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate and . This article will cover the 10K view of kedro, future articles will dive deper into each one. kedro is an open-source data pipeline framework. It pr

Custom Scrollbar Design

Getting a custom scrollbar on your site makes it stand out a bit compared to the very plain stock one that are on most sites. This is how I set mine up on my gatsby site. Inspired by Wes Bos's new uses.tech I wanted a custom scrollbar on my personal site. I had tried to do it in the past, but gave up after it was not working. Looking at the Source Since uses.tech is open source I jumped on github, searched for scroll and found this layout.js . Copy it to my own component My first step was

2020 waylonwalker.com rebrand

Moving into 2020 I have been really leaning on using purple as my theme color everywhere more and more. Its time for an update to my personal site, not just because it feels plain, not just because the cover art I am using for dev.to doesn't fit my current card layout, but because I feel inspired and I want to. Starting point  This is what we are working with. It has been my card design for

I just added react-headroom to my site

It was so easy to get a professional looking navbar with just 3 lines of code. This package seriously is so usable on mobile it is ridiculous. I found this package from day-4 of the 100 days of gatsby challenge. It is by the wonderful man who brought us gatsbyjs Kyle Mathews, so you know its gotta be good. install react-headroom installation is easy Import Headroom There was no instructions for es6 style imports that are common with gatsbyjs sites like mine, but it was intuitive to figure out.

Serverless things to investigate

Social Automation Zappier Free only one step is free can take rss to social Automate.io Form submission https://getform.io Free to get started https://formkeep.com/?#pricing Free 7 day trial $4.99 cheapest after

I finally fixed my Styled-Components in gatsby.js

I finally fixed my Styled-Components in gatsby.js. I am starting a redesign of my website. I have started cross posting to dev.to more regularly. With that I have been making more detailed cover images at the recommended . These images get cut off on my own site, which is a bit ridiculous to have my own content not look right on my own site. But before we start a heavy redesign I have a small issue that has plagued the site for at least a year! I have a small issue that has plagued the si

Should I switch to Zeit Now

Netlify I have happily had my personal site waylonwalker.com hosted on netlify for nearly 2 years now. In fact I have hosted about a dozen different toy projects to play with on there, 4 of which have gone far enough to get a custom domain name. They are fast to deploy and consistently do so on every to main. Zeit I have recently started playing with zeit again. I really like their cli tool, its dead simple and makes sense. I tried the netlify one early on and dont think I really gave it

Personal URL shortener with Netlify Redirects

I love using URL shorteners to easily share links without hitting character limits, but they loose their meaning. Services like bit.ly will save my links for me so that I can find them, but I would rather them to be easy to remember. https://bit.ly/2ruLwQz does not roll of the tongue so well. 301 🤸‍♀️ I recently discovered a really cool feature of netlify that I have always looked past, . It is so simple cool and powerful, every netlify site should do this! But how 🤷‍♀️ simply add a file to

simple click

cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Example Ideas Things a cli can do to enhance your library. 🆚 print version 🕶 print readme 📝 print changelog 📃 print config ✏ change config 👩‍🎓 run a tutorial 🏗 scaffold a project with cookiecutter 🖱 Click Click is the most popular python cli tool framework for python. There are others, some old, some new c

SqlAlchemy Models

Make a connection Make a session Make a Base Class Make your First Model Make your own Base Class to inherit From Use the Custom Base Class

Building Cli apps in Python

Packages Click Inputs Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option typically aliased with a shorthand ('-v', '--verbose') **From the Docs To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. Argument positional required no help text supplied by click Yaspin Yaspin Gif Cl

Kedro

See all of my kedro related posts in [[ kedro-feed ]]. #kedrotips I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips . 🗣 Heads up Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful mo

📝 Packages to Investigate Notes

jmespath Tabnine Bulwark |-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example

Debugging Python

Using pdb

Just Use Pathlib

Pathlib is an amazing cross-platform path tool. Import Create path object Current Directory Users Home Directory module directory Others Let's create a path relative to our current module. Check if files exist Make Directories rename files List files Glob Files recursively Write

Filtering Pandas

query Good for method chaining, i.e. adding more methods or filters without assigning a new variable. masking general purpose, this is probably the most common method you see in training/examples isin capable of including multiple strings to include contains Good For partial matches MASKS anything that we put inside of square brackets can be set as a variable then passed in. Operators & - and ~ - not | - or AVAILABLE and NAME AVAILABLE or NAME AVAILABLE and not NAME

Quick Progress Bars in python using TQDM

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. for more gifs like these follow me

Clean up Your Data Science with Named Tuples

If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing , , and data. You may need to calculate total revenue, inventory

Background Tasks in Python for Data Science

This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth R

📝 Bash Notes

Bash is super powerful. File System Full Show Remaining Space on Drives show largest files in current directory Move files then symlink them Fuzzy One Liners edit in vim cat a file bash execute git add git reset Kill a process Finding things Files fd-find is amazing for finding files, it even respects your file 😲. Install with . ++Vanilla Bonus Content ** show matching text ** ++Vanilla Bonus ** show file names only ** ++Vanilla Bonus Recursively Replace text ++Vanilla Bon

Autoreload in Ipython

I have used for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library co

Forestry.io

Testing out forestry.io Sorry Netlify CMS I still ♥️ your product dont be forestry is simple I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really struggle configuring it. Forestry is so simple to setup. My favorite part is that I can code up my gatsby.js site, storing all editable text in markdown, and come back later and add the CMS based on existing documents. Configuration is Simple Forestry.io has this amazing feature to create cre

D3 Day 5

select rowid, key, raw, store_time, expire_time, access_time, access_count, tag, size, mode, filename from Cache where key='e3cd37c97980ab2c'; 'e3cd37c97980ab2c' 'e3cd37c97980ab2c' select rowid, key, raw, store_time, expire_time, access_time, access_count, tag, size, mode, filename from Cache where key='c14050404a107d5e'; Learn D3 in 5 days For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere! These are just stepping stones into real custom

D3 Day 4

Learn D3 in 5 days For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere! These are just stepping stones into real custom visualizations that cannot be done in any other tool today. I still cannot explain how excited I am to say "I created that in d3!!!" Todays Result Today I will be learning about d3 scales, and adding them to the bar chart that we created yesterday. Follow along as I try to create something interesting. today's_result R

D3 Day 3

Learn D3 in 5 days I recently subscribed to Ben Clinkinbeard's learn D3.js in 5 days, and am currently on day 3. I read through the first 2 days, and felt fairly comfortable with selecting elements, so I did not follow along on the first two days. I probably should have, but there are only so many hours in the day. Why Learn D3 D3 is the ubiquitous dynamic visualization library for building custom interactive visualizations on the web. It is a bit low level, and more verbose than many other libr

FlexBox

Flexbox-zombies I recently finished up the flexbox-zombies course to learn more about flexbox, and to become proficient with it. I can truly say that this course has changed the way that I create layouts. Flexbox is very intuitive now. What this course does really well at is explaining the concepts and hitting you with a ton of examples that you can work through really quickly. flexbox-zombies A clip from the final round against Dave Basic Setup Flexbox requires a wrapper container to work I