Post Archive

Packages I Maintain

I open sourced the static site framework that I use to build {% gh {% gh I am a heavy user of the {% gh I use vim for all of my text editing needs. It brings me joy to make any part {% gh

I was on Talk Python

After years of listening to I was quite nervous to talk on a show that I helped shape my career in such a During the episode I tried my best to let Yetu and Ivan take the spotlight as https://youtu.be/WTcjvwkXoY0 Michael made the call available on y

The one reason I switched to arch

The community, that I am a tinkerer, I am not going to run a stock desktop manager, mostly becuase I love minimal installs. I love just building up my system from the bottom up I spend a lot of my time in the terminal. I Now by far this is the thin

LIVE-REPLAY - Python dev | Markata todoui | 4/6/2022

https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love

Copier < 6.0.0b0 considered dangerous

Copier is a fantastic templating library written in python, but older versions As of May 15, 2022, the stable release of copier now includes these changes, if I Use copier several times per day and get fantastic benefit from this project, https://git

Smoother Python with automatic imports | pyflyby

This is not a flaky works half the time kind of plugin, it I can https://youtu.be/2QW5DJiEJH4 Give the video a watch, I did not have noise-cancelling on in obs. My pyflyby is hosted on pypi, so you can get it with pip. I have had no issues If you Se

You must use augroup with autocmd in vim | Here's how

If you are running vim autocmd https://youtu.be/2ITTn4Dl0lc For Lets create a new file called I What you need to do is clear out all commands in the augroup with Now this is what I have in my dotfiles for a silky smooth setup that does not

Code Review from the comfort of vim | Diffurcate

I often review Pull requests from the browser as it just makes it so easy to see https://youtu.be/5NKaZFavM0E This all stems from the great plugin by First to quickly checkout PR Next I have a few aliases setup for checking diffs. The first one chec

Setup a yaml schema | yamlls for a silky smooth setup

I https://youtu.be/xo4HrFoKF4c The video for this one is part of a I have my You will need the following plugins. I use plug, if you don Make sure that you have nvim Again if you don Next up is the heart of this post, the lsp-config.lua. This one

Open files FAST from zsh | or bash if thats your thing

https://youtu.be/PQw I am often in a set of tmux splits flying back and forth, accidentally close my Make sure you check out the YouTube video to see all of my improvements. If you have never heard of an alias before it Here are the new aliases that

How linux users install a text editor

In honor of the neovim 0.6.0 release, I decided to do a funny skit installing https://youtu.be/64oKLphhBuo The thing that took me the longest to realize was.... I had a path issue https://neovim.io/

30 days dotfile ricing

https://youtu.be/Jq1Y48F I am challenging myself to 30 days of dotfile ricing. I have been on linux This is primarily going to be focused on productivity, but may have a few Follow the

JUT | Read Notebooks in the Terminal

Trying to read a .ipynb file without starting a jupyter server? jut has you https://youtu.be/t8AvImnwor0 watch the video version of this post on jut ! Take a look at the help of the There is some good information on the projects Don jut nbconvert

kedro catalog create

I use https://youtu.be/ https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. The command to ensure there are catalog entries for every dataset in the passed Create Fills in new dataset entries with the default datase

nvim conf 2021 | IDE's are slow | Waylon Walker

https://youtu.be/E18m4KkJUnI Using Vim as a Team Lead I 💜 Tmux Why I stopped using @code Get there fast How I vim Use a graphical IDE if it works for you. vim is so well integrated into the terminal, take advantage As a team lead I bounce betweeen a

Just Ask Ipython for help

We can https://youtu.be/TZrRAP-9UMk In any python repl you can access the docstring of a function by calling for In Ipython we can even get some syntax highlighting with the Sometimes the docstrings are not good enough, and don The more common way

I made a neovim plugin

I https://youtu.be/8m5ipBuopPU Check out the project I want a simple way to make remaps into shell commands that can open new tmux For example I want to make remaps to do things like open the current file in lookatme. The main goal here is that rema

Setting Parameters in kedro

Parameters are a place for you to store variables for your pipeline that can be https://youtu.be/Jj5cQ5bqcjg https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. You can have multiple parameters files and choose which

Writing your first kedro Nodes

https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let You will need to import node from kedro.pipeline to start creating nodes. The Inputs and outputs can be None, a single catalog entry as a string, mutiple Sometimes in our pipeli

Running your Kedro Pipeline from the command line

Running your kedro pipeline from the command line could not be any easier to https://youtu.be/ZmccpLy-OEI https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. To run the whole darn project all we need to do is fire up

kedro Virtual Environment

Avoid serious version conflict issues, and use a virtual environment anytime https://youtu.be/ZSxc5VVCBhM conda venv pipenv I prefer to use conda as my virtual environment manager of choice as it give me stores environment in a root directory i.e. c

Kedro Install

Kedro comes with an https://youtu.be/IWimEs-hHQg You must start by having your kedro project either cloned down https://waylonwalker.com/kedro-new/ this post covers kedro new https://waylonwalker.com/kedro-environment/ This post covers creating your

Kedro Git Init

Immediately after https://youtu.be/IGba3ytf Its as simple as these three commands to get started. I don

Kedro New

https://youtu.be/uqiv5LAiJe0 Kedro new is simply a wrapper around the cookiecutter templating library. The https://waylonwalker.com/what-is-kedro/ Unsure what kedro is, Check out yesterdays post on What is Kedro. I reccomend using The kedro team do

What is Kedro

Kedro is an unopinionated Data Engineering framework that comes with a somewhat https://youtu.be/Wf4rnFsaFFU https://waylonwalker.com/what-is-kedro-1/ This is an updated version of my original what-is-kedro article If you are doing a series of operat

How I Kedro

https://youtu.be/bw5 I recently switched over to using Ubuntu, it works well pretty much out of the I am still using the built in default gnome terminal, it just works. It does One Dark Theme You can find my I use zsh as my default shell. I like to

tmux show-messages

https://youtu.be/LLk94fKpGg4 As we push the limits of tmux further and further you are bound to end up in a show-messages In case you wnat more information about show-messages, here is the man page. https://waylonwalker.com/tmux-nav-2021/ for more in

tmux ta

https://youtu.be/nT0FA1RNZEs Now your creating, jumping, and killing sessions like a boss. You are slicing This script is simply my fork of Chris Toomey My version of the By default I have my projects open with a vertical split, vim is on top, with

tmux display-message

https://youtu.be/utfLA6L8o5s You Here is a snippet of With Without setting the target-pane https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post Also check out the full YouTube

killing tmux

https://youtu.be/QWPyYx54JbE Now it One viable option is to nuke the whole dang thing. I actually do this more save and commit your work diligently before A more reasonable option might be to kill a single session. Killing sessions one by one from

tmux has-session

https://youtu.be/XucVVgGmesM I see you there, trying to script out your tmux layouts. Tryig to get each The The command is pretty straight forward, you simply ask tmux if the session name https://waylonwalker.com/tmux-nav-2021/ for more information

tmux list-keys

https://youtu.be/Y1MYmL8ZolE Tmux list keys can be a useful tool to help remind you of what kebindings you You can call list-keys from the command line but the interface is not very Running By default tmux comes with You can see the additional flag

tmux status-bar

https://youtu.be/mvgM8UH11 The tmux status bar can be a handy tool to remind yourself where you are within You can set a hotkey to show or hide the status bar. I really want a minimal status bar with very little bling, I want it to get out I want my

tmux source-file

https://youtu.be/dDq0depPrTs So you have been tricking out that Let We can run this command from your shell to re-source your changed It also works from the tmux command line. It This is my preferred way of re-sourcing my https://waylonwalker.com/

tmux fzf session jumper

https://youtu.be/DkJ9rb85LC0 Quickly getting between tmux splits is critical skill for productivity. You I have used this fzf one keybinding for quite awhile, honestly I did not make Like with many of my keybindings I have swapped this one out for

tmux next/prev session

https://youtu.be/8kZnjHPYnKU Now that we are splitting up work into their own sessions, lets talk about how https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post Also check out the full YouTube

tmux start application

https://youtu.be/c-a2Bnv Scripting tmux to open up specific applications can be intimidating your first I used this one for a number of years to take a quick peek into my systems 🗒️ note that the With the new tmux popup windows I really like the flo

tmux rename session

https://youtu.be/WRLRiQDjVIA So you have been working on your tmux workflow, you Let There is a default keybinding that you can use I https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post Also

tmux targeted session

https://youtu.be/5KE7Il7SOEk This is something that I made up but use every single day, this is what keeps https://waylonwalker.com/tmux-new-session/ This one is building off of yeserday https://waylonwalker.com/tmux-nav-2021/ for more information on

tmux new-session

https://youtu.be/LbQNdCAUogE This one starts a new chapter in our series that is going to open up a whole https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post Also check out the full YouTube

Git in Depth Notes

These are my notes from taking @nnja git --version

tmux attach

https://youtu.be/JQ0yDCVu44E attach is one of the most useful features of tmux. If you have no interest in this command will simply attach back to tmux if you are ever disconnected If you ever run long running tasks on a remote machine by sshing int

tmux ls

https://youtu.be/LY41GLn tmux ls will list the sessions that you have running within the tmux server if https://waylonwalker.com/tmux-attach/ https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full pos

tmux command line

https://youtu.be/SNu-4IrkjAs So far we have covered a lot of tmux commands and how they map to keybindings Let Or we can open the tmux command line and run it from tmux 🗒️ note that the tmux command is called by default when inside of tmux. Finally w

tmux copy-mode

https://youtu.be/-ypY tmux copy-mode is a tmux mode that lets you scroll, search, copy, and jump your Default keybinding to get into copy mode is If you are a vim user you will likely want to use vi style keys, add this to full list of copy-mode key

tmux join-pane

https://youtu.be/Vm5rRtcVXLw Join-pane allows you to join panes that you have broken away from your window, Before you can join a pane you must first have a pane marked to join. Once you My keybindings, you must add this to your https://waylonwalke

tmux break-pane

https://youtu.be/ICL609F2xnc Break-pane is a handy tmux command when your layout gets too cramped and you Default key binding for https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post

tmux zoom

https://youtu.be/Rn6mOarCQ-Y Zooming into the current split in tmux is a valuable tool to give yourself some Default key bindings for zooming the current split I have rebound this to match the default binding with mod+z rather so that I https://waylo

tmux new-window

https://youtu.be/YRPZBv-iYyE New window as it sounds makes new windows in tmux. Windows are kind of like Default key bindings for creating and navigating windows in tmux. As always I have rebound these keys because I generally prefer a single When I

tmux slect-pane

https://youtu.be/CPZJZjN9YTY These are my MOST often used keybindings that I use in tmux. They allow me to https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post

tmux rotate-window

https://youtu.be/06z5qf81ofo Rotate window is the main way that I navigated tmux before I learned Default keybindings My keybindings look just a bit different than the default ones, I do not like https://waylonwalker.com/tmux-nav-2021/ for more infor

tmux select-layout

https://youtu.be/F0mHnwTrNNc When you get many splits going in tmux sometimes its time for a new layout. https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post

tmux resize-panes

https://youtu.be/hpFYE2LU7xc Resizing panes in tmux can be quite difficult in default tmux, I Most often when I need to resize panes I just grab the edge of the pane with my https://waylonwalker.com/tmux-nav-2021/ for more information on how I naviga

tmux choose-tree

https://youtu.be/79Y-kqAiMpw Choose tree is a powerful tmux utility that provides a graphical interface to The default keybinding my preferred keybinding to open sessions and windows collapsed and Zoomed in. From the man page. https://waylonwalker.co

tmux prefix

https://youtu.be/BMkpbfhbkKM The prefix key is an essential part of tmux, by default all of tmux A few of the essential default key-bindings. A more complete list of key-bindings can be found in this gist https://gist.github.com/mzmonsour/8791835. ht

tmux splitting panes

https://youtu.be/kzgyiHap1nQ splitting panes is a core feature of tmux. It allows us to split the terminal 🗒️ note that https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post

tmux last session

https://youtu.be/RB87EEnnMnU An ultimate productivity key-binding in tmux is one to switch to the last session. I use this to quickly get between sessions really quick. Often I am working and need to lookup a quick note, or copy something into my n

tmux floating popups

https://youtu.be/2ZqFDsJywt8 Tmux popups are actually floating windows that you can drag around the screen. They always open in the middle (by default) when you open them, no matter where you leave them. Here are a couple of keybindings I use to ope

tmux popups

https://youtu.be/2I8fB28zfB4 Tmux-popups are a great feature that is relatively new to tmux, many repos such https://waylonwalker.com/tmux-nav-2021/ for more information on how I navigate tmux, check out this full post I use popups quite a bit in my

Incremental Versioned Datasets in Kedro

Kedro versioned datasets can be mixed with incremental and partitioned datasets https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. This was a question presented to me at work. We had some plots being produces To en

I Started Streaming on Twitch

I recently started streaming on python kedro Data Science Data Engineering webdev digital gardening It all started with My introduction to twitch came from After taking a jump in with the spaceflights tutorial I realized that it really I started stor

Kedro Spaceflights - part 2 | Stream replay June 7, 2021

This was my seconf time ever streaming on

🌱 Digital Gardening | gif to Mp4 | Stream replay June 4, 2021

https://youtu.be/I4VenHqIEng Doing some Digital Gardening on stream Ahrefs Errors ahrefs large images Automatic gif to mp4 After this stream all gifs on my site are converted to mp4/webm if they exist.

Kedro Spaceflights - part 1 | Stream replay June 4, 2021

This was my first time ever streaming on https://youtu.be/Y07UBr9Ccjs It all started with

Using Kedro In Scripts

With the latest releases of kedro https://waylonwalker.com/what-is-kedro/ If your just learning about kedro check out this post walking through it I To get kedro running, you will need a pipeline, catalog, and 👆 Above is the minimal setup to get a k

Silence Kedro Logs

Kedro can have a chatty logger. While this is super nice in production First, how does one silence a python log? Python loggers can be retrieved by | Level | Numeric value | Getting a python logger is straightforward if we know the name of the l

How I navigate tmux in 2021

In 2021 I changed the way I navigate between tmux sessions big time. Now I can Since making this post, I have made ~20 other posts in short form that all have a YouTube video to go along with them you can find them all on my I took Chris I am now ke

What is if __name__ == "__main___", and how do I use it.

When a python module is called it is assigned the Let I have set this module up to execute one of two if statements based on whether Note it is not common to have a Running a python script with the command This will print out https://waylonwalker

Trim unused git branches

Create a Virtual File Gallery with Symlinks

Creating a directory that is a union of several directories can be achieved Here is how I am creating a virtual directory of all my projects that is a ⚠ Notice that first I am recreating the directory each time. This will ensure Since links are alway

How to Install micromamba on linux (from the comamnd line only)

I really like using conda ( Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used i

Vim Wsl Clipboard

I wsl can access clip.exe. You can do some cool things with it, such as Let pipe streams of text into clip.exe I recently made mine feel a bit more natural by aliasing it to clip. pop this in your ~/.bashrc or ~/.zshrc I use neovim as my daily text

kedro replit

I am trying to see what an embeded replit

Pytest capsys

Testing print/log statements in pytest can be a bit tricky, capsys makes it capsys is a builtin pytest fixture that can be passed into any test to capture Simply create a test function that accepts capsys as an argument and pytest

Building Rich a Dev Server

I I am working on a project in which I want to have a dev automatic port number auto-restart display I am very novice at best when it comes to sockets, the following function came I am going super basic here and just running Optionally if you wante

Kedro pipeline_registry.py

With the realease of create a register_pipelines register hook_impl class You should now have something that looks like this in your pipeline I was not able to find any official documentation on how conflict resolution kedro automatically merges pi

Stand With Your Team

People who are quick to toss team members under a bus are not well trusted or While collaborating on any project there are going to be decisions made that During a review, leadership showed interest in the option that I wanted to go I walked out so

Blogging For Me

I create this blog with one person in mind, me. This is not completely selfish, as there are likely many others out there that I am as accurate as possible. I don Generally I know that I am directionally accurate for most of the article. In General

Minimal Kedro Pipeline

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. I have everything for this po

Markdown Cli

This is a post that may be a work in progress for awhile, Its a collections of posts tags draft posts frontmatter filepath content template html Markdown.Markdown support extentsions provide ways to hook in or clean up the frontmatter load render sav

My Content Strategy For 2021

I am making another push in 2021 to get my content out in the world and meeting users where they are. See how I plan to execute. waylonwalker.com Twitter DEV hashnode Medium LinkedIn Anchor My content is written in markdown, all markdown. I find th

Quickly Edit Posts

Recently I automated starting new posts with a python script. Today I want to https://waylonwalker.com/automating-my-post-starter Check out this post about setting up my posts with python 🐍 For the process of editing a post I just need to open the f

Gitui is a blazing fast terminal git interface

Gitui is a terminal-based git user interface (TUI) that will change the way Go to their It opens blazing fast. Sometimes I edit a number of files and want to commit them one at a time, this By default, Generally, I pop open w/s: to toggle focus

Kedro - My Data Is Not A Table

In python data science/engineering most of our data is in the form of some sort These containers for data contain many convenient methods to manipulate table https://waylonwalker.com/what-is-kedro/ unfamiliar with kedro, check out this post There are

Quickly Change Conda Env With Fzf

Changing conda environments is a bit verbose, I use a function with fzf that I have used conda as a virtual environment tool for years now. I started using What environment manager do you use for python? Conda environments are stored in a central lo

Minimal Python Package

What does it take to create an installable python package that can be hosted on pypi? setup.py my This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in o

Vim Replace Visual Star

Replacing text based on whats in the current search register is a quite handy https://www.youtube.com/watch?v=fP If there is one thing that I Like most about vim it Vim can often be a bit verbose, but that I have a keybinding in my In command mode

If Tmux

I do much of my work from tmux, I love it so much that I want to setup some Bash function to check if the shell is in a tmux session. I often open up vim to do some quite edits, but before I know it I have several Using I am not quite sure if this i

Save Vim Macro

If you are like me, you have created a macro or two that is pure glory, and you forget how you made it after a day or so, or you immediately want to store it away as a custom keybinding. As with most things with vim, it One of the earliest things we

Live Substitution In Neovim

Replacing text in vim can be quite frustrating especially since it doesn https://twitter.com/ I had to do a bit of searching and found a great post from I believe that this is a neovim only feature, add it into your Check out Josh Branchaud https://

Newsboat

Web browsers are a black hole of productivity. I try to use them as little as I am going to give a terminal rss reader a try for a bit and see how that goes I installed with the reccomended snap for Ubuntu. Running help for newsboat directed me towa

Large Refactor At The Command Line

As projects grow patterns that worked early on break and we need to change things to make the project easier to work with, and more welcoming to new developers. Before you start mucking up a project with wild commands at the terminal check that you h

Custom Ipython Prompt

I If you already have an ipython config you can move on otherwise check out this https://waylonwalker.com/ipython-config/ I want something similar to the starship prompt I am using in the python version active environment git branch This is my zsh pr

Ipython-Config

I use my ipython terminal daily. It Activate your virtual environment of choice and pip install it. Any time you You are using a virtual environment right? Virtual environments like venv or When you install ipython you start out with no config at a

Automating my Post Starter

One thing we all dread is mundane work of getting started, and all the hoops it When I start a new post I need to cd into my blog directory, start neovim in a frontmatter template Title slug tags date cover description create markdown file open in ne

Windowing Python Lists

In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools This post is a walkthrough of me solving a problem wi

Adding Audio to my blog posts

This is episode 1 of the Waylon Walker Audio experience, posts from So I have had this idea for awhile to add audio to my blog posts. The idea Honestly I don I will say that the time that I have available for clean audio is a bit limited One thing t

Expand One Line Links

I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML into the markdown. While this works, it The new card shou

gatsby-remark-embedder

Inspired by discourse https://twitter.com/kylemathews/status/1329817928666005504 This covers a couple of use cases I have with very little effort. Twitter YouTube This was super quick and simple to setup, the only thing that was extra was to Thats it

Find and Replace in the Terminal.

Resume Tips

customize for the job Why are you a good fit? What will you bring to the role? Give real outcomes give real experience Stop tech vomiting if you link to GitHub Make a profile readme Guide me to your best work have some activity if you link to LinkedI

Codeit Bro Interview

use this profile image Please share your professional role as a data scientist? I graduated with a Mechanical Engineering Degree 8 years ago. Much of my work Three years ago I made the switch to Data Science and have loved every minute of What are

reasons-to-kedro

There are many reasons that you should be using kedro. If you are on a team of https://waylonwalker.com/what-is-kedro Kedro makes it super easy to get started with their cli that utilizes https://waylonwalker.com/create-new-kedro-project read more a

Reasons to Kedro

collaboration Sharable catalog small nodes over monolithic notebooks catalog easily load anything without needing to run No need to write read/write code pipeline No need to keep execution order in your head easily run a slice of a pipeline plugins p

Reading List

https://waylonwalker.com/latest/ STOP LEAVING Browser Tabs open and save them here Jungle Scout - Julia Evans - git course - https://ianthehenry.com/posts/how-to-learn-nix/python3-alpha/ https://twitter.com/willmcgugan/status/1531294412696956930 h

What's New in Kedro 0.16.6

Kedro This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Most of them hinge on a sweet combination of the ke

A brain dump of stories

I started making stories as kind of a brain dump a few times per day and I store all the stories on my website with the hopes of doing something with

Fix git commit author

I was 20 commits into a hackoberfest PR when I suddenly realized they they all had my work email on them instead of my personal email 😱. This is the story of how I corrected my email address on 19 individual commits after already submitting for a PR

Designing a "Router" for kedro

I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called One thing

Reclaim memory usage in Jupyter

Today I ran into an issue where we had a one-off script that just needed to It started with a colleague asking me How do I clear the memory in a Jupyter How do I clear the memory in a Jupyter notebook? There are a number of ways that you can check th

Strip Trailing Whitespace from Git projects

A common linting error thrown by various linters is for trailing whitespace. I read more about pre-commit

Chrome Extensions I use

There are many useful chrome extensions out there. I probably have way too many installed, here are four that I am currently using. This post was inspired from Chris over at Love it or hate it passwords are hard to manage. Everyone needs a passwor

Creating Reusable Bash Scripts

Bash is a language that is quite useful for automation no matter what language I have three techniques that help me write more composable bash scripts. Functions in bash are quite simple. They are something that I wish I would have Source the file to

Three things to Automate with Python using Pandas

Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an altern

How to Install miniconda on linux (from the command line only)

miniconda is a python distribution from continuum. It Installing miniconda on Linux can be a bit tricky the first time you do it completely from the terminal. The following snippet will create a directory to install miniconda into, download the lates

Kedro Basics

Learn Kedro in 5 days vm install python editor kedro new kedro viz catalog filter catalog load data fsspec pipeline nodes filter pipeline run partial pipeline kedro docker GitHub Actions hooks custom datasets modular pipelines

How to crush amazing posts on DEV

This post was inspired by a comment I left on @dsteenman {% post dsteenman/how-long-should-a-blogpost-be-2k6n %} Most of the time I prefer short as I am more likely to read the whole thing. If its setup as a series I am more likely to work my way th

Black Tech Pipeline

I was particularly inspired by @chantastic episode 103 of the react podcast with @ParissAthena. They spoke about the black tech pipeline as well as Diversity, Equity, and Inclusion. Pariss is quite an inspiration. She has done so much work to create

Review of the git-auto-commit-action

It Check out the repo for It This has been by far the easiest way to commit back to a repo that I have seen. Other patterns often require fully setting up the git user and everything. While it You must give it a commit message and thats it. Optio

What's New in Kedro 0.16.4

If we take a look at the This one comes a bit surprising as it was just casually mentioned in As mentioned in It feels a bit crazy that simply installing a package will change the way that your pipeline gets executed. I do like that it requires ju

Integration testing with Python, TestProject.io, and GitHub Actions

As I continue to build out Running integration tests will not prevent bugs from happening completely, but it will allow me to quickly spot them and rollback. The very first thing that comes to my mind is anything that is loaded or ran client-side.

🐍 Practice Python Online

When learning a new skill it When learning a new skill it Here are three different options that I have used in the past to try out something at some various levels. I am sure there are plenty more, but these are three that I have tried. I am not cove

New Machine for developing Tests with TestProject.io

Today I setup a new machine on Digital Ocean to use with TestProject.io, Here are my installation notes. envsubst

Kedro Catalog

I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. metadata will be attached to the dataset object under a metadata will be updated metadata will be empty until a pipeline is ran with the hook on optionall

How python tools configure

Mypy only uses pyproject.toml only uses pyproject.toml

Edit On GitHub

I recently added a button to my blog, and subsequently my posts on I refer back to my old posts quite a bit, sometimes I find errors in them. The slug that I am getting from gatsby is formatted as GitHub makes it super easy to form a URL that puts y

Why use a cms

When first learning to code its very common to hard code everything right into Hard coding everything right into your code makes it really hard for The cms generally come with a rich content editor that feels more like Even when you are developing fo

🐍 Parsing RSS feeds with Python

I am looking into a way to replace my google reader experience that I had back This is how I used python to parse rss and setup my own custom feed. Install the feedparser library. The feed is a feedparser.FeedParserDict. For all intents and purposes

Reader

The input will be a yaml file containing a list of rss feed (primary source) youtube feed Stack Overflow tags GitHub repo activity pypi release dev.to post Twitter Search # user will need an api key Each url will be pulled in and parsed into a stand

🤓 What's on your GitHub Profile

I ran this post on dev.to and got a great response of great examples, The GitHub profile feature just went live for a subset of users. Simply creating a repo named after your username, and clicking share to Profile on the sidebar will create a cust

🙋‍♂️ Can Anyone Explain Twitter Cards to me?

Can someone explain how or why twitter cards render differently from device to device? I do understand that twitter cards a built from meta tags, the full list can be found in their Mobile Looks fine. On Desktop it is not picking up the image. The

How I Built My GitHub Profile

I ran a discussion on dev that collected quite a list of examples in the comment section. So many great calls to action, animations, memes, and weird tricks. Upload all of your icons to the repo in a directory such as Aligning things in the center

SLIDES - understanding python \*args and \*\*kwargs

Python I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog See the full article

Gracefully adopt kedro, the catalog

While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont e

How to find things in your kedro catalog

kedro 0.16.2 just dropped last week with a long-awaited feature... The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections

My first eight years as a working professional.

This day 8 years ago I started my first day as a Mechanical Engineer. I am so My initial career plans were down a completely different path. I have been very flexible in taking on a new career path. I have been eager to learn new things and respond t

How Kedro handles your inputs

Passing inputs into kedro is a key concept. Understanding how it accepts a Check out this post for a review of how https://waylonwalker.com/python-args-kwargs/ When kedro runs your pipeline it uses the catalog to imperatively load your data, meanin

Master No More

It https://waylonwalker.com/refactor-in-cli/ this post follows my method of refactoring code bases from the command line, First off browsing through the content of my blog I found many references to master. I cannot completely whole-sale find and re

Refactoring your blog urls

I just did a quick refactoring of my JAMStack blog urls. Some didn https://waylonwalker.com/refactor-in-cli When refactorings similar to this get really big I often need to do some My post urls are based on the file name of my markdown file, so I ca

understanding python \*args and \*\*kwargs

Python Python * When recieving variables as a Never add Generally I find If your Here Here the function signature makes it clear what Inversely we can send a list of things as individual arguments by Just like Here is a function accepting

pre-commit is awesome

I recently discovered the ✨ awesomeness that is pre-commit. I steered away I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it It comes wi

Building kedro.dev

Follow along the Journey as I build out I have really enjoyed my own personal journey as I have started to build all of my data pipeline projects with the kedro framework. I want to start building a place to share resources with the community. I w

Kedro Static Viz 0.3.0 is out with Hooks Support

Even though it is a static site there is no functionality lost. The only thing that Even though it Kedro viz is a fantastic kedro plugin that allows you to visualize your data pipeline. Kedro allows you to quickly build production-ready pipelines w

Create Configurable Kedro Hooks

There are two main ways to create kedro hooks, with modules and classes. Each Class hooks seem a bit special as they give you a way to configure them so that they are a bit more generally useful. https://waylonwalker.com/what-is-kedro If you are com

Brainstorming Kedro Hooks

This post is a 🧠 branstorming work in progress. I will likely use it as a If you are completely unsure what kedro is be sure to check out my filepath replacer bucket replacer preflight check that data exists run run mypy run interrogate run flake8

Gatsby Scripts with onload

This might be useful

How to get Dev Comments from an article Url

I want to incorporate some of the wonderful comments, I want to incorporate some of the wonderful comments, 💕, 🦄, and 🔖 dev.to has an open API that allows us to easily get comments as HTML. They have their API hosted at Here we can see that going

Four github actions for your website

GitHub GitHub actions can give you confidence that your site is up and running, with the latest JavaScript packages, does not have broken links, and can even take screenshots of what your website looks like on different screen sizes and operating sys

Adding google fonts to a gatsbyjs site

Create Custom Kedro Dataset

Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When run

Interrogate is a pretty awesome, brand new, cli for Python packages

As usual while listening to This thing is 🔥 hot off the press folks, we Nothing I have tried has come close to being this good It runs documentation coverage for your python project. It allows you to set the minimum amount of docstring coverage for

drawing ascii boxes

When creating cli I rarely get very complex with my semi-manual ASCII art. I can do 98% of what I need with bars and corners. Using some simple full-width characters can really give your cli a nice clean look. I As I was putting this together I stu

creating the kedro-preflight hook

Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro kedro hooks are an exciting upcoming feature of kedro https://waylonwalker.com/what-is-kedro/ If you are completely unsure what kedro is be sure to check out my what is kedro

📝 Kedro Preflight Notes

This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources. check that inputs exist or are of a type to skip (sql) check that all input and o

Maintianing multiple git remotes

📢 Announcing find-kedro

find-kedro kedro When working on larger pipeline projects, it is advisable to break your project down into different sub-modules which requires knowledge of building python libraries, and knowing how to import each module correctly. While this is no

Explicit vs Implicit Returns in Javascript

Often when reading through javascript examples you will find some arrow Arrow functions are one-liner functions in javascript that have two main syntactical ways to create the code block. with parentheses and braces. Let Here is an example of an ar

Twitter deepdives

Inspired by Chris Achard List comps Classes Inheritance Background Click Lambdas Cataloging Custom datasets Reusable pipelines find-kedro Email course inspired by learn d3 in 5 days Share your knowledge Practice Practice in public Make practice easy

TIL: Bind arguments to dynamically generated lambdas in python

This past week I had a really weird bug in my After a few days of looking at it off and on, I pinpointed that it was all the way down in the raw layer. Right as data is coming off of the database. For this I already had existing I have dynamically

python-deepwatch

Is it possible to deep watch a single python function for changes? keeping track of a python functions hash is quite simple. There is a Using hashlib provides a consistent hash. Now we have a consistent way to hash function code. setup a function in

Four Github Actions for Python

If you are developing python packages and using GitHub here are four actions If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have suc

Variables names don't need their type

So often I see a variables Pandas Sometimes vanilla structures too It Always name your containers plural, so that naming while iterating is simple. Before I start fights 🥊 in code review, am I inline here or just being pedantic?

Send Emails with GitHub Actions

Here is one useful thing that you can do with GitHub actions no matter what language you use, send email. You might want to know right away when your ci passes. You might want to give your team a nice pat on the back when a new release is deployed.

What Are GitHub Actions

GitHub actions are an amazing tool that allows us to run code based on triggers I have been diving deep into Github actions for about a month now and they are wicked good some interaction to your repository The online editor for actions is pretty am

Getting Started with GitHub Actions

Github actions are written in configuration files using the YAML syntax. YAML Can I just say that I learned more than I realized about YAML by writing this In YAML or JSON, the most basic containers for data are The start of an array container is s

Today I learned `git diff feature..main`

Today I learned how to diff between two branches. Sometimes we get a little Let After all of that, we have a git repository on our local machine with a single file Let At this point we have one commit. Things are really straightforward, and our di

Create New Kedro Project

This is a quickstart to getting a new 🧹 Install 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization I use conda to control my virtual environments and will create a new environment called Options I try to keep my base

What is YOUR Advice for New Data Scientists

Learn the business Learn Git Your code does not need to be amazing Keep Learning You dont have to start out as a git wizard with the cleanest possible commit history. At first dont let yourself get too wrapped up in it, the most important part is th

Do You Hoist

I am working through Wes Bos Do you have any use cases that you use hoising? Why? It seems like a really cool feature in any language that uses it, but I dont really notice it in use. There are many articles that cover this in far more depth, but i

What is Kedro

https://waylonwalker.com/what-is-kedro This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set Currently, 💎 📉 🏗 Inside this core library is a data catalo

Custom Scrollbar Design

Getting a custom scrollbar on your site makes it stand out a bit compared to Inspired by Wes Bos Since My first step was to take his css and copy it into a styled component for my entire layout, but it failed. I do not fully understand why. None o

Don’t waste your time learning everything

" Inspired by this I felt that this comment was very powerful. Here are my 2 cents. Stick to what you know, and learn a little bit of something new every day. If what you know is how to use Excel like a boss, don Take small steps enhance what you kn

2020 waylonwalker.com rebrand

Moving into 2020 I have been really leaning on using purple as my theme color ![ This is what we are working with. It has been my card design for at least a plain cover art does not fit I am not digging the date on cover art that also has text I hav

I just added react-headroom to my site

It was so easy to get a professional looking navbar with just 3 lines of code. installation is easy There was no instructions for es6 style imports that are common with gatsbyjs Simply wrap your existing content, Nav in my case, with the I think thi

Serverless things to investigate

Free only one step is free can take rss to social Free to get started Free 7 day trial $4.99 cheapest after

I finally fixed my Styled-Components in gatsby.js

I finally fixed my Styled-Components in gatsby.js. I am starting a redesign of I have a small issue that has plagued the site for at least a year I have been using styled components in my gatsby.js site for about a year now. And it has been plagued

Should I switch to Zeit Now

I have happily had my personal site I have recently started playing with zeit again. I really like their cli tool, its dead simple and makes sense. I tried the netlify one early on and dont think I really gave it much of a chance. I was able to ba

git push without setting upstream

Finally after years of hand typing out a full This one setting will now

Out of Space

This morning I logged into my machine and was nearly out of space 64GB miniconda3 5GB conda cache 4GM pip cache 34GB docker Photo by These are the commands that I often use to reclaim space. Its so easy to fill up small vm This shows us where to st

Personal URL shortener with Netlify Redirects

I love using URL shorteners to easily share links without hitting character I recently discovered a really cool feature of netlify that I have always looked past, simply add a Now with shorter links we have more space for our content without needin

simple click

cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Things a cli can do to enhance your library. 🆚 print versio

cmd.exe tips

I spend a lot of my time at the terminal for my daily work, mostly in Linux or wsl. One big reason for using wsl over cmd.exe is the ease of walking through history that fzf provides. This week we had a windows bug in a cli and I was stuck in vanil

What is something you should have learned or understood earlier?

Mine is the python debugger. I was a long holdout thinking that print statements were sufficient. That was untill I started having errors crop up in functions that took minutes to run. The thing that I most notably wish I would have known about is po

Supercharge Zsh Startup

I have been using oh-my-zsh successfully for about 2 years now. But lately my I believe the real issue is io speed on wsl. I have some remote servers with Quick side note: your zsh config is controled by your Time the Move your Time the fastest s

Keep Location List Closed

Vim Location List eating up the screen while I am zoomed in and trying to live code Through some google search I found the culprit was syntastic. It has an I want to keep the location list open automatically most of the time, but when I don First,

SqlAlchemy Models

Building Cli apps in Python

Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. typically aliased with a shorthand ( ** To get the Python argument name, the chosen name is conver

Kedro

I am tweeting out most of these snippets as I add them, you can find them all here Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro

📝 Packages to Investigate Notes

jmespath Tabnine |-|-| I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+.

Debugging Python

Just Use Pathlib

Pathlib is an amazing cross-platform path tool.

Custom Python Exceptions

Filtering Pandas

Good for method chaining, i.e. adding more methods or filters without assigning a new variable. general purpose, this is probably the most common method you see in training/examples capable of including multiple strings to include Good For partial ma

Digital Ocean

I love digital ocean for it

Quick Progress Bars in python using TQDM

tqdm is one of my favorite general purpose utility libraries in python. It for more gifs like these follow me on twitter TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically. There i

Clean up Your Data Science with Named Tuples

If you are a regular listener of As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis

Background Tasks in Python for Data Science

This post is intended as an extension/update from This post is intended as an extension/update from I use it in more places than I probably should Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth

Autoreload in Ipython

I have used 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code

📝 Bash Notes

Bash is super powerful. * * There are so many options inside of https://waylonwalker.com/refactor-in-cli/ I use these replace commands heavily when doing large refactorings. I like this one when there is not a good cli into config files and I need t

Keyboard Driven VSCode

Throw that mouse Away its time to setup some keyboard shortcuts. These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on the hackabili

Realistic Git Workflow

My git workflow based on real life. Its not always clean and simple. As complicated as that seems it is pretty straight forward. When you sit down to work the first thing you do is to Next create a new branch with a name that will remind you of w

Forestry.io

Testing out forestry.io I have been playing with the netlify cms for a while now, and it has been a decent experience, but I Forestry.io has this amazing feature to create By far my favorite feature is I have a use case for a photography site wher

Rewrite History with Git

rebase git commit --amend rage quit git reset HEAD~n removes modifications keeps hitsory of changes and undoes them git checkout HEAD~n -- keeps modifications removes history --SOFT --HARD --Mixed locally before push after push after push

Rewrite History with Git

rebase git commit --amend rage quit git reset HEAD~n removes modifications keeps hitsory of changes and undoes them git checkout HEAD~n -- keeps modifications removes history --SOFT --HARD --Mixed locally before push after push after push

It's not all about winning

I am addicted to the process of learning and improving my skills nearly to a fault. The reason I say nearly is because my addiction is fueled with results. I crave the output of my work enhance the work of others. I jump with joy as I see users gain

Update Git User

This morning I log into my VCS and check activity on my projects to find that Clone the repo, note it must be a Curl down the Run the script, and push the updates. Delete the I hope this helps someone, or future me who needs to fix their informat

2019 goals

This is a follow up to my previous post This year I will become more productive, by intensionally working on a well thought out plan, learning the right technologies, and leave behind a positive legacy. Last year I was able to make some great stride

2018 Retrospective

2018 was a year of many ups and downs, and learning to deal with a whole new At home the doctor appointments keep piling in. While I am learning to deal Here are the goals I had listed out for 2018 and how I faired at completing Continue Take 2 10m

Do More of What Brings You Joy

Today I want to take some time to talk about the things that make me happy in my work environment. This is completely free-flow off the cuff, but are things that I do that make me happy, not having them would definitely be a deal breaker for me . A

D3 Day 5

For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere

D3 Day 4

For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere Today I will be learning about d3 scales, and adding them to the bar chart that we created yesterday. Follow along as I try to create somethin

D3 Day 3

I recently subscribed to Ben Clinkinbeard D3 is the ubiquitous dynamic visualization library for building custom interactive visualizations on the web. It is a bit low level, and more verbose than many other libraries that build upon it, but if you

FlexBox

I recently finished up the flexbox-zombies course to learn more about flexbox, and to become proficient with it. I can truly say that this course has changed the way that I create layouts. Flexbox is very intuitive now. What this course does reall