Just starred machfiles [1] by ChristianChiarulli [2]. It’s an exciting project with a lot to offer.
The dotfiles you see in all my videos
References:
[1]: https://github.com/ChristianChiarulli/machfiles
[2]: https://github.com/ChristianChiarulli
Publishing rhythm
Check out LunarVim [1] and their project LunarVim [2].
🌙 LunarVim is an IDE layer for Neovim. Completely free and community driven.
References:
[1]: https://github.com/LunarVim
[2]: https://github.com/LunarVim/LunarVim
Looking for inspiration? joelhooks-com [1] by joelhooks [2].
playing with static pages
References:
[1]: https://github.com/joelhooks/joelhooks-com
[2]: https://github.com/joelhooks
Automating my Post Starter
One thing we all dread is mundane work of getting started, and all the hoops it
takes to get going. This year I want to post more often and I am taking some
steps towards making it easier for myself to just get started.
When I start a new post I need to cd into my blog directory, start neovim in a
markdown file with a clever name, copy some frontmatter boilerplate, update the
post date, add tags, a description, and a cover.
Todo List for starting a post # [1]
- frontmatter template
- Title
- slug
- tags
- date
- cover
- description
- create markdown file
- open in neovim
Lets Automate this # [2]
This aint no proper cli # [3]
hot and fast
As with many thing running behind the scenes on this site, I am the one and
only user, I have limited time, so this is going to be a bit hot and fast.
Let’s create a file called new-post.
start the script new-post
#!python
# new-post
👆 Works on my machine
If this were something that had more users than me I would probably use
some...
Windowing Python Lists
In python data science we often will reach for pandas a bit more than
necessary. While pandas can save us so much there are times where there are
alternatives that are much simpler. The itertoolsandmore-itertools` are
full of cases of this.
This post is a walkthrough of me solving a problem with more-itertools rather
than reaching for a for loop, or pandas.
I am working on a one-line-link expander for my blog. I ended up doing it,
just by modifying the markdown with python. I first split the post into lines
with content.split('\n'), then look to see if the line appears to be just a
link. One more safety net that I wanted to add was to check if there was
whitespace around the line, this could not simply be done in a list
comprehension by itself. I need just a bit of knowledge of the surrounding
lines, enter more-itertools.
simplified rendering function # [1]
I have a function that will check to see if the line should be expanded, then
render the correct template. Fist step is to ...
WaylonWalker [1] has done a fantastic job with devtainer [2]. Highly recommend taking a look.
🐳 (dotfiles) My personal development docker container base image
References:
[1]: https://github.com/WaylonWalker
[2]: https://github.com/WaylonWalker/devtainer
WaylonWalker [1] has done a fantastic job with WaylonWalker [2]. Highly recommend taking a look.
Learning in public
References:
[1]: https://github.com/WaylonWalker
[2]: https://github.com/WaylonWalker/WaylonWalker
Check out aoc [1] by ThePrimeagen [2]. It’s a well-crafted project with great potential.
2020
References:
[1]: https://github.com/ThePrimeagen/aoc
[2]: https://github.com/ThePrimeagen
Check out ZaxR [1] and their project bulwark [2].
Bulwark is a package for convenient property-based testing of pandas dataframes.
References:
[1]: https://github.com/ZaxR
[2]: https://github.com/ZaxR/bulwark
mariokostelac [1] has done a fantastic job with sagemaker-setup [2]. Highly recommend taking a look.
Useful scripts for making AWS SageMaker better
References:
[1]: https://github.com/mariokostelac
[2]: https://github.com/mariokostelac/sagemaker-setup
I like pypeaday’s [1] project aoc-2020 [2].
Advent of Code 2020
References:
[1]: https://github.com/pypeaday
[2]: https://github.com/pypeaday/aoc-2020
I’m really excited about auto-editor [1], an amazing project by WyattBlue [2]. It’s worth exploring!
Auto-Editor: Efficient media analysis and rendering
References:
[1]: https://github.com/WyattBlue/auto-editor
[2]: https://github.com/WyattBlue
Adding Audio to my blog posts
This is episode 1 of the Waylon Walker Audio experience, posts from
waylonwalker.com [1]{.hoverlink} in audio form.
So I have had this idea for awhile to add audio to my blog posts. The idea
partly comes from the aws blog, if you have ever been on their blog you will
have noticed that they have a voiced by amazon polly section.
What to Expect # [2]
Honestly I don’t know this is all new to me and I dont have much to go off of.
For now its a test that may or may not work out.
I will say that the time that I have available for clean audio is a bit limited
so expect these to come out in batches as I get time to go back and record.
What Not to Expect # [3]
One thing that makes the aws blog really hard to listen to is the robotic
voice, I definitely don’t want that. This will be voiced by a real human, Me.
At the same time written text doesn’t translate directly to audio well so don’t
necessarily expect the audio to be
word for word.
Code blocks # [4]
There are a lot of code block...
Check out yetudada [1] and their project yetudada [2].
No description available.
References:
[1]: https://github.com/yetudada
[2]: https://github.com/yetudada/yetudada
I’m impressed by quickpython [1] from timothycrosley [2].
A retro interactive coding environment powered by Python and nostalgia
References:
[1]: https://github.com/timothycrosley/quickpython
[2]: https://github.com/timothycrosley
gatsby-remark-embedder
Inspired by discourse’s link expansion I am rolling out expansions for one line
links on the blog waylonwalker [1]. I was able to find
a gatsby plugin
gatsby-remark-embedder [2]
that expands one line links for social cards for popular platforms like twitter
and YouTube through a repose from Kyle Mathews to my tweet.
https://twitter.com/kylemathews/status/1329817928666005504
Use Cases # [3]
This covers a couple of use cases I have with very little effort.
- Twitter
- YouTube
install # [4]
npm i gatsby-remark-embedder gatsby-plugin-twitter
This was super quick and simple to setup, the only thing that was extra was to
install the gatsby-plugin-twitter plugin as well as the
gatsby-remark-embedder.
enable # [5]
// In your gatsby-config.js
module.exports = {
// Find the 'plugins' array
plugins: [
`gatsby-plugin-twitter`,
{
resolve: `gatsby-transformer-remark`,
options: {
plugins: [
{
resolve: `gatsby-remark-embedder`,
options: {
customTransformers: [
// Your custom t...
Expand One Line Links
I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML [1] into the markdown. While this works, it’s more manual/difficult for me does not look the best, and does not read well as
Goals for new card # [2]
The new card should be fully automated to expand with title, description, and
cover image. Bonus if I am able to attach a comment behind it.
- fully automated
- card expansion
- Title
- description
- cover image
Old Card # [3]
If you can call it a card 🤣. This card was just an image wrapped in an anchor tag and a paragraph tag. I found this was the most consistent way to get an image narrower and centered in both GitHub and dev.to.
<p style='text-align: center'>
<a href='https://waylonwalker.com/notes/eight-years-cat/'>
<img
style='width:500px; max-width:80%; margin: auto;'
src="https://dropper.wayl.one/file/99f80283-2f27-4664-8c4a-d1...
astronomer [1] has done a fantastic job with dag-factory [2]. Highly recommend taking a look.
Dynamically generate Apache Airflow DAGs from YAML configuration files
References:
[1]: https://github.com/astronomer
[2]: https://github.com/astronomer/dag-factory
orchest [1] by orchest [2] is a game-changer in its space. Excited to see how it evolves.
Build data pipelines, the easy way 🛠️
References:
[1]: https://github.com/orchest/orchest
[2]: https://github.com/orchest
gvanrossum [1] has done a fantastic job with patma [2]. Highly recommend taking a look.
Pattern Matching
References:
[1]: https://github.com/gvanrossum
[2]: https://github.com/gvanrossum/patma
mingrammer [1] has done a fantastic job with diagrams [2]. Highly recommend taking a look.
🎨 Diagram as Code for prototyping cloud system architectures
References:
[1]: https://github.com/mingrammer
[2]: https://github.com/mingrammer/diagrams
Just starred svelte-actions [1] by swyxio [2]. It’s an exciting project with a lot to offer.
prototype official actions for Svelte
References:
[1]: https://github.com/swyxio/svelte-actions
[2]: https://github.com/swyxio
reasons-to-kedro
There are many reasons that you should be using kedro. If you are on a team of
Data Scientists/Data Engineers processing DataFrames from many data sources
should be considering a pipeline framework. Kedro is a great option that
provides many benefits for teams to collaborate, develop, and deploy data
pipelines
What is Kedro [1]
Starter Template # [2]
Kedro makes it super easy to get started with their cli that utilizes
cookiecutter under the hood.
conda create -n my-new-project -y python=3.8
kedro new
kedro install
kedro run
Create New Kedro Project [3]
read more about how to start your first kedro project here
Collaboration # [4]
Kedro provides many tools that help teams collaborate on a single codebase.
While writing monolithic scripts it can be easy to pin yourself in a corner
where it is difficult to have multiple people making changes to the
notebook/script at the same time. Kedro helps guide your team to break your
project down into small pieces that different members o...
Just starred Second-Brain [1] by KasperZutterman [2]. It’s an exciting project with a lot to offer.
A curated list of awesome Public Zettelkastens 🗄️ / Second Brains 🧠 / Digital Gardens 🌱
References:
[1]: https://github.com/KasperZutterman/Second-Brain
[2]: https://github.com/KasperZutterman
Just starred Repo-Roster [1] by nastyox [2]. It’s an exciting project with a lot to offer.
Shout-out supporters in your GitHub README file.
References:
[1]: https://github.com/nastyox/Repo-Roster
[2]: https://github.com/nastyox
What's New in Kedro 0.16.6
Kedro 0.16.6 [1] is out! Let’s take a look through the release notes
Deployment Docs # [2]
This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible.
- Argo [3]
- Prefect [4]
- Kubeflow [5]
- Batch [6]
- SageMaker [7]
Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies.
Argo, Prefect, and Kubeflow have an interesting technique where they translate the pipeline and its dependencies from kedro to their language.
Batch uses the aws cli to submit jobs, one node per job, and listen for them to complete. It will submit all nodes with completed dependencies at once, meaning that we can get some massive parallelization.
I did a quick and dirty test of one of these by simulating the technique in a bash script and saw a 40 hr pipeline finish in about 1 hour. I am excited to get thi...
mkdocs [1] by mkdocs [2] is a game-changer in its space. Excited to see how it evolves.
Project documentation with Markdown.
References:
[1]: https://github.com/mkdocs/mkdocs
[2]: https://github.com/mkdocs
A brain dump of stories
I started making stories as kind of a brain dump a few times per day and
posting them to
[LinkedIn](https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/).
Here are the last 11 days of stories.
I store all the stories on my website with the hopes of doing something with
them on my own platform eventually. For now it makes it easy to make these
posts.
cd static/stories
ls | xargs -I {} echo ''
Stories 10-10-2020 - 10-21-2020 # [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
References:
[1]: #stories-10-10-2020---10-21-2020
[2]: https://waylonwalker.com/stories/TIL-kedro-sorts-nodes.png
[3]: https://waylonwalker.com/stories/disable-base-pip.png
[4]: https://waylonwalker.com/stories/discovered-social-cards.png
[5]: https://waylonwalker.com/stories/find-kedro-de1-contributor.png
[6]: https://waylonwalker.com/stories/hacktoberfest-2020-kedro-538-tests-pass.png
[7]: https://waylonwalk...
Check out mmchougule [1] and their project kedro-grpc-server [2].
Kedro gRPC Server is a Kedro plugin that creates a gRPC server for triggering and monitoring pipeline runs using a general-purpose RPC framework gRPC
References:
[1]: https://github.com/mmchougule
[2]: https://github.com/mmchougule/kedro-grpc-server
Check out yetudada [1] and their project kedro-user-testing [2].
Discovery prototypes for user testing
References:
[1]: https://github.com/yetudada
[2]: https://github.com/yetudada/kedro-user-testing
The work on flynt [1] by ikamensh [2].
A tool to automatically convert old string literal formatting to f-strings
References:
[1]: https://github.com/ikamensh/flynt
[2]: https://github.com/ikamensh
charmbracelet [1] has done a fantastic job with glow [2]. Highly recommend taking a look.
Render markdown on the CLI, with pizzazz! 💅🏻
References:
[1]: https://github.com/charmbracelet
[2]: https://github.com/charmbracelet/glow
Check out mytechnotalent [1] and their project Python-For-Kids [2].
A FREE comprehensive online Python development tutorial FOR KIDS utilizing an official BBC micro:bit Development Board going step-by-step into the world of Python for microcontrollers.
References:
[1]: https://github.com/mytechnotalent
[2]: https://github.com/mytechnotalent/Python-For-Kids
I’m impressed by pycon_pybadge_2020 [1] from nnja [2].
Initial code for Microsoft’s PyBadge at PyCon 2020
References:
[1]: https://github.com/nnja/pycon_pybadge_2020
[2]: https://github.com/nnja
Fix git commit author
I was 20 commits into a hackoberfest PR when I suddenly realized they they all had my work email on them instead of my personal email 😱. This is the story of how I corrected my email address on 19 individual commits after already submitting for a PR.
- Change the email for this repo [1]
- Prepare for rebasing [2]
- start the rebase [3]
- 🛠 Fix First wrong Commit [4]
- Fix all commits [5]
- Done [6]
- ReCap [7]
Change the email for this repo # [1]
stop the bleeding
Before anything else set the email correctly!
cd kedro
git config user.name "Waylon Walker"
git config user.email [email protected]
Prepare for rebasing # [2]
First thing is to find how many commits back this mistake goes. I opened up the git [8] log, and saw mine went back 19 commits. I rolled back 20 just to be sure.
$ git log
...
commit a355926b9d7ec4c05659adaa254beefbdb036332
Author: WaylonWalker <[email protected]>
Date: Sat Oct 17 10:28:59 2020 -0500
give name of function inside incorrect parameters erro...
I like muesli’s [1] project duf [2].
Disk Usage/Free Utility - a better ‘df’ alternative
References:
[1]: https://github.com/muesli
[2]: https://github.com/muesli/duf
Designing a "Router" for kedro
nodes_global # [1]
I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called nodes_global. It worked fairly well but did have some issues around being set as a global variable.
But…
One thing in particular that it did not lend itself well to was being able to create a packagable pipeline that I could pip install and append into any of my existing pipelines. Something I am still trying to work out, maybe I don’t need this. I think I have it working for our internal pipelines and it seems like the way to go, but we don’t necessarily end up using it.
Also…
With this pattern all of the nodes needed to be importable by the module containing nodes_global. I find that this becomes a big hurdle for new pipelines coming from jupyter to overcome and can be most infuriating when their nodes aren’t getting ran af...
I came across python_training [1] from AnkurDedania [2], and it’s packed with great features and ideas.
Intro to Python
References:
[1]: https://github.com/AnkurDedania/python_training
[2]: https://github.com/AnkurDedania
github [1] has done a fantastic job with renaming [2]. Highly recommend taking a look.
Guidance for changing the default branch name for GitHub repositories
References:
[1]: https://github.com/github
[2]: https://github.com/github/renaming
Reclaim memory usage in Jupyter
Today I ran into an issue where we had a one-off script that just needed to
work, but it was just chewing threw memory like nothing.
It started with a colleague asking me How do I clear the memory in a Jupyter
notebook, these are the steps we took to debug the issue and free up some
memory in their notebook.
How do I clear the memory in a Jupyter notebook?
Pre check the status of memory # [1]
There are a number of ways that you can check the amount of memory on your
system. The easiest is not necessarily my first go to is free… literally
free.
check for free space
$ free -h
total used free shared buffers cached
Mem: 15G 15G 150M 0B 59M 8.7G
Generally my first go to is a bit more graphical, and not available on a stock
stystem, but far more useful…. htop. htop [2] is a
terminal process explorer that shows cpu usage, mem usage, and running
processes.
htop
sudo apt-get install htop # install it from your package repo
htop
[3]
First step throw more swap at it # [4]
Often be...
Strip Trailing Whitespace from Git projects
A common linting error thrown by various linters is for trailing whitespace. I
most often use flake8. I generally have
[pre-commit](https://waylonwalker.com/pre-commit-is-awesome
hooks setup to strip this,
but sometimes I run into situations where I jump into a project without it, and
my editor lights up with errors. A simple fix is to run this one-liner.
One-Liner to strip whitespace # [1]
bash
git grep -I --name-only -z -e '' | xargs -0 sed -i -e 's/[ \t]\+\(\r\?\)$/\1/'
pre-commit is awesome
I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but...
Jun 5, 2020 [2]
References:
[1]: #one-liner-to-strip-whitespace
[2]: /pre-commit-is-awesome/
tpope [1] has done a fantastic job with vim-sleuth [2]. Highly recommend taking a look.
sleuth.vim: Heuristically set buffer options
References:
[1]: https://github.com/tpope
[2]: https://github.com/tpope/vim-sleuth
actions [1] has done a fantastic job with setup-python [2]. Highly recommend taking a look.
Set up your GitHub Actions workflow with a specific version of Python
References:
[1]: https://github.com/actions
[2]: https://github.com/actions/setup-python
I came across starter-workflows [1] from actions [2], and it’s packed with great features and ideas.
Accelerating new GitHub Actions workflows
References:
[1]: https://github.com/actions/starter-workflows
[2]: https://github.com/actions
checkout [1] by actions [2] is a game-changer in its space. Excited to see how it evolves.
Action for checking out a repo
References:
[1]: https://github.com/actions/checkout
[2]: https://github.com/actions
Looking for inspiration? dotfiles [1] by nicknisi [2].
vim, zsh, git [3], homebrew, neovim - my whole world
References:
[1]: https://github.com/nicknisi/dotfiles
[2]: https://github.com/nicknisi
[3]: /glossary/git/
Just starred zk [1] by sirupsen [2]. It’s an exciting project with a lot to offer.
Zettelkasten on the command-line 📚 🔍
References:
[1]: https://github.com/sirupsen/zk
[2]: https://github.com/sirupsen
The work on napkin-math [1] by sirupsen [2].
Techniques and numbers for estimating system’s performance from first-principles
References:
[1]: https://github.com/sirupsen/napkin-math
[2]: https://github.com/sirupsen
deepyaman [1] has done a fantastic job with kedro-accelerator [2]. Highly recommend taking a look.
Kedro-Accelerator speeds up pipelines by parallelizing I/O in the background.
References:
[1]: https://github.com/deepyaman
[2]: https://github.com/deepyaman/kedro-accelerator
Chrome Extensions I use
There are many useful chrome extensions out there. I probably have way too many installed, here are four that I am currently using.
This post was inspired from Chris over at daily-dev-tips [1]
- LastPass [2]
- Stylus [3]
- Vimium [4]
- hypothesis [5]
---
LastPass [6] # [7]
Love it or hate it passwords are hard to manage. Everyone needs a password manager to avoid the dreaded password reuse, and to be able to quickly rotate them with a service. I use lastpass, thus it’s browser extension is my most used extension.
[6]
---
Stylus [8] # [9]
Stylus is an extension that allows you to add your own CSS to style pages how you want. There seems to be a full community of folks that really use this to the nth degree to style all of their commonly used sites somewhat similarly or add dark mode to sites without it.
Personally I mostly use it to add my favorite syntax highlighting theme to jupyter, onedark. I’ve long lost the original author, but have posted the CSS I use in this gi...