mingrammer [1] has done a fantastic job with diagrams [2]. Highly recommend taking a look.
🎨 Diagram as Code for prototyping cloud system architectures
References:
[1]: https://github.com/mingrammer
[2]: https://github.com/mingrammer/diagrams
Publishing rhythm
Just starred svelte-actions [1] by swyxio [2]. It’s an exciting project with a lot to offer.
prototype official actions for Svelte
References:
[1]: https://github.com/swyxio/svelte-actions
[2]: https://github.com/swyxio
Codeit Bro Interview
[1]
use this profile image
Please share your professional role as a data scientist? [Also feel free to
share about your personal projects, publications, etc.]
I graduated with a Mechanical Engineering Degree 8 years ago. Much of my work
early in my career [2] was wrapped
around analyzing larger datasets for my group to understand quality, drive
changes to improve quality or prove that quality was already good.
My first eight years as a working professional.
This day 8 years ago I started my first day as a Mechanical Engineer. I am so grateful for this journey that I have been able to have. There is no way that I...
Jun 19, 2020 [3]
Three years ago I made the switch to Data Science and have loved every minute of
it. It is a very dynamic field that is continually changing and there are
always a new set of skills to learn and hone in on. I talk a lot about the
mindset of always learning, sharing knowledge, and communicating in my
newsletter [4]
What are the most difficult chall...
reasons-to-kedro
There are many reasons that you should be using kedro. If you are on a team of
Data Scientists/Data Engineers processing DataFrames from many data sources
should be considering a pipeline framework. Kedro is a great option that
provides many benefits for teams to collaborate, develop, and deploy data
pipelines
What is Kedro [1]
Starter Template # [2]
Kedro makes it super easy to get started with their cli that utilizes
cookiecutter under the hood.
conda create -n my-new-project -y python=3.8
kedro new
kedro install
kedro run
Create New Kedro Project [3]
read more about how to start your first kedro project here
Collaboration # [4]
Kedro provides many tools that help teams collaborate on a single codebase.
While writing monolithic scripts it can be easy to pin yourself in a corner
where it is difficult to have multiple people making changes to the
notebook/script at the same time. Kedro helps guide your team to break your
project down into small pieces that different members o...
Reasons to Kedro
Reasons to Kedro # [1]
- collaboration
- Sharable catalog
- small nodes over monolithic notebooks
- catalog
- easily load anything without needing to run
- No need to write read/write code
- pipeline
- No need to keep execution order in your head
- easily run a slice of a pipeline
- plugins
- pip install
- make your own
- hooks
- flexible expandable cli
Reasons Not to Kedro # [2]
- Already utilizing another DAG framework
- Data is not in a widely supported format
- Micro short-lived project
- Large Project / Deadline
- Use a lower profile project to learn first
- Team not willing to change
- Need minimal dependencies
- God Project - kedro owns everything??
References:
[1]: #reasons-to-kedro
[2]: #reasons-not-to-kedro
Just starred Second-Brain [1] by KasperZutterman [2]. It’s an exciting project with a lot to offer.
A curated list of awesome Public Zettelkastens 🗄️ / Second Brains 🧠 / Digital Gardens 🌱
References:
[1]: https://github.com/KasperZutterman/Second-Brain
[2]: https://github.com/KasperZutterman
Reading List
Latest Post # [1]
latest [2]
STOP LEAVING Browser Tabs open and save them here!
- https://nesbitt.io/2026/03/04/package-managers-need-to-cool-down.html
- https://mariozechner.at/posts/2026-03-25-thoughts-on-slowing-the-fuck-down/
- https://danielmiessler.com/blog/ai-stops-being-artificially-cheap
---
- jbrancha til [3]
- The Video Course Launch that Made Me Think [4]
- photo prism [5]
- box python library [6]
- kedro on hn [7]
- How can a Data Scientist refactor Jupyter notebooks towards production-quality code? [8]
- Sourcing vs executing in Bash [9]
- Should We Follow The Open-Closed Principle? [10]
- Create multi-dimensional arrays in pure Python: The Correct Way [11]
- Beware of These 9 Red Flags in a Developer Interview [12]
- How to Overcome Impostor Syndrome as a Developer [13]
- lazy load youtube videos [14]
- lite youtube embeds [15]
- full subtitle youtube search [16]
---
- Jungle Scout - Kedro Case Study [17]
- Kedro Sessions [18]
- Julia Evans - A...
Just starred Repo-Roster [1] by nastyox [2]. It’s an exciting project with a lot to offer.
Shout-out supporters in your GitHub README file.
References:
[1]: https://github.com/nastyox/Repo-Roster
[2]: https://github.com/nastyox
What's New in Kedro 0.16.6
Kedro 0.16.6 [1] is out! Let’s take a look through the release notes
Deployment Docs # [2]
This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible.
- Argo [3]
- Prefect [4]
- Kubeflow [5]
- Batch [6]
- SageMaker [7]
Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies.
Argo, Prefect, and Kubeflow have an interesting technique where they translate the pipeline and its dependencies from kedro to their language.
Batch uses the aws cli to submit jobs, one node per job, and listen for them to complete. It will submit all nodes with completed dependencies at once, meaning that we can get some massive parallelization.
I did a quick and dirty test of one of these by simulating the technique in a bash script and saw a 40 hr pipeline finish in about 1 hour. I am excited to get thi...
mkdocs [1] by mkdocs [2] is a game-changer in its space. Excited to see how it evolves.
Project documentation with Markdown.
References:
[1]: https://github.com/mkdocs/mkdocs
[2]: https://github.com/mkdocs
A brain dump of stories
I started making stories as kind of a brain dump a few times per day and
posting them to
[LinkedIn](https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/).
Here are the last 11 days of stories.
I store all the stories on my website with the hopes of doing something with
them on my own platform eventually. For now it makes it easy to make these
posts.
cd static/stories
ls | xargs -I {} echo ''
Stories 10-10-2020 - 10-21-2020 # [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
References:
[1]: #stories-10-10-2020---10-21-2020
[2]: https://waylonwalker.com/stories/TIL-kedro-sorts-nodes.png
[3]: https://waylonwalker.com/stories/disable-base-pip.png
[4]: https://waylonwalker.com/stories/discovered-social-cards.png
[5]: https://waylonwalker.com/stories/find-kedro-de1-contributor.png
[6]: https://waylonwalker.com/stories/hacktoberfest-2020-kedro-538-tests-pass.png
[7]: https://waylonwalk...
Check out mmchougule [1] and their project kedro-grpc-server [2].
Kedro gRPC Server is a Kedro plugin that creates a gRPC server for triggering and monitoring pipeline runs using a general-purpose RPC framework gRPC
References:
[1]: https://github.com/mmchougule
[2]: https://github.com/mmchougule/kedro-grpc-server
Check out yetudada [1] and their project kedro-user-testing [2].
Discovery prototypes for user testing
References:
[1]: https://github.com/yetudada
[2]: https://github.com/yetudada/kedro-user-testing
The work on flynt [1] by ikamensh [2].
A tool to automatically convert old string literal formatting to f-strings
References:
[1]: https://github.com/ikamensh/flynt
[2]: https://github.com/ikamensh
charmbracelet [1] has done a fantastic job with glow [2]. Highly recommend taking a look.
Render markdown on the CLI, with pizzazz! 💅🏻
References:
[1]: https://github.com/charmbracelet
[2]: https://github.com/charmbracelet/glow
Check out mytechnotalent [1] and their project Python-For-Kids [2].
A FREE comprehensive online Python development tutorial FOR KIDS utilizing an official BBC micro:bit Development Board going step-by-step into the world of Python for microcontrollers.
References:
[1]: https://github.com/mytechnotalent
[2]: https://github.com/mytechnotalent/Python-For-Kids
I’m impressed by pycon_pybadge_2020 [1] from nnja [2].
Initial code for Microsoft’s PyBadge at PyCon 2020
References:
[1]: https://github.com/nnja/pycon_pybadge_2020
[2]: https://github.com/nnja
Fix git commit author
I was 20 commits into a hackoberfest PR when I suddenly realized they they all had my work email on them instead of my personal email 😱. This is the story of how I corrected my email address on 19 individual commits after already submitting for a PR.
- Change the email for this repo [1]
- Prepare for rebasing [2]
- start the rebase [3]
- 🛠 Fix First wrong Commit [4]
- Fix all commits [5]
- Done [6]
- ReCap [7]
Change the email for this repo # [1]
stop the bleeding
Before anything else set the email correctly!
cd kedro
git config user.name "Waylon Walker"
git config user.email [email protected]
Prepare for rebasing # [2]
First thing is to find how many commits back this mistake goes. I opened up the git [8] log, and saw mine went back 19 commits. I rolled back 20 just to be sure.
$ git log
...
commit a355926b9d7ec4c05659adaa254beefbdb036332
Author: WaylonWalker <[email protected]>
Date: Sat Oct 17 10:28:59 2020 -0500
give name of function inside incorrect parameters erro...
I like muesli’s [1] project duf [2].
Disk Usage/Free Utility - a better ‘df’ alternative
References:
[1]: https://github.com/muesli
[2]: https://github.com/muesli/duf
Designing a "Router" for kedro
nodes_global # [1]
I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called nodes_global. It worked fairly well but did have some issues around being set as a global variable.
But…
One thing in particular that it did not lend itself well to was being able to create a packagable pipeline that I could pip install and append into any of my existing pipelines. Something I am still trying to work out, maybe I don’t need this. I think I have it working for our internal pipelines and it seems like the way to go, but we don’t necessarily end up using it.
Also…
With this pattern all of the nodes needed to be importable by the module containing nodes_global. I find that this becomes a big hurdle for new pipelines coming from jupyter to overcome and can be most infuriating when their nodes aren’t getting ran af...