astronomer [1] has done a fantastic job with dag-factory [2]. Highly recommend taking a look.
Dynamically generate Apache Airflow DAGs from YAML configuration files
References:
[1]: https://github.com/astronomer
[2]: https://github.com/astronomer/dag-factory
Publishing rhythm
orchest [1] by orchest [2] is a game-changer in its space. Excited to see how it evolves.
Build data pipelines, the easy way 🛠️
References:
[1]: https://github.com/orchest/orchest
[2]: https://github.com/orchest
Find and Replace in the Terminal.
grepr # [1]
grepr() {grep -iRl "$1" | xargs sed -i "s/$1/$2/g"}
```bash
grepr() {grep -iRl "$1" | xargs sed -i "s/$1/$2/g"}
grepd # [2]
grepd() {grep -iRl "$1" | xargs sed -i "/^$1/d"}
CocSearch # [3]
:CocSearch published: false -g *.md
References:
[1]: #grepr
[2]: #grepd
[3]: #cocsearch
gvanrossum [1] has done a fantastic job with patma [2]. Highly recommend taking a look.
Pattern Matching
References:
[1]: https://github.com/gvanrossum
[2]: https://github.com/gvanrossum/patma
Resume Tips
- customize for the job
- Why are you a good fit?
- What will you bring to the role?
- Give real outcomes
- give real experience
- Stop tech vomiting
- if you link to GitHub
- Make a profile readme
- Guide me to your best work
- have some activity
- if you link to LinkedIn
- Provide some benefit that is not on your resume
- Have a logical flow of experience (dont make me hunt for past experience)
- Keep it under 2 pages
- Who you know.
- Reference real experience
- Deployed 12 data pipelines with over 500 nodes to process 200GB of data at a Fortune 100 company
- vs
- Knowledge of Data Engineering methodology with python EC2
- Dont be so fluffy
mingrammer [1] has done a fantastic job with diagrams [2]. Highly recommend taking a look.
🎨 Diagram as Code for prototyping cloud system architectures
References:
[1]: https://github.com/mingrammer
[2]: https://github.com/mingrammer/diagrams
Just starred svelte-actions [1] by swyxio [2]. It’s an exciting project with a lot to offer.
prototype official actions for Svelte
References:
[1]: https://github.com/swyxio/svelte-actions
[2]: https://github.com/swyxio
Codeit Bro Interview
[1]
use this profile image
Please share your professional role as a data scientist? [Also feel free to
share about your personal projects, publications, etc.]
I graduated with a Mechanical Engineering Degree 8 years ago. Much of my work
early in my career [2] was wrapped
around analyzing larger datasets for my group to understand quality, drive
changes to improve quality or prove that quality was already good.
My first eight years as a working professional.
This day 8 years ago I started my first day as a Mechanical Engineer. I am so grateful for this journey that I have been able to have. There is no way that I...
Jun 19, 2020 [3]
Three years ago I made the switch to Data Science and have loved every minute of
it. It is a very dynamic field that is continually changing and there are
always a new set of skills to learn and hone in on. I talk a lot about the
mindset of always learning, sharing knowledge, and communicating in my
newsletter [4]
What are the most difficult chall...
reasons-to-kedro
There are many reasons that you should be using kedro. If you are on a team of
Data Scientists/Data Engineers processing DataFrames from many data sources
should be considering a pipeline framework. Kedro is a great option that
provides many benefits for teams to collaborate, develop, and deploy data
pipelines
What is Kedro [1]
Starter Template # [2]
Kedro makes it super easy to get started with their cli that utilizes
cookiecutter under the hood.
conda create -n my-new-project -y python=3.8
kedro new
kedro install
kedro run
Create New Kedro Project [3]
read more about how to start your first kedro project here
Collaboration # [4]
Kedro provides many tools that help teams collaborate on a single codebase.
While writing monolithic scripts it can be easy to pin yourself in a corner
where it is difficult to have multiple people making changes to the
notebook/script at the same time. Kedro helps guide your team to break your
project down into small pieces that different members o...
Reasons to Kedro
Reasons to Kedro # [1]
- collaboration
- Sharable catalog
- small nodes over monolithic notebooks
- catalog
- easily load anything without needing to run
- No need to write read/write code
- pipeline
- No need to keep execution order in your head
- easily run a slice of a pipeline
- plugins
- pip install
- make your own
- hooks
- flexible expandable cli
Reasons Not to Kedro # [2]
- Already utilizing another DAG framework
- Data is not in a widely supported format
- Micro short-lived project
- Large Project / Deadline
- Use a lower profile project to learn first
- Team not willing to change
- Need minimal dependencies
- God Project - kedro owns everything??
References:
[1]: #reasons-to-kedro
[2]: #reasons-not-to-kedro
Just starred Second-Brain [1] by KasperZutterman [2]. It’s an exciting project with a lot to offer.
A curated list of awesome Public Zettelkastens 🗄️ / Second Brains 🧠 / Digital Gardens 🌱
References:
[1]: https://github.com/KasperZutterman/Second-Brain
[2]: https://github.com/KasperZutterman
Reading List
Latest Post # [1]
latest [2]
STOP LEAVING Browser Tabs open and save them here!
- https://nesbitt.io/2026/03/04/package-managers-need-to-cool-down.html
- https://mariozechner.at/posts/2026-03-25-thoughts-on-slowing-the-fuck-down/
- https://danielmiessler.com/blog/ai-stops-being-artificially-cheap
---
- jbrancha til [3]
- The Video Course Launch that Made Me Think [4]
- photo prism [5]
- box python library [6]
- kedro on hn [7]
- How can a Data Scientist refactor Jupyter notebooks towards production-quality code? [8]
- Sourcing vs executing in Bash [9]
- Should We Follow The Open-Closed Principle? [10]
- Create multi-dimensional arrays in pure Python: The Correct Way [11]
- Beware of These 9 Red Flags in a Developer Interview [12]
- How to Overcome Impostor Syndrome as a Developer [13]
- lazy load youtube videos [14]
- lite youtube embeds [15]
- full subtitle youtube search [16]
---
- Jungle Scout - Kedro Case Study [17]
- Kedro Sessions [18]
- Julia Evans - A...
Just starred Repo-Roster [1] by nastyox [2]. It’s an exciting project with a lot to offer.
Shout-out supporters in your GitHub README file.
References:
[1]: https://github.com/nastyox/Repo-Roster
[2]: https://github.com/nastyox
What's New in Kedro 0.16.6
Kedro 0.16.6 [1] is out! Let’s take a look through the release notes
Deployment Docs # [2]
This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible.
- Argo [3]
- Prefect [4]
- Kubeflow [5]
- Batch [6]
- SageMaker [7]
Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies.
Argo, Prefect, and Kubeflow have an interesting technique where they translate the pipeline and its dependencies from kedro to their language.
Batch uses the aws cli to submit jobs, one node per job, and listen for them to complete. It will submit all nodes with completed dependencies at once, meaning that we can get some massive parallelization.
I did a quick and dirty test of one of these by simulating the technique in a bash script and saw a 40 hr pipeline finish in about 1 hour. I am excited to get thi...
mkdocs [1] by mkdocs [2] is a game-changer in its space. Excited to see how it evolves.
Project documentation with Markdown.
References:
[1]: https://github.com/mkdocs/mkdocs
[2]: https://github.com/mkdocs
A brain dump of stories
I started making stories as kind of a brain dump a few times per day and
posting them to
[LinkedIn](https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/).
Here are the last 11 days of stories.
I store all the stories on my website with the hopes of doing something with
them on my own platform eventually. For now it makes it easy to make these
posts.
cd static/stories
ls | xargs -I {} echo ''
Stories 10-10-2020 - 10-21-2020 # [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
References:
[1]: #stories-10-10-2020---10-21-2020
[2]: https://waylonwalker.com/stories/TIL-kedro-sorts-nodes.png
[3]: https://waylonwalker.com/stories/disable-base-pip.png
[4]: https://waylonwalker.com/stories/discovered-social-cards.png
[5]: https://waylonwalker.com/stories/find-kedro-de1-contributor.png
[6]: https://waylonwalker.com/stories/hacktoberfest-2020-kedro-538-tests-pass.png
[7]: https://waylonwalk...
Check out mmchougule [1] and their project kedro-grpc-server [2].
Kedro gRPC Server is a Kedro plugin that creates a gRPC server for triggering and monitoring pipeline runs using a general-purpose RPC framework gRPC
References:
[1]: https://github.com/mmchougule
[2]: https://github.com/mmchougule/kedro-grpc-server
Check out yetudada [1] and their project kedro-user-testing [2].
Discovery prototypes for user testing
References:
[1]: https://github.com/yetudada
[2]: https://github.com/yetudada/kedro-user-testing
The work on flynt [1] by ikamensh [2].
A tool to automatically convert old string literal formatting to f-strings
References:
[1]: https://github.com/ikamensh/flynt
[2]: https://github.com/ikamensh
charmbracelet [1] has done a fantastic job with glow [2]. Highly recommend taking a look.
Render markdown on the CLI, with pizzazz! 💅🏻
References:
[1]: https://github.com/charmbracelet
[2]: https://github.com/charmbracelet/glow