Check out csurfer [1] and their project pypette [2].
Ridiculously simple flow controller for building complex pipelines
References:
[1]: https://github.com/csurfer
[2]: https://github.com/csurfer/pypette
Archive
All published posts
2469 posts
latest post 2026-05-08
Publishing rhythm
Kedro
See all of my kedro related posts in [[ tag/kedro ]].
#kedrotips [1] # [2]
I am tweeting out most of these snippets as I add them, you can find them all
here #kedrotips [3].
🗣 Heads up # [4]
Below are some quick snippets/notes for when using kedro to build data
pipelines. So far I am just compiling snippets. Eventually I will create
several posts on kedro. These are mostly things that I use In my everyday with
kedro. Some are a bit more essoteric. Some are helpful when writing production
code, some are useful more usefule for exploration.
📚 Catalog # [5]
[6]
Photo by jesse orrico on Unsplash
CSVLocalDataSet # [7]
python
import pandas as pd
iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv')
data_set = CSVLocalDataSet(filepath="test.csv",
load_args=None,
save_args={"index": False})
iris_data_set.save(iris)
reloaded_iris = iris_data_se...
Check out requests [1] by psf [2]. It’s a well-crafted project with great potential.
A simple, yet elegant, HTTP library.
References:
[1]: https://github.com/psf/requests
[2]: https://github.com/psf
Check out vscode-git-semantic-commit [1] by nitayneeman [2]. It’s a well-crafted project with great potential.
💬 A Visual Studio Code extension which enables to commit simply by the semantic message conventions
References:
[1]: https://github.com/nitayneeman/vscode-git-semantic-commit
[2]: https://github.com/nitayneeman
awesome-streamlit [1] by MarcSkovMadsen [2] is a game-changer in its space. Excited to see how it evolves.
The purpose of this project is to share knowledge on how awesome Streamlit is and can be
References:
[1]: https://github.com/MarcSkovMadsen/awesome-streamlit
[2]: https://github.com/MarcSkovMadsen
I’m impressed by js13k-2019 [1] from bencoder [2].
xx142-b2.exe. An entry for js13kgames 2019
References:
[1]: https://github.com/bencoder/js13k-2019
[2]: https://github.com/bencoder
Just starred death-to-ie11 [1] by gabLaroche [2]. It’s an exciting project with a lot to offer.
Countdown for IE11 end of support
References:
[1]: https://github.com/gabLaroche/death-to-ie11
[2]: https://github.com/gabLaroche
📝 Packages to Investigate Notes
- jmespath
- Tabnine
Bulwark # [1]
|-|-|
|github: |https://github.com/zaxr/bulwark|
I definitely want to try this out with kedro.
Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+.
Example # [2]
import bulwark.decorators as dc
@dc.IsShape((-1, 10))
@dc.IsMonotonic(strict=True)
@dc.HasNoNans()
def compute(df):
# complex operations to determine result
...
return result_df
References:
[1]: #bulwark
[2]: #example
I came across awesome-data-engineering [1] from igorbarinov [2], and it’s packed with great features and ideas.
A curated list of data engineering tools for software developers
References:
[1]: https://github.com/igorbarinov/awesome-data-engineering
[2]: https://github.com/igorbarinov
I’m really excited about vscode-python [1], an amazing project by microsoft [2]. It’s worth exploring!
Python extension for Visual Studio Code
References:
[1]: https://github.com/microsoft/vscode-python
[2]: https://github.com/microsoft
Debugging Python
Using pdb # [1]
References:
[1]: #using-pdb
Just Use Pathlib
Pathlib is an amazing cross-platform path tool.
Import # [1]
from pathlib import Path
Create path object # [2]
Current Directory
cwd = Path('.').absolute()
Users Home Directory
home = Path.home()
module directory
module_path = Path(__file__)
Others
Let’s create a path relative to our current module.
data_path = Path(__file__) / 'data'
Check if files exist # [3]
Make Directories # [4]
data_path.mkdir(parents=True, exists_ok=True)
rename files # [5]
Path(data_path /'example.csv').rename('real.csv')
List files # [6]
Glob Files # [7]
data_path.glob('*.csv')
recursively
data_path.rglob('*.csv')
Write # [8]
Path(data_path / 'meta.txt').write_text(f'created on {datetime.datetime.today()})
References:
[1]: #import
[2]: #create-path-object
[3]: #check-if-files-exist
[4]: #make-directories
[5]: #rename-files
[6]: #list-files
[7]: #glob-files
[8]: #write
Custom Python Exceptions
Custom Exceptions # [1]
class ProjectNameError(NameError):
pass
class UserNameError(NameError):
pass
class CondaEnvironmentError(RuntimeError):
pass
class BucketNotDefinedError(NameError):
pass
References:
[1]: #custom-exceptions
Filtering Pandas
query # [1]
Good for method chaining, i.e. adding more methods or filters without assigning a new variable.
# is
skus.query('AVAILABILITY == " AVAILABLE"')
# is not
skus.query('AVAILABILITY != " AVAILABLE"')
masking # [2]
general purpose, this is probably the most common method you see in training/examples
# is
skus[skus['AVAILABILITY'] == 'AVAILABLE']
# is not
skus[~skus['AVAILABILITY'] == 'AVAILABLE']
isin # [3]
capable of including multiple strings to include
# is in
df[df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])]
# is not in
df[~df.AVAILABILITY.isin(['AVAILABLE', 'AVL'])]
contains # [4]
Good For partial matches
# contains
df[df.AVAILABILITY.str.contains('AVA')]
# not contains
df[~df.AVAILABILITY.str.contains('AVA')]
MASKS # [5]
anything that we put inside of square brackets can be set as a variable then passed in.
service_mask = skus['AVAILABILITY'] == 'AVAILABLE'
name_mask = skus['NAME'] == 'Dell chromebook 11'
Operators # [6]
& - and
~ - not
| - or
AVAILABLE and ...
Digital Ocean
I love digital ocean for it’s simplicity and its commitment to open source.
If you’re into interesting projects, don’t miss out on Recreation-of-Nature [1], created by Kashu7100 [2].
ALife simulation with Python: patterns, behavior, and cognition.
References:
[1]: https://github.com/Kashu7100/Recreation-of-Nature
[2]: https://github.com/Kashu7100
Quick Progress Bars in python using TQDM
tqdm is one of my favorite general purpose utility libraries in python. It
allows me to see progress of multipart processes as they happen. I really like
this for when I am developing something that takes some amount of time and I am
unsure of performance. It allows me to be patient when the process is going
well and will finish in sufficient time, and allows me to 💥 kill it and find a
way to make it perform better if it will not finish in sufficient time.
[1]
for more gifs like these follow me on twitter
@waylonwalker [2]
Add a simple Progress bar!
from tqdm import tqdm
from time import sleep
for i in tqdm(range(10)):
sleep(1)
convenience
TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically.
from tqdm import trange
from time import sleep
for i in trange(range(10)):
sleep(1)
notebook support
There is also notebook support. If you are bouncing between ipython and jupyter I recomend importing from the auto m...
I’m impressed by bake [1] from kennethreitz [2].
Bake — the strangely familiar workflow utility.
References:
[1]: https://github.com/kennethreitz/bake
[2]: https://github.com/kennethreitz
Check out terminal [1] by microsoft [2]. It’s a well-crafted project with great potential.
The new Windows Terminal and the original Windows console host, all in the same place!
References:
[1]: https://github.com/microsoft/terminal
[2]: https://github.com/microsoft
Clean up Your Data Science with Named Tuples
If you are a regular listener of TalkPython [1] or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow.
Example # [2]
As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes.
Load data # [3]
Here we setup functions that can load data from the sales database. Assume that we also have similar functions to get_inventory and get_pricing.
def get_engine():
engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase')
def get_sales():
'''
gets sales history from the sales database
'''
engine = ge...
Background Tasks in Python for Data Science
This post is intended as an extension/update from background tasks in
python [1]. I started using background
the week that Kenneth Reitz released it. It takes away so much boilerplate
from running background tasks that I use it in more places than I probably
should. After taking a look at that post today, I wanted to put a better data
science example in here to help folks get started.
This post is intended as an extension/update from background tasks in python [1]. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.
I use it in more places than I probably should
Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many w...
If you’re into interesting projects, don’t miss out on starship [1], created by starship [2].
☄🌌️ The minimal, blazing-fast, and infinitely customizable prompt for any shell!
References:
[1]: https://github.com/starship/starship
[2]: https://github.com/starship
alttch [1] has done a fantastic job with rapidtables [2]. Highly recommend taking a look.
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
References:
[1]: https://github.com/alttch
[2]: https://github.com/alttch/rapidtables
📝 Bash Notes
Bash is super powerful.
File System Full # [1]
Show Remaining Space on Drives
df -h
show largest files in current directory
du . -h --max-depth=1
Move files then symlink them
mkdir /mnt/mounted_drive
mv ~/bigdir /mnt/mounted_drive
ln -s /mnt/mounted_drive/bigdir ~/bigdir
Fuzzy One Liners # [2]
a() {source activate "$(conda info --envs | fzf | awk '{print $
edit in vim
vf() { fzf | xargs -r -I % $EDITOR % ;}
cat a file
vf() { fzf | xargs -r -I % $EDITOR % ;}
bash execute
bf() { bash "$(fzf)" }
git [3] add
gadd() { git status -s | fzf -m | awk '{print $2}' | xargs git add && git status -s}
git reset
greset() { git status -s | fzf -m | awk '{print $2}' |xargs git reset && git status -s}
Kill a process
fkill() {kill $(ps aux | fzf | awk '{print($2)}')}
Finding things # [4]
Files # [5]
fd-find [6] is amazing for finding files, it even respects your .gitignore file 😲. Install with apt install fd-find.
fd md
ag -g python
find . -n "*.md"
++Vanilla Bonus
Content # [7]
** sh...
Autoreload in Ipython
I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated.
What I like about autoreload # [1]
🔥 Blazing Fast
💥 Keeps me in the comfort of my text editor
👏 Allows me to use Jupyter when I need
👟 Extremely Reliable
One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code inside of a terminal/notebook. Now I primarily use jupyter notebooks for the presentation aspect. I develop code from the comfort of my editor with all of the tools I have setup, and run the functions in a notebook to get the output. From there I might do some aggregations or plots, but the 🥩 meat of development is done outside of jupyter.
Now I primarily use jupyter notebooks for the presentation aspect.
Enabling Autoreload # [2]
📐 config
This is a sh...
If you’re into interesting projects, don’t miss out on psutil [1], created by giampaolo [2].
Cross-platform lib for process and system monitoring in Python
References:
[1]: https://github.com/giampaolo/psutil
[2]: https://github.com/giampaolo
If you’re into interesting projects, don’t miss out on promote-open-source-project [1], created by zenika-open-source [2].
📄 How to promote my open source project?
References:
[1]: https://github.com/zenika-open-source/promote-open-source-project
[2]: https://github.com/zenika-open-source
Check out watchtower [1] by kislyuk [2]. It’s a well-crafted project with great potential.
Python CloudWatch Logging: Log Analytics and Application Intelligence
References:
[1]: https://github.com/kislyuk/watchtower
[2]: https://github.com/kislyuk
I recently discovered arrow [1] by apache [2], and it’s truly impressive.
Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
References:
[1]: https://github.com/apache/arrow
[2]: https://github.com/apache
Just starred shell-functools [1] by sharkdp [2]. It’s an exciting project with a lot to offer.
Functional programming tools for the shell
References:
[1]: https://github.com/sharkdp/shell-functools
[2]: https://github.com/sharkdp
Keyboard Driven VSCode
Throw that mouse Away its time to setup some keyboard shortcuts.
These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on the hackability of tmux. tmux allows me to rapidly fire up a workspace, create new windows and splits. Then When I switch tasks I can leave that workspace open and and jump right back in later exactly where I left off. There is nothing quite like it. The shortcuts listed here make the transition a bit better. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. This first set of keybindings solve that issue.
The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse.
!!! see-also
I have an updated article in my tmux workflow How I navigate tmux in 2021 [1]
Alt+[hjkl] # [2]
navigation
⬅ jump to left split alt+h
⬇ j...
Looking for inspiration? Jupyter-Atom-Dark-Theme [1] by burglarbenson [2].
A dark theme for Jupyter Lab
References:
[1]: https://github.com/burglarbenson/Jupyter-Atom-Dark-Theme
[2]: https://github.com/burglarbenson
tarpas [1] has done a fantastic job with pytest-testmon [2]. Highly recommend taking a look.
Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
References:
[1]: https://github.com/tarpas
[2]: https://github.com/tarpas/pytest-testmon
If you’re into interesting projects, don’t miss out on vim-flog [1], created by rbong [2].
A blazingly fast, stunningly beautiful, exceptionally powerful git [3] branch viewer for Vim/Neovim.
References:
[1]: https://github.com/rbong/vim-flog
[2]: https://github.com/rbong
[3]: /glossary/git/
I like mcfunley’s [1] project pugsql [2].
A HugSQL-inspired database library for Python
References:
[1]: https://github.com/mcfunley
[2]: https://github.com/mcfunley/pugsql
I like ggreer’s [1] project the_silver_searcher [2].
A code-searching tool similar to ack, but faster.
References:
[1]: https://github.com/ggreer
[2]: https://github.com/ggreer/the_silver_searcher
Realistic Git Workflow
My git [1] workflow based on real life. Its not always clean and simple.
sometimes things get messy
The Clean Path # [2]
[3]
pull 👉 branch 👉 format 👉 work👉 add 👉 commit 👉 pull 👉 rebase 👉 push
Pull # [4]
As complicated as that seems it is pretty straight forward. When you sit down to work the first thing you do is to pull down the teams latest working “develop” branch from git.
git checkout develop
git pull
Branch # [5]
Next create a new branch with a name that will remind you of what you are working on. For your own sanity choose something descriptive. It is easy to get too many similar branches going and forget which branch is which.
git checkout -b ingest_product_id_table
Format # [6]
If you know which files in existance that you will be editing before you start work it is a good idea to format them in a commit early on to keep your working commits separate from formatting. This will make it easier for reviewers to distinguish from your changes and formatting fixes.
...
Just starred kedro [1] by kedro-org [2]. It’s an exciting project with a lot to offer.
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
References:
[1]: https://github.com/kedro-org/kedro
[2]: https://github.com/kedro-org
Check out forestryio [1] and their project forestry.io [2].
Forestry.io website
References:
[1]: https://github.com/forestryio
[2]: https://github.com/forestryio/forestry.io
Check out maildown [1] by chris104957 [2]. It’s a well-crafted project with great potential.
A super simple CLI for sending emails
References:
[1]: https://github.com/chris104957/maildown
[2]: https://github.com/chris104957
Forestry.io
Testing out forestry.io
Sorry Netlify CMS # [1]
I still ♥️ your product dont be
forestry is simple
I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really struggle configuring it. Forestry is so simple to setup. My favorite part is that I can code up my gatsby.js site, storing all editable text in markdown, and come back later and add the CMS based on existing documents.
Configuration is Simple # [2]
Forestry.io has this amazing feature to create create based on existing document 🤯. This is great because it sets up the .yml config for me without error. And If I really want to come back later to customize it more I have that option, too.
[3]
By far my favorite feature is create based on existing document
Multi-File Gallery # [4]
I have a use case for a photography site where the owner wants to be able to show off sample photos of each type of work she does. I got it working in the netlify cms, although it was not very user...
Just starred eslint-config-wesbos [1] by wesbos [2]. It’s an exciting project with a lot to offer.
No-Sweat™ Eslint and Prettier Setup - with or without VS Code
References:
[1]: https://github.com/wesbos/eslint-config-wesbos
[2]: https://github.com/wesbos
Check out ydataai [1] and their project ydata-profiling [2].
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
References:
[1]: https://github.com/ydataai
[2]: https://github.com/ydataai/ydata-profiling
I came across ydata-profiling [1] from Data-Centric-AI-Community [2], and it’s packed with great features and ideas.
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
References:
[1]: https://github.com/Data-Centric-AI-Community/ydata-profiling
[2]: https://github.com/Data-Centric-AI-Community
mdbartos [1] has done a fantastic job with tabview [2]. Highly recommend taking a look.
Python curses command line CSV viewer
References:
[1]: https://github.com/mdbartos
[2]: https://github.com/mdbartos/tabview
TabViewer [1] has done a fantastic job with tabview [2]. Highly recommend taking a look.
Python curses command line CSV and tabular data viewer
References:
[1]: https://github.com/TabViewer
[2]: https://github.com/TabViewer/tabview
I came across voidrice [1] from LukeSmithxyz [2], and it’s packed with great features and ideas.
My dotfiles (deployed by LARBS)
References:
[1]: https://github.com/LukeSmithxyz/voidrice
[2]: https://github.com/LukeSmithxyz
I like FormidableLabs’s [1] project webpack-dashboard [2].
A CLI dashboard for webpack dev server
References:
[1]: https://github.com/FormidableLabs
[2]: https://github.com/FormidableLabs/webpack-dashboard
Rewrite History with Git
- rebase
- git [1] commit –amend
Unstage # [2]
git reset -- <file>
rage unstage to wipte out history of staged commit
git reset --hard <file>
Undo file # [3]
- rage quit
- git reset HEAD~n
- removes modifications
- keeps hitsory of changes and undoes them
- git checkout HEAD~n –
-
keeps modifications
-
removes history
-
–SOFT
-
–HARD
-
–Mixed
undo n commits back # [4]
locally before push
git reset HEAD~n
after push
git revert HEAD~n
update .gitignore # [5]
after push
git rm -r --cached .
git commit -am "Updated .gitignore"
References:
[1]: /glossary/git/
[2]: #unstage
[3]: #undo-file
[4]: #undo-n-commits-back
[5]: #update-gitignore
It's not all about winning
This is my story into data science.
The Journey Begins # [1]
I am addicted to the process of learning and improving my skills nearly to a fault. The reason I say nearly is because my addiction is fueled with results. I crave the output of my work enhance the work of others. I jump with joy as I see users gain insights they could have never imagined before. My mouth starts watering as I see their boring repetitive data mining activites be completed in a matter of seconds, opening up their mind to focus on their expertise. The day I stop learning will be the day that I start looking for another career path.
It’s not all about winning.
~Mom
This happened to me in 2014. I have a mechanical engineering degree and had a really good position at the time. I owned full engineering control of a small subset of engine components. The problem was that Everything is so proprietary and hardly documented the process of learning did not click with me. I felt like I had learned a lot about the ...