Create New Kedro Project
This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro. You can learn more about this Hello World Example in the docs
...
Posts tagged: python
All posts with the tag "python"
313 posts
latest post 2026-05-06
Publishing rhythm
What is YOUR Advice for New Data Scientists
You dont have to start out as a git wizard with the cleanest possible commit history. At first dont let yourself get too wrapped up in it, the most important part is that you make commits. You will find needs for more advanced stuff later.
git add . git commit -m "FEAT added new function to calculate revenue by product family" git push
Get comfortable with this, then learn how to branch, rebase, stash, etc…
Get the job done. Keep it in small bite size pieces. Make readable function definitions and variable names. You will thank yourself for naming things well later. Readability counts more than performance in most cases of data science. If it gets the job done try not to over worry about things like performance. A few extra seconds to clean a dataset or build a model is not worth hours of your time. As you go you will have cases that performance is more critical and you will learn what to do from the start to avoid...
What is Kedro
This is my original what-is-kedro article. There is a brand new one
Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate catalogs and pipelines. This article will cover the 10K view of kedro, future articles will dive deper into each one.
...
Long variable names are good
🏷️ Long variable names are a good thing. Self documenting code is more important than poorly documented code. Simply adding a few characters to your variable names can go a long ways.
Scope is important
simple click
cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library.
Things a cli can do to enhance your library.
🆚 print version 🕶 print readme 📝 print changelog 📃 print config ✏ change config 👩🎓 run a tutorial 🏗 scaffold a project with cookiecutter
...
SqlAlchemy Models
Make a connection # from sqlalchemy import create_engine def get_engine(): return create_engine("sqlite:///mode_examples.sqlite") Make a session # from sqlalchemy.orm import sessionmaker def get_session(): con = get_engine() Base.bind = con Base.metadata.create_all() Session = sessionmaker(bind=con) session = Session() return session Make a Base Class # from sqlalchemy.ext.declarative import declarative_base Base = declarative_base() Base.metadata.bind = get_engine() Make your First Model # class User(Base): __tablename__ = "users" username = Column('username', Text()) firstname = Column('firstname', Text()) lastname = Column('lastname', Text()) Make your own Base Class to inherit From # class MyBaseHelper: def to_dict(self): return {k: v for k, v in self.__dict__.items() if k[0] != "_"} def update(self, **attrs): for key, value in attrs.items(): if hasattr(self, key): setattr(self, key, value) Use the Custom Base Class # class User(Base, MyBaseHelper):...
Building Cli apps in Python
Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments.
**From the Docs
To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores.
...
Kedro
See all of my kedro related posts in [[ tag/kedro ]].
I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips.
Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration.
...
📝 Packages to Investigate Notes
|-|-| |github: |https://github.com/zaxr/bulwark|
I definitely want to try this out with kedro.
Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+.
Debugging Python
Using pdb #
Just Use Pathlib
Pathlib is an amazing cross-platform path tool.
from pathlib import Path
Create path object #
Current Directory
cwd = Path('.').absolute()
Users Home Directory
...
Filtering Pandas
Good for method chaining, i.e. adding more methods or filters without assigning a new variable.
# is skus.query('AVAILABILITY == " AVAILABLE"') # is not skus.query('AVAILABILITY != " AVAILABLE"')
masking #
general purpose, this is probably the most common method you see in training/examples
# is skus[skus['AVAILABILITY'] == 'AVAILABLE'] # is not skus[~skus['AVAILABILITY'] == 'AVAILABLE']
isin #
capable of including multiple strings to include
...
Pyspark
I have been using pyspark since March 2019, here are my thoughts.
Making good documentation in python
I just started using portray and it is amazingly simple to use!
Quick Progress Bars in python using TQDM
tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time.
for more gifs like these follow me on twitter @waylonwalker
Add a simple Progress bar!
...
Clean up Your Data Science with Named Tuples
If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow.
As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes.
Here we setup functions that can load data from the sales database. Assume that we also have similar...
...
Background Tasks in Python for Data Science
This post is intended as an extension/update from background tasks in python. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.
This post is intended as an extension/update from background tasks in python. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started.
I use it in more places than I probably should
...
📝 Bash Notes
Bash is super powerful.
Show Remaining Space on Drives
df -h
show largest files in current directory
...
Autoreload in Ipython
I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated.
🔥 Blazing Fast
💥 Keeps me in the comfort of my text editor
...