Posts tagged: kedro Archive

💭 Migration guide for config loaders — kedro 0.19.11 documentation

2025-02-05T18:16:44Z

Migrating from kedro 0.18.4 to the latest version involves handling the deprecated OmegaConf loader. Switching over does not look as bad as I originally thought.

installing kedro 0.18.5+
set the CONFIG_LOADER_CLASS in settings.py
swap out import statements
config must be yaml or json
getting values from config must be done with bracket __getattr__ style not with .get
any Exceptions caught from Templated config loader will need to be swapped to OmegaConfig exceptions, similar to #3
templated values must lead with an _
Globals are handled different
OmegaConfig does not support jinja2 sytax, but rather a ${variable} syntax

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Make Kedro Runs Beautiful

2022-03-11T13:14:51Z

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.

Install kedro rich #

There is no pypi package yet, but it’s on github. You can pip install it with the git url.

pip install git+https://github.com/datajoely/kedro-rich

Kedro run #

You can run your pipeline just as you normally would, except you get progress bars and pretty prints.

kedro run

Kedro catalog #

Listing out catalog entries from the command line now print out a nice pretty table.

kedro catalog list

Give it a star #

Go to the GitHub repo and give it a star, Joel deserves it.

Lambda Function as a Kedro Node

2022-01-31T21:52:36Z

I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas.

Examples #

Here are two examples, the first one lambda x: x is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow standard layering conventions, without skipping a layer, overthinking if you should have the layer or not, and leaves a good placholder to fill in later when you need it.

Many times I just want to get the data in as fast as possible, learn about it, then go back and tidy it up.

from kedro.pipeline import node

my_first_node = node(
   func=lambda x: x,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

my_first_node = node(
   func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'),
   inputs='raw_cars',
   output='int_cars',
   tags=['pri',]
   )

Note: try not to take the idea of a one liner too far. If your one line function wraps several lines down it probably deserves to be a real function for readability and a good docstring.

Add New Dependencies to Your Kedro Project

2022-01-28T21:30:48Z

As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command pip-compile under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let’s take a look at it.

git-status">git status #

Before you start mucking around with any changes to dependencies make sure that your git status is clean. I’d even reccomend starting a new branch for this, and if you are working on a team potentially submit this as its own PR for clarity.

git status
git checkout main
git checkout -b add-rich-dependency

requirements.in #

New requirements get added to a requirements.in file. If you need to specify an exact version, or a minimum version you can do that, but if all versions generally work you can leave it open.

# requirements.in
rich

Here I added the popular rich package to my requirements.in file. Since I am ok with the latest version I am not going to pin anything, I am going to let the pip resolver pick the latest version that does not conflict with any of my dependencies for me.

build-reqs #

The command kedro build-reqs will tell kedro to recompile the requirements.txt file that has all of our dependencies pinned down to exact versions. This ensures that all of our teammates and production workflows use the same exact versions of packages even if new ones are released after we installed on our development machines.

kedro build-reqs

git add #

Now that we have our new dependencies ready to go commit those to git, and submit a PR for them if you are working on a team. This is a good way to document the discussion of adding new dependencies to your teams project.

git add requirements.in
git add requirements.txt
git status
git commit -m "FEAT updated dependencies with rich"
git push
# go make a pr
gh pr create --title "feat add rich to dependencies" --body "I added rich as a dependency, and ran pip-compile"

Practice making pipelines with kedro

2022-01-27T21:07:37Z

I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity.

What is Kedro

Start practicing #

practice building pipelines with #kedro today

Go to your playground directory, and if you don’t have one, make one.

cd ~/playground

get pipx #

Install pipx in your system python. This is one of the very few, and possibly the only python library that deserves to be installed in your system directory, primarily because its used to sanbox clis in their own virtual environment automatically for you.

pip install pipx

make a new project #

From inside your playground directory, start your new kedro project. This is quite simple and painless. So much so that if you mess this one up doing something wild, it might be easier to make a new one that fixing the wild one.

pipx run kedro new
# answer the questions it asks

I use this quite often to try out new things in a safe place.

Make a virtual environment #

Using Conda #

Conda is a fine choice to manage your virtual environments. It used to make things so much easier on windows that it was almost required. Nowadays getting python running on windows has become so much easier that this is less so.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

one great benefit of conda is that it lets you choose the interpreter to go with your virtual environment.

Your new environment will be listed in your list of conda env here.

conda info --envs

Using venv #

venv is what I use now. Nothing against conda, it works great. venv just feels a bit lighter and more common. I’ve actually grown to appreciate that the venv is right where I put it, most often in the project directory.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

using pipenv #

pipenv is another fine choice. I like how in one command it makes the environment and activates it for you. pipenv also puts virtual environments in the global directory.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

Make pipelines #

Now go make some pipelines with your new project, try something wild, break it, and make another.

Running Kedro on Ubuntu 21.10 Impish Indri

2022-01-02T20:24:48Z

I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran

pipx run kedro new --starter spaceflights

But what I got back was not what I expected!

Fatal error from pip prevented installation. Full pip output in file:
    /home/walkers/.local/pipx/logs/cmd_2022-01-01_20.42.16_pip_errors.log

Some possibly relevant errors from pip install:
    ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
    ERROR: No matching distribution found for kedro
Error installing kedro.

This is weird, why cant I run kedro new with pipx? Lets try pip.

pip install kedro

Same issue.

ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
ERROR: No matching distribution found for kedro

What is Kedro

Curious what kedro is? Check out this article.

What’s up #

wrong python version

The issue is that kedro only runs on up to python 3.8, and on Ubuntu 21.10 when you apt install python3 you get python 3.9 and the standard repos don’t have an old enough version to run kedro.

How to fix this? #

Theres a couple of ways you can fix this? They all involve installing a distribution that does not come from the standard repo.

Where Can I get the right version #

Anaconda
Python.org
deadsnakes
pyenv
miniconda

I have two articles that can help you #

How to Install miniconda on linux (from the command line only)

Using miniconda

conda create -n myenv python=3.8

My first impressions with pyenv

Using pyenv

pyenv install 3.8.12

kedro catalog create

2021-11-15T07:18:55Z

I use kedro catalog create to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key.

https://youtu.be/_22ELT4kja4

What is Kedro

👆 Unsure what kedro is? Check out this post.

Running Kedro Catalog Create #

The command to ensure there are catalog entries for every dataset in the passed in pipeline.

kedro catalog create --pipeline history_nodes

Create’s new yaml file, if needed
Fills in new dataset entries with the default dataset
Keeps existing datasets untouched
it will reformat your yaml file a bit
- default sorting will be applied
- empty newlines will be removed

CONF_ROOT #

Kedro will respect your CONF_ROOT settings when it creates a new catalog file, or looks for existing catalog files. You can change the location of your configuration files by editing your CONF_ROOT variable in your projects.

settings.py.

# settings.py
# default settings
CONF_ROOT = "conf"

# I like to package my configuration
CONF_ROOT = str(Path(__file__).parent / "conf")

I prefer to keep my configuration packaged inside of my project. This is partly due to how my team operates and deploys pipelines.

File Location #

The kedro catalog create command will look for a yaml file based on the name of the pipeline (CONF_ROOT/catalog/<pipeline-name>.yml). If it does not find one it will create one and make entries for each dataset in the pipeline. It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline. If you are going to use this command its important that you follow this pattern or copy what it generates into your own catalog file of choice.

⚠️ It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline.

MemoryDataSet’s #

When you run kedro catalog create you get MemoryDataSet, that’s it. As of 0.17.4 its hard coded into the library and not configurable.

range12:
  type: MemoryDataSet

Your free to use what you want though #

Let’s switch this dataset over to a pandas.CSVDataSet so that the file gets stored and we can pick up and read the file without re-running the whole pipeline.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv

Continue adding nodes #

As we work we will keep adding nodes to our kedro pipeline, in this case we added another node that created a dataset called range13.

kedro catalog create --pipeline history_nodes

After telling kedro to create new catalog entries for us we will see that it left our range12 entry alone and created range13 for us.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range13:
  type: MemoryDataSet

Formatting is not worthwhile #

If we decide this is too cramped for us we could add some space between datasets. The next time we run kedro catalog create empty lines will be removed.

range12:
  type: pandas.CSVDataSet

range13:
  type: MemoryDataSet

Continuing to work #

If we coninue adding new nodes, and tell kedro to create catalog entries again, all of our effort given to formatting will be lost. I wouldn’t worry about it unless you have an autoformatter that you can run on your yaml files. The productivity gains in an semi-automated catalog are worth it.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range121:
  type: MemoryDataSet
range13:
  type: MemoryDataSet

Sorting Order #

Notice the sorting order in the last entry, range121 comes before range13. This is all based on how pythons yaml.safe_dump works, kedro has set the default_flow_style to False. You can see where they write your file in the source code currently here

nvim conf 2021 | IDE's are slow | Waylon Walker

2021-10-29T00:00:00Z

https://youtu.be/E18m4KkJUnI

Slides 👇 #

welcome #

Other possible titles #

Using Vim as a Team Lead
I 💜 Tmux
Why I stopped using @code
Get there fast
How I vim

It’s ok #

Use a graphical IDE if it works for you.

Trick it out #

vim is so well integrated into the terminal, take advantage

It wasn’t working for me anymore #

dozens of instances #

As a team lead I bounce betweeen a dozen projects a per day

https://pbs.twimg.com/media/FAEmRjYUcAUk2eR?format=jpg&name=large

Move With Intent #

Running vim inside tmux lets me move swiftly between the exact project I need.

https://twitter.com/_WaylonWalker/status/1438849269407047686/photo/1 //: <> (__)

Hub and Spoke #

direct link to specific projects
fuzzy into all projects
fuzzy into open projects

How I navigate tmux in 2021#hub-and-spoke

Other Things That Make this Possible #

tmux
direnv

vim adjacent things

yes, vim is ugly, make it yours #

@rook

command! Q :q

@_waylonwalker

nnoremap <leader>6 <c-^>

lsp #

lua vim.lsp.buf.definition()

treesitter #

Plug 'nvim-treesitter/nvim-treesitter-textobjects'

telescope #

fuzzy matching like a boss

fzf is ok too

Check Messages #

Kedro-Broken-Urls

2021-10-20T00:00:00Z

Broken Urls #

Setting Parameters in kedro

2021-09-19T00:00:00Z

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files.

https://youtu.be/Jj5cQ5bqcjg

What is Kedro

👆 Unsure what kedro is? Check out this post.

parameters files #

You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a base and local parameters file.

conf/base/parameters.yml
conf/local/parameters.yml

base #

The base environment should contain all of the default values you want to run.

# /conf/base/parameters.yml
test_size: 0.2
random_state: 3
features:
  - engines
  - passenger_capacity
  - crew
  - d_check_complete
  - moon_clearance_complete
  - iata_approved
  - company_rating
  - review_scores_rating

NOTE base will always be loaded first.

accessing parameters #

Parameters can be accessed through context or through the catalog. Generally when you are working with nodes it will be loaded through the catalog.

Loding with the context.

context = session.load_context()
context.params

Loading with the catalog.

catalog.load('parameters')

Loading a specific key with the catalog.

catalog.load('params:test_size')

using parameters in nodes #

Here is an example from the complete spaceflights demo. The entire parameters dict is passed in, then the features key is accessed.

def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
    """Splits data into features and targets training and test sets.

    Args:
        data: Data containing features and target.
        parameters: Parameters defined in parameters.yml.
    Returns:
        Split data.
    """
    X = data[parameters["features"]]
    y = data["price"]
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=parameters["test_size"], random_state=parameters["random_state"]
    )
    return X_train, X_test, y_train, y_test

Comprehensive guide to creating kedro nodes

👆 Check out this complete guide to creating kedro nodes.

local #

The local parameters by default are in conf/local/parameters.yml. They will override the base parameters in a shallow fashion. If a top level key exists in local, it will override that entire key in your parameters.

# /conf/local/parameters.yml
env: local
features:
  - company_rating
  - review_scores_rating

NOTE If you have not explicitly set your environment, local will be the default environment selected to override base.

env #

You can also have other environments that override the base environment.

# /conf/new/parameters.yml
env: new

NOTE if you use an env local will not be applied

Activating this environment #

Following the configuration docs we can activate the environment by setting an environment variable in our shell or passing in –env to our kedro cli command.

Setting an environment variable.

export KEDRO_ENV=test

Passing in the env to a kedro cli command.

kedro run --env=test

Setting the Environment Variable in python.

import os
os.environ['KEDRO_ENV'] = 'new'

Links #

Writing your first kedro Nodes

2021-09-13T22:40:45Z

https://youtu.be/-gEwU-MrPuA

Before we jump in with anything crazy, let’s make some nodes with some vanilla data structures.

import node #

You will need to import node from kedro.pipeline to start creating nodes.

from kedro.pipeline import node

func #

The func is a callable that will take the inputs and create the outputs.

inputs / outputs #

Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is the catalog entry to use for that keyword.

our first node #

Sometimes in our pipelines our data is coming from an api where we already have python functions built to pull with. Thats ok, kedro supposrts that with inputs=None.

def create_range():
    return range(100)

make_range = node(
    func=create_range,
    inputs=None,
    outputs='range'
    )

second node #

Now we have some data to work from, lets use that as our input.

def square_range():
    return [i**2 for i in range]

square_range = node(
    func=square_range,
    inputs='range',
    outputs='range_squared'
    )

Multiple Inputs #

Kedro can take lists or dicts as either input or output when your function needs more than one input or output.

def concat(range, range_two):
    return [*range, *range_two]

concat_ranges = node(
    func=concat,
    inputs=['range', 'range_squared']
    outputs='concat'
    )

## inputs could also be defined as a dict
concat_ranges = node(
    func=concat,
    inputs={'range': 'range', 'range_two': 'range_squared'}
    outputs='concat'
    )

Links #

Running your Kedro Pipeline from the command line

2021-08-24T22:40:45Z

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli.

https://youtu.be/ZmccpLy-OEI

What is Kedro

👆 Unsure what kedro is? Check out this post.

Kedro run #

To run the whole darn project all we need to do is fire up a terminal, activate our environment, and tell kedro to run.

kedro run

Specific Pipelines #

Running a sub pipeline that we have created is as easy as telling kedro which one we want to run.

kedro run --pipeline dp

Single Nodes #

While developing a node or a small list of nodes in a larger pipeline its handy to be able to run them one at a time. Besides the use case of developing a single node I would not reccomend leaning very heavy on running single nodes, let the DAG do the work of figuring out which nodes to run for you.

kedro run --pipeline dp --node create_model_input_table_node
kedro run --pipeline dp -n create_model_input_table_node

Some DAG concepts #

We will cover more of the benefits that we get from the graph nature of the DAG in the future, but here is a quick peek at some things we can do.

kedro run --pipeline dp --to-outputs preprocessed_shuttles
kedro run --pipeline dp --from-inputs preprocessed_shuttles
kedro run --pipeline dp --to-nodes create_model_input_table_node

Multiple things #

You can stack up multiple kedro dag concepts into a single run command.

kedro run --pipeline dp --to-nodes create_model_input_table_node --to-nodes preprocess_shuttles_node

kedro Virtual Environment

2021-08-23T20:40:45Z

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment.

https://youtu.be/ZSxc5VVCBhM

conda
venv
pipenv

conda #

I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don’t have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

conda info --envs

stores environment in a root directory i.e. ~/miniconda3
conda can use its own way to manage environments environment.yml
the python interpreter is packaged with the environment

virtualenv #

Virtual env (venv) is another very respectable option that is built right into python, and requires no additional installs or using a different distribution of pytyhon.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

environments are typically stored in the project directory
does not package the interpreter

pipenv #

Pipenv is another virtual enviroment tool that comes with its own system for managing dependencies using a pipfile. It’s main benefit is that it creates a lockfile that will allow users to replicate the exact version of all their packages. The typical requirements.txt workflow can easily break as new version of dependecies are released between testing and deplpoyment.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

stores environment in a root directory i.e. ~/.local/share/virtualenvs/
pipenv can use its own way to manage environments pipfile
does not package the interpreter

Kedro Pipeline Create

2021-08-22T22:40:45Z

Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself.

https://youtu.be/HtyIKqlEoNw

creating a new pipeline #

The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your pipeline_registry, to be covered later, you will need to add it yourself.

kedro pipeline create example

results #

The directory structure that it creates looks like this.

tree src/kedro_conda/pipelines
src/kedro_conda/pipelines
├── __init__.py
└── example
    ├── __init__.py
    ├── nodes.py
    ├── pipeline.py
    └── README.md

Kedro Install

2021-08-21T20:40:45Z

Kedro comes with an install command to install and manage all of your projects dependencies.

https://youtu.be/IWimEs-hHQg

cd into your project directory and activate env #

You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment.

Kedro New

this post covers kedro new

kedro Virtual Environment

This post covers creating your virtual environment for kedro

install kedro #

Make sure you have kedro installed in your current environment, if you dont already have it.

pip install kedro==0.17.4

pip-tools #

Kedro uses the pip-tools package under the hood to pin dependencies in a very robust way to ensure that the project will continue to work on everyone’s machine day, including production, day in and day out. No matter what happens to the dependencies you have installed.

pip-compile #

The command that kedro uses from pip-tools is pip-compile. It will look at what you have in a requirements.in file, compile the dependencies down to exact versions, and create a requirements.txt that is fully pinned down, and updatable by re-running pip-compile.

requirements.in #

If kedro does not see a requirements.in file it will automatically move your requirements.txt to requirements.in and run pip-compile.

No requirements.in found. Copying contents from requirements.txt...

kedro install #

Lets go ahead and run kedro install on one of the projects we already create and environment for in a previous post, kedro-conda.

kedro install

The first time you run this on a new repo it is likely that you will run into this warning about creating a new requirements.in file.

No requirements.in found. Copying contents from requirements.txt...

kedro install flags #

Kedro does let you avoid pip-compile all together, by using the --no-build-reqs flag.

kedro install --no-build-reqs

It also lets you upgrade all of your dependencies with build-reqs. I would reccomend doing this on its own branch, and own pull request. If you are working on a team you want everyone to be on the same page when it comes to dependencies. If you are not, you surely do not want something to break with a new set of dependencies without a way of rolling back.

kedro install --build-reqs

Kedro Git Init

2021-08-20T22:40:45Z

Immediately after kedro new, before you start running kedro install or your first line of code the first thing you should always do after getting a new kedro template created is to git init.

https://youtu.be/IGba3ytf_6U

git-init">git init #

Its as simple as these three commands to get started.

git init
git add .
git commit -m init

I don’t care if this project is for learning, if it will never have a remote or not, use git.

Kedro New

2021-08-18T20:40:45Z

https://youtu.be/uqiv5LAiJe0

Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template.

What is Kedro

Unsure what kedro is, Check out yesterdays post on What is Kedro.

pipx #

I reccomend using pipx when running kedro new. pipx is designed for system level cli tools so that you do not need to maintain a virtual environment or worry about version conflicts, pipx manages the environment for you.

The kedro team does not reccomend pipx in their docs as they already feel like there is a bit of a tool overload for folks that may be less familiar with

pipx kedro new

I like using pipx as it gives you better control over using a specific version or always the latest version, unlike when you run what you have on your system depends on when you last installed or upgraded.

Kedro New #

The kedro team also has a set of starters, by passing in --starter you can start with a different template. Here is an example with the kedro spaceflights starter.

pipx run kedro new --starter spaceflights

=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
 [New Kedro Project]: Spaceflights Complete

Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
 [spaceflights-complete]:

Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter
or underscore.
 [spaceflights_complete]:

Change directory to the project generated in /home/u_walkews/git/spaceflights-complete

A best-practice setup includes initialising git and creating a virtual environment before running ``kedro install`` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

Other versions of kedro with pipx #

pipx not only ensures that you run the latest version, it can also run a very specific version.

pipx run --spec kedro==0.16.6 kedro new

kedro Virtual Environment

The next post in this series will help you create your virtual environment for your new kedro project.

What is Kedro

2021-08-17T12:48:00Z

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract DataSets that the user specifies through Catalog entries. These Catalog entries are loaded, ran through a function, and saved by Nodes. The order that these Nodes are executed are determined by the Pipeline, which is a DAG. It’s the runner’s job to manage the execution of the Nodes.

https://youtu.be/Wf4rnFsaFFU

What is Kedro

This is an updated version of my original what-is-kedro article

Hot Take #

If you are doing a series of operations to data with python, especially if you are using something as supported as pandas, you should be using a framework that gives you a pipeline as a DAG and abstracts io.

Orchestrators #

Like I said, kedro is unopinionated it does determine where or how your data should be ran. The kedro team does support the following Orchestrators with very little add on to the base template.

DataSets #

Did I say kedro is unopionated? Datasets are what allow kedro too be so flexible accross a number of different python objects. Any python object can be made into a kedro dataset. Kedro comes out of the box with many purpose built DataSets like storing pandas DataFrames to parquet, csv, or a sql table. If kedro does not come with support for the type of python objects you work with don’t worry, you can for the closest option they support and build your own. Or if you do not want to build your own, you can use a PickleDataSet for anything.

Catalog #

You will not often be creating your own datasets, most of what you need whould already be taken care of by the kedro framework. What you will need to do is to use the existing DataSets to build your data catalog.

Kedro takes care of all fo the file io for you, you simply need to use the catalog to tell kedro what type of DataSet to use and any extra information that DataSet needs. Much of the time this is simply a filepath.

Typically the catalog is specified in yaml format. If you are not familiar with yaml, I suggest learnxinyminutes.com/docs/yaml/ as a resource of examples.

test:
  type: pandas.CSVDataSet
  filepath: s3://your_bucket/test.csv #

Here is the most basic yaml catalog entry taken from the kedro docs

cars:
  type: pandas.CSVDataSet
  filepath: data/01_raw/company/cars.csv
    sep: ','
    load_args:
  save_args:
    index: False
    date_format: '%Y-%m-%d %H:%M'
    decimal: .

Here is a bit more complex example that takes in load_args and save_args docs

Nodes #

Nodes are a very core part of kedro to build the DAG. These nodes are what provides the definition of what catalog entries, get passed into which function, and output to another catalog entry.

import pandas as pd
import numpy as np

def clean_data(cars: pd.DataFrame,
               boats: pd.DataFrame) -> Dict[str, pd.DataFrame]:
    return dict(cars_df=cars.dropna(), boats_df=boats.dropna())

def halve_dataframe(data: pd.DataFrame) -> List[pd.DataFrame]:
    return np.array_split(data, 2)

nodes = [
    node(clean_data,
         inputs=['cars2017', 'boats2017'],
         outputs=dict(cars_df='clean_cars2017',
                      boats_df='clean_boats2017')),
    node(halve_dataframe,
         'clean_cars2017',
         ['train_cars2017', 'test_cars2017']),
    node(halve_dataframe,
         dict(data='clean_boats2017'),
         ['train_boats2017', 'test_boats2017'])
]

Here is an example of three nodes taken from their docs

Pipeline #

The kedro Pipeline, is a DAG (Directed Acyclic Graph). It is a graph object that flows in one direction. You can slice into the pipeline using a few built in graph method to_nodes, from_nodes, to_outputs, and from_inputs. You can chain up these method calls since each one returns a new Pipeline object. You can also ask a pipline for its edges with inputs, and outputs. You can also list every dataset along the way with all_inputs or all_outputs. Lastly you can convert it back into a list of nodes with nodes.

from kedro.pipeline import Pipeline, node

# using our nodes from last tim
Pipeline(nodes)

Runner #

The runner is the bridge between kedro and the orchestrators. The kedro team provides some basic runners for running pipelines locally, built right into the framework, but adding on new runners for different orchestrators is done through the use of adding in a new runner to your project.

Hooks #

Kedro allows you to hook into a number of lifecycle methods through the use of the pluggy framework. Yes the one that pytest is built on. There are a number of different lifecycle methods that allow us to hook in around where kedro is running such as before_pipeline_run or after_catalog_loaded.

Links #

How I Kedro

2021-08-16T22:34:45Z

https://youtu.be/bw5_FWDVRpU

Ubuntu #

I recently switched over to using Ubuntu, it works well pretty much out of the box for me. I am using gnome with a dark theme.

Gnome Terminal #

I am still using the built in default gnome terminal, it just works. It does all the things that I need it to do. It supports transparency renders my fonts and allows me to highlight things well.

One Dark Theme

dotfiles #

You can find my dotfiles on github. Feel free to read through and take anything that you find useful. I would encourage you not to steal them, but to integrate the parts that you want into your own dotfiles. dotfiles are a very personal thing. They are an extension of ones fingertips designed for how you think and type.

zsh #

I use zsh as my default shell. I like to use it as my interactive shell. It works, and does a bit better with things like tab completion out of the box.

starship #

I use the starship prompt for my shell. It works well out of the box. It looks good and includes all of the information that I would ever need.

tmux #

As a team lead I am in and out of many projects per day, tmux allows me to get in and out of these files with super speed. I was using a mix of vscode and tmux in until October 2020. At this point I got moved development machines and pushed myself to use only the terminal. I felt that vscode was just getting slower and slower, and I was getting less benefit from it. Especially now that the lsp is a part of nvim.

neovim #

I really like the raw speed and customizability of neovim. You can see all the customization, and plugins I have added in my dotfiles.

OneBuddy Theme
pylsp
kedro-lsp

ipython #

I really like ipython, it lets me edit code in my code editor, then import it or run it quickly. Ipython gives me the right level of tooling. I don’t need markdown mixed in my code, I put those notes into docstrings, a readme, or wiki. When I need to see plots I just store them as png or html and view them in my browser.

I do a bit of customization to my ipython session that you can find in my dotfiles repo. I use a custom prompt and use rich formatting and tracebacks if rich is installed.

custom prompt
rich traceback

Links #

Incremental Versioned Datasets in Kedro

2021-07-08T11:12:25Z

Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work.

What is Kedro

👆 Unsure what kedro is? Check out this post.

How does our dataset change over time?? #

This was a question presented to me at work. We had some plots being produces as the output of our pipeline and the user wanted the ability to compare results over time. Luckily this was asked early in the project so we were able to proactively setup versioning on the right datasets.

To enable this all we needed to do now was to add versioned: true and we will be able to compare results over time. Yes kedro makes it that easy to setup.

set up a project #

Set up a new project just as usual. note I like using pipx for global cli packages. You can pick a specific version of kedro or opt for the latest while simply globally installing kedro and running kedro new is purely dependent on the last time you chose to update kedro.

pip install pipx
pipx run kedro new

cd versioned-partitioned-kedro-example
conda create -n versioned-partitioned-kedro-example python=3.8  -y
conda activate versioned-partitioned-kedro-example

pip install kedro
kedro install

git init
git add .
git commit -m "init project from pipx run kedro new"

I called my project versioned-partitioned-kedro-example. You can call your project whatever you like. If you try to use some special characters where they don’t belong, kedro will catch you. Under the hood, kedro is using a library called cookiecutter

⚠️ Please do not skip out on using a virtual environment. You may use whichever virtual environment tool you prefer, but please do not skip out. Wrecking a running project for learning is not fun.

update dependencies #

I popped open my dependencies, added kedro[pandas] and find-kedro. Since those are extra packages our example will require.

aiohttp
black==21.5b1
find-kedro
flake8>=3.7.9, <4.0
ipython
isort~=5.0
jupyter_client>=5.1, <7.0
jupyterlab~=3.0
jupyter~=1.0
kedro-telemetry~=0.1.0
kedro==0.17.4
kedro[pandas]
nbstripout~=0.4
pytest-cov~=2.5
pytest-mock>=1.7.1, <2.0
pytest~=6.2
requests
wheel>=0.35, <0.37

note I created find-kedro, and I like using it to create my pipeline object. Think of how pytest automatically picks up everything named test, find-kedro does the same thing for kedro. It picks up everything with node or pipeline in the name and creates pipelines out of it.

Install new dependencies #

After adding our additional dependencies to the requirements.in, we can tell kedro to install everything and compile the dependencies. Behind the scenes --build-reqs uses a library called pip-compile to create a requirements.txt file with hard pinned dependencies, which is ideal for creating reproducible projects. You and your future colleagues may not thank you for this, but they sure as heck won’t be cursing your name when they can’t get the project to run.

kedro install --build-reqs

git add .
git commit -m "added additional dependencies"

create a node #

For this example, we need a node to do much. This node will pass the cars.csv from a URL to a parquet file. I am going to use a lambda to build my identity function inline.

# pipelines/cars_nodes.py

from kedro.pipeline import node

nodes = []

nodes.append(
        node(
            func=lambda x:x,
            inputs='raw_cars',
            outputs='int_cars',
            name='create_int_cars',
            )
        )

🗒️ notefind-kedro' will automatically pick up these nodes for us after we set up our pipeline_registry.py`.

git add .
git commit -m "add create_int_cars node"

implement find-kedro #

Next, we need to tell kedro where our nodes are. This is where find-kedro comes in. Once we point to the directory where our modules of nodes/pipelines are, it creates the pipelines dictionary for us automatically. It will even separate each module into a pipeline and stitch them all into one default pipeline.

# pipeline_registry.py

"""Project pipelines."""
from typing import Dict
from pathlib import Path

from kedro.pipeline import Pipeline

from find_kedro import find_kedro


def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    Returns:
        A mapping from a pipeline name to a "Pipeline "object.
    """
    pipeline_dir = Path(__file__).parent / 'pipelines'
    return find_kedro(directory= pipeline_dir)

🗒️ This is very similar to the default ` pipeline_registry’except the last two lines.

git add .
git commit -m "implement find-kedro"

create a baseline catalog #

Once we have a pipeline setup, the kedro cli can automatically fill in missing catalog entries with MemoryDataSet’s. Thus, using the cli helps consistently scaffold the catalog and ensure we don’t end up with a typo in our dataset name.

kedro catalog create --pipeline cars_nodes

Kedro will kick out the following catalog file to base/catalog/cars_nodes.yml for us to get started with.

raw_cars:
  type: MemoryDataSet
int_cars:
  type: MemoryDataSet

🔥 use the kedro cli to fill in any missing datasets from the automatically catalog.

make a versioned dataset #

Kedro has scaffolded MemoryDataSet ’s for us. We will convert them to the appropriate dataset type and turn on versioning for our int layer, which is the first point we save in our environment.

raw_cars:
  type: pandas.CSVDataSet
  filepath: https://waylonwalker.com/cars.csv
int_cars:
  type: pandas.ParquetDataSet
  filepath: data/int_cars.parquet
  versioned: true

Commit your changes to the catalog.

git add .
git commit -m "create catalog"

run the pipeline #

Once we have the nodes and catalog setup, we can run the pipeline a few times to get some versioned data. Each time we run, it will save a new version inside the int_cars.parquet directory.

kedro run
kedro run
kedro run
kedro run
kedro run

🗒️ we put our data in the data directory. By default, this directory is included in the .gitignore and will not be picked up by git.

inspect the data #

Listing the files in data/int_cars.parquet shows that I now have five different datasets available. I can load old ones, but by default, kedro will load the latest one.

ls data/int_cars.parquet

2021-07-05T15.24.53.164Z
2021-07-05T15.29.56.144Z
2021-07-05T15.30.23.101Z
2021-07-05T15.30.26.555Z
2021-07-05T15.31.12.688Z

🗒️ kedro sets the version at the timestamp that the session starts. All datasets created within the same run will have the same version.

stack on an incremental dataset #

This is where things get interesting. Kedro comes with an incremental dataset that will load all of the files from a particular directory into a dictionary where the keys are the filename of the dataset. To load up all datasets into this dictionary all we need to do is add a new catalog entry that is a type: PartitionedDataSet, with a path pointing to the same place as the original, and a dataset type the same as the original.

int_cars_partitioned:
  type: PartitionedDataSet
  dataset: pandas.ParquetDataSet
  path: data/int_cars.parquet

catalog list #

Listing the catalog entries confirms that we have successfully added our new PartitionedDataSet.

In [17]: context.catalog.list()
Out[17]:
['raw_cars',
 'int_cars',
 'int_cars_partitioned',
 'parameters']

loading an incremental dataset #

Now we can easily load the datasets from every run we just did into a single dictionary, simply by running context.catalog.load('int_cars_incremental').

In [19]: context.catalog.load('int_cars_incremental')
2021-07-05 11:32:40,534 - kedro.io.data_catalog - INFO - Loading data from `int_cars_incremental` (IncrementalDataSet)...
Out[19]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.30.23.101Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.30.26.555Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.31.12.688Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2}

👆 notice that incremental datasets are all loaded for you, its a dict of filepath:dataset

stack on a partitioned dataset #

Let’s take a look at a similar type of dataset called PartitionedDataSet. We can add it to the catalog in a very similar way to how we added the IncrementalDataSet.

int_cars_incremental:
  type: IncrementalDataSet
  dataset: pandas.ParquetDataSet
  path: data/int_cars.parquet

loading a partitioned dataset #

Note that we get a dict with the same keys as before, but this time the values are a load function rather than loaded data. Partitioned datasets can be helpful if you are operating on datasets that take up more memory than you have available. In our case of coupling this with versioned datasets, its likely to grow quite large, so PartitionedDataSet ’s are likely a better option for this use.

In [18]: context.catalog.load('int_cars_partitioned')
2021-07-05 11:31:11,253 - kedro.io.data_catalog - INFO - Loading data from `int_cars_partitioned` (PartitionedDataSet)...
Out[18]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570820>>,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570850>>,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570910>>,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb15709a0>>}

incremental vs. partitioned #

IncrementalDataSet ’s and PartitionedDataSet ’s are very similar as they give you access to a whole directory of data that uses the same underlying dataset loader. The significant difference is whether you want your data pre-loaded or if you want to load and dispose of it as you iterate over it.

incremental loads the data
partitioned give a load function

creating nodes with partitioned datasets #

Let’s create a node with this PartitionedDataSet to collect stats on our dataset over time. This node does a dict comprehension to get the length of each version that we pulled.

def timeseries_partitioned(cars: Dict):
    return {k:len(car()) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_partitioned,
            inputs='int_cars_partitioned',
            outputs='int_cars_timeseries_partitioned',
            name='create_int_cars_timeseries_partitioned',
            )
        )

🗒️ note that inside of the dict comprehension car is a load function that we need to call.

creating nodes with incremental datasets #

Doing the same node with our IncrementalDataSet looks very similar, except this time car is loaded data inside of the dict comprehension, not a function that we need to call.

def timeseries_incremental(cars: Dict):
    return {k:len(car) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_incremental,
            inputs='int_cars_incremental',
            outputs='int_cars_timeseries_incremental',
            name='create_int_cars_timeseries_incremental',
            )
        )

More catalog entries #

After adding those nodes, we can add the catalog entries again with the command line. This will not overwrite any of the datasets we just created. It will only add to it.

kedro catalog create --pipeline cars_nodes

int_cars_timeseries_partitioned:
  type: MemoryDataSet
int_cars_timeseries_incremental:
  type: MemoryDataSet

int_cars_timeseries_partitioned:
  type: pickle.PickleDataSet
  filepath: data/int_cars_timeseries_partitioned.parquet
int_cars_timeseries_incremental:
  type: pickle.PickleDataSet
  filepath: data/int_cars_timeseries_incremental.parquet

Loading the new datasets #

Loading the two dtasets that we just created show that we have the ended up with the same result using both incremental and partitioned datasets. This result is a dictionary of filepaths mapped to the size of the dataset. Since the default filepaths are timestamps we could start doing some time series analysis to see how our dataset is changing over time.

In [32]: context.catalog.load('int_cars_timeseries_incremental')
2021-07-05 12:00:55,014 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_incremental` (PickleDataSet)...
Out[32]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
 '2021-07-05T16.43.43.088Z/int_cars.parquet': 32}

In [33]: context.catalog.load('int_cars_timeseries_partitioned')
2021-07-05 12:01:03,223 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_partitioned` (PickleDataSet)...
Out[33]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
 '2021-07-05T16.43.43.088Z/int_cars.parquet': 32,
 '2021-07-05T16.50.46.686Z/int_cars.parquet': 32}

Kedro - My Data Is Not A Table

☝️ I have a full article on creating datasets that are not tabular datasets using pickle.

cross posted to dev #

If you like this article give it some ❤️🦄🏷️ on dev https://dev.to/waylonwalker/incremental-versioned-datasets-in-kedro-3ajn

This post was primarily built live on https://twitch.tv/waylonwalker, give me a follow and join in the live show if that is something that interests you.

I Started Streaming on Twitch

2021-06-12T10:11:00Z

I recently started streaming on twitch.tv/waylonwalker and it’s been a blast so far.

python
kedro
Data Science
Data Engineering
webdev
digital gardening

Kedro Spaceflights #

It all started with kedro/issues/606, Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me.

Inspiration #

My introduction to twitch came from twitch.tv/theprimeagen. I watched him on YouTube, and then decided to drop into a stream. It was so fun to watch him live that I started following others in the science and tech category.

twitch.tv/teej_dv Brilliant neovim core dev, I learn a bunch about nvim every time I watch.
twitch.tv/cmgriffing Super Chill and engaging chat.
twitch.tv/cassidoo Fantastic discussion/chat.
twitch.tv/anthonywritescode Building the python ecosystem.
twitch.tv/kierisi Exploring data in r-studio, taking xgboost to Sliced.

Streaming What I Do #

After taking a jump in with the spaceflights tutorial I realized that it really was not that hard to do, and I can stream things that I am doing anyways. I will be streaming while I build open source kedro plugins, pipelines, and while I work on my website.

YouTube #

I started storing all the replay, and some clips on YouTube. If that is more of your thing be sure to check it out. To be completely honest the editing/ clips will likely come at a minimum, as they take me a lot more time to edit. I would greatly appreciate anyone who subscribes to help me get to the 100 subscriber baseline to get my custom url.

Come join me while I try to figure out what I am doing.

Deploying a kedro dataset to netlify

Here is a small clip from my latest

Upcoming Stream

2021-06-08T20:27:53Z

I'm no longer streaming

As much as I would really love to make streaming work, its really hard for

my family situation to make large blocks of time work for me.

https://stackoverflow.com/questions/16720541/python-string-replace-regular-expression

I am starting to stream 3 days per week, before I start work in the morning. These streams will likely be me just talking through things I am already doing.

Making DAGs do 🔮Magical Things | Open Source 🐍Python | kedro plugins | #

Science & Technology | Every Monday • 7:00 AM - 9:00 AM CDT

On Monday’s I am going to be working on open source packages/plugins for kedro.

kedro-diff
test kedro-diff on piplines with history
setup deploy pipeline
deply to pypi

🌱 Digital Gardening | Blogging with 🐍Python | Building 🔮Markata a static site generator in python for waylonwalker.com #

Science & Technology | Every Wednesday • 7:00 AM - 9:00 AM CDT

On Wednesday morning I will be working on my personal website and the static site generator that I built for it, Markata.

Features #

DONE ~~one_line_gif~~
DONE automate mp4’s
Fix big images
Fix Amp issues
speed up build times
measure plugin performance
speed up images.waylonwalker builds
build footer

Error Correction #

redirect detector
detect multiple H1
title too short (50-70 characters)
missing alt text

Building DAGS with 🐍 Python | 🛢 Writing data pipelines wih Kedro #

Science & Technology | Every Friday • 7:00 AM - 9:00 AM CDT

Friday will be all about building DAG’s or data related things in python. I might reach for something like tidy tuesday, makeover monday, or just do my own thing. I have a few ideas of things to do on my own first.

open pypi package names

Kedro Spaceflights - part 2 | Stream replay June 7, 2021

2021-06-07T16:15:04Z

This was my seconf time ever streaming on twitch.tv/waylonwalker, and I completely botched my mic 2x.

https://youtu.be/_7MwgKu-844

Links #

Notes to get started #

pipx run kedro new
cd project
python -m venv .venv
source .venv/bin/activate
pip install kedro
kedro install

Kedro Spaceflights - part 1 | Stream replay June 4, 2021

2021-06-04T16:15:04Z

This was my first time ever streaming on twitch.tv/waylonwalker. I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go.

https://youtu.be/Y07UBr9Ccjs

Kedro Spaceflights #

Notes #

pipx run kedro new
cd project
python -m venv .venv
source .venv/bin/activate
pip install kedro
kedro install

Comprehensive guide to creating kedro nodes

2021-06-03T21:30:35Z

The Kedro node is an essential part of the pipeline. It defines what catalog entries get passed in, what function gets ran, and the catalog entry to save the results under.

does this link work? #

https://waylonwalker.com/what-is-kedro/

👆 Unsure what kedro is? Check out this post.

The node function #

The node function is the most common and reccomended way to define kedro nodes. It is a function that constructs and returns Node objects for you.

Creating your first kedro node #

from kedro.pipeline import node

def identity(df):
    "a function that returns itself"
    return df

my_first_node = node(
   func=identity,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

function #

The func passed into node can be any callable that accepts the inputs yout have specified, and returns the correct output that you specify as your output.

any callable
a function you write
a function from a library
class constructor
lambda function
partial function
literally any callable

https://waylonwalker.com/kedro-inputs/

For more information on how kedro passes inputs into your functions check out this post

import pandas as pd
from kedro.pipeline import node

range_node = node(lambda: range(100), None, "range", name="range"),
dataframe_node = node(pd.DataFrame, "range", "df"),

Using a lambda as a function #

from kedro.pipeline import node

my_first_node = node(
   func=lambda x: x,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

my_first_node = node(
   func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'),
   inputs='raw_cars',
   output='int_cars',
   tags=['pri',]
   )

Using a partial function #

I prefer the simplicity of lambdas, but many others prefer using a partial as it can yield a better docstring, node name, and easier to reuse. I name all of my nodes anyways, never look at the docstring of a partial, and almost always only use them on a single node or set of nodes constructed together. So I prefer the readablility of the lambda most of the time, but if you like partials better, or need to assign it to a variable and reuse it, here are some partial examples.

from kedro.pipeline import node
from functools import partial, update_wrapper

def divide(array, by):
    return [i/by for i in array]

halfer = update_wrapper(partial(divide, by=2), divide)

my_halfer_node = node(
   func=halfer,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

We can achieve the same results that the partial gave with this lambda.

my_halfer_node = node(
   func=lambda array: divide(array, 2),
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

To further show the point that any callable can be out node’s func, I have made a partial from the pd.DataFrame class that has column names pre populated.

from kedro.pipeline import node
import pandas as pd
from functools import partial, update_wrapper

MyDataFrame = update_wrapper(partial(pd.DataFrame, columns=["mycol"]), pd.DataFrame)

range_node = node(lambda: range(100), None, "range", name="range"),
dataframe_node = node(MyDataFrame, "range", "df"),

inputs/outputs #

kedro inputs and outputs can be None, a catalog entry, or a dict mapping the functions keyword arguments to catalog entries. Catalog entries are always represented as a string matching the key of the catalog entry you want to load.

None #

no catalog entries

Sometimes you may want to have a node without any inputs. This node may be used to generate some data from scratch, or fetch some data that does not have an existing DataSet type setup. DataSets are easy to setup, simply fork one of kedros built in ones and use it, but for one or two nodes the setup may not be worth it.

from kedro.pipeline import node

random_100_node = node(
   func=lambda: random.sample(range(0, 100), 100),
   inputs=None,
   output='random_100',
   name='create_random_100',
   )

str #

one catalog entry

This is by far the most common input that you will use. This will simply tell kedro what dataset to load behind the scenes and passin to the function that you provide.

from kedro.pipeline import node

random_100_node = node(
   func=lambda random_100: [x**2 for x in random_100],
   inputs='random_100',
   output='random_squared',
   name='create_random_squared',
   )

Note, I am using a lot of lambdas here for simplicity as each function so far is a simple one-liner. These could also be a regular function if you are uncomfortable with lambdas.

list #

several catalog entries, passed in by position

In order to start passing in more than one DataSet into a kedro node you need to use a list or dictionary as the input. Using a list is convenient for a small number of inputs.

from kedro.pipeline import node

random_100_node = node(
   func=lambda random_100, random_squared: list(zip(random_100, random_squared)
   inputs=['random_100', 'random_squared'],
   output='random_join',
   name='create_random_join',
   )

dict #

several catalog entries, passed in by name

kedro will unpack dictionaries into your function if you pass in a dictionary. In code review I start suggesting converting from a list to dict at 3 and require it above 5. It gets way too hard to refactor and move things while keeping track of the order of really long sets of inputs. Passing them in by name, as a dictionary, makes it such that order no longer matters.

from kedro.pipeline import node

random_100_node = node(
   func=lambda x, y: list(zip(x, y)),
   inputs={'x': 'random_100', 'y':'random_squared'},
   output='random_join',
   name='create_random_join',
   )

Switch from list to dict inputs between 3 and five inputs to improve readability and prevent ordering mistakes.

tags #

Tags provide an easy way to add a label nodes for something to interact with them. This may be a node that we want a plugin to modify or a set of nodes that we want quick access to during development.

Tags are always passed in as a list of strings. They must be a 1-d data structure. You may create that data structure however you want, but its still just a list of strings. Below I have set a global variable TAGS that I want to apply to every node within a given module, then I splat it into every nodes tags. This lets me easily apply a whole set of tags to an entire module of nodes. I can easily modify that list of nodes if I wanted to, but its acually rare that I do.

TAGS = ['cars']

my_first_node = node(
   func=identity,
   inputs='raw_cars',
   output='int_cars',
   tags=['int', *TAGS]
   )

# get a pipeline of one node
one_node = pipeline.only_nodes_with_tags('int')

# get a pipeline of a few nodes
one_node = pipeline.only_nodes_with_tags('raw', 'int')

name #

The name attribute is simple, it’s the name of the node. Later you can use the name to find the node or all nodes named a particular way. This name will also show up in the logs provided by kedro or a plugin so naming things well makes everything much easier to read.

Consistent naming makes it easier to do things like extracting nodes out of a pipeline, running them, and making pipelines from them.

# get a pipeline of one node
one_node = pipeline.only_nodes('a_raw_cars')

# get a pipeline of a few nodes
one_node = pipeline.only_nodes('a_raw_cars', 'a_raw_trains')

# create a pipeline by nodes that contain 'raw' in the name
raw_nodes = Pipeline([node for node in pipeline.nodes if 'raw' in node.name])

⚠️ filtering by name requires a bit o diligence and consistency by the team, it’s a fantastic way to grab some nodes adhoc, but for production you probably want something a bit more robust.

Using PartitionedDataSets #

def timeseries_partitioned(cars: Dict):
    return {k:len(car()) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_partitioned,
            inputs='int_cars_partitioned',
            outputs='int_cars_timeseries_partitioned',
            name='create_int_cars_timeseries_partitioned',
            )
        )

Using IncrementalDataSets #

def timeseries_incremental(cars: Dict):
    return {k:len(car) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_incremental,
            inputs='int_cars_incremental',
            outputs='int_cars_timeseries_incremental',
            name='create_int_cars_timeseries_incremental',
            )
        )

https://waylonwalker.com/kedro-incremental-versioned-datasets/

for more information in incremental and partitioned datasets see this post

This may be a separate post on the pipeline object

Execution order #

Execution order is set by resolving catalog dependencies. I imagine kedro taking a razer blade tool and slicing out all nodes with completed dependencies, throwing those in a bag drawing them out one by one randomly, then when the bag is full it slices more out and repeats until there are no more nodes. This idea of randomness can be really maddening when there are two separate issues on your pipeline. If you are debugging an error in your pipeline run the erroring node by itself.

🔥 If you are debugging an error in your pipeline run the erroring node by itself.

Running specific nodes #

the power of the DAG

One of the greatest benefits of using kedro is that it gives you a Pipeline object that is a DAG. This is a powerful object that can quickly pull decide a set of nodes to run when you tell it to run to or from somewhere. Somewhere being either a catalog entry or node.

DAG (directed acyclic graph) is a fancy word for a data structure that may branch and join, but may not loop

by name #

by tag #

to_inputs #

to_outputs #

from_inputs #

from outputs #

Creating pypi-list with kedro

2021-05-23T22:02:16Z

I had an idea come to me via twitter. Short one word name packages are becoming hard to find on pypi. Short one word readable package names that are not a play on words are easy to remember, easy to spell correctly, and quick to type out.

Simple index #

I started with the simple index. Pypi provides a single page listing to every single package hosted on pypi via the simple-index

Using Kedro In Scripts

2021-05-23T11:00:44Z

With the latest releases of kedro 0.17.x, it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects.

New to Kedro #

What is Kedro

If your just learning about kedro check out this post walking through it

No More Rabbit Hole of Errors #

as of 0.17.2

I’ve tried to do this in kedro 0.16.x, and it turned into a rabbit hole of errors. First kedro needed a conf directory, if you tried to fake one in it would then ask for logging setup. These errors just kept coming to the point it wasnt worth doing and I might as well use a proper template for real projects and stick to simple function calls for things that are not a kedro project.

Kedro in a script #

To get kedro running, you will need a pipeline, catalog, and runner at a minimum. Those who have used kedro before the pipeline will look very similar to what you are familiar with, but the catalog will not be loaded from yaml and you will need to bring your own runner.

from kedro.pipeline import Pipeline, node
from kedro.io import DataCatalog
from kedro.runner.sequential_runner import SequentialRunner


# additional datasets you want to use
from kedro.extras.datasets.pandas.csv_dataset import CSVDataSet
from kedro.extras.datasets.pandas.parquet_dataset import ParquetDataSet

# the sequential runner is the simplest. It runs one node at a time.
runner = SequentialRunner()

# this is a super simple example pipeline
pipeline = Pipeline(
    [
        node(lambda: range(100), None, "range"),
        node(lambda x: [i ** 2 for i in x], "range", "range**2"),
        node(lambda x: [i for i in x if i > 5000], "range**2", "range>5k"),
        node(lambda x: x[:5], "range>5k", "range>5k-head"),
        node(lambda x: sum(x) / len(x), "range>5k", "range>5k-mean"),
    ]
)

# to get up and running, you can use an empty catalog
catalog = DataCatalog()

runner.run(pipeline, catalog)

👆 Above is the minimal setup to get a kedro pipeline running

more practically #

More often, your kedro pipelines are going to use a function rather than a lambda, and pandas DataFrames.

def clean_columns(df: pd.DataFrame):
    df.columns = [col.lower().strip() for col in df.columns]

pipeline = Pipeline(
    [
        node(clean_columns, "raw_data", "clean_columns", name="create_clean_columns"),
    ]
)

catalog = DataCatalog(
    {
        "raw_data": ParquetDataSet(filepath=f"data/raw_data.parquet")
        "clean_columns": ParquetDataSet(filepath=f"data/clean_columns.parquet")
    }
)

One single node pipeline to get you started

Semi-automatic catalog #

For some reason, when I tried to use the DataCatalogWithDefault it did not pick up my datasets right. I suspect this has something to do with not setting up a proper session, so this is what I did in a pinch to get that catalog goodness for my DataFrames without setting up each one manually.

catalog = DataCatalog(
    {
        name: ParquetDataSet(filepath=f"data/{name}.parquet")
        for name in pipeline.all_outputs()
    }
)

⚠ If all of your datasets are pandas dataframes

For the example above that does not use DataFrames, I would pickle all of my outputs to enable re-loading them later.

catalog = DataCatalog(
    {
        name: PickleDataSet(filepath=f"data/{name}.pkl")
        for name in pipeline.all_outputs()
    }
)

🔥 for use with non-pandas datasets

Logging #

Once you explicitly add datasets, kedro will start logging when it’s loading, running, or saving each node. Things will begin to look a bit more familiar to anyone who has used kedro before.

ww3 ↪main ©kedro-in-scripts v3.8.8 ipython
❯ runner.run(pipeline, catalog)
2021-04-18 09:30:58,099 - kedro.pipeline.node - INFO - Running node: <lambda>(None) -> [range]
2021-04-18 09:30:58,100 - kedro.io.data_catalog - INFO - Saving data to `range` (PickleDataSet)...
2021-04-18 09:30:58,104 - kedro.runner.sequential_runner - INFO - Completed 1 out of 5 tasks
2021-04-18 09:30:58,105 - kedro.io.data_catalog - INFO - Loading data from `range` (PickleDataSet)...
2021-04-18 09:30:58,105 - kedro.pipeline.node - INFO - Running node: <lambda>([range]) -> [range**2]
2021-04-18 09:30:58,105 - kedro.io.data_catalog - INFO - Saving data to `range**2` (PickleDataSet)...
2021-04-18 09:30:58,111 - kedro.runner.sequential_runner - INFO - Completed 2 out of 5 tasks
2021-04-18 09:30:58,111 - kedro.io.data_catalog - INFO - Loading data from `range**2` (PickleDataSet)...
2021-04-18 09:30:58,112 - kedro.pipeline.node - INFO - Running node: <lambda>([range**2]) -> [range>5k]
2021-04-18 09:30:58,112 - kedro.io.data_catalog - INFO - Saving data to `range>5k` (PickleDataSet)...
2021-04-18 09:30:58,115 - kedro.runner.sequential_runner - INFO - Completed 3 out of 5 tasks
2021-04-18 09:30:58,115 - kedro.io.data_catalog - INFO - Loading data from `range>5k` (PickleDataSet)...
2021-04-18 09:30:58,115 - kedro.pipeline.node - INFO - Running node: <lambda>([range>5k]) -> [range>5k-mean]
2021-04-18 09:30:58,115 - kedro.io.data_catalog - INFO - Saving data to `range>5k-mean` (PickleDataSet)...
2021-04-18 09:30:58,118 - kedro.runner.sequential_runner - INFO - Completed 4 out of 5 tasks
2021-04-18 09:30:58,119 - kedro.io.data_catalog - INFO - Loading data from `range>5k` (PickleDataSet)...
2021-04-18 09:30:58,119 - kedro.pipeline.node - INFO - Running node: <lambda>([range>5k]) -> [range>5k-head]
2021-04-18 09:30:58,119 - kedro.io.data_catalog - INFO - Saving data to `range>5k-head` (PickleDataSet)...
2021-04-18 09:30:58,122 - kedro.runner.sequential_runner - INFO - Completed 5 out of 5 tasks
2021-04-18 09:30:58,122 - kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully.

Kedro Viz #

I was not able to quickly get kedro viz up and running for my use case. If you really wanted to you could start modifying their format_pipelines_data function in server.py. Or you could render a new template and put your pipeline there for viz purposes.

It’s possible, but might as well stick to the template

cli #

For something that I would be using this on, I am probably not going to put much effort into the cli as it is not likely something that we will have a team of developers interacting with constantly. I would just put together the minimum necessary to run my application how I need.

if __name__ == "__main__":
    import sys

    if '--skip-raw' in sys.argv:
        runner.run(pipeline.from_inputs('range**2'), catalog)
    else:
        runner.run(pipeline, catalog)

Keeping it simple

If I want to go down the route of having a full cli built out I am probably going to use the full kedro template, or something very similar.

It’s a bit Rough #

While I might use this in production somewhere, it will be inside of some other not kedro application. I will still be using something quite similar to their template for my pipelining projects. It misses some excellent things that bring me to kedro like hooks, plugins, credentials, catalog, logging config, cli, and viz.

Silence Kedro Logs

2021-05-20T13:13:38Z

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output.

Silence a Python log #

First, how does one silence a python log? Python loggers can be retrieved by the logging module’s getLogger function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways.

python logging levels #

Level	Numeric value
CRITICAL	50
ERROR	40
WARNING	30
INFO	20
DEBUG	10
NOTSET	0

Get or Create a logger #

Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in.

logger = logging.getLogger('kedro')

🔥 If a logger doesn’t exist under the passed in name, it will create one for you.

Set Level #

Once we get the logger, we need to silence it by setting the log level. Typically it’s not appropriate to completely turn off loggers as you would still want information in the case of a complete failure. If you are building a cli such as one that prints out the pipelines to the console, you may not want to see logs that happen during regular operation as this would make it more challenging to integrate with other shell applications.

logger.setLevel(logging.ERROR)

⚠ Be sure to leave some logging left. After the point of error, you are not going to get a clean output anyways. So let the user see what happened.

It is possible to set the log level before kedro even registers the logger, if there is no logger currently setup under getLogger, it will create one.

Silent all kedro loggers #

As of kedro==0.17.3 this function covers every logger issued by kedro. I generated this list of known_kedro_loggers by looking through their codebase and filling in a few others I found by running it.

def silent_loggers() -> None:
    """All logs need to be silent in order for a clean kedro diff output."""
    known_kedro_loggers = [
        "ProfileTimeTransformer",
        "hooks_handler",
        "kedro.__init__",
        "kedro",
        "kedro.config",
        "kedro.config.config",
        "kedro.extras.decorators.memory_profiler",
        "kedro.framework.cli",
        "kedro.framework.session.session",
        "kedro.framework.session.store",
        "kedro.framework.session",
        "kedro.io.cached_dataset",
        "kedro.io.data_catalog",
        "kedro.io",
        "kedro.journal",
        "kedro.pipeline",
        "kedro.pipeline.decorators",
        "kedro.pipeline.node",
        "kedro.pipeline.pipeline",
        "kedro.runner",
        "kedro.runner.runner",
        "kedro.versioning.journal",
        "py4",
    ]
    for logger in [
        *known_kedro_loggers,
        *list(logging.root.manager.loggerDict.keys()),  # type: ignore
    ]:
        logging.getLogger(logger).setLevel(logging.ERROR)

This function comes right from a plugin I am currently working on kedro-diff. Check it out, give it a star, and watch it for release.

What is Kedro

Not familiar with kedro, check out this article to see what it’s all about.

Master the log #

Python logs can seem super confusing at first, understanding how to get a logger and set its level are the first steps to mastering it.

Vim Fugitive

2021-05-08T22:51:53Z

:G
:G status
:G commit
:G add %
:Gdiff
:G push
:Glog

Add current file and commit with diff in a split #

function! s:GitAdd()
    exe "G add %"
    exe "G diff --staged"
    exe "only"
    exe "G commit"
endfunction
:command! GitAdd :call s:GitAdd()
nnoremap gic :GitAdd<CR>

:on[ly] #

C-W o

:on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default.

C-I C-O #

cycle through the jumplist

This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize C-i and C-o, I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific commit, then no way to get back where I was in the log.

C-i jump

:jump[s] #

show the jumplist

The jumplist is sorted Oldest to newest

:Telescope jumplist #

When navigating the jumplist with :Telescope jumplist, it will add a new entry to the jumplist and let you get back to where you were with a C-O.

:Telescope jumplist adds to the jumplist

C-W J / C-W L #

:G log #

:G log
:G log -p
:Glog

Ggrep #

:Ggrep python **/*md

Unlike :vim[grep] you don’t need to specify a file glob.

:Ggrep python

Staging Hunks #

In the file you want to stage hunks of run :Gdiff. Then use vim’s diff commands (dp, do, :diffput, :diffget) to move the hunk between the stage/index. Then write the file to stage.

:Gdiff
dp
:w
:Gcommit

Custom Kedro Logger

2021-05-02T09:45:22Z

DRAFT -

formatters:
    mine:
        format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s - %(me)s"

handlers:

    mine_handler:
        class: logging.StreamHandler
        level: INFO
        formatter: mine
        stream: ext://sys.stdout

loggers:
    me:
        level: DEBUG
        handlers: [mine_handler]

root:
    level: INFO
    handlers: [console, info_file_handler, error_file_handler]

kedro replit

2021-04-15T12:37:29Z

I am trying to see what an embeded replit

Kedro pipeline_registry.py

2021-03-20T00:00:00Z

With the realease of kedro==0.17.2 came a new module in the project template pipeline_registry.py. Here are some notes that I learned while playing with this new module.

migrating to `pipeline_registry.py` #

create a src/<package-name>/pipeline_registry.py file create a
register_pipelines function in pipeline_registry.py that mirrors the
register_pipelines method from your hooks.py module do not bring the
hook_impl decorator remove register_pipelines method on your ProjectHooks
class

You should now have something that looks like this in your src/<package-name>/pipeline_registry.py.

"""Project pipelines."""
from typing import Dict

from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    Returns: A mapping from a pipeline name to a ``Pipeline`` object.
    """
    return {"__default__": Pipeline([])}

pipeline_registry only works in kedro>=0.17.2

Conflict Resolution #

What happens If I register pipelines in both places

I was not able to find any official documentation on how conflict resolution worked so I stepped into a project and added to both my hooks.py and pipeline_registry.py file. I noticed that it would pick up pipelines from both modules, but pipelines from hooks.py always take precedence. The entire duplicate pipeline will be over written by the one from hooks.py.

kedro automatically merges pipelines from both hooks.py takes precedence

Ready to update #

In my experience there were no issues upgrading from 0.17.1 to 0.17.2. I would reccomend only having one register_pipelines so decide to migrate to the new pipeline_registry.py or keep it in your hooks.py, but both is only going to lead to confusion.

https://twitter.com/datajoely/status/1375193511264456705

Minimal Kedro Pipeline

2021-01-20T00:00:00Z

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It’s only a total of 35 lines of python, 8 in setup.py and 27 in mini_kedro_pipeline.py.

📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner.

Minimal Kedro Pipeline #

I have everything for this post hosted in this gihub repo, you can fork it, clone it, or just follow along.

Installation #

pip install git+https://github.com/WaylonWalker/mini-kedro-pipeline

Caveats #

This repo represents the minimal amount of structure to build a kedro pipeline that can be shared across projects. Its installable, and drops right into your hooks.py or run.py modules. It is not a runnable pipeline. At this point I think the config loader requires to have a logging config file.

This is a sharable pipeline that can be used across many different projects.

Usage #

# hooks.py

import mini_kedro_project as mkp

class ProjectHooks:
    @hook_impl
    def register_pipelines(self) -> Dict[str, Pipeline]:
        """Register the project's pipeline.

        Returns:
            A mapping from a pipeline name to a ``Pipeline`` object.

        """

        return {"__default__": Pipeline([]), "mkp": mkp.pipeline}

Implemantation #

This builds on another post that I made about creating the minimal python package. I am not sure if it should be called a package, it’s a module, but what do you call it after you build it and host it on pypi?

[[ minimal-python-package ]]

Directory structure #

.
├── .gitignore
├── README.md
├── setup.py
└── my_pipeline.py

setup.py #

This is a very minimal setup.py. This is enough to get you started with a package that you can share across your team. In practice, there is a bit more that you might want to include as your project grows.

from setuptools import setup

setup(
    name="MiniKedroPipeline",
    version="0.1.0",
    py_modules=["mini_kedro_pipeline"],
    install_requires=["kedro"],
)

mini_kedro_pipeline.py #

The mini kedro pipeline looks like any set of nodes in your project. Many projects will separate nodes and functions, I prefer to keep them close together. The default recommendation is also to have a create_pipelines function that returns the pipeline.

This pattern creates a singleton, if you were to reference the same pipeline in multiple places within the same running interpreter and modify the one you would run into issues. I don’t foresee myself running into this issue, but maybe as more features become available I will change my mind.

"""
An example of a minimal kedro pipeline project
"""
from kedro.pipeline import Pipeline, node

__version__ = "0.1.0"
__author__ = "Waylon S. Walker"

nodes = []


def create_data():
    "creates a dictionary of sample data"
    return {"beans": range(10)}


nodes.append(node(create_data, None, "raw_data", name="create_raw_data"))


def mult_data(data):
    "multiplies each record of each item by 100"
    return {item: [i * 100 for i in data[item]] for item in data}


nodes.append(node(mult_data, "raw_data", "mult_data", name="create_mult_data"))

pipeline = Pipeline(nodes)

Kedro in scripts #

Using Kedro In Scripts

If you enjoyed this one check out this companion article where I build a fully runnable kedro project in a single script.

Go forth and share your pipelines across projects. Let me know, do you share pipelines or catalogs across projects?

Kedro Dependency Management

2021-01-19T00:00:00Z

Docs #

https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/01_dependencies.html?highlight=install

pip-tools #

pip-compile #

requirements #

requirements.in
requirements.txt

Kedro - My Data Is Not A Table

2021-01-14T00:00:00Z

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.

DataFrames are the heart of most pipelines #

These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types.

What is Kedro

unfamiliar with kedro, check out this post

Sometimes datasets are not tables #

There are times when our data doesn’t fit nicely into a DataFrame. Lucky for us Kedro has pickle support out of the box. Pickle is a way to store any python object to disk. Beware that pickle files coming from an unknown source can run malicous code and are considered unsafe. For the most part though when you read and write your own pickle files they are a good tool to consider.

See more about pickle from python.org.

Cataloging Pickle #

I may have a dictionary that describes some cars.

{
  'truck-012-abc': {
    'type': 'truck'
    'sales': [12, 2, 3, 4, 8]
    'weight': 9024,
    'accesories': ['leather', 'audio-1']
}

In the catalog we will simply set the type as pickle.PickleDataSet and give it a filepath.

cars:
  filepath: data/cars.pkl
  type: pickle.PickleDataSet

This filepath does not have to be on the local filesystem it can be on the cloud thanks to how kedro utilizes fsspec for each of its datasets.

Loading the dataset #

The benefit of cataloging this dataset compared to leaving it as a MemoryDataSet is that you can easily load this data back into memory for further development or debugging without running any of the pipeline.

catalog.load('cars')

Testing Data Pipelines

2020-11-16T06:00:00Z

Lint/Format/Doc

black
flake8
interrogate
mypy

Pipeline Assertions

pipeline constructs
pipeline as expected nodes
pipeline has minimum nodes
test minimum tags
test alternate tags

Catalog Assertions

test catalog follows naming structure

Node Tests

test function does the correct operations on test data

Great Expectations

reasons-to-kedro

2020-11-01T05:00:00Z

There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines

What is Kedro

Starter Template #

Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood.

conda create -n my-new-project -y python=3.8
kedro new
kedro install
kedro run

Create New Kedro Project

read more about how to start your first kedro project here

Collaboration #

Kedro provides many tools that help teams collaborate on a single codebase. While writing monolithic scripts it can be easy to pin yourself in a corner where it is difficult to have multiple people making changes to the notebook/script at the same time. Kedro helps guide your team to break your project down into small pieces that different members of the team can work on in parallel.

sharable catalog #

Kedro makes it easy to collaborate with members who aren’t even working on the pipeline. I often see team members who want to investigate datasets from different points in the pipeline. Kedro makes it really easy for them to load it into python.

for python users

Share catalog entries with folks doing EDA.

catalog.load('main_table')

for non-python users

For those who may not be using python, we can easily kick out a CSV version of that main_table that they can get from s3 or your cloud storage solution of choice.

master_table:
  type: pandas.CSVDataSet
  filepath: s3://bucket/data/03_primary/master_table.csv
  layer: primary

for the SQL folks

We aren’t even constrained to those who only use python or excel, we can kick out any kind of dataset that python can output. Kedro even comes with many DataSet types out of the box so that we don’t have to write any read/write code.

master_table:
  type: SQLTableDataSet
  table_name: master_table
  credentials: postgres

small nodes over monolithic scripts #

As I said before single notebooks/scripts are really hard to collaborate on. I have seen Data Engineers sitting idle waiting to get their changes manually added into the master notebook. When you find yourself in this situation, find a better solution. It’s time to break things down into individual modules and utilize a version control system that can automatically merge changes in.

Kedro encourages the use of git version control and storing all node functions inside of modules while still making it really easy to load data into a notebook/shell and start trying out new things.

No More read and write code #

As I said earlier kedro comes with datasets for the most popular output formats. It is also backed by a really amazing library called fsspec, this library makes the filesystem that you are storing agnostic to how you write to it. This means that the kedro library utilizes fsspec under the hood and writes to the file as if it was to disk, but based on the prefix to the file it may actually be writing to the local filesystem, gcp, azure blob, or s3.

custom DataSets

If kedro does not have a DataSet for the format that you need to read or write you can easily create your own custom DataSet all you need to do is inherit from kedro.io.AbstractDataSet and create methods for __init__, _load, _save, _exists, and _describe.

Check out this example from their docs. I removed the docstrings for brevity, you can see the entire DataSet in their docs.

The complete example all in one was only available in an older version, more up to date docs have a good writeup that walks through everything separately.

from os.path import isfile
from typing import Any, Union, Dict

import pandas as pd

from kedro.io import AbstractDataSet

class ExcelLocalDataSet(AbstractDataSet):

    def _describe(self) -> Dict[str, Any]:
        return dict(filepath=self._filepath,
                    engine=self._engine,
                    load_args=self._load_args,
                    save_args=self._save_args)

    def __init__(
        self,
        filepath: str,
        engine: str = "xlsxwriter",
        load_args: Dict[str, Any] = None,
        save_args: Dict[str, Any] = None,
    ) -> None:

        self._filepath = filepath
        default_save_args = {}
        default_load_args = {"engine": "xlrd"}

        self._load_args = {**default_load_args, **load_args} \
            if load_args is not None else default_load_args
        self._save_args = {**default_save_args, **save_args} \
            if save_args is not None else default_save_args
        self._engine = engine

    def _load(self) -> Union[pd.DataFrame, Dict[str, pd.DataFrame]]:
        return pd.read_excel(self._filepath, **self._load_args)

    def _save(self, data: pd.DataFrame) -> None:
        writer = pd.ExcelWriter(self._filepath, engine=self._engine)
        data.to_excel(writer, **self._save_args)
        writer.save()

    def _exists(self) -> bool:
        return isfile(self._filepath)

Execution order is taken care of #

As you build up complex pipelines containing 10’s or 100’s of nodes it becomes difficult to splice in new nodes/steps without messing up or a framework to help. Kedro simply needs a set of nodes that each takes in catalog entries as input and output to catalog entries and it will figure out the order for you.

These nodes can be made for one-off purposes, take in functions from reusable libraries, and even be dynamically generated from a configuration. There is no need to worry about hand curating the execution order, that’s all taken care of.

Easily slice up a pipeline #

Since kedro is a DAG that takes in a pile of nodes and figures out all of the dependencies for you it knows a lot about your pipeline. You can slice it up to only the specific pieces that you need.

# single nodes
pipeline.only_nodes("node1")

# single nodes and all of thier dependencies
pipeline.to_nodes("node1", "node2")

# from a dataset to all of its dependants
pipeline.from_inputs("dataset1", "dataset2")

# to a an outputs with all of its dependencies
pipeline.to_outputs("dataset6", "dataset7")

plugins/hooks #

Creating your own modifications to how kedro behaves is made really simple through the use of hooks. There are several hooks that happen at different points in the kedro lifecycle. For instance, you can hook in before pipeline run or after pipeline run to do whatever your project needs.

creating the kedro-preflight hook

pip install plugin #

There is a growing list of plugins available from pypi that is only a pip install away. Most of them are on GitHub and tagged as a kedro-plugin topic.

flexible cli #

In the end, you have a cli for your project that can run your pipeline in all sorts of cool ways since it knows about each node’s dependencies. This makes running and scheduling production a breeze.

# single nodes
kedro run --node node1

# single nodes and all of their dependencies
kedro run --to-nodes node1,node2

# from a dataset to all of its dependents
kedro run --from-inputs dataset1,dataset2

# to outputs with all of their dependencies
kedro run --to-outputs dataset6,dataset7

Try it out #

Hopefully this post gave you the inspiration to get started today, if it did pip install kedro and run kedro new to try it out.

Reasons to Kedro

2020-10-31T05:00:00Z

Reasons to Kedro #

collaboration
- Sharable catalog
- small nodes over monolithic notebooks
catalog
- easily load anything without needing to run
- No need to write read/write code
pipeline
- No need to keep execution order in your head
- easily run a slice of a pipeline
plugins
- pip install
- make your own
hooks
flexible expandable cli

Reasons Not to Kedro #

Already utilizing another DAG framework
Data is not in a widely supported format
Micro short-lived project
Large Project / Deadline
- Use a lower profile project to learn first
Team not willing to change
Need minimal dependencies
God Project - kedro owns everything??

What's New in Kedro 0.16.6

2020-10-25T05:00:00Z

Kedro 0.16.6 is out! Let’s take a look through the release notes

Deployment Docs #

This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible.

Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies.

Argo, Prefect, and Kubeflow have an interesting technique where they translate the pipeline and its dependencies from kedro to their language.

Batch uses the aws cli to submit jobs, one node per job, and listen for them to complete. It will submit all nodes with completed dependencies at once, meaning that we can get some massive parallelization.

I did a quick and dirty test of one of these by simulating the technique in a bash script and saw a 40 hr pipeline finish in about 1 hour. I am excited to get this working in my production workflow.

Spaceflight starter #

They have officially added the spaceflights pipeline as a starter. I have not yet had a chance to try this out, but I anticipate this will be a great pipeline to teach from as it is a bit more complex than the iris pipeline.

I tell folks all the time the best way to learn something new like kedro is to practice, practice, practice. Having this at their fingertips will give an easy way to fire up a pipeline that is ok to break try out some new ideas and leave it. I see this helping me testing out plugins on more complex pipelines and writing blog posts with examples that readers can more easily follow along with.

practice
testing
blog posts
tutorials

Better Error Messages #

Shameless pug, my PR landed in kedro, but fell off of the release.md in a merge issue. I was frustrated working on large projects when the wrong arguments were passed into a node and it did not give you any information in the Error to figure out where the error was thrown. Now it will at least give the name of the function that caused the error. This was a simple fix as all of the information was already there.

Overall #

Love the progress that the project is seeing and the focus on all of the deployment options.

Thanks for supporting contributions #

Deepyaman Datta, Bhavya Merchant, Lovkush Agarwal, Varun Krishna S, Sebastian Bertoli, noklam, Daniel Petti, Waylon Walker

A brain dump of stories

2020-10-21T05:00:00Z

I started making stories as kind of a brain dump a few times per day and posting them to [LinkedIn](https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/). Here are the last 11 days of stories.

I store all the stories on my website with the hopes of doing something with them on my own platform eventually. For now it makes it easy to make these posts.

cd static/stories
ls | xargs -I {} echo '![](https://waylonwalker.com/stories/{})'

Stories 10-10-2020 - 10-21-2020 #

Kedro Basics

2020-08-08T05:00:00Z

Learn Kedro in 5 days

Day 0 Setup #

vm
install
python
editor

Day 1 #

kedro new
kedro viz

Day 2 #

catalog
filter catalog
load data
fsspec

Day 3 #

pipeline
nodes

Day 4 #

filter pipeline
run partial pipeline

Day 5 #

kedro docker
GitHub Actions

Advanced Kedro #

hooks
custom datasets
modular pipelines

What's New in Kedro 0.16.4

2020-08-01T05:00:00Z

If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks.

## Major features and improvements

* Enabled auto-discovery of hooks implementations coming from installed plugins.

This one comes a bit surprising as it was just casually mentioned in #435

Think pytest #

As mentioned in #435 this is the model that pytest uses. Not all plugins automatically start doing things right out of the box but require a CLI argument.

simplicity #

It feels a bit crazy that simply installing a package will change the way that your pipeline gets executed. I do like that it requires just a bit less reaching into the framework stuff for the average user. Most folks will be able to write in the catalog and nodes without much change to the rest of the project.

Implementation #

Reading through the docs, they show us that we can make our hooks automatically register by adding a kedro.hooks endpoint that points to a singleton instance of our hook.

from the docs

setup(
    ...
    entry_points={"kedro.hooks": ["plugin_name = plugin_name.plugin:hooks"]},
)

import logging

from kedro.framework.hooks import hook_impl

class MyHooks:
    @hook_impl
    def after_catalog_created(self, catalog): # pylint: disable=unused-argument
        logging.info("Reached after_catalog_created hook")

hooks = MyHooks()

Careful with the singletons #

hook authors beware

I will be a bit cautious before installing a plugin that is automatically registered. I know its not a common pattern, but if you were to leverage any part of two kedro projects at the same time, and project-specific data was stored in the instance of the hook it will likely be broken.

As long as the hook doesn’t store data on the instance you will be ok. Hooks like what they have in the examples will be ok. They generally just take some information from the lifecycle arguments and do something at their prescribed lifecycle point.

Many of the hooks I am seeing in the wild are already more complicated and require the hooks author to utilize an __init__ method and store data on the instance. If you were to do this on two pipelines simultaneously it would break.

Can my hook be auto-discovered #

If your hook doesn’t include a __init__ method its a fairly easy yes, otherwise be aware of the potential dangers of passing singleton on to your users.

Use Virtual environments #

Whatever virtual environment manager you use, it is more important than ever to make sure you DO NOT install plugins in your global environment. Generally, you should always run projects even toys or tests in a virtual environemnt.

I use conda

conda create -n my-sample-env python=3.8 -y

Overall #

I think this is a really interesting direction for the project to go to. Hooks are still really early. The implementation is good, but I foresee us getting some more functionality that may require us to rely on the __init__ method a little less. I think there are going to be some really cool hooks that can leverage the simplicity of auto-discoverability.

Kedro Catalog

2020-07-24T05:00:00Z

I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking.

Process #

metadata will be attached to the dataset object under a .metadata attribute
metadata will be updated after_node_run
metadata will be empty until a pipeline is ran with the hook on
optionally a function to add metadata will be added
metadata will be stored in a file next to the filepath
meta

Problems This Hook Should solve #

what datasets have a columns with sales in the name
what datasets were updated after last tuesday
which pipeline node created this dataset
how many rows are in this dataset (without reloading all datasets)

implementation details #

metadata will be attached to each dataset as a dictionary
list/dict comprehensions can be used to make queries

Metadata to Capture #

try pandas method -> try spark -> try dict/list -> none

column names
length
Null count
created_by node name

Database? #

Is there an easy way to create a nosql database in memory from a a list of dictionaries?

Gracefully adopt kedro, the catalog

2020-06-29T03:00:00Z

Why use kedro catalog? #

While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all.

Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different for each data and storage type. You just don’t need to think about it.

iperitive loading style
organizes your data
all file locations can be quickly identified
can be dropped into kedro later

“can be dropped into kedro later” Let’s talk a bit more about that

2 Ways to Gracefully adopt the catalog #

How do I get started with the kedro catalog

add with the code api
load from yaml (recommended)

1. Adding to the catalog with the code api #

how to use the kedro catalog code api

It is possible to keep everything inside of one single file if you want by utilizing the code api, which defines the kedro catalog inside of your script with python. I personally like this method as it is a bit more scriptable to create many layers of datasets with a for loop. It can still be carried right into a kedro project, but is not the normal way that other kedro users will be used to seeing. I’ll leave it up to you which technique to use.

from kedro.io import DataCatalog
from kedro.extras.datasets.pandas import CSVDataSet

io = DataCatalog(
    {
        "bikes": CSVDataSet(filepath="../data/01_raw/bikes.csv"),
    }
)

taken right from the kedro docs

👆 This can be done inside of a single Jupyter cell, and pulled out later.

2. Creating a catalog config file #

recommended

This reccommended method it great since it will simply drop right in to a full kedro project if you were ever ready to adopt the framework as a whole. The downside to gracefully adopting the framework is that you need to have a bit of an understanding of the internals to do it. When using the framework as a whole it seemlessly takes care of everything for you.

# conf/base/catalog.yml
# Example 1: Loads / saves a CSV file from / to a local file system

bikes:
  type: pandas.CSVDataSet
  filepath: data/01_raw/bikes.csv

from kedro.config import ConfigLoader
from kedro.io import DataCatalog

conf_loader = ConfLoader(['conf/base'])
conf_catalog = conf_loader.get('catalog*', 'catalog/**')
catalog = DataCatalog.from_config(conf_catalog)

You can even do this 👆 from a Jupyter notebook

If you do happen to be in a module deeper into an existing library I tend to leverage the use of the __file__ magic. __file__ is a string containing the location of the current file. You can initiate a Path object from this and roll up the necessary number of directories with the parents attribute then into the conf/base directory. I find this more more robust as it does not depend on your current working directory.

from pathlib import Path
conf_loader = ConfLoader([Path(__file__).parents[1] / 'conf/base'])
conf_catalog = conf_loader.get("catalog*", "catalog/**")
new_catalog = DataCatalog.from_config(conf_catalog)

I find leveraging the __file__ magic a bit more robust when possible as it does not depend on your current working directory.

Adhoc adding to an existing kedro catalog #

Let’s say that you are already using a kedro project, but you are wanting to develop new features entirely in the context of a notebook. You can easily create a catalog as shown above. Since the kedro catalog variable is typically catalog, we will call it new_catalog and add it to the existing catalog below. If you are not quite sure what that **catalog syntax is doing check out my article on **kwargs.

catalog.add_feed_dict(new_catalog.datasets.__dict__)

python args and kwargs article by @_waylonwalker

The framework makes appending

catalog = DataCatalog({**catalog.datasets.__dict__, **new_catalog.datasets.__dict__})
``



## One word of caution


If you have any hooks running `after_catalog_created`, they will not be run on the appended catalog entries.

How to find things in your kedro catalog

2020-06-22T03:00:00Z

kedro 0.16.2 just dropped last week with a long-awaited feature… catalog search! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful.

The Catalog #

The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place make it so easy to pick your project up and move it to a completely new environment. That sweet imperative loading style saves so much read/write overhead. I can load all my data with a single command whether it’s in amazon s3, google cloud platform, or a local file.

Kick start a toy project #

Just like with most of these articles, I am going to create a conda environment so that I don’t break any existing projects and scaffold up a toy project to learn from.

conda create -n kedro0162 python=3.8 -y
activate kedro0162
pip install kedro
kedro new # call it Kedro 0162 and click-through
cd kedro-0162
kedro install

Expect this set of commands to take a few minutes depending on your system, connection speed, and amount of packages already in your local cache.

Create some catalog #

Now the power of the catalog search really starts to shine when your projects grow legs. You have groups of many datasets containing patterns of data including layer, or source among other things.

vim conf/base/catalog.yml

In the catalog, you will see a few lines of instructions followed by

example_iris_data:
  type: pandas.CSVDataSet
  filepath: data/01_raw/iris.csv

This gives us one stored catalog entry called example_iris_data, it is a CSV file stored in data/01_raw/iris.csv.

Let’s make up a transportation company that is siloed into three different divisions and it is our job to bring their sales and product metadata into a single report. This company makes lifted-trucks, primium-scoots, and luxy-yahts. and we know that we will want raw, int, pri and modin layers to start our project so let’s scaffold up that catalog real quick.

# ――――――――― lifted-truck ―――――――――

raw_lifted_truck_sales:
  type: pandas.CSVDataSet
  filepath: data/01_raw/sales/lifted-truck.csv

int_lifted_truck_sales:
  type: pandas.CSVDataSet
  filepath: data/01_int/sales/lifted-truck.csv

pri_lifted_truck_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/lifted-truck.csv

raw_lifted_truck_info:
  type: pandas.CSVDataSet
  filepath: data/01_raw/info/lifted-truck.csv

int_lifted_truck_info:
  type: pandas.CSVDataSet
  filepath: data/01_int/info/lifted-truck.csv

pri_lifted_truck_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/lifted-truck.csv

# ――――――――― primium-scoot ―――――――――

raw_primium_scoot_sales:
  type: pandas.CSVDataSet
  filepath: data/01_raw/sales/primium-scoot.csv

int_primium_scoot_sales:
  type: pandas.CSVDataSet
  filepath: data/01_int/sales/primium-scoot.csv

pri_primium_scoot_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/primium-scoot.csv

raw_primium_scoot_info:
  type: pandas.CSVDataSet
  filepath: data/01_raw/info/primium-scoot.csv

int_primium_scoot_info:
  type: pandas.CSVDataSet
  filepath: data/01_int/info/primium-scoot.csv

pri_primium_scoot_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/primium-scoot.csv

# ――――――――― luxy-yaht ―――――――――

raw_luxy_yaht_sales:
  type: pandas.CSVDataSet
  filepath: data/01_raw/sales/luxy-yaht.csv

int_luxy_yaht_sales:
  type: pandas.CSVDataSet
  filepath: data/01_int/sales/luxy-yaht.csv

pri_luxy_yaht_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/luxy-yaht.csv

raw_luxy_yaht_info:
  type: pandas.CSVDataSet
  filepath: data/01_raw/info/luxy-yaht.csv

int_luxy_yaht_info:
  type: pandas.CSVDataSet
  filepath: data/01_int/info/luxy-yaht.csv

pri_luxy_yaht_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/luxy-yaht.csv


# ――――――――― combined ―――――――――
pri_combined_sales:
  type: pandas.CSVDataSet
  filepath: data/01_pri/sales/combined.csv

pri_combined_info:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/combined.csv

# ――――――――― modin ―――――――――

modin_main:
  type: pandas.CSVDataSet
  filepath: data/01_pri/info/combined.csv

Some examples of common regex uses #

regex gets really complicated fast, but these basic examples are very common use cases and will get you a long way without being very complicated.

term - all catalog entries that include term in the catalog entry
^term - all catalog entries that include term at the beginning of the catalog entry
term$ - all catalog entries that include term at the end of the catalog entry
term1.*term2 - include anything in between term1 and term2.
term1|term2 - all catalog entries that include term1 or term2

Let’s Search this thing #

kedro has long included the catalog.list() feature that returns a list of all datasets. Now the list command takes in a regex_search keyword argument. By default, it is empty and returns the entire catalog.

kedro ipython

list out all of the luxy-yahts #

>>> catalog.list('luxy_yaht`)
['raw_luxy_yaht_sales',
 'int_luxy_yaht_sales',
 'pri_luxy_yaht_sales',
 'raw_luxy_yaht_info',
 'int_luxy_yaht_info',
 'pri_luxy_yaht_info']

List out data by layer #

Easy just search for the layer name.

raw #

>>> catalog.list('raw')
['raw_lifted_truck_sales',
 'raw_lifted_truck_info',
 'raw_primium_scoot_sales',
 'raw_primium_scoot_info',
 'raw_luxy_yaht_sales',
 'raw_luxy_yaht_info']

pri #

 >>> catalog.list('pri')
['pri_lifted_truck_sales',
 'pri_lifted_truck_info',
 'raw_primium_scoot_sales',
 'int_primium_scoot_sales',
 'pri_primium_scoot_sales',
 'raw_primium_scoot_info',
 'int_primium_scoot_info',
 'pri_primium_scoot_info',
 'pri_luxy_yaht_sales',
 'pri_luxy_yaht_info',
 'pri_combined_sales',
 'pri_combined_info']

😲 We just included every primium-scoot dataset!

Here we just encountered our first need for regex. I’ll be the first to admit that I am really bad at regex, it’s incredibly confusing, becomes read-only with much complexity, but is super powerful and used in a lot of places.

`^term` #

beginning of catalog entry

The ^ regex operator searches for catalog entries that include the search term at the very beginning.

 >>> catalog.list('^pri')
['pri_lifted_truck_sales',
 'pri_lifted_truck_info',
 'pri_primium_scoot_sales',
 'pri_primium_scoot_info',
 'pri_luxy_yaht_sales',
 'pri_luxy_yaht_info',
 'pri_combined_sales',
 'pri_combined_info']

`term$` #

end of catalog entry

The $ operator is the opposite of the ^ operator. It means give me all that matches that occur at the end of the catalog entry.

>>> catalog.list('info$')
['raw_lifted_truck_info',
 'int_lifted_truck_info',
 'pri_lifted_truck_info',
 'raw_primium_scoot_info',
 'int_primium_scoot_info',
 'pri_primium_scoot_info',
 'raw_luxy_yaht_info',
 'int_luxy_yaht_info',
 'pri_luxy_yaht_info',
 'pri_combined_info']

`term1.*term2` #

The .* operator in regex means give me all the datasets that include the two terms no matter what is between them. There is also a .? to only allow one character between them. More often than not I really just want the two patterns to exist in the dataset entry.

>>> catalog.list('raw.*info$')
['raw_lifted_truck_info',
 'raw_primium_scoot_info',
  'raw_luxy_yaht_info']

Some real things that we can do with search #

Let’s look at a few examples beyond the obvious of just searching for the dataset that we want to load.

Check Raw Data #

While migrating pipelines between environments it’s important to know if your raw datasets are available. I will argue that you should also consider looking at pipeline.inputs as it cannot lie and gives you a true reading of the pipeline inputs. But another easy check might be to check all the datasets that the Data Engineers have labeled raw.

>>> {dataset: catalog.exists(dataset) for dataset in catalog.list('^raw')}
{'raw_lifted_truck_sales': False,
 'raw_lifted_truck_info': False,
 'raw_primium_scoot_sales': False,
 'raw_primium_scoot_info': False,
 'raw_luxy_yaht_sales': False,
 'raw_luxy_yaht_info': False}

Since we just created a dummy catalog the data does not exist in this example.

Create a new catalog #

Let’s say that we have someone on the team who is from the land division of our company and they want a simplified catalog readily available that does not include any marine data.

To do this we will need to reach a bit into the kedro internals for the DataCatalog class and utilize a new regex operator |.

>>> from kedro.io import DataCatalog
>>> land_catalog = DataCatalog(
    {
        dataset: getattr(catalog.datasets, dataset)
        for dataset in catalog.list('truck|scoot')
        }
    )
>>> land_catalog.list()
['raw_lifted_truck_sales',
 'int_lifted_truck_sales',
 'pri_lifted_truck_sales',
 'raw_lifted_truck_info',
 'int_lifted_truck_info',
 'pri_lifted_truck_info',
 'raw_primium_scoot_sales',
 'int_primium_scoot_sales',
 'pri_primium_scoot_sales',
 'raw_primium_scoot_info',
 'int_primium_scoot_info',
 'pri_primium_scoot_info']

regex recap #

^term - beginning
term$ - end
term1.*term2 - anything in between
term1|term2 - or

How Kedro handles your inputs

2020-06-19T03:00:00Z

Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing.

*args/**args review #

Check out this post for a review of how *args **kwargs work in python.

understanding python *args and **kwargs

python args and kwargs article by @_waylonwalker

All Kedro inputs are catalog Entries #

When kedro runs your pipeline it uses the catalog to imperatively load your data, meaning that you don’t tell kedro how to load your data, you tell it where your data is and what type it is. These catalog entries are like a key-value store. You just need to give the key when setting up a node.

Single Inputs #

These are fairly straightforward to understand. In the example below when kedro runs the pipeline it will load the input from the catalog, then pass that input to the func, then save the returned value to the output catalog entry.

from kedro.pipeline import node

def create_int_sales(sales):
    "cleans up raw sales data"
    ...
    return cleaned_sales

my_node = node(
    func=create_int_sales,
    inputs='raw_sales',
    output='int_sales',
    )

List of inputs #

Let’s look at an example node that combines more than one dataset. When kedro has sees a list of catalog entries it will load up each catalog entry sequentially then pass them in order to the create_sales_report function.

from kedro.pipeline import node

def create_sales_report(sales, products):
    "adds product metadata to the sales data"
    ...


my_node = node(
    func=create_sales_report,
    inputs=['pri_sales', 'pri_products'],
    output='sales_report',
    )

simulating pipeline run using 2 inputs #

Here We can simulate what kedro does during the pipeline run by using *args.

# inputs you gave kedro
inputs=['pri_sales', 'pri_products']
# load data
input_data = [catalog.load(entry) for entry in  inputs]
# run the node
sales_report = create_sales_report(*input_data)
# save the data to the output
catalog.datasets.sales_report.save(sales_report)

More generalizable functions #

We can also use *args to make our functions a little bit more generalizable. The first that comes to my mind is a unioner. The second

def unioner(*dfs: pd.DataFrame): -> pd.DataFrame
    pd.concat(dfs)

Now we can pass any number of DataFrames into our kedro node to get unioned together, but do we really need a function for a one-liner… No we can use an inline function for this case.

my_node = node(
    func=lambda *dfs: pd.concat(dfs),
    input=['sales_2017', 'sales_2018'],
    output='sales',
)

`*args` scares the crap out of me #

It’s great for the unioner example where its a collection of similar things where order does not matter. But for the create_sales_report function. Those are distinctly different inputs. If someone does some refactoring and changes the order in one place or another it’s going to turn into a bad day real fast.

**kwargs are a bit better #

Let’s refactor the create_sales_report before someone tries to ruin our day. We can easily do this by passing a dictionary (keys are the argument name, values are the catalog key) of arguments to kedro instead of a list.

from kedro.pipeline import node

def create_sales_report(sales, products):
    "adds product metadata to the sales data"
    ...

my_node = node(
    func=create_sales_report,
    inputs={'sales': 'pri_sales', 'products': 'pri_products'},
    output='sales_report',
    )

Now if someone tries to refactor the order of arguments we are safe!

Simulating the pipeline run with `**kwargs` #

Pretty much the same as before, except with **kwargs and dictionaries keeping us a bit safer.

# inputs you gave kedro
inputs={'sales': 'pri_sales', 'products': 'pri_products'},
# load data
input_data = {arg: catalog.load(entry) for arg, entry in inputs.items()}
# run the node
sales_report = create_sales_report(**input_data)
# save the data to the output
catalog.datasets.sales_report.save(sales_report)

Stay Safe #

Kedro inputs are quite easy to understand if you already have a grasp of *args and **kwargs and if you don’t it is still fairly intuitive to pick up. Stay on the safe side, if your collection of inputs are clearly different things, use a dictionary for safety.

004

2020-05-31T19:22:07Z

🔥 #kedrotips use find-kedro to assembly your pipelines

from kedro.context import KedroContext
from find_kedro import find_kedro

class ProjectContext(KedroContext):
    def _get_pipelines(self) -> Pipeline:
        return find_kedro()

kedro-hooks/src/kedro-hooks/preflight.py

2020-05-31T04:37:25Z

🔥 #kedrotips hooks can be created using modules

# kedro-hooks/src/kedro-hooks/preflight.py
from kedro.hooks import hook_impl
from kedro.io.core import DataSetNotFoundError
from colorama import Fore
import textwrap


@hook_impl
def before_pipeline_run(run_params, pipeline, catalog):
    missing_input = [i for i in pipeline.inputs() if not getattr(catalog.datasets, i)._exists()]
    if len(missing_input) != 0:
        raise DataSetNotFoundError(textwrap.dedent(f'''

    {Fore.LIGHTBLACK_EX}――――――――  {Fore.RED}PREFLIGHT ERROR {Fore.LIGHTBLACK_EX}―――――――――
    {Fore.RESET} preflight of pipeline failed due to {Fore.YELLOW}missing datasets
    {Fore.BLUE} {missing_input}{Fore.RESET}
    '''))

002

2020-05-28T23:48:00Z

kedro-static-viz 0.3.0 just launched with hooks support 🎉

from kedro_static_viz.hooks import StaticViz

class ProjectContext(KedroContext):
   project_name = "kedro0160"
   project_version = "0.16.1"
   package_name = "kedro0160"
   hooks = [ StaticViz() ]

Kedro Static Viz 0.3.0 is out with Hooks Support

2020-05-28T05:00:00Z

kedro-static-viz is out with support for the newly released hooks feature. This means that you can have kedro-static-viz automatically deploy a full gatsby site before_pipeline_run keeping your visualization always up to date.

Even though it is a static site there is no functionality lost. The only thing that’s missing is the flask server. With kedro-static-viz you can deploy your visualization to a number of static hosting providers such as GitHub pages free of charge with wicked fast performance

⚡ It’s Fast #

Even though it’s built on gatsbyjs the full site builds in under 2s even on slower hardware. This is because the site is already pre-rendered and stripped of any excess. It’s zipped up right into the python package and is typically used with the cli, but now can be used with python, or as a hook as well.

What is kedro-viz 🤔 #

Kedro viz is a fantastic kedro plugin that allows you to visualize your data pipeline. Kedro allows you to quickly build production-ready pipelines where you just configure a catalog, then toss python functions into a big pile. Kedro figures out the order everything needs ran in for you, allows you to run a datasets dependencies or dependents only. kedro-viz gives you a great way to see this ordering visually.

kedro visualization from the projects readme

Check out a live running example #

Using the power of GitHub actions the I have built a kedro iris pipeline visualization that can be found on https://static-viz.kedro.dev/

Itching to get started with kedro #

You can be up and running in a matter of minutes if you already have python running on your machine.

Make a virtual environment with your environment manager of choice.

conda create -n kedro-practice python=3.8 -y
conda activate kedro-practice

Install kedro. Then create a new project with their awesome cli template built on cookiecutter. Make sure to answer y to get a prebuilt example pipeline with data.

pip install kedro kedro-static-viz
kedro new

Vizualize your pipeline with the cli 〽 #

For local use when you already have the full project kedro viz is a great tool to use, but this is an article about kedro-static-viz.

kedro-static-viz static-viz

Since we used kedro-static-viz you will have a new directory called public that you can host on any static web hosting service, like GitHub pages or Netlify.

Ready to try out the new hooks feature 🙋‍♀️ #

Open up your <project>/src/run.py and add the hook to your ProjectContext class. Next time you run your pipeline you will have an updated pipeline.

from kedro_static_viz.hooks import StaticViz

class ProjectContext(KedroContext):
   project_name = "kedro0160"
   project_version = "0.16.1"
   package_name = "kedro0160"
   hooks = [ StaticViz() ]

Now Run that pipeline 🏃‍♀️ #

Run your pipeline and enjoy that fresh kedro viz each and every time you run your pipeline.

kedro run

Want to make your own hooks 🎣 #

Check out some of my other articles on building kedro hooks.

Check out the example 👉 https://static-viz.kedro.dev/

make sure to answer yes to adding example data your first time

2020-05-26T12:48:00Z

practice building pipelines with #kedro today

conda create -n learning-kedro-1 python=3.8 -y
conda activate learning-kedro-1
pip install kedro kedro-viz
kedro new
# make sure to answer yes to adding example data your first time

Brainstorming Kedro Hooks

2020-05-22T22:02:00Z

This post is a 🧠 branstorming work in progress. I will likely use it as a storage location/brain dump of hook ideas.

What is Kedro 🤔 #

If you are completely unsure what kedro is be sure to check out my what is kedro post

after_catalog_created #

filepath replacer
bucket replacer

before_pipeline_run #

preflight
check that data exists
run kedro_static_viz
run mypy
run interrogate
run flake8

after_pipeline_run #

Great Expectations
send email
send slack

before_node_run #

after_node_run #

Great Expectations
save stats/meta data

Execution Order #

hooks are executed in reverse order of the hooks list.

hooks with tryfirst will be moved to the end of the list hooks with trylast will be moved to the end of the list

after_catalog_created
before_pipeline_run

args
- run_params = run_params = {‘run_id’: ‘2020-05-23T15.24.23.958Z’, ‘project_path’: ‘/mnt/c/temp/kedro0160’, ’env’: ’local’, ‘kedro_version’: ‘0.15.9’, ’tags’: (), ‘from_nodes’: [], ’to_nodes’: [], ’node_names’: (), ‘from_inputs’: [], ’load_versions’: {}, ‘pipeline_name’: None, ’extra_params’:
- pipeline
- catalog

before_node_run
after_node_run

When does data get saved??? #

before or after node hook?

??Unsure?? #

does before catalog load have access to parameters?
- Yes

🥾steel toes #

I was way too excited about this one and already created it

prevents pain from stepping on your teammates toes

Kedro is so amazing at promoting collaboration between team members. Each team member can check out the code, branch, and start work on their own section of the pipeline. Issues can arrise if the team members section of the pipeline happen to cross. Breaking changes happen, BREAKS during development happen and can completely kill a teammates workflow.

is there a way to prevent toe stepping?
try to load filepath_<branch>
if load fails try filepath
save data to filepath_<branch>

how

on after_catalog_load check for existing “branch” data
if “branch” data exists load that
otherwise keep default

Run only nodes that have changed #

store a deephash of functions code
store a hash of the inputs
if neither code or inputs changed run function, otherwise skip.
- How could a hook choose to skip the node?

Static viz hook #

Before pipeline run

make site
Set node status to queued

Before node run

Set running status

After node run

Set running status

On pipeline error

Set run status

On node error

Set error status

After pipeline run

Set complete status

After node run

set complete

Create Custom Kedro Dataset

2020-05-15T05:00:00Z

Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key.

But what is happening behind the scenes #

Under the hood there is an AbstractDataSet that each connector inherits from. It sets up a lot of the behind the scenes structure for us so that we dont have to. For the most part kedro has connectors for about anything that you want to load, csv, parquet, sql, json, from about anywhere, http, s3, localfile system are just some of the examples.

Here is a DataSet implementation from their docs. Here you can see the barebones example straight from the docs. Parameters from the yaml catalog will get passed in

from pathlib import Path

import pandas as pd

from kedro.io import AbstractDataSet


class MyOwnDataSet(AbstractDataSet):
    def __init__(self, param1, param2, filepath, version):
        super().__init__(Path(filepath), version)
        self._param1 = param1
        self._param2 = param2

    def _load(self) -> pd.DataFrame:
        load_path = self._get_load_path()
        return pd.read_csv(load_path)

    def _save(self, df: pd.DataFrame) -> None:
        save_path = self._get_save_path()
        df.to_csv(save_path)

 	def _exists(self) -> bool:
        path = self._get_load_path()
        return path.is_file()

    def _describe(self):
        return dict(version=self._version, param1=self._param1, param2=self._param2)

creating the kedro-preflight hook

2020-05-10T07:12:00Z

Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro 0.16.0. They allow you to hook into catalog_created,pipeline_run, and node_run(nouns). With a before, or after (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability.

kedro hooks are an exciting upcoming feature of kedro 0.16.0. They allow you to hook into catalog_created,pipeline_run, and node_run(nouns). With a before, or after (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability.

What is Kedro

If you are completely unsure what kedro is be sure to check out my what is kedro post

Docs #

a work in progress

As this is a part of an upcoming release you will need to look in the latest docs, not stable and you will find a 15_hoooks page. As these docs are still in development they are not very complete at this point and do require a bit more existing kedro knowledge to understand. I am sure they will get much better as we approach the realease of hooks.

This doesn’t mean that we can’t still install the latest/unstable version and have some fun learning!

Installation #

Straight from GitHub

As this is part of an upcoming release you will need to get the library straight from github. Since this is not a stable release of kedro I cannot express the importance of using a virtual environment enough. Trying to install this version in the same place that you are trying to develop a pipeline potentially break your existing working development environment.

conda create -n kedro0160 -y
conda activate kedro0160 # may also be source activate kedro0160 or activate kedro0160
pip install git+https://github.com/kedro-org/kedro.git
pip install colorama

note the version is still somewhere between 0.15.9 and 0,16.0. kedro.__version__ will still be 0.15.9 and wiill not roll until the official release.

Create a sample project #

Kedro new #

For more details check out my full post on kedro new

For this post I really just want a working pipeline as fast as possible. For this I am going to use iris pipeline that is generated from the kedro new command in the cli. It’s important that you answer y to create an example pipeline.

Hold On ✋ #

Did you create a separate environment for this? Please do.

kedro new

After you run the kedro new command it will ask a series of questions. 👇 Here is how I answered them.

Project Name:
=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
 [New Kedro Project]: Kedro Hooks
Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
 [kedro-hooks]:
Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter or underscore.
 [kedro_hooks]:
Generate Example Pipeline:
==========================
Do you want to generate an example pipeline in your project?
Good for first-time users. (default=N)
 [y/N]: y
Change directory to the project generated in /mnt/c/temp/kedro-hooks/
A best-practice setup includes initialising git and creating a virtual environment before running `kedro install` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

Install the Project #

Next install the project itself and all of its dependencies with the kedro install command.

cd kedro-hooks
kedro install

🏃‍♀️ Run the pipeline #

Before we start developing any hooks lets make sure everything is setup correctly by running the pipeline with kedro run.

kedro run

Let’s make a hook #

getting to the meat of the post

Now that we have a project scaffolded up and running we can develop a hook for it. As far as I can tell hooks can be implemented one of two ways. As a function inside of a module, then import that module and pass it into the hooks list, or implemented as a method on a class, then the class is passed into the hooks list. Either method must follow the naming convention with the @hook_impl decorator. Each module/class can implement more than one hook, but not more than one of the same type. One of each kind will be created below.

Full list of hooks available #

before_catalog_created

after_catalog_created

before_pipeline_run

after_pipeline_run

before_node_run

after_node_run

debug_hook (class) #

quick and dirty

I highly recommend this as your first hook. It’s super easy to make and lets you explore the arguments passed into the hook. For this one I am going to pop the following class right into kedro-hooks/src/kedro-hooks/run.py, remember that I chose kedro-hooks as my project name. Your path might be slightly different. If you wanted to make a real hook it might make sense to put it in its own module, but for simplicity of your first hook you can put it directly in the same module that it gets implemented.

class debug_hook:
    @hook_impl
    def before_pipeline_run(run_params, pipeline, catalog):
        "pops into a debugger before pipeline run"
        print('I hooked in right before the pipeline run')
        breakpoint()

It is really that easy to create a kedro hook! Now lets apply it to our project. All we need to do is add one line (hooks = [debug_hook]) to the existing ProjectContext class within kedro-hooks/src/kedro-hooks/run.py. Once we do that our ProjectContext will look like this.

class ProjectContext(KedroContext):
    """Users can override the remaining methods from the parent class here,
    or create new ones (e.g. as required by plugins)
    """

    project_name = "kedro-hooks"
    # `project_version` is the version of kedro used to generate the project
    project_version = "0.15.9"
    package_name = "kedro-hooks"

    hooks = [ debug_hook ] # 👈 This is where you implement the hook

    def _get_pipelines(self) -> Dict[str, Pipeline]:
        return create_pipelines()

Run it! While you are in the debugger, explore what the run_params, pipeine, and catalog arguments give you. This will give you some insight to what to expect when creating your next hook.

preflight hook (module) #

giving it a bit more flair

Create a new file kedro-hooks/src/kedro-hooks/preflight.py and place the following content into the file. This will raise a DataSetNotFoundError before wasting time running any of the pipeline. This could be useful to save some developer time for long running pipelines by warning them that they don’t have all of the raw data they need before running.

# kedro-hooks/src/kedro-hooks/preflight.py
from kedro.hooks import hook_impl
from kedro.io.core import DataSetNotFoundError
from colorama import Fore
import textwrap


@hook_impl
def before_pipeline_run(run_params, pipeline, catalog):
    missing_input = [i for i in pipeline.inputs() if not getattr(catalog.datasets, i)._exists()]
    if len(missing_input) != 0:
        raise DataSetNotFoundError(textwrap.dedent(f'''

    {Fore.LIGHTBLACK_EX}――――――――  {Fore.RED}PREFLIGHT ERROR {Fore.LIGHTBLACK_EX}―――――――――
    {Fore.RESET} preflight of pipeline failed due to {Fore.YELLOW}missing datasets
    {Fore.BLUE} {missing_input}{Fore.RESET}
    '''))

Once we are happy with this hook, it can live anywhere. It can be a module inside our project. It can be a separate libarary that gets handed out as a back ally wheel, or we can even publish it as its own package to pypi so that anyone can easily pip install it.

One Step Back #

a bit of explanation of preflight

If you are not familiar, pipeline.inputs() gives us all of the edge inputs into the pipeline. kedro does also have a pipeline.all_inputs() that tells us all of the edge and internal pipeline inputs that will be called throughout the pipeline run. For this hook we are just concerned with the edge inputs as internal inputs will be generated during the run.

Also each one of the kedro datasets have an _exists() method attached to them to check if the dataset exists or not. For a more robust implementation of preflight it would probably be best to ignore AttributeErrors, i.e the dataset type does not have an implementation of _exists. It would probably also be a good idea to filter for types such as SQLQueryDataSets that assume _exists is False by default.

Ideas #

Now that the juices are flowing what ideas do you have for kedro hooks?

📝 Kedro Preflight Notes

2020-05-09T15:01:00Z

This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources.

Must Haves #

check that inputs exist or are of a type to skip (sql)

Good to haves

check that all input and output databases are accessible with good credentials
check for s3 bucket access
check for spark install

Implementation #

@hook_spec
def before_pipeline_run(run_params, pipeline, catalog):

run params #

{
  "run_id": str
  "project_path": str,
  "env": str,
  "kedro_version": str,
  "tags": Optional[List[str]],
  "from_nodes": Optional[List[str]],
  "to_nodes": Optional[List[str]],
  "node_names": Optional[List[str]],
  "from_inputs": Optional[List[str]],
  "load_versions": Optional[List[str]],
  "pipeline_name": str,
  "extra_params": Optional[Dict[str, Any]]
}

📢 Announcing find-kedro

2020-05-04T11:53:00Z

find-kedro is a small library to enhance your kedro experience. It looks through your modules to find kedro pipelines, nodes, and iterables (lists, sets, tuples) of nodes. It then assembles them into a dictionary of pipelines, each module will create a separate pipeline, and __default__ being a combination of all pipelines. This format is compatible with the kedro _create_pipelines format.

#

kedro is a ✨ fantastic project that allows for super-fast prototyping of data pipelines, while yielding production-ready pipelines. find-kedro enhances this experience by adding a pytest like node/pipeline discovery eliminating the need to bubble up pipelines through modules.

When working on larger pipeline projects, it is advisable to break your project down into different sub-modules which requires knowledge of building python libraries, and knowing how to import each module correctly. While this is not too difficult, in some cases, it can trip up even the most senior engineers, losing precious feature development time to debugging a library.

#

find-kedro is deployed to pypi and can easily be pip installed.

pip install find-kedro

#

The recommended usage of find-kedro is to implement it directly into your projects run.py module

from kedro.context import KedroContext
from find_kedro import find_kedro

class ProjectContext(KedroContext):
    def _get_pipelines(self) -> Pipeline:
        return find_kedro()

Creating nodes #

find-kedro will not execute any functions. It will simply look for variables that match the pattern and identify if they are a kedro.pipeline.Pipeline, kedro.pipeline.nodes.Node, or a list of kedro.pipeline.nodes. Node’s. If so, it will collect them into the dictionary of pipelines.

There are typically three ways that pipelines are constructed with find-kedro; lists, single-nodes, and pipelines.

Lists #

Any pattern matched list will be flattened and collected into the pipeline. Nodes can be created all at once in the list definition.

# my-proj/pipelinies/data_engineering/pipeline
from kedro.pipeline import node
from .nodes import split_data

pipeline = [
    node(
        split_data,
        ["example_iris_data", "params:example_test_data_ratio"],
        dict(
            train_x="example_train_x",
            train_y="example_train_y",
            test_x="example_test_x",
            test_y="example_test_y",
        ),
    )
]

It is also convenient many times to keep the node definition close to the function definition. Many times I define the list at the top of the file, then append to it as I go.

# my-proj/pipelinies/data_engineering/pipeline
from kedro.pipeline import node
from .nodes import split_data

nodes = []
nodes.append(
    node(
        split_data,
        ["example_iris_data", "params:example_test_data_ratio"],
        dict(
            train_x="example_train_x",
            train_y="example_train_y",
            test_x="example_test_x",
            test_y="example_test_y",
        ),
    )
)

Nodes #

All pattern matched kedro.pipeline.node.Node objects will get collected into the pipeline.

# my-proj/pipelinies/data_engineering/pipeline
from kedro.pipeline import node
from .nodes import split_data

split_node = node(
        split_data,
        ["example_iris_data", "params:example_test_data_ratio"],
        dict(
            train_x="example_train_x",
            train_y="example_train_y",
            test_x="example_test_x",
            test_y="example_test_y",
        ),
    )

Pipeline #

All pattern matched kedro.pipeline.Pipeline objects will get collected into the pipeline.

# my-project/pipelinies/data_engineering/pipeline
from kedro.pipeline import node, Pipeline
from .nodes import split_data

split_node = Pipeline(
    [
        node(
            split_data,
            ["example_iris_data", "params:example_test_data_ratio"],
            dict(
                train_x="example_train_x",
                train_y="example_train_y",
                test_x="example_test_x",
                test_y="example_test_y",
            ),
        )
    ]
)

Fully Qualified imports #

When using fully qualified imports from my_proj.pipelines.data_science.nodes import split_data instead of relative imports from .nodes split_data you will need to make sure that your project is installed, in your current path, or you set the directory

#

The CLI provides a handy interface to search your project for nodes

Usage: find-kedro [OPTIONS]

Options:
  --file-patterns TEXT       glob-style file patterns for Python node module
                             discovery

  --patterns TEXT            prefixes or glob names for Python pipeline, node,
                             or list object discovery

  -d, --directory DIRECTORY  Path to save the static site to
  --version                  Prints version and exits
  -v, --verbose              Prints extra information for debugging
  --help                     Show this message and exit.

Example ran with a slightly modified default kedro new project.

❯ find-kedro
{
  "__default__": [
    "split_data([example_iris_data,params:example_test_data_ratio]) -> [example_test_x,example_test_y,example_train_x,example_train_y]",
    "train_model([example_train_x,example_train_y,parameters]) -> [example_model]",
    "predict([example_model,example_test_x]) -> [example_predictions]",
    "report_accuracy([example_predictions,example_test_y]) -> None"
  ],
  "src.default_kedro_159.pipelines.data_engineering.pipeline": [
    "split_data([example_iris_data,params:example_test_data_ratio]) -> [example_test_x,example_test_y,example_train_x,example_train_y]"
  ],
  "src.default_kedro_159.pipelines.data_science.pipeline": [
    "train_model([example_train_x,example_train_y,parameters]) -> [example_model]",
    "predict([example_model,example_test_x]) -> [example_predictions]",
    "report_accuracy([example_predictions,example_test_y]) -> None"
  ]
}

#

You’re Awesome for considering a contribution! Contributions are welcome, please check out the Contributing Guide for more information. Please be a positive member of the community and embrace feedback

#

We use SemVer for versioning. For the versions available, see the tags on this repository.

#

- Waylon Walker - Original Author

- Zain Patel

#

This project is licensed under the MIT License - see the LICENSE.md file for details

Create New Kedro Project

2020-03-02T12:09:00Z

This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro. You can learn more about this Hello World Example in the docs

🧹 Install Kedro

🛢 Create the Example Pipeline

💨 Run the example

📉 Show the pipeline visualization

Create a Virtual Environment #

I use conda to control my virtual environments and will create a new environment called kedro_iris with the following command. note the latest compatible version of python is 3.7.

EDIT: as of kedro 0.16.0 kedro supports up to 3.8

conda create -n kedro_iris python=3.8 -y

Options

Activate your conda environment #

I try to keep my base environment as clean as possible. I have ran into too many issues installing things in the base environment. Almost always its some dependency that starts causing issues making it even harder to realize where its coming from as I never even installed it in base.

source activate kedro_iris

Install Kedro #

Currently kedro==0.15.5 is available on pypi and can be pip installed.

EDIT kedro is up to

pip install kedro

Make sure you are in the directory that you want your project in #

cd /mnt/c/temp

Create a new Kedro project #

kedro new
cd kedro-iris
git init
kedro install

Run the pipeline #

This will tell kedro to run your pipeline. It will look at all of your nodes and determine the correct execution order for you, then run each one of them. You can do this from a python script, python terminal session, or from the kedro cli.

✨ It will look at all of your nodes and determine the correct execution order for you

Lets run from the cli while in the same directory as kedro-iris

kedro run

Viz #

kedro-viz is a priceless feature of kedro. It’s like x-ray vision into your pipeline. I can’t imagine working without it after having it over the past year. Unlike traditional documentation kedro-viz cannot lie to you. It will help guarantee your changes line up properly, plan out adding nodes, and identify dependencies of deprecating nodes.

Unlike traditional documentation kedro-viz cannot lie to you.

Install kedro-viz #

kedro-viz is also on pypi and can be installed just like any other python package with pip.

pip install kedro-viz

Visualize the pipeline #

kedro-viz is ran from the command line in the same directory as your kedro project. There are ways to store your pipeline data as json, then load them from outside your project, but we will follow the standard practice for now.

kedro viz

🏗 Docker #

There is another package that makes creating docker images from kedro projects super simple kedro-docker.

If you dont already have docker installed on your machine, feel free to skip this section.

install kedro-docker #

pip install kedro-docker

build the image #

kedro docker build

run the image #

kedro docker run

Simple Huh #

Getting up and going with a brand new kedro project is super simple, thanks to the help of the kedro new command. The ability to add an example pipeline from the start makes it that much easier to get going and have a template to follow for your own projects.

Recap #

conda create -n kedro_iris python=3.7 -y
source activate kedro_iris
pip install kedro
cd /mnt/c/temp
kedro new
# give it a project name Kedro Iris
# accept default package name kedro_iris
# addept default directory name kedro-iris
# yes for an example pipeline
cd kedro-iris
git init
git add .
git commit -m "initialized new kedro project"
kedro install
kedro run
pip install kedro-viz
kedro viz
pip install kedro-docker
kedro docker build
kedro docker run

Other resources #

The kedro docs have a ton of great resources. They are searchable, but can be a bit of an overwhelming amount of data.

I keep adding to my kedro notes as I find new and interesting things.

I tweet out most of those snippets as I add them, you can find them all here #kedrotips.

More to come #

I am planning to do more articles like this, you can stay up to date with them by following me on dev.to, subscribing to my rss feed.

What is Kedro

2020-02-24T12:48:00Z

What is Kedro

This is my original what-is-kedro article. There is a brand new one

Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate catalogs and pipelines. This article will cover the 10K view of kedro, future articles will dive deper into each one.

kedro is an open-source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to set up your own python library for data pipelining. It includes great ways to manipulate catalogs and pipelines. This article will cover the 10K view of kedro, future articles will dive deeper into each one.

Libraries #

Currently, kedro is broken down into 3 different libraries.

💎 kedro

📉 kedro-viz

🏗 kedro-docker

kedro #

kedro is the core of the ecosystem. It provides the docs, getting started, kedro new templates, and the core library including the catalog and pipeline.

Catalog #

Inside this core library is a data catalog object. This allows you to specify attributes about your data, then load and save it without ever writing a single line of read/write code, which can become very cumbersome. Older versions would load this into the io variable, currently it loads into the catalog. The power of the catalog is that it allows you to read and write data by just referencing its name. Typically this is done inside of a YAML file, but can be done in python.

Here is an example of a CSV dataset stored locally

# Example 1: Loads a local csv file
bikes:
  type: CSVLocalDataSet
  filepath: "data/01_raw/bikes.csv"

This dataset can be loaded by name

catalog.load('bikes')

Though it’s not typical practice it is possible to save data to a catalog entry ad-hoc. Typically the pipeline is used to run functions and save data for you.

import pandas as pd
bikes_df = pd.DataFrame({...<bikes_data>...})
catalog.datasets.bikes.save(bikes_df)

Pipeline #

The pipeline object is the brains of kedro. When working with kedro you simply define functions that take in data as arguments, manipulate it, and return a new dataset. The pipeline will decide what order to execute these functions ini based on their dependencies. It will then work with the catalog to load the data from the catalog pass it to your function, the save the returned data in the catalog.

Here is an example pipeline from the docs.

import pandas as pd
import numpy as np
from kedro.pipeline import Pipeline
from kedro.pipeline import node

def clean_data(cars: pd.DataFrame,
               boats: pd.DataFrame) -> Dict[str, pd.DataFrame]:
    return dict(cars_df=cars.dropna(), boats_df=boats.dropna())

def halve_dataframe(data: pd.DataFrame) -> List[pd.DataFrame]:
    return np.array_split(data, 2)

nodes = [
    node(clean_data,
         inputs=['cars2017', 'boats2017'],
         outputs=dict(cars_df='clean_cars2017',
                      boats_df='clean_boats2017')),
    node(halve_dataframe,
         'clean_cars2017',
         ['train_cars2017', 'test_cars2017']),
    node(halve_dataframe,
         dict(data='clean_boats2017'),
         ['train_boats2017', 'test_boats2017'])
]

pipeline = Pipeline(nodes)

kedro-viz #

kedro-viz is a priceless component to the kedro ecosystem. It gives you x-ray vision into your project. You can see exactly how data flows through your pipeline. Since it is fully automated it is always up to date and never lies to you. kedro-viz is an integral part of my daily debugging and refactoring toolbelt.

Starting the viz from the command line is super easy

cd my-kedro-project
kedro viz

kedro-docker #

kedro-docker is a simple way to set up your project for production. It provides a few simple cli commands

cd my-kedro-project
kedro docker build
kedro docker run

Other resources #

The kedro docs have a ton of great resources. They are searchable, but can be a bit of an overwhelming amount of data.

I keep adding to my kedro notes as I find new and interesting things.

I tweet out most of those snippets as I add them, you can find them all here #kedrotips.

More to come #

I am planning to do more articles like this, you can stay up to date with them by following me on dev.to, subscribing to my rss feed.

Kedro

2019-11-02T05:00:00Z

See all of my kedro related posts in [[ tag/kedro ]].

#kedrotips #

I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips.

🗣 Heads up #

Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful more usefule for exploration.

📚 Catalog #

Photo by jesse orrico on Unsplash

CSVLocalDataSet #

python

import pandas as pd
iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv')

data_set = CSVLocalDataSet(filepath="test.csv",
                                 load_args=None,
                                 save_args={"index": False})

iris_data_set.save(iris)
reloaded_iris = iris_data_set.load()

yaml

test_data:
   type: CSVLocalDataset
   filepath: test.csv
   load_args: None
   save_args:
      index: False

CSVHTTPDataSet #

cities = CSVHTTPDataSet(
    fileurl="https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv",
    auth=None,
    load_args=None)

iris = iris_data_set.load()

cities:
   type: CSVHTTPDataSet
    fileurl: https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv
    auth: None
    load_args: None

HDFLocalDataSet #

import pandas as pd
from kedro.io import HDFLocalDataSet

iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv')
iris_data_set = HDFLocalDataSet(filepath="iris.hdf",
                           key="test_hdf_key",
                           load_args=None,
                           save_args=None)

iris_data_set.save(iris)
reloaded_iris = iris_data_set.load()

cars:
   type: HDFLocalDataSet
   filepath: test.hdf
   key: test_hdf_key

HDFS3LocalDataSet #

import pandas as pd
from kedro.io import HDFS3DataSet

iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv')
iris_data_set = HDFS3DataSet(filepath="iris.hdf",
                        bucket_name="bucket-us-west-1",
                        key="test_hdf_key",
                        load_args=None,
                        save_args=None)

iris_data_set.save(iris)
reloaded_iris = iris_data_set.load()

cars:
   type: HDFS3DataSet
   filepath: cars.hdf
   bucket_name: bucket-us-west-1
   key: test_hdf_key

JSONLocalDataSet #

import pandas as pd
from kedro.io import JSONLocalDataSet

iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv')
cars = JSONLocalDataSet(filepath="iris.json",
                        load_args=None,
                        save_args=None)

iris_data_set.save(iris)
reloaded_iris = iris_data_set.load()

cars:
   type: JSONLocalDataSet
   filepath: iris.json

ParquetLocalDataSet #

import pandas as pd
from kedro.io import ParquetLocalDataSet

iris = pd.read_csv('https://raw.githubusercontent.com/kedro-org/kedro/d3218bd89ce8d1148b1f79dfe589065f47037be6/kedro/template/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/01_raw/iris.csv')

iris_data_set = ParquetLocalDataSet('iris',
                           engine='auto',
                           load_args=None,
                           save_args=None,
                           version=None)

iris_data_set.save(iris)
reloaded_iris = iris_data_set.load()

cars:
   type: JSONLocalDataSet
   filepath: cars

PickleS3DataSet

SQLTableDataSet

SQLQueryDataSet

TextLocalDataSet

ExcelLocalDataSet

⏳ Loading Data #

Photo by Battlecreek Coffee Roasters on Unsplash

Simple Loading #

df = catalog.load('cars')

list all datasets #

catalog.list()

Saving Data #

catalog.save('cars', cars)

🔍 Finding data #

simple keyword search

query = 'raw'
[data for data in catalog.list() if query in data]

see on #kedrotips

multi keyword serch

query = 'raw sales'
data_sets = catalog.list()
for word in query.split():
 data_sets = [
       data
       for data in data_sets
       if query in data
       ]

see on #kedrotips

🐒 monkey patch it

def query(*search_terms):
     data_sets = catalog.list()
     for search in search_terms:
         data_sets = [
         data
         for data in data_sets
         if search in data
         ]
     return data_sets

catalog.query = query

_see on #kedrotips

🤙 YOLO #

You Only Load Once

simple

data = [catalog.load(d)
        for d in
        catalog.query('c_pri', 'cars')
        ]

more refined

data = {
   d: catalog.load(d)
   for d in catalog.query('c_pri', 'cars')
   }

🍷 refined like a fine wine

from types import SimpleNamespace
data = SimpleNamespace**{
   d: catalog.load(d)
   for d in catalog.query('c_pri', 'cars')
   })

🧀 Make it a function getting funcy

from types import SimpleNamespace

def yolo(*search_terms):
   """you only load once
   using query method from previous tip"""
   data = SimpleNamespace(**{
       d: catalog.load(d)
   for d in catalog.query(*search_terms)
    })
    return data

all_pri = yolo('c_pri')

🐒 monkey patch it

from functools import partial

catalog.yolo = yolo
catalog.yolo.__doc__ = "you only load once"

all_pri = catalog.yolo('c_pri')

adding catalogs together #

from kedro.io import DataCatalog
DataCatalog({**cat1.__dict__['_data_sets'], **cat2.__dict__['_data_sets']})

🛢 Building pipelines #

Photo by roman pentin on Unsplash

📍 Creating Nodes #

from kedro.pipeline import node
node = node(lambda x: x.dropna(), inputs='raw_cars', outputs='int_cars')

from kedro.pipeline import node

def drop_columns(df, *columns):
   for column in columns:
      df = df.drop(columns=column)
   return df

node = node(
   lambda x: drop_columns(x, 'vs', 'am', 'gear', 'carb'),
   inputs='int_cars',
   outputs='pri_cars'
   )

🛢 Creating a pipeline #

Don’t be so verbose #

Create similar nodes dynamically

def halve_dataframe(data: pd.DataFrame) -> List[pd.DataFrame]:
   """ splits a dataframe in half """
    return np.array_split(data, 2)

nodes = []
datasets = [
   'cars', 'trucks', 'boats', 'motorcycles', 'planes',
   'ships', 'busses', 'trains', 'subways'
   ]

# creates a pipeline node for every dataset in the datasets list
for dataset in datasets
   nodes.append(
       node(halve_dataframe,
            'e_modin_{dataset}',
            ['train_{dataset}', 'test_{dataset}']),
   )

🏃‍♂️ Running Pipelines #

Photo by Rodion Kutsaev on Unsplash

🔖 filter by tags

nodes = pipeline.only_nodes_with_tags('cars')

see on #kedrotips

filter by node

nodes = pipeline.only_nodes('b_int_cars')

_see on #kedrotips

filter nodes like

query_string = 'cars'
nodes = [
   node.name
   for node in pipeline.nodes
   if query_string in node.name
   ]
pipeline.only_nodes(*nodes)

see on #kedrotips

only nodes with tags or

nodes = pipeline.only_nodes_with_tags('cars', 'trains')

only nodes with tags and

raw_nodes = pipeline.only_nodes_with_tags('raw')
car_nodes = pipeline.only_nodes_with_tags('cars')
raw_car_nodes = raw_nodes & car_nodes

raw_nodes = (
   pipeline
   .only_nodes_with_tags('raw')
   .only_nodes_with_tags('cars')
   )

add pipelines

car_nodes = pipeline.only_nodes_with_tags('cars')
train_nodes = pipeline.only_nodes_with_tags('trains')
transportation_nodes = car_nodes + train_nodes

ensure nodes are attached

cars_attached = len(
   pipeline
   .only_nodes_with_tags('cars')
   .grouped_nodes
   ) == 1

🎂 Pipeline Decorators #

example - log_time

from kedro.pipeline.decorators import log_time, mem_profile
pipeline.decorate(log_running_time)

Pipeline IO #

pipleine.all_inputs() and pipeline.all_outputs() return sets of pipeline inputs and outputs and you can do set operations on them. This is particularly useful to find the upper and lower edges of your pipeline or subset of pipeline. The pipeline object here is any kedro pipeline including a filtered subset.

Find all raw data #

pipeline.all_inputs() - pipeline.all_outputs()

Find all final data #

pipeline.all_outputs() - pipeline.all_inputs()

Find all nodes that do not raw #

This one is probably one that is pushing the limits of what I would do in a list comprehension that I use in prod or even put into a text editor, but I commonly use ipython for my adhoc work and keeping it all in one line is very handy. Complex list comprehensions kinda start becoming like regex in a way that they are really easy to write and really hard to read. I don’t think this one quite hits that point but its getting close.

I find this one super useful to help me either move data beween environments, or avoid unnecessary database calls.

raw_inputs = pipeline.all_inputs() - pipeline.all_outputs()
raw_nodes = [node for node in pipeline.nodes if [i for i in raw_inputs if i in set(node.inputs)] != []]

Posts tagged: kedro Archive

💭 Migration guide for config loaders — kedro 0.19.11 documentation

Make Kedro Runs Beautiful

Install kedro rich #

Kedro run #

Kedro catalog #

Give it a star #

Lambda Function as a Kedro Node

Examples #

Add New Dependencies to Your Kedro Project

git-status">git status #

requirements.in #

build-reqs #

git add #

Practice making pipelines with kedro

Start practicing #

get pipx #

make a new project #

Make a virtual environment #

Using Conda #

Using venv #

using pipenv #

Make pipelines #

Running Kedro on Ubuntu 21.10 Impish Indri

What’s up #

How to fix this? #

Where Can I get the right version #

I have two articles that can help you #

kedro catalog create

Running Kedro Catalog Create #

CONF_ROOT #

File Location #

MemoryDataSet’s #

Your free to use what you want though #

Continue adding nodes #

Formatting is not worthwhile #

Continuing to work #

Sorting Order #

nvim conf 2021 | IDE's are slow | Waylon Walker

Slides 👇 #

welcome #

Other possible titles #

It’s ok #

Trick it out #

It wasn’t working for me anymore #

dozens of instances #

Move With Intent #

Hub and Spoke #

Other Things That Make this Possible #

yes, vim is ugly, make it yours #

lsp #

treesitter #

telescope #

Check Messages #

Kedro-Broken-Urls

Broken Urls #

Setting Parameters in kedro

parameters files #

base #

accessing parameters #

using parameters in nodes #

local #

env #

Activating this environment #

Links #

Writing your first kedro Nodes

import node #

func #

inputs / outputs #

our first node #

second node #

Multiple Inputs #

Links #

Running your Kedro Pipeline from the command line

Kedro run #

Specific Pipelines #

Single Nodes #

Some DAG concepts #

Multiple things #

Related Links #