Posts tagged: kedro

💭 Migration guide for config loaders — kedro 0.19.11 documentation

2025-02-05T18:16:44Z

Migrating from kedro 0.18.4 to the latest version involves handling the deprecated OmegaConf loader. Switching over does not look as bad as I originally thought.

installing kedro 0.18.5+
set the CONFIG_LOADER_CLASS in settings.py
swap out import statements
config must be yaml or json
getting values from config must be done with bracket __getattr__ style not with .get
any Exceptions caught from Templated config loader will need to be swapped to OmegaConfig exceptions, similar to #3
templated values must lead with an _
Globals are handled different
OmegaConfig does not support jinja2 sytax, but rather a ${variable} syntax

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Make Kedro Runs Beautiful

2022-03-11T13:14:51Z

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.

Install kedro rich #

There is no pypi package yet, but it’s on github. You can pip install it with the git url.

pip install git+https://github.com/datajoely/kedro-rich

Kedro run #

You can run your pipeline just as you normally would, except you get progress bars and pretty prints.

kedro run

Kedro catalog #

Listing out catalog entries from the command line now print out a nice pretty table.

kedro catalog list

Give it a star #

Go to the GitHub repo and give it a star, Joel deserves it.

Lambda Function as a Kedro Node

2022-01-31T21:52:36Z

I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas.

Examples #

Here are two examples, the first one lambda x: x is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow standard layering conventions, without skipping a layer, overthinking if you should have the layer or not, and leaves a good placholder to fill in later when you need it.

Many times I just want to get the data in as fast as possible, learn about it, then go back and tidy it up.

from kedro.pipeline import node

my_first_node = node(
   func=lambda x: x,
   inputs='raw_cars',
   output='int_cars',
   tags=['int',]
   )

my_first_node = node(
   func=lambda cars: cars[['mpg', 'cyl', 'disp',]].query('disp>200'),
   inputs='raw_cars',
   output='int_cars',
   tags=['pri',]
   )

Note: try not to take the idea of a one liner too far. If your one line function wraps several lines down it probably deserves to be a real function for readability and a good docstring.

Add New Dependencies to Your Kedro Project

2022-01-28T21:30:48Z

As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command pip-compile under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let’s take a look at it.

git-status">git status #

Before you start mucking around with any changes to dependencies make sure that your git status is clean. I’d even reccomend starting a new branch for this, and if you are working on a team potentially submit this as its own PR for clarity.

git status
git checkout main
git checkout -b add-rich-dependency

requirements.in #

New requirements get added to a requirements.in file. If you need to specify an exact version, or a minimum version you can do that, but if all versions generally work you can leave it open.

# requirements.in
rich

Here I added the popular rich package to my requirements.in file. Since I am ok with the latest version I am not going to pin anything, I am going to let the pip resolver pick the latest version that does not conflict with any of my dependencies for me.

build-reqs #

The command kedro build-reqs will tell kedro to recompile the requirements.txt file that has all of our dependencies pinned down to exact versions. This ensures that all of our teammates and production workflows use the same exact versions of packages even if new ones are released after we installed on our development machines.

kedro build-reqs

git add #

Now that we have our new dependencies ready to go commit those to git, and submit a PR for them if you are working on a team. This is a good way to document the discussion of adding new dependencies to your teams project.

git add requirements.in
git add requirements.txt
git status
git commit -m "FEAT updated dependencies with rich"
git push
# go make a pr
gh pr create --title "feat add rich to dependencies" --body "I added rich as a dependency, and ran pip-compile"

Practice making pipelines with kedro

2022-01-27T21:07:37Z

I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity.

What is Kedro

Start practicing #

practice building pipelines with #kedro today

Go to your playground directory, and if you don’t have one, make one.

cd ~/playground

get pipx #

Install pipx in your system python. This is one of the very few, and possibly the only python library that deserves to be installed in your system directory, primarily because its used to sanbox clis in their own virtual environment automatically for you.

pip install pipx

make a new project #

From inside your playground directory, start your new kedro project. This is quite simple and painless. So much so that if you mess this one up doing something wild, it might be easier to make a new one that fixing the wild one.

pipx run kedro new
# answer the questions it asks

I use this quite often to try out new things in a safe place.

Make a virtual environment #

Using Conda #

Conda is a fine choice to manage your virtual environments. It used to make things so much easier on windows that it was almost required. Nowadays getting python running on windows has become so much easier that this is less so.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

one great benefit of conda is that it lets you choose the interpreter to go with your virtual environment.

Your new environment will be listed in your list of conda env here.

conda info --envs

Using venv #

venv is what I use now. Nothing against conda, it works great. venv just feels a bit lighter and more common. I’ve actually grown to appreciate that the venv is right where I put it, most often in the project directory.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

using pipenv #

pipenv is another fine choice. I like how in one command it makes the environment and activates it for you. pipenv also puts virtual environments in the global directory.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

Make pipelines #

Now go make some pipelines with your new project, try something wild, break it, and make another.

Running Kedro on Ubuntu 21.10 Impish Indri

2022-01-02T20:24:48Z

I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran

pipx run kedro new --starter spaceflights

But what I got back was not what I expected!

Fatal error from pip prevented installation. Full pip output in file:
    /home/walkers/.local/pipx/logs/cmd_2022-01-01_20.42.16_pip_errors.log

Some possibly relevant errors from pip install:
    ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
    ERROR: No matching distribution found for kedro
Error installing kedro.

This is weird, why cant I run kedro new with pipx? Lets try pip.

pip install kedro

Same issue.

ERROR: Could not find a version that satisfies the requirement kedro (from versions: none)
ERROR: No matching distribution found for kedro

What is Kedro

Curious what kedro is? Check out this article.

What’s up #

wrong python version

The issue is that kedro only runs on up to python 3.8, and on Ubuntu 21.10 when you apt install python3 you get python 3.9 and the standard repos don’t have an old enough version to run kedro.

How to fix this? #

Theres a couple of ways you can fix this? They all involve installing a distribution that does not come from the standard repo.

Where Can I get the right version #

Anaconda
Python.org
deadsnakes
pyenv
miniconda

I have two articles that can help you #

How to Install miniconda on linux (from the command line only)

Using miniconda

conda create -n myenv python=3.8

My first impressions with pyenv

Using pyenv

pyenv install 3.8.12

kedro catalog create

2021-11-15T07:18:55Z

I use kedro catalog create to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key.

https://youtu.be/_22ELT4kja4

What is Kedro

👆 Unsure what kedro is? Check out this post.

Running Kedro Catalog Create #

The command to ensure there are catalog entries for every dataset in the passed in pipeline.

kedro catalog create --pipeline history_nodes

Create’s new yaml file, if needed
Fills in new dataset entries with the default dataset
Keeps existing datasets untouched
it will reformat your yaml file a bit
- default sorting will be applied
- empty newlines will be removed

CONF_ROOT #

Kedro will respect your CONF_ROOT settings when it creates a new catalog file, or looks for existing catalog files. You can change the location of your configuration files by editing your CONF_ROOT variable in your projects.

settings.py.

# settings.py
# default settings
CONF_ROOT = "conf"

# I like to package my configuration
CONF_ROOT = str(Path(__file__).parent / "conf")

I prefer to keep my configuration packaged inside of my project. This is partly due to how my team operates and deploys pipelines.

File Location #

The kedro catalog create command will look for a yaml file based on the name of the pipeline (CONF_ROOT/catalog/<pipeline-name>.yml). If it does not find one it will create one and make entries for each dataset in the pipeline. It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline. If you are going to use this command its important that you follow this pattern or copy what it generates into your own catalog file of choice.

⚠️ It will not look in all of your existing catalog files for entries, only the one in the exact file for your pipeline.

MemoryDataSet’s #

When you run kedro catalog create you get MemoryDataSet, that’s it. As of 0.17.4 its hard coded into the library and not configurable.

range12:
  type: MemoryDataSet

Your free to use what you want though #

Let’s switch this dataset over to a pandas.CSVDataSet so that the file gets stored and we can pick up and read the file without re-running the whole pipeline.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv

Continue adding nodes #

As we work we will keep adding nodes to our kedro pipeline, in this case we added another node that created a dataset called range13.

kedro catalog create --pipeline history_nodes

After telling kedro to create new catalog entries for us we will see that it left our range12 entry alone and created range13 for us.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range13:
  type: MemoryDataSet

Formatting is not worthwhile #

If we decide this is too cramped for us we could add some space between datasets. The next time we run kedro catalog create empty lines will be removed.

range12:
  type: pandas.CSVDataSet

range13:
  type: MemoryDataSet

Continuing to work #

If we coninue adding new nodes, and tell kedro to create catalog entries again, all of our effort given to formatting will be lost. I wouldn’t worry about it unless you have an autoformatter that you can run on your yaml files. The productivity gains in an semi-automated catalog are worth it.

range12:
  type: pandas.CSVDataSet
  filepath: data/range12.csv
range121:
  type: MemoryDataSet
range13:
  type: MemoryDataSet

Sorting Order #

Notice the sorting order in the last entry, range121 comes before range13. This is all based on how pythons yaml.safe_dump works, kedro has set the default_flow_style to False. You can see where they write your file in the source code currently here

nvim conf 2021 | IDE's are slow | Waylon Walker

2021-10-29T00:00:00Z

https://youtu.be/E18m4KkJUnI

Slides 👇 #

welcome #

Other possible titles #

Using Vim as a Team Lead
I 💜 Tmux
Why I stopped using @code
Get there fast
How I vim

It’s ok #

Use a graphical IDE if it works for you.

Trick it out #

vim is so well integrated into the terminal, take advantage

It wasn’t working for me anymore #

dozens of instances #

As a team lead I bounce betweeen a dozen projects a per day

https://pbs.twimg.com/media/FAEmRjYUcAUk2eR?format=jpg&name=large

Move With Intent #

Running vim inside tmux lets me move swiftly between the exact project I need.

https://twitter.com/_WaylonWalker/status/1438849269407047686/photo/1 //: <> (__)

Hub and Spoke #

direct link to specific projects
fuzzy into all projects
fuzzy into open projects

How I navigate tmux in 2021#hub-and-spoke

Other Things That Make this Possible #

tmux
direnv

vim adjacent things

yes, vim is ugly, make it yours #

@rook

command! Q :q

@_waylonwalker

nnoremap <leader>6 <c-^>

lsp #

lua vim.lsp.buf.definition()

treesitter #

Plug 'nvim-treesitter/nvim-treesitter-textobjects'

telescope #

fuzzy matching like a boss

fzf is ok too

Check Messages #

Kedro-Broken-Urls

2021-10-20T00:00:00Z

Broken Urls #

Setting Parameters in kedro

2021-09-19T00:00:00Z

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files.

https://youtu.be/Jj5cQ5bqcjg

What is Kedro

👆 Unsure what kedro is? Check out this post.

parameters files #

You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a base and local parameters file.

conf/base/parameters.yml
conf/local/parameters.yml

base #

The base environment should contain all of the default values you want to run.

# /conf/base/parameters.yml
test_size: 0.2
random_state: 3
features:
  - engines
  - passenger_capacity
  - crew
  - d_check_complete
  - moon_clearance_complete
  - iata_approved
  - company_rating
  - review_scores_rating

NOTE base will always be loaded first.

accessing parameters #

Parameters can be accessed through context or through the catalog. Generally when you are working with nodes it will be loaded through the catalog.

Loding with the context.

context = session.load_context()
context.params

Loading with the catalog.

catalog.load('parameters')

Loading a specific key with the catalog.

catalog.load('params:test_size')

using parameters in nodes #

Here is an example from the complete spaceflights demo. The entire parameters dict is passed in, then the features key is accessed.

def split_data(data: pd.DataFrame, parameters: Dict) -> Tuple:
    """Splits data into features and targets training and test sets.

    Args:
        data: Data containing features and target.
        parameters: Parameters defined in parameters.yml.
    Returns:
        Split data.
    """
    X = data[parameters["features"]]
    y = data["price"]
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=parameters["test_size"], random_state=parameters["random_state"]
    )
    return X_train, X_test, y_train, y_test

Comprehensive guide to creating kedro nodes

👆 Check out this complete guide to creating kedro nodes.

local #

The local parameters by default are in conf/local/parameters.yml. They will override the base parameters in a shallow fashion. If a top level key exists in local, it will override that entire key in your parameters.

# /conf/local/parameters.yml
env: local
features:
  - company_rating
  - review_scores_rating

NOTE If you have not explicitly set your environment, local will be the default environment selected to override base.

env #

You can also have other environments that override the base environment.

# /conf/new/parameters.yml
env: new

NOTE if you use an env local will not be applied

Activating this environment #

Following the configuration docs we can activate the environment by setting an environment variable in our shell or passing in –env to our kedro cli command.

Setting an environment variable.

export KEDRO_ENV=test

Passing in the env to a kedro cli command.

kedro run --env=test

Setting the Environment Variable in python.

import os
os.environ['KEDRO_ENV'] = 'new'

Links #

Writing your first kedro Nodes

2021-09-13T22:40:45Z

https://youtu.be/-gEwU-MrPuA

Before we jump in with anything crazy, let’s make some nodes with some vanilla data structures.

import node #

You will need to import node from kedro.pipeline to start creating nodes.

from kedro.pipeline import node

func #

The func is a callable that will take the inputs and create the outputs.

inputs / outputs #

Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is the catalog entry to use for that keyword.

our first node #

Sometimes in our pipelines our data is coming from an api where we already have python functions built to pull with. Thats ok, kedro supposrts that with inputs=None.

def create_range():
    return range(100)

make_range = node(
    func=create_range,
    inputs=None,
    outputs='range'
    )

second node #

Now we have some data to work from, lets use that as our input.

def square_range():
    return [i**2 for i in range]

square_range = node(
    func=square_range,
    inputs='range',
    outputs='range_squared'
    )

Multiple Inputs #

Kedro can take lists or dicts as either input or output when your function needs more than one input or output.

def concat(range, range_two):
    return [*range, *range_two]

concat_ranges = node(
    func=concat,
    inputs=['range', 'range_squared']
    outputs='concat'
    )

## inputs could also be defined as a dict
concat_ranges = node(
    func=concat,
    inputs={'range': 'range', 'range_two': 'range_squared'}
    outputs='concat'
    )

Links #

Running your Kedro Pipeline from the command line

2021-08-24T22:40:45Z

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli.

https://youtu.be/ZmccpLy-OEI

What is Kedro

👆 Unsure what kedro is? Check out this post.

Kedro run #

To run the whole darn project all we need to do is fire up a terminal, activate our environment, and tell kedro to run.

kedro run

Specific Pipelines #

Running a sub pipeline that we have created is as easy as telling kedro which one we want to run.

kedro run --pipeline dp

Single Nodes #

While developing a node or a small list of nodes in a larger pipeline its handy to be able to run them one at a time. Besides the use case of developing a single node I would not reccomend leaning very heavy on running single nodes, let the DAG do the work of figuring out which nodes to run for you.

kedro run --pipeline dp --node create_model_input_table_node
kedro run --pipeline dp -n create_model_input_table_node

Some DAG concepts #

We will cover more of the benefits that we get from the graph nature of the DAG in the future, but here is a quick peek at some things we can do.

kedro run --pipeline dp --to-outputs preprocessed_shuttles
kedro run --pipeline dp --from-inputs preprocessed_shuttles
kedro run --pipeline dp --to-nodes create_model_input_table_node

Multiple things #

You can stack up multiple kedro dag concepts into a single run command.

kedro run --pipeline dp --to-nodes create_model_input_table_node --to-nodes preprocess_shuttles_node

kedro Virtual Environment

2021-08-23T20:40:45Z

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment.

https://youtu.be/ZSxc5VVCBhM

conda
venv
pipenv

conda #

I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don’t have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

conda info --envs

stores environment in a root directory i.e. ~/miniconda3
conda can use its own way to manage environments environment.yml
the python interpreter is packaged with the environment

virtualenv #

Virtual env (venv) is another very respectable option that is built right into python, and requires no additional installs or using a different distribution of pytyhon.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

environments are typically stored in the project directory
does not package the interpreter

pipenv #

Pipenv is another virtual enviroment tool that comes with its own system for managing dependencies using a pipfile. It’s main benefit is that it creates a lockfile that will allow users to replicate the exact version of all their packages. The typical requirements.txt workflow can easily break as new version of dependecies are released between testing and deplpoyment.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

stores environment in a root directory i.e. ~/.local/share/virtualenvs/
pipenv can use its own way to manage environments pipfile
does not package the interpreter

Kedro Pipeline Create

2021-08-22T22:40:45Z

Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself.

https://youtu.be/HtyIKqlEoNw

creating a new pipeline #

The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your pipeline_registry, to be covered later, you will need to add it yourself.

kedro pipeline create example

results #

The directory structure that it creates looks like this.

tree src/kedro_conda/pipelines
src/kedro_conda/pipelines
├── __init__.py
└── example
    ├── __init__.py
    ├── nodes.py
    ├── pipeline.py
    └── README.md

Kedro Install

2021-08-21T20:40:45Z

Kedro comes with an install command to install and manage all of your projects dependencies.

https://youtu.be/IWimEs-hHQg

cd into your project directory and activate env #

You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment.

Kedro New

this post covers kedro new

kedro Virtual Environment

This post covers creating your virtual environment for kedro

install kedro #

Make sure you have kedro installed in your current environment, if you dont already have it.

pip install kedro==0.17.4

pip-tools #

Kedro uses the pip-tools package under the hood to pin dependencies in a very robust way to ensure that the project will continue to work on everyone’s machine day, including production, day in and day out. No matter what happens to the dependencies you have installed.

pip-compile #

The command that kedro uses from pip-tools is pip-compile. It will look at what you have in a requirements.in file, compile the dependencies down to exact versions, and create a requirements.txt that is fully pinned down, and updatable by re-running pip-compile.

requirements.in #

If kedro does not see a requirements.in file it will automatically move your requirements.txt to requirements.in and run pip-compile.

No requirements.in found. Copying contents from requirements.txt...

kedro install #

Lets go ahead and run kedro install on one of the projects we already create and environment for in a previous post, kedro-conda.

kedro install

The first time you run this on a new repo it is likely that you will run into this warning about creating a new requirements.in file.

No requirements.in found. Copying contents from requirements.txt...

kedro install flags #

Kedro does let you avoid pip-compile all together, by using the --no-build-reqs flag.

kedro install --no-build-reqs

It also lets you upgrade all of your dependencies with build-reqs. I would reccomend doing this on its own branch, and own pull request. If you are working on a team you want everyone to be on the same page when it comes to dependencies. If you are not, you surely do not want something to break with a new set of dependencies without a way of rolling back.

kedro install --build-reqs

Kedro Git Init

2021-08-20T22:40:45Z

Immediately after kedro new, before you start running kedro install or your first line of code the first thing you should always do after getting a new kedro template created is to git init.

https://youtu.be/IGba3ytf_6U

git-init">git init #

Its as simple as these three commands to get started.

git init
git add .
git commit -m init

I don’t care if this project is for learning, if it will never have a remote or not, use git.

Kedro New

2021-08-18T20:40:45Z

https://youtu.be/uqiv5LAiJe0

Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template.

What is Kedro

Unsure what kedro is, Check out yesterdays post on What is Kedro.

pipx #

I reccomend using pipx when running kedro new. pipx is designed for system level cli tools so that you do not need to maintain a virtual environment or worry about version conflicts, pipx manages the environment for you.

The kedro team does not reccomend pipx in their docs as they already feel like there is a bit of a tool overload for folks that may be less familiar with

pipx kedro new

I like using pipx as it gives you better control over using a specific version or always the latest version, unlike when you run what you have on your system depends on when you last installed or upgraded.

Kedro New #

The kedro team also has a set of starters, by passing in --starter you can start with a different template. Here is an example with the kedro spaceflights starter.

pipx run kedro new --starter spaceflights

=============
Please enter a human readable name for your new project.
Spaces and punctuation are allowed.
 [New Kedro Project]: Spaceflights Complete

Repository Name:
================
Please enter a directory name for your new project repository.
Alphanumeric characters, hyphens and underscores are allowed.
Lowercase is recommended.
 [spaceflights-complete]:

Python Package Name:
====================
Please enter a valid Python package name for your project package.
Alphanumeric characters and underscores are allowed.
Lowercase is recommended. Package name must start with a letter
or underscore.
 [spaceflights_complete]:

Change directory to the project generated in /home/u_walkews/git/spaceflights-complete

A best-practice setup includes initialising git and creating a virtual environment before running ``kedro install`` to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

Other versions of kedro with pipx #

pipx not only ensures that you run the latest version, it can also run a very specific version.

pipx run --spec kedro==0.16.6 kedro new

kedro Virtual Environment

The next post in this series will help you create your virtual environment for your new kedro project.

What is Kedro

2021-08-17T12:48:00Z

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract DataSets that the user specifies through Catalog entries. These Catalog entries are loaded, ran through a function, and saved by Nodes. The order that these Nodes are executed are determined by the Pipeline, which is a DAG. It’s the runner’s job to manage the execution of the Nodes.

https://youtu.be/Wf4rnFsaFFU

What is Kedro

This is an updated version of my original what-is-kedro article

Hot Take #

If you are doing a series of operations to data with python, especially if you are using something as supported as pandas, you should be using a framework that gives you a pipeline as a DAG and abstracts io.

Orchestrators #

Like I said, kedro is unopinionated it does determine where or how your data should be ran. The kedro team does support the following Orchestrators with very little add on to the base template.

DataSets #

Did I say kedro is unopionated? Datasets are what allow kedro too be so flexible accross a number of different python objects. Any python object can be made into a kedro dataset. Kedro comes out of the box with many purpose built DataSets like storing pandas DataFrames to parquet, csv, or a sql table. If kedro does not come with support for the type of python objects you work with don’t worry, you can for the closest option they support and build your own. Or if you do not want to build your own, you can use a PickleDataSet for anything.

Catalog #

You will not often be creating your own datasets, most of what you need whould already be taken care of by the kedro framework. What you will need to do is to use the existing DataSets to build your data catalog.

Kedro takes care of all fo the file io for you, you simply need to use the catalog to tell kedro what type of DataSet to use and any extra information that DataSet needs. Much of the time this is simply a filepath.

Typically the catalog is specified in yaml format. If you are not familiar with yaml, I suggest learnxinyminutes.com/docs/yaml/ as a resource of examples.

test:
  type: pandas.CSVDataSet
  filepath: s3://your_bucket/test.csv #

Here is the most basic yaml catalog entry taken from the kedro docs

cars:
  type: pandas.CSVDataSet
  filepath: data/01_raw/company/cars.csv
    sep: ','
    load_args:
  save_args:
    index: False
    date_format: '%Y-%m-%d %H:%M'
    decimal: .

Here is a bit more complex example that takes in load_args and save_args docs

Nodes #

Nodes are a very core part of kedro to build the DAG. These nodes are what provides the definition of what catalog entries, get passed into which function, and output to another catalog entry.

import pandas as pd
import numpy as np

def clean_data(cars: pd.DataFrame,
               boats: pd.DataFrame) -> Dict[str, pd.DataFrame]:
    return dict(cars_df=cars.dropna(), boats_df=boats.dropna())

def halve_dataframe(data: pd.DataFrame) -> List[pd.DataFrame]:
    return np.array_split(data, 2)

nodes = [
    node(clean_data,
         inputs=['cars2017', 'boats2017'],
         outputs=dict(cars_df='clean_cars2017',
                      boats_df='clean_boats2017')),
    node(halve_dataframe,
         'clean_cars2017',
         ['train_cars2017', 'test_cars2017']),
    node(halve_dataframe,
         dict(data='clean_boats2017'),
         ['train_boats2017', 'test_boats2017'])
]

Here is an example of three nodes taken from their docs

Pipeline #

The kedro Pipeline, is a DAG (Directed Acyclic Graph). It is a graph object that flows in one direction. You can slice into the pipeline using a few built in graph method to_nodes, from_nodes, to_outputs, and from_inputs. You can chain up these method calls since each one returns a new Pipeline object. You can also ask a pipline for its edges with inputs, and outputs. You can also list every dataset along the way with all_inputs or all_outputs. Lastly you can convert it back into a list of nodes with nodes.

from kedro.pipeline import Pipeline, node

# using our nodes from last tim
Pipeline(nodes)

Runner #

The runner is the bridge between kedro and the orchestrators. The kedro team provides some basic runners for running pipelines locally, built right into the framework, but adding on new runners for different orchestrators is done through the use of adding in a new runner to your project.

Hooks #

Kedro allows you to hook into a number of lifecycle methods through the use of the pluggy framework. Yes the one that pytest is built on. There are a number of different lifecycle methods that allow us to hook in around where kedro is running such as before_pipeline_run or after_catalog_loaded.

Links #

How I Kedro

2021-08-16T22:34:45Z

https://youtu.be/bw5_FWDVRpU

Ubuntu #

I recently switched over to using Ubuntu, it works well pretty much out of the box for me. I am using gnome with a dark theme.

Gnome Terminal #

I am still using the built in default gnome terminal, it just works. It does all the things that I need it to do. It supports transparency renders my fonts and allows me to highlight things well.

One Dark Theme

dotfiles #

You can find my dotfiles on github. Feel free to read through and take anything that you find useful. I would encourage you not to steal them, but to integrate the parts that you want into your own dotfiles. dotfiles are a very personal thing. They are an extension of ones fingertips designed for how you think and type.

zsh #

I use zsh as my default shell. I like to use it as my interactive shell. It works, and does a bit better with things like tab completion out of the box.

starship #

I use the starship prompt for my shell. It works well out of the box. It looks good and includes all of the information that I would ever need.

tmux #

As a team lead I am in and out of many projects per day, tmux allows me to get in and out of these files with super speed. I was using a mix of vscode and tmux in until October 2020. At this point I got moved development machines and pushed myself to use only the terminal. I felt that vscode was just getting slower and slower, and I was getting less benefit from it. Especially now that the lsp is a part of nvim.

neovim #

I really like the raw speed and customizability of neovim. You can see all the customization, and plugins I have added in my dotfiles.

OneBuddy Theme
pylsp
kedro-lsp

ipython #

I really like ipython, it lets me edit code in my code editor, then import it or run it quickly. Ipython gives me the right level of tooling. I don’t need markdown mixed in my code, I put those notes into docstrings, a readme, or wiki. When I need to see plots I just store them as png or html and view them in my browser.

I do a bit of customization to my ipython session that you can find in my dotfiles repo. I use a custom prompt and use rich formatting and tracebacks if rich is installed.

custom prompt
rich traceback

Links #

Incremental Versioned Datasets in Kedro

2021-07-08T11:12:25Z

Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work.

What is Kedro

👆 Unsure what kedro is? Check out this post.

How does our dataset change over time?? #

This was a question presented to me at work. We had some plots being produces as the output of our pipeline and the user wanted the ability to compare results over time. Luckily this was asked early in the project so we were able to proactively setup versioning on the right datasets.

To enable this all we needed to do now was to add versioned: true and we will be able to compare results over time. Yes kedro makes it that easy to setup.

set up a project #

Set up a new project just as usual. note I like using pipx for global cli packages. You can pick a specific version of kedro or opt for the latest while simply globally installing kedro and running kedro new is purely dependent on the last time you chose to update kedro.

pip install pipx
pipx run kedro new

cd versioned-partitioned-kedro-example
conda create -n versioned-partitioned-kedro-example python=3.8  -y
conda activate versioned-partitioned-kedro-example

pip install kedro
kedro install

git init
git add .
git commit -m "init project from pipx run kedro new"

I called my project versioned-partitioned-kedro-example. You can call your project whatever you like. If you try to use some special characters where they don’t belong, kedro will catch you. Under the hood, kedro is using a library called cookiecutter

⚠️ Please do not skip out on using a virtual environment. You may use whichever virtual environment tool you prefer, but please do not skip out. Wrecking a running project for learning is not fun.

update dependencies #

I popped open my dependencies, added kedro[pandas] and find-kedro. Since those are extra packages our example will require.

aiohttp
black==21.5b1
find-kedro
flake8>=3.7.9, <4.0
ipython
isort~=5.0
jupyter_client>=5.1, <7.0
jupyterlab~=3.0
jupyter~=1.0
kedro-telemetry~=0.1.0
kedro==0.17.4
kedro[pandas]
nbstripout~=0.4
pytest-cov~=2.5
pytest-mock>=1.7.1, <2.0
pytest~=6.2
requests
wheel>=0.35, <0.37

note I created find-kedro, and I like using it to create my pipeline object. Think of how pytest automatically picks up everything named test, find-kedro does the same thing for kedro. It picks up everything with node or pipeline in the name and creates pipelines out of it.

Install new dependencies #

After adding our additional dependencies to the requirements.in, we can tell kedro to install everything and compile the dependencies. Behind the scenes --build-reqs uses a library called pip-compile to create a requirements.txt file with hard pinned dependencies, which is ideal for creating reproducible projects. You and your future colleagues may not thank you for this, but they sure as heck won’t be cursing your name when they can’t get the project to run.

kedro install --build-reqs

git add .
git commit -m "added additional dependencies"

create a node #

For this example, we need a node to do much. This node will pass the cars.csv from a URL to a parquet file. I am going to use a lambda to build my identity function inline.

# pipelines/cars_nodes.py

from kedro.pipeline import node

nodes = []

nodes.append(
        node(
            func=lambda x:x,
            inputs='raw_cars',
            outputs='int_cars',
            name='create_int_cars',
            )
        )

🗒️ notefind-kedro' will automatically pick up these nodes for us after we set up our pipeline_registry.py`.

git add .
git commit -m "add create_int_cars node"

implement find-kedro #

Next, we need to tell kedro where our nodes are. This is where find-kedro comes in. Once we point to the directory where our modules of nodes/pipelines are, it creates the pipelines dictionary for us automatically. It will even separate each module into a pipeline and stitch them all into one default pipeline.

# pipeline_registry.py

"""Project pipelines."""
from typing import Dict
from pathlib import Path

from kedro.pipeline import Pipeline

from find_kedro import find_kedro


def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    Returns:
        A mapping from a pipeline name to a "Pipeline "object.
    """
    pipeline_dir = Path(__file__).parent / 'pipelines'
    return find_kedro(directory= pipeline_dir)

🗒️ This is very similar to the default ` pipeline_registry’except the last two lines.

git add .
git commit -m "implement find-kedro"

create a baseline catalog #

Once we have a pipeline setup, the kedro cli can automatically fill in missing catalog entries with MemoryDataSet’s. Thus, using the cli helps consistently scaffold the catalog and ensure we don’t end up with a typo in our dataset name.

kedro catalog create --pipeline cars_nodes

Kedro will kick out the following catalog file to base/catalog/cars_nodes.yml for us to get started with.

raw_cars:
  type: MemoryDataSet
int_cars:
  type: MemoryDataSet

🔥 use the kedro cli to fill in any missing datasets from the automatically catalog.

make a versioned dataset #

Kedro has scaffolded MemoryDataSet ’s for us. We will convert them to the appropriate dataset type and turn on versioning for our int layer, which is the first point we save in our environment.

raw_cars:
  type: pandas.CSVDataSet
  filepath: https://waylonwalker.com/cars.csv
int_cars:
  type: pandas.ParquetDataSet
  filepath: data/int_cars.parquet
  versioned: true

Commit your changes to the catalog.

git add .
git commit -m "create catalog"

run the pipeline #

Once we have the nodes and catalog setup, we can run the pipeline a few times to get some versioned data. Each time we run, it will save a new version inside the int_cars.parquet directory.

kedro run
kedro run
kedro run
kedro run
kedro run

🗒️ we put our data in the data directory. By default, this directory is included in the .gitignore and will not be picked up by git.

inspect the data #

Listing the files in data/int_cars.parquet shows that I now have five different datasets available. I can load old ones, but by default, kedro will load the latest one.

ls data/int_cars.parquet

2021-07-05T15.24.53.164Z
2021-07-05T15.29.56.144Z
2021-07-05T15.30.23.101Z
2021-07-05T15.30.26.555Z
2021-07-05T15.31.12.688Z

🗒️ kedro sets the version at the timestamp that the session starts. All datasets created within the same run will have the same version.

stack on an incremental dataset #

This is where things get interesting. Kedro comes with an incremental dataset that will load all of the files from a particular directory into a dictionary where the keys are the filename of the dataset. To load up all datasets into this dictionary all we need to do is add a new catalog entry that is a type: PartitionedDataSet, with a path pointing to the same place as the original, and a dataset type the same as the original.

int_cars_partitioned:
  type: PartitionedDataSet
  dataset: pandas.ParquetDataSet
  path: data/int_cars.parquet

catalog list #

Listing the catalog entries confirms that we have successfully added our new PartitionedDataSet.

In [17]: context.catalog.list()
Out[17]:
['raw_cars',
 'int_cars',
 'int_cars_partitioned',
 'parameters']

loading an incremental dataset #

Now we can easily load the datasets from every run we just did into a single dictionary, simply by running context.catalog.load('int_cars_incremental').

In [19]: context.catalog.load('int_cars_incremental')
2021-07-05 11:32:40,534 - kedro.io.data_catalog - INFO - Loading data from `int_cars_incremental` (IncrementalDataSet)...
Out[19]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.30.23.101Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.30.26.555Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2,
 '2021-07-05T15.31.12.688Z/int_cars.parquet':              Unnamed: 0   mpg  cyl   disp   hp  drat     wt   qsec  vs  am  gear  carb
 0             Mazda RX4  21.0    6  160.0  110  3.90  2.620  16.46   0   1     4     4
 1         Mazda RX4 Wag  21.0    6  160.0  110  3.90  2.875  17.02   0   1     4     4
 2            Datsun 710  22.8    4  108.0   93  3.85  2.320  18.61   1   1     4     1
 3        Hornet 4 Drive  21.4    6  258.0  110  3.08  3.215  19.44   1   0     3     1
 4     Hornet Sportabout  18.7    8  360.0  175  3.15  3.440  17.02   0   0     3     2
 5               Valiant  18.1    6  225.0  105  2.76  3.460  20.22   1   0     3     1
 6            Duster 360  14.3    8  360.0  245  3.21  3.570  15.84   0   0     3     4
 7             Merc 240D  24.4    4  146.7   62  3.69  3.190  20.00   1   0     4     2
 8              Merc 230  22.8    4  140.8   95  3.92  3.150  22.90   1   0     4     2
 9              Merc 280  19.2    6  167.6  123  3.92  3.440  18.30   1   0     4     4
 10            Merc 280C  17.8    6  167.6  123  3.92  3.440  18.90   1   0     4     4
 11           Merc 450SE  16.4    8  275.8  180  3.07  4.070  17.40   0   0     3     3
 12           Merc 450SL  17.3    8  275.8  180  3.07  3.730  17.60   0   0     3     3
 13          Merc 450SLC  15.2    8  275.8  180  3.07  3.780  18.00   0   0     3     3
 14   Cadillac Fleetwood  10.4    8  472.0  205  2.93  5.250  17.98   0   0     3     4
 15  Lincoln Continental  10.4    8  460.0  215  3.00  5.424  17.82   0   0     3     4
 16    Chrysler Imperial  14.7    8  440.0  230  3.23  5.345  17.42   0   0     3     4
 17             Fiat 128  32.4    4   78.7   66  4.08  2.200  19.47   1   1     4     1
 18          Honda Civic  30.4    4   75.7   52  4.93  1.615  18.52   1   1     4     2
 19       Toyota Corolla  33.9    4   71.1   65  4.22  1.835  19.90   1   1     4     1
 20        Toyota Corona  21.5    4  120.1   97  3.70  2.465  20.01   1   0     3     1
 21     Dodge Challenger  15.5    8  318.0  150  2.76  3.520  16.87   0   0     3     2
 22          AMC Javelin  15.2    8  304.0  150  3.15  3.435  17.30   0   0     3     2
 23           Camaro Z28  13.3    8  350.0  245  3.73  3.840  15.41   0   0     3     4
 24     Pontiac Firebird  19.2    8  400.0  175  3.08  3.845  17.05   0   0     3     2
 25            Fiat X1-9  27.3    4   79.0   66  4.08  1.935  18.90   1   1     4     1
 26        Porsche 914-2  26.0    4  120.3   91  4.43  2.140  16.70   0   1     5     2
 27         Lotus Europa  30.4    4   95.1  113  3.77  1.513  16.90   1   1     5     2
 28       Ford Pantera L  15.8    8  351.0  264  4.22  3.170  14.50   0   1     5     4
 29         Ferrari Dino  19.7    6  145.0  175  3.62  2.770  15.50   0   1     5     6
 30        Maserati Bora  15.0    8  301.0  335  3.54  3.570  14.60   0   1     5     8
 31           Volvo 142E  21.4    4  121.0  109  4.11  2.780  18.60   1   1     4     2}

👆 notice that incremental datasets are all loaded for you, its a dict of filepath:dataset

stack on a partitioned dataset #

Let’s take a look at a similar type of dataset called PartitionedDataSet. We can add it to the catalog in a very similar way to how we added the IncrementalDataSet.

int_cars_incremental:
  type: IncrementalDataSet
  dataset: pandas.ParquetDataSet
  path: data/int_cars.parquet

loading a partitioned dataset #

Note that we get a dict with the same keys as before, but this time the values are a load function rather than loaded data. Partitioned datasets can be helpful if you are operating on datasets that take up more memory than you have available. In our case of coupling this with versioned datasets, its likely to grow quite large, so PartitionedDataSet ’s are likely a better option for this use.

In [18]: context.catalog.load('int_cars_partitioned')
2021-07-05 11:31:11,253 - kedro.io.data_catalog - INFO - Loading data from `int_cars_partitioned` (PartitionedDataSet)...
Out[18]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570820>>,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570850>>,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb1570910>>,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': <bound method AbstractVersionedDataSet.load of <kedro.extras.datasets.pandas.parquet_dataset.ParquetDataSet object at 0x7f4bb15709a0>>}

incremental vs. partitioned #

IncrementalDataSet ’s and PartitionedDataSet ’s are very similar as they give you access to a whole directory of data that uses the same underlying dataset loader. The significant difference is whether you want your data pre-loaded or if you want to load and dispose of it as you iterate over it.

incremental loads the data
partitioned give a load function

creating nodes with partitioned datasets #

Let’s create a node with this PartitionedDataSet to collect stats on our dataset over time. This node does a dict comprehension to get the length of each version that we pulled.

def timeseries_partitioned(cars: Dict):
    return {k:len(car()) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_partitioned,
            inputs='int_cars_partitioned',
            outputs='int_cars_timeseries_partitioned',
            name='create_int_cars_timeseries_partitioned',
            )
        )

🗒️ note that inside of the dict comprehension car is a load function that we need to call.

creating nodes with incremental datasets #

Doing the same node with our IncrementalDataSet looks very similar, except this time car is loaded data inside of the dict comprehension, not a function that we need to call.

def timeseries_incremental(cars: Dict):
    return {k:len(car) for k, car in cars.items()}

nodes.append(
        node(
            func=timeseries_incremental,
            inputs='int_cars_incremental',
            outputs='int_cars_timeseries_incremental',
            name='create_int_cars_timeseries_incremental',
            )
        )

More catalog entries #

After adding those nodes, we can add the catalog entries again with the command line. This will not overwrite any of the datasets we just created. It will only add to it.

kedro catalog create --pipeline cars_nodes

int_cars_timeseries_partitioned:
  type: MemoryDataSet
int_cars_timeseries_incremental:
  type: MemoryDataSet

int_cars_timeseries_partitioned:
  type: pickle.PickleDataSet
  filepath: data/int_cars_timeseries_partitioned.parquet
int_cars_timeseries_incremental:
  type: pickle.PickleDataSet
  filepath: data/int_cars_timeseries_incremental.parquet

Loading the new datasets #

Loading the two dtasets that we just created show that we have the ended up with the same result using both incremental and partitioned datasets. This result is a dictionary of filepaths mapped to the size of the dataset. Since the default filepaths are timestamps we could start doing some time series analysis to see how our dataset is changing over time.

In [32]: context.catalog.load('int_cars_timeseries_incremental')
2021-07-05 12:00:55,014 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_incremental` (PickleDataSet)...
Out[32]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
 '2021-07-05T16.43.43.088Z/int_cars.parquet': 32}

In [33]: context.catalog.load('int_cars_timeseries_partitioned')
2021-07-05 12:01:03,223 - kedro.io.data_catalog - INFO - Loading data from `int_cars_timeseries_partitioned` (PickleDataSet)...
Out[33]:
{'2021-07-05T15.29.56.144Z/int_cars.parquet': 32,
 '2021-07-05T15.30.23.101Z/int_cars.parquet': 32,
 '2021-07-05T15.30.26.555Z/int_cars.parquet': 32,
 '2021-07-05T15.31.12.688Z/int_cars.parquet': 32,
 '2021-07-05T16.43.43.088Z/int_cars.parquet': 32,
 '2021-07-05T16.50.46.686Z/int_cars.parquet': 32}

Kedro - My Data Is Not A Table

☝️ I have a full article on creating datasets that are not tabular datasets using pickle.

cross posted to dev #

If you like this article give it some ❤️🦄🏷️ on dev https://dev.to/waylonwalker/incremental-versioned-datasets-in-kedro-3ajn

This post was primarily built live on https://twitch.tv/waylonwalker, give me a follow and join in the live show if that is something that interests you.

Posts tagged: kedro

💭 Migration guide for config loaders — kedro 0.19.11 documentation

Make Kedro Runs Beautiful

Install kedro rich #

Kedro run #

Kedro catalog #

Give it a star #

Lambda Function as a Kedro Node

Examples #

Add New Dependencies to Your Kedro Project

git-status">git status #

requirements.in #

build-reqs #

git add #

Practice making pipelines with kedro

Start practicing #

get pipx #

make a new project #

Make a virtual environment #

Using Conda #

Using venv #

using pipenv #

Make pipelines #

Running Kedro on Ubuntu 21.10 Impish Indri

What’s up #

How to fix this? #

Where Can I get the right version #

I have two articles that can help you #

kedro catalog create

Running Kedro Catalog Create #

CONF_ROOT #

File Location #

MemoryDataSet’s #

Your free to use what you want though #

Continue adding nodes #

Formatting is not worthwhile #

Continuing to work #

Sorting Order #

nvim conf 2021 | IDE's are slow | Waylon Walker

Slides 👇 #

welcome #

Other possible titles #

It’s ok #

Trick it out #

It wasn’t working for me anymore #

dozens of instances #

Move With Intent #

Hub and Spoke #

Other Things That Make this Possible #

yes, vim is ugly, make it yours #

lsp #

treesitter #

telescope #

Check Messages #

Kedro-Broken-Urls

Broken Urls #

Setting Parameters in kedro

parameters files #

base #

accessing parameters #

using parameters in nodes #

local #

env #

Activating this environment #

Links #

Writing your first kedro Nodes

import node #

func #

inputs / outputs #

our first node #

second node #

Multiple Inputs #

Links #

Running your Kedro Pipeline from the command line

Kedro run #

Specific Pipelines #

Single Nodes #

Some DAG concepts #

Multiple things #

Related Links #