-
💭 Migration guide for config loaders — kedro 0.19....
Here's my thought on 💭 Migration guide for config loaders — kedro 0.19.11 documentation Migrating from kedro 0.18.4 to the latest version involves handling…
-
Make Kedro Runs Beautiful
Kedro rich is a very new and unstable (it's good, just not ready) plugin for kedro to make the command line prettier. Install kedro rich There is no pypi package yet, but it's on github. You can pip install it with the git url. Kedro run You can run your pipeline just as you normally would, except you get progress bars and pretty prints. kedro rich pretty run Kedro catalog Listing out catalog entries from the command line now print out a nice pretty table. kedro rich catalog list table output G
-
Lambda Function as a Kedro Node
I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas. Examples Here are two examples, the first one is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow sta
-
Add New Dependencies to Your Kedro Project
As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let's take a look at it. git status Before you start mucking around with any changes to dependencies make sure that your git status is clean. I'd even reccomend starting a n
-
Practice making pipelines with kedro
I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity. [[ what-is-kedro ]] Start practicing practice building pipelines with #kedro t
-
Running Kedro on Ubuntu 21.10 Impish Indri
I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran But what I got back was not what I expected! This is weird, why cant I run kedro new with pipx? Lets try pip. Same issue. [[ what-is-kedro ]] Curious what kedro is? Check out this article. What's up wrong python version The issue is that kedro only runs on up to , and on Ubuntu 21.10 when you you get and the standard rep
-
kedro catalog create
I use to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key. https://youtu.be/_22ELT4kja4 {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Running Kedro Catalog Create The command to ensure there are catalog entries for every dataset in the pass
-
nvim conf 2021 | IDE's are slow | Waylon Walker
https://youtu.be/E18m4KkJUnI {.youtube-embed} Slides 👇 welcome Other possible titles Using Vim as a Team Lead I 💜 Tmux Why I stopped using @code Get there fast How I vim It's ok Use a graphical IDE if it works for you. Trick it out vim is so well integrated into the terminal, take advantage It wasn't working for me anymore dozens of instances As a team lead I bounce betweeen a dozen projects a per day https://pbs.twimg.com/media/FAEmRjYUcAUk2eR?format=jpg&name=large {.hoverlink} Move With Intent
-
Kedro-Broken-Urls
Broken Urls [ ] https://github.com/josephhaaga ) [ ] https://example.com/file.h5 [ ] https://raw.githubusercontent.com/kedro-org/kedro/develop/static/img/pipeline_visualisation.png [ ] https://example.com/file.txt [ ] https://github.com/jmespath/jmespath.py . [ ] https://github.com/tsanikgr ) [ ] https://example.com/file.csv [ ] https://kedro.readthedocs.io/en/latest/04_user_guide/15_hooks.html [ ] https://kedro.readthedocs.io/en/stable/07_extend_kedro/04_hooks.html [ ] https://github.
-
Setting Parameters in kedro
Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files. https://youtu.be/Jj5cQ5bqcjg {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. parameters files You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a and para
-
Writing your first kedro Nodes
https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla data structures. import node You will need to import node from kedro.pipeline to start creating nodes. func The is a callable that will take the and create the . inputs / outputs Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is
-
Running your Kedro Pipeline from the command line
Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli. https://youtu.be/ZmccpLy-OEI {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Kedro run To run the whole darn proj
-
kedro Virtual Environment
Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment. https://youtu.be/ZSxc5VVCBhM {.youtube-embed} conda venv pipenv conda I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don't have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool. st
-
Kedro Pipeline Create
Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself. https://youtu.be/HtyIKqlEoNw creating a new pipeline The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your , to be covered later, you will need to add it yourself. results The directory structure that it creates looks like this.
-
Kedro Install
Kedro comes with an command to install and manage all of your projects dependencies. https://youtu.be/IWimEs-hHQg cd into your project directory and activate env You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment. [[ kedro-new ]] this post covers kedro new [[ kedro-environment ]] This post covers creating your virtual environment for kedro install kedro Make sure you have kedro installed in your cur
-
Kedro Git Init
Immediately after , before you start running or your first line of code the first thing you should always do after getting a new kedro template created is to . https://youtu.be/IGba3ytf_6U git init Its as simple as these three commands to get started. I don't care if this project is for learning, if it will never have a remote or not, use git.
-
Kedro New
https://youtu.be/uqiv5LAiJe0 {.youtube-embed} Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template. [[ what-is-kedro ]] Unsure what kedro is, Check out yesterdays post on What is Kedro. pipx I reccomend using when running kedro new. is designed for system level cli tools so that you do not
-
What is Kedro
Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract that the user specifies through entries. These entries are loaded, ran through a function, and saved by . The order that these are executed are determined by the , which is a DAG . It's the 's job to manage the execution of the . https://youtu.be/Wf4rnFsaFFU {.youtube-embed}
-
How I Kedro
https://youtu.be/bw5_FWDVRpU Ubuntu I recently switched over to using Ubuntu, it works well pretty much out of the box for me. I am using gnome with a dark theme. Gnome Terminal I am still using the built in default gnome terminal, it just works. It does all the things that I need it to do. It supports transparency renders my fonts and allows me to highlight things well. One Dark Theme dotfiles You can find my dotfiles on github. Feel free to read through and take anything that you find use
-
Incremental Versioned Datasets in Kedro
Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work. [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. How
-
I Started Streaming on Twitch
I recently started streaming on twitch.tv/waylonwalker and it's been a blast so far. python kedro Data Science Data Engineering webdev digital gardening Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. kedro-issue-606 Inspiration My introduction to twitch c
-
Upcoming Stream
!!! Caution I'm no longer streaming As much as I would really love to make streaming work, its really hard for my family situation to make large blocks of time work for me. https://stackoverflow.com/questions/16720541/python-string-replace-regular-expression {.hoverlink} I am starting to stream 3 days per week, before I start work in the morning. These streams will likely be me just talking through things I am already doing. Making DAGs do 🔮Magical Things | Open Source 🐍Python | kedro plugins |
-
Kedro Spaceflights - part 2 | Stream replay June 7...
This was my seconf time ever streaming on twitch.tv/waylonwalker , and I completely botched my mic 2x. https://youtu.be/_7MwgKu-844 Links Spaceflights Tutorial my spaceflights repo Notes to get started
-
Kedro Spaceflights - part 1 | Stream replay June 4...
This was my first time ever streaming on twitch.tv/waylonwalker . I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go. https://youtu.be/Y07UBr9Ccjs Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time co
-
Comprehensive guide to creating kedro nodes
The Kedro node is an essential part of the pipeline. It defines what catalog entries get passed in, what function gets ran, and the catalog entry to save the results under. does this link work? https://waylonwalker.com/what-is-kedro/ {.hoverlink} 👆 Unsure what kedro is? Check out this post. The node function The node function is the most common and reccomended way to define kedro nodes. It is a function that constructs and returns objects for you. Creating your first kedro node function The
-
Using Kedro In Scripts
With the latest releases of kedro , it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro [[ what-is-kedro ]] If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors as of 0.17.2 I've tried to do this in kedro and it turned into a rabbit hole of erro
-
Creating pypi-list with kedro
I had an idea come to me via twitter. Short one word name packages are becoming hard to find on pypi. Short one word readable package names that are not a play on words are easy to remember, easy to spell correctly, and quick to type out. Simple index I started with the simple index. Pypi provides a single page listing to every single package hosted on pypi via the simple-index
-
Silence Kedro Logs
Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log First, how does one silence a python log? Python loggers can be retrieved by the module's function. Then their log level can be changed. Much of kedro's chattiness comes from INFO level logs. I don't want to hear about anything for my current use case unless i
-
Vim Fugitive
Add current file and commit with diff in a split :on[ly] C-W o :on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default. C-I C-O cycle through the jumplist This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize and , I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific c
-
Custom Kedro Logger
DRAFT -
-
kedro replit
I am trying to see what an embeded replit
-
Kedro pipeline_registry.py
With the realease of came a new module in the project template . Here are some notes that I learned while playing with this new module. migrating to create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your . pipeline_registry only works in Conflict Resolution What happens If I register pipelines in both places I w
-
Minimal Kedro Pipeline
How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in and 27 in . 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline I have everything for this post hosted in this gihub repo , you can fork it, clone it, or just follow along. Installation Caveats This repo represents the minimal amount of structure to build a ked
-
Kedro Dependency Management
Docs https://kedro.readthedocs.io/en/stable/04_kedro_project_setup/01_dependencies.html?highlight=install pip-tools pip-compile requirements requirements.in requirements.txt
-
Kedro - My Data Is Not A Table
In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. DataFrames are the heart of most pipelines These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types. [[ what-is-kedro ]] unfamiliar with kedro, check out this post Sometimes datasets are not t
-
Testing Data Pipelines
Lint/Format/Doc black flake8 interrogate mypy Pipeline Assertions pipeline constructs pipeline as expected nodes pipeline has minimum nodes test minimum tags test alternate tags Catalog Assertions test catalog follows naming structure Node Tests test function does the correct operations on test data Great Expectations
-
reasons-to-kedro
There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines [[ what-is-kedro ]] Starter Template Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. [[ create-new-kedro-project ]] read more about how
-
Reasons to Kedro
Reasons to Kedro collaboration Sharable catalog small nodes over monolithic notebooks catalog easily load anything without needing to run No need to write read/write code pipeline No need to keep execution order in your head easily run a slice of a pipeline plugins pip install make your own hooks flexible expandable cli Reasons Not to Kedro Already utilizing another DAG framework Data is not in a widely supported format Micro short-lived project Large Project / Deadline Use a lower profile proje
-
What's New in Kedro 0.16.6
Kedro 0.16.6 is out! Let's take a look through the release notes Deployment Docs This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Argo Prefect Kubeflow Batch SageMaker Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies. Argo, Prefect, and Kubeflow have an interesting technique where
-
A brain dump of stories
I started making stories as kind of a brain dump a few times per day and posting them to [LinkedIn](https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/). Here are the last 11 days of stories. I store all the stories on my website with the hopes of doing something with them on my own platform eventually. For now it makes it easy to make these posts. Stories 10-10-2020 - 10-21-2020
-
Kedro Basics
Learn Kedro in 5 days Day 0 Setup vm install python editor Day 1 kedro new kedro viz Day 2 catalog filter catalog load data fsspec Day 3 pipeline nodes Day 4 filter pipeline run partial pipeline Day 5 kedro docker GitHub Actions Advanced Kedro hooks custom datasets modular pipelines
-
What's New in Kedro 0.16.4
If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks. This one comes a bit surprising as it was just casually mentioned in #435 auto enabled plugins mentioned in issue 435 Think pytest As mentioned in #435 this is the model that pytest uses. Not all plugins automatically start doing things right out of the box but require a CLI argument. simplicity It feels a bit crazy that simply installing a package will change the way that your
-
Kedro Catalog
I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. Process metadata will be attached to the dataset object under a attribute metadata will be updated metadata will be empty until a pipeline is ran with the hook on optionally a function to add metadata will be added metadata will be stored in a file next to the meta Problems This Hook Should solve what datasets have a columns with in the name what datasets were updated after last tuesday which pipe
-
Gracefully adopt kedro, the catalog
Why use kedro catalog? While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all. Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different fo
-
How to find things in your kedro catalog
kedro 0.16.2 just dropped last week with a long-awaited feature... catalog search ! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful. The Catalog The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place
-
How Kedro handles your inputs
Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing. *args/**args review Check out this post for a review of how work in python. [[ python-args-kwargs ]] python args and kwargs article by @_waylonwalker All Kedro inputs are catalog Entries When kedro runs your pipeline it uses the catalog to imperatively load your data, mea
-
004
🔥 #kedrotips use find-kedro to assembly your pipelines
-
003
🔥 #kedrotips hooks can be created using modules
-
002
kedro-static-viz 0.3.0 just launched with hooks support 🎉
-
Kedro Static Viz 0.3.0 is out with Hooks Support
kedro-static-viz is out with support for the newly released hooks feature. This means that you can have automatically deploy a full gatsby site keeping your visualization always up to date. Even though it is a static site there is no functionality lost. The only thing that's missing is the flask server. With kedro-static-viz you can deploy your visualization to a number of static hosting providers such as GitHub pages free of charge with wicked fast performance ⚡ It's Fast Even though
-
001
practice building pipelines with #kedro today
-
Brainstorming Kedro Hooks
This post is a 🧠 branstorming work in progress. I will likely use it as a storage location/brain dump of hook ideas. What is Kedro 🤔 If you are completely unsure what kedro is be sure to check out my what is kedro post after_catalog_created filepath replacer bucket replacer before_pipeline_run preflight check that data exists run run mypy run interrogate run flake8 after_pipeline_run Great Expectations send email send slack before_node_run after_node_run Great Expectations save stats/meta da
-
Create Custom Kedro Dataset
Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key. But what is happening behind the scenes Under the hood there is an that each connector inherits from. It sets up a lot of the behind the scenes structure for us so
-
creating the kedro-preflight hook
Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and (nouns). With a , or (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability. kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and
-
📝 Kedro Preflight Notes
This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources. Must Haves check that inputs exist or are of a type to skip (sql) Good to haves check that all input and output databases are accessible with good credentials check for s3 bucket access check for spark install Implementation run params
-
📢 Announcing find-kedro
is a small library to enhance your kedro experience. It looks through your modules to find kedro pipelines, nodes, and iterables (lists, sets, tuples) of nodes. It then assembles them into a dictionary of pipelines, each module will create a separate pipeline, and being a combination of all pipelines. This format is compatible with the kedro format. Python package Test Build-Docs Motivation is a ✨ fantastic project that allows for super-fast prototyping of data pipelines, while yielding
-
Create New Kedro Project
This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro . You can learn more about this Hello World Example in the docs 🧹 Install Kedro 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization Create a Virtual Environment I use conda to control my virtual environments and will create a new environment called with the following command. note the latest compatible vers
-
What is Kedro
[[ what-is-kedro ]] This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate and . This article will cover the 10K view of kedro, future articles will dive deper into each one. kedro is an open-source data pipeline framework. It pr
-
Kedro
See all of my kedro related posts in [[ kedro-feed ]]. #kedrotips I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips . 🗣 Heads up Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful mo