Posts tagged: kedro Archive

💭 Migration guide for config loaders — kedro 0.19.11 documentation

hello@waylonwalker.com — Wed, 05 Feb 2025 18:16:44 +0000

!https://docs.kedro.org/en/stable/configuration/config_migration.html

Make Kedro Runs Beautiful

hello@waylonwalker.com — Fri, 11 Mar 2022 13:14:51 +0000

Kedro rich is a very new and unstable (it's good, just not ready) plugin for kedro to make the command line prettier.

Lambda Function as a Kedro Node

hello@waylonwalker.com — Mon, 31 Jan 2022 21:52:36 +0000

I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to...

Add New Dependencies to Your Kedro Project

hello@waylonwalker.com — Fri, 28 Jan 2022 21:30:48 +0000

As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command under the hood to...

Practice making pipelines with kedro

hello@waylonwalker.com — Thu, 27 Jan 2022 21:07:37 +0000

I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In...

Running Kedro on Ubuntu 21.10 Impish Indri

hello@waylonwalker.com — Sun, 02 Jan 2022 20:24:48 +0000

I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command...

kedro catalog create

hello@waylonwalker.com — Mon, 15 Nov 2021 07:18:55 +0000

I use to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing...

nvim conf 2021 | IDE's are slow | Waylon Walker

hello@waylonwalker.com — Fri, 29 Oct 2021 00:00:00 +0000

https://youtu.be/E18m4KkJUnI

Kedro-Broken-Urls

hello@waylonwalker.com — Wed, 20 Oct 2021 00:00:00 +0000

[ ] ) [ ] - [ ] - [ ] - [ ] - [ ] ) - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] - [ ] . - [ ]

Setting Parameters in kedro

hello@waylonwalker.com — Sun, 19 Sep 2021 00:00:00 +0000

Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing...

Writing your first kedro Nodes

hello@waylonwalker.com — Mon, 13 Sep 2021 22:40:45 +0000

https://youtu.be/-gEwU-MrPuA

Running your Kedro Pipeline from the command line

hello@waylonwalker.com — Tue, 24 Aug 2021 22:40:45 +0000

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on...

kedro Virtual Environment

hello@waylonwalker.com — Mon, 23 Aug 2021 20:40:45 +0000

Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual...

Kedro Pipeline Create

hello@waylonwalker.com — Sun, 22 Aug 2021 22:40:45 +0000

Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself.

Kedro Install

hello@waylonwalker.com — Sat, 21 Aug 2021 20:40:45 +0000

Kedro comes with an command to install and manage all of your projects dependencies.

Kedro Git Init

hello@waylonwalker.com — Fri, 20 Aug 2021 22:40:45 +0000

Immediately after , before you start running or your first line of code the first thing you should always do after getting a new kedro template created is to .

Kedro New

hello@waylonwalker.com — Wed, 18 Aug 2021 20:40:45 +0000

https://youtu.be/uqiv5LAiJe0

What is Kedro

hello@waylonwalker.com — Tue, 17 Aug 2021 12:48:00 +0000

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that...

How I Kedro

hello@waylonwalker.com — Mon, 16 Aug 2021 22:34:45 +0000

https://youtu.be/bw5_FWDVRpU Ubuntu # I recently switched over to using Ubuntu, it works well pretty much out of the box for me. I am using gnome with a dark theme. Gnome Terminal # I am still using the built in default gnome terminal, it just works. It does all the things that I need it to do. It supports transparency renders my fonts and allows me to highlight things well. One Dark Theme dotfiles # You can find my dotfiles on github. Feel free to read through and take anything that...

Incremental Versioned Datasets in Kedro

hello@waylonwalker.com — Thu, 08 Jul 2021 11:12:25 +0000

Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is...

I Started Streaming on Twitch

hello@waylonwalker.com — Sat, 12 Jun 2021 10:11:00 +0000

I recently started streaming on twitch.tv/waylonwalker and it's been a blast so far.

Upcoming Stream

hello@waylonwalker.com — Tue, 08 Jun 2021 20:27:53 +0000

!!! Caution I'm no longer streaming As much as I would really love to make streaming work, its really hard for my family situation to make large blocks of...

Kedro Spaceflights - part 2 | Stream replay June 7, 2021

hello@waylonwalker.com — Mon, 07 Jun 2021 16:15:04 +0000

This was my seconf time ever streaming on twitch.tv/waylonwalker, and I completely botched my mic 2x.

Kedro Spaceflights - part 1 | Stream replay June 4, 2021

hello@waylonwalker.com — Fri, 04 Jun 2021 16:15:04 +0000

This was my first time ever streaming on twitch.tv/waylonwalker. I am excited to get going. I have been streaming early in the morning while I am still...

Comprehensive guide to creating kedro nodes

hello@waylonwalker.com — Thu, 03 Jun 2021 21:30:35 +0000

The Kedro node is an essential part of the pipeline. It defines what catalog entries get passed in, what function gets ran, and the catalog entry to save the...

Creating pypi-list with kedro

hello@waylonwalker.com — Sun, 23 May 2021 22:02:16 +0000

I had an idea come to me via twitter. Short one word name packages are becoming hard to find on pypi. Short one word readable package names that are not a...

Using Kedro In Scripts

hello@waylonwalker.com — Sun, 23 May 2021 11:00:44 +0000

With the latest releases of kedro , it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it...

Silence Kedro Logs

hello@waylonwalker.com — Thu, 20 May 2021 13:13:38 +0000

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while...

Vim Fugitive

hello@waylonwalker.com — Sat, 08 May 2021 22:51:53 +0000

Custom Kedro Logger

hello@waylonwalker.com — Sun, 02 May 2021 09:45:22 +0000

DRAFT -

kedro replit

hello@waylonwalker.com — Thu, 15 Apr 2021 12:37:29 +0000

I am trying to see what an embeded replit

Kedro pipeline_registry.py

hello@waylonwalker.com — Sat, 20 Mar 2021 00:00:00 +0000

With the realease of came a new module in the project template . Here are some notes that I learned while playing with this new module.

Minimal Kedro Pipeline

hello@waylonwalker.com — Wed, 20 Jan 2021 00:00:00 +0000

How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in...

Kedro Dependency Management

hello@waylonwalker.com — Tue, 19 Jan 2021 00:00:00 +0000

https://kedro.readthedocs.io/en/stable/04_project_dependencies.html?highlight=install

Kedro - My Data Is Not A Table

hello@waylonwalker.com — Thu, 14 Jan 2021 00:00:00 +0000

In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask.

Testing Data Pipelines

hello@waylonwalker.com — Mon, 16 Nov 2020 06:00:00 +0000

Thinking through testing best practices for Data Pipelines

reasons-to-kedro

hello@waylonwalker.com — Sun, 01 Nov 2020 05:00:00 +0000

There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines What is Kedro Starter Template # Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. conda create -n my-new-project -y python=3.8 kedro...

Reasons to Kedro

hello@waylonwalker.com — Sat, 31 Oct 2020 05:00:00 +0000

collaboration - Sharable catalog - small nodes over monolithic notebooks - catalog - easily load anything without needing to run - No need to write...

What's New in Kedro 0.16.6

hello@waylonwalker.com — Sun, 25 Oct 2020 05:00:00 +0000

This version of kedro releases a new set of supported deployment options and the spaceflights pipeline is officially added as a starter alias.

A brain dump of stories

hello@waylonwalker.com — Wed, 21 Oct 2020 05:00:00 +0000

I started making stories as kind of a brain dump a few times per day and posting them to LinkedIn. Here are the last 11 days of stories.

Kedro Basics

hello@waylonwalker.com — Sat, 08 Aug 2020 05:00:00 +0000

In my upcoming free kedro course, you can learn how to start building pipelines in 5 days.

What's New in Kedro 0.16.4

hello@waylonwalker.com — Sat, 01 Aug 2020 05:00:00 +0000

If we take a look at the release notes I see one ** feature improvement on the list, auto-discovery of hooks.

Kedro Catalog

hello@waylonwalker.com — Fri, 24 Jul 2020 05:00:00 +0000

I am exploring a kedro catalog meta data hook

Gracefully adopt kedro, the catalog

hello@waylonwalker.com — Mon, 29 Jun 2020 03:00:00 +0000

While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually....

How to find things in your kedro catalog

hello@waylonwalker.com — Mon, 22 Jun 2020 03:00:00 +0000

kedro 0.16.2 just dropped last week with a long-awaited feature... **! I went as far as monkey patching this into each of my projects. I work jump between a...

How Kedro handles your inputs

hello@waylonwalker.com — Fri, 19 Jun 2020 03:00:00 +0000

Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing...

004

hello@waylonwalker.com — Sun, 31 May 2020 19:22:07 +0000

🔥 #kedrotips use find-kedro to assembly your pipelines

kedro-hooks/src/kedro-hooks/preflight.py

hello@waylonwalker.com — Sun, 31 May 2020 04:37:25 +0000

🔥 #kedrotips hooks can be created using modules

002

hello@waylonwalker.com — Thu, 28 May 2020 23:48:00 +0000

** 0.3.0 just launched with _ support 🎉

Kedro Static Viz 0.3.0 is out with Hooks Support

hello@waylonwalker.com — Thu, 28 May 2020 05:00:00 +0000

kedro-static-viz is out with support for the newly released hooks feature. This means that you can have automatically deploy a full gatsby site keeping your...

make sure to answer yes to adding example data your first time

hello@waylonwalker.com — Tue, 26 May 2020 12:48:00 +0000

** building pipelines with _ today

Brainstorming Kedro Hooks

hello@waylonwalker.com — Fri, 22 May 2020 22:02:00 +0000

This post is a 🧠 branstorming work in progress. I will likely use it as a storage location/brain dump of hook ideas.

Create Custom Kedro Dataset

hello@waylonwalker.com — Fri, 15 May 2020 05:00:00 +0000

Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For...

creating the kedro-preflight hook

hello@waylonwalker.com — Sun, 10 May 2020 07:12:00 +0000

Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into ,, and (nouns). With a , or (adjective). This really...

📝 Kedro Preflight Notes

hello@waylonwalker.com — Sat, 09 May 2020 15:01:00 +0000

run checks before running the pipeline

📢 Announcing find-kedro

hello@waylonwalker.com — Mon, 04 May 2020 11:53:00 +0000

kedro is an amazing project that allows for super-fast prototyping of data pipelines, yet yielding production-ready pipelines. find-kedro enhances this experience by adding a pytest-like node discovery eliminating the need to bubble up pipelines through modules.

Create New Kedro Project

hello@waylonwalker.com — Mon, 02 Mar 2020 12:09:00 +0000

This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro. You...

What is Kedro

hello@waylonwalker.com — Mon, 24 Feb 2020 12:48:00 +0000

what-is-kedro

Kedro

hello@waylonwalker.com — Sat, 02 Nov 2019 05:00:00 +0000

My Notes about using kedro