Kedro pipeline_registry.py

2021-03-20

edit✏️

With the realease of kedro==0.17.2 came a new module in the project template pipeline_registry.py. Here are some notes that I learned while playing with this new module.

migrating to pipeline_registry.py

• create a src/<package-name>/pipeline_registry.py file create a
• register_pipelines function in pipeline_registry.py that mirrors the
• register_pipelines method from your hooks.py module do not bring the
• hook_impl decorator remove register_pipelines method on your ProjectHooks
• class

You should now have something that looks like this in your src/<package-name>/pipeline_registry.py.

"""Project pipelines."""
from typing import Dict

from kedro.pipeline import Pipeline


def register_pipelines() -> Dict[str, Pipeline]:
    """Register the project's pipelines.

    Returns: A mapping from a pipeline name to a ``Pipeline`` object.
    """
    return {"__default__": Pipeline([])}

pipeline_registry only works in kedro>=0.17.2

Conflict Resolution

What happens If I register pipelines in both places

I was not able to find any official documentation on how conflict resolution worked so I stepped into a project and added to both my hooks.py and pipeline_registry.py file. I noticed that it would pick up pipelines from both modules, but pipelines from hooks.py always take precedence. The entire duplicate pipeline will be over written by the one from hooks.py.

kedro automatically merges pipelines from both hooks.py takes precedence

Ready to update

In my experience there were no issues upgrading from 0.17.1 to 0.17.2. I would reccomend only having one register_pipelines so decide to migrate to the new pipeline_registry.py or keep it in your hooks.py, but both is only going to lead to confusion.

https://twitter.com/datajoely/status/1375193511264456705