Running your Kedro Pipeline from the command line


Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli.

👆 Unsure what kedro is? Check out this post.

Kedro run

To run the whole darn project all we need to do is fire up a terminal, activate our environment, and tell kedro to run.

kedro run 

Specific Pipelines

Running a sub pipeline that we have created is as easy as telling kedro which one we want to run.

kedro run --pipeline dp

Single Nodes

While developing a node or a small list of nodes in a larger pipeline its handy to be able to run them one at a time. Besides the use case of developing a single node I would not reccomend leaning very heavy on running single nodes, let the DAG do the work of figuring out which nodes to run for you.

kedro run --pipeline dp --node create_model_input_table_node
kedro run --pipeline dp -n create_model_input_table_node

Some DAG concepts

We will cover more of the benefits that we get from the graph nature of the DAG in the future, but here is a quick peek at some things we can do.

kedro run --pipeline dp --to-outputs preprocessed_shuttles
kedro run --pipeline dp --from-inputs preprocessed_shuttles
kedro run --pipeline dp --to-nodes create_model_input_table_node

Multiple things

You can stack up multiple kedro dag concepts into a single run command.

kedro run --pipeline dp --to-nodes create_model_input_table_node --to-nodes preprocess_shuttles_node