Published

Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli. https://youtu.be/ZmccpLy-OEI What is Kedro [1] 👆 Unsure what kedro is? Check out this post. Kedro run # [2] To run the whole darn project all we need to do is fire up a terminal, activate our environment, and tell kedro to run. kedro run Specific Pipelines # [3] Running a sub pipeline that we have created is as easy as telling kedro which one we want to run. kedro run --pipeline dp Single Nodes # [4] While developing a node or a small list of nodes in a larger pipeline its handy to be able to run them one at a time. Besides the use case of developing a single node I would not reccomend leaning very heavy on running single nodes, le...

Avoid serious version conflict issues, and use a virtual environment [1] anytime you are running python, here are three ways you can setup a kedro virtual environment. https://youtu.be/ZSxc5VVCBhM - conda - venv - pipenv conda # [2] I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don’t have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool. conda create -n my-project python=3.8 -y conda activate my-project python -m pip install --upgrade pip pip install -e src conda info --envs - stores environment in a root directory i.e. ~/miniconda3 - conda can use its own way to manage environments environment.yml - the python interpreter is packaged with the environment virtualenv # [3] Virtual env (venv) is another very respectable option that is built right into python, and requires no additional installs or using a different dis...

Kedro comes with an install command to install and manage all of your projects dependencies. https://youtu.be/IWimEs-hHQg cd into your project directory and activate env # [1] You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment. Kedro New [2] this post covers kedro new kedro Virtual Environment [3] This post covers creating your virtual environment [4] for kedro install kedro # [5] Make sure you have kedro installed in your current environment, if you dont already have it. pip install kedro==0.17.4 pip-tools # [6] Kedro uses the pip-tools package under the hood to pin dependencies in a very robust way to ensure that the project will continue to work on everyone’s machine day, including production, day in and day out. No matter what happens to the dependencies you have installed. pip-compile # [7] The command that kedro uses from pip-tools is pip-compile. It will look at what yo...

Immediately after kedro new, before you start running kedro install or your first line of code the first thing you should always do after getting a new kedro template created is to git init. https://youtu.be/IGba3ytf_6U git init # [2] Its as simple as these three commands to get started. git init git add . git commit -m init I don’t care if this project is for learning, if it will never have a remote or not, use git. References: [1]: /glossary/git/ [2]: #git-init

https://youtu.be/uqiv5LAiJe0 Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template. What is Kedro [1] Unsure what kedro is, Check out yesterdays post on What is Kedro. pipx # [2] I reccomend using pipx when running kedro new. pipx is designed for system level cli tools so that you do not need to maintain a virtual environment [3] or worry about version conflicts, pipx manages the environment for you. The kedro team does not reccomend pipx in their docs as they already feel like there is a bit of a tool overload for folks that may be less familiar with pipx kedro new I like using pipx as it gives you better control over using a specific version or always the latest version, unlike when you run what you have on your system depends on when you last installed or upgraded. Kedro Ne...

Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract DataSets that the user specifies through Catalog entries. These Catalog entries are loaded, ran through a function, and saved by Nodes. The order that these Nodes are executed are determined by the Pipeline, which is a DAG. It’s the runner’s job to manage the execution of the Nodes. https://youtu.be/Wf4rnFsaFFU --- What is Kedro [1] This is an updated version of my original what-is-kedro article --- Hot Take # [2] If you are doing a series of operations to data with python, especially if you are using something as supported as pandas, you should be using a framework that gives you a pipeline as a DAG and abstracts io. Orchestrators # [3] Like I said, kedro is unopinionated it does determine where or how your data should be ran. The kedro team does support the following ...

https://youtu.be/bw5_FWDVRpU Ubuntu # [1] I recently switched over to using Ubuntu, it works well pretty much out of the box for me. I am using gnome with a dark theme. Gnome Terminal # [2] I am still using the built in default gnome terminal, it just works. It does all the things that I need it to do. It supports transparency renders my fonts and allows me to highlight things well. - One Dark Theme dotfiles # [3] You can find my dotfiles [4] on github. Feel free to read through and take anything that you find useful. I would encourage you not to steal them, but to integrate the parts that you want into your own dotfiles. dotfiles are a very personal thing. They are an extension of ones fingertips designed for how you think and type. zsh # [5] I use zsh as my default shell. I like to use it as my interactive shell. It works, and does a bit better with things like tab completion out of the box. starship # [6] I use the starship prompt for my shell. It works well out of the...

https://youtu.be/LLk94fKpGg4 As we push the limits of tmux further and further you are bound to end up in a situation where you are mashing down a hotkey and it’s just not doing what you want it to do, and you have no idea why. show-messages is a tmux command that can be used to show what tmux is actually doing behind the scenes. This might highlight any hot key conflicts you might have in your ~/.tmux.conf. man page for show-messages # [1] In case you wnat more information about show-messages, here is the man page. show-messages [-JT] [-t target-client] (alias: showmsgs) Show server messages or information. Messages are stored, up to a maximum of the limit set by the message-limit server option. -J and -T show debugging information about jobs and terminals. How I navigate tmux in 2021 [2] for more information on how I navigate tmux, check out this full post Also check out the full YouTube tmux-playlist [3] to see all of the videos in this series. References: [1]: #man-...

https://youtu.be/nT0FA1RNZEs Now your creating, jumping, and killing sessions like a boss. You are slicing through projects with ease, let me show you one more thing that can be the cream on top of this silky smooth setup we have been working towards. Chris Toomey’s [1] Tmux Course # [2] This script is simply my fork of Chris Toomey’s tat script straight out of his course. It helps us create or jump to project specific sessions with ease. a directory of projects # [3] My version of the ta script will let you pass it a directory, and it will give you a fuzzy popup. ta ~/git setting up a keybinding # [4] bind C-g display-popup -E "ta ~/git" [5] default layout # [6] By default I have my projects open with a vertical split, vim is on top, with my file finder open and the lower split is set to just my terminal. This is what I do 90% of the time that I open a project anyways. [7] More projects # [8] I also have a directory setup that is a symlink-gallery of all of my projec...

https://youtu.be/utfLA6L8o5s You’ve got some long running tasks, and you’re trying to stay productive and knock tasks off that board, but you keep finding that your processes finish and you stay on other tasks for longer than you should. You were in the flow and just did not check back in on it. With display-message you can send yourself a notification when that long running task is complete. from the man page # [1] Here is a snippet of display-message from the tmux man page. I rarely need to do anything other than just display message, but there are other flags for it. display-message [-aINpv] [-c target-client] [-d delay] [-t target-pane] [message] (alias: display) Display a message. If -p is given, the output is printed to stdout, otherwise it is displayed in the target-client status line for up to delay milliseconds. If delay is not given, the message-time option is used; a delay of zero waits for a key press. ‘N’ ignores key presses and closes only after the delay e...

https://youtu.be/QWPyYx54JbE Now it’s time to switch gears, we are onto a different part of our day and there are just too many sessions running and we need to clean up shop. kill-server # [1] One viable option is to nuke the whole dang thing. I actually do this more than you might think. tmux kill-server save and commit your work diligently before kill-server kill-session # [2] A more reasonable option might be to kill a single session. # kills the current session tmux kill-session # kills the session named scratch tmux kill-session -t scratch choose-tree # [3] Killing sessions one by one from the command line can be a bit tedious, and involve more keystrokes than necessary. Another option built right into tmux is choose-tree. By default choose-tree is bound to prefix+s, that’s pressing control+b then s. Once you are in choose-tree, you can navigate around with your configured navigation scheme, press x to kill a session, or pane or window then y to confirm. You can also ...