Published

With the latest releases of kedro 0.17.x, it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro # [1] What is Kedro [2] If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors # [3] as of 0.17.2 I’ve tried to do this in kedro 0.16.x, and it turned into a rabbit hole of errors. First kedro needed a conf directory, if you tried to fake one in it would then ask for logging setup. These errors just kept coming to the point it wasnt worth doing and I might as well use a proper template for real projects and stick to simple function calls for things that are not a kedro project. Kedro in a script # [4] To get kedro running, you will need a pipeline, catalog, and runner at a minimum. Those who have used kedro before the pipeline will look v...

Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log # [1] First, how does one silence a python log? Python loggers can be retrieved by the logging module’s getLogger function. Then their log level can be changed. Much of kedro’s chattiness comes from INFO level logs. I don’t want to hear about anything for my current use case unless it’s essential, i.e., a failure. In this case, I set the log levels to ERROR as most errors should stop execution anyways. python logging levels # [2] Level Numeric value CRITICAL 50 ERROR 40 WARNING 30 INFO 20 DEBUG 10 NOTSET 0 Get or Create a logger # [3] Getting a python logger is straightforward if we know the name of the logger. The following block will grab the logger object for the logger currently registered under the name passed in. logger = logging.getLog...

change_speed = (speed) => [...document.querySelectorAll('video')].map(v => v.playbackRate=v.playbackRate+speed) Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error # [1] Minimum reporducible error is one of my superpowers in development. I do this very often to sus out what is really happening. My day to day work is processing data with python, I keep a number of very small data sets handy to break and fix. This helps separate complexities of the project and the problem. Let’s break it # [2] Markata has a lot going on. It’s a plugins all the way down static site generator built in python. Trying to find the root cause through the layers ...

change_speed = (speed) => [...document.querySelectorAll('video')].map(v => v.playbackRate=v.playbackRate+speed) In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into logical groups. Update # [1] Since making this post, I have made ~20 other posts in short form that all have a YouTube video to go along with them you can find them all on my tmux-playlist [2]. Chris Toomey’s [3] Tmux Course # [4] I took Chris’s tmux course [5] in December 2020 and it was fantastic. Even as a seasoned tmux user, I learned quite a bit. Before the course, I was proficient in navigating within each of my tmux sessions but rarely started more than one session. A few months later, I have adopted a lot of what I learned from Chris and made it my own. I am now keeping projects to their own session and can move between them fluidly with just a few keystrokes. For high-traffic projects, I have them bound to a si...

Trim branches no longer on origin # [1] git remote prune origin --dry-run git remote prune origin Find branches already merged # [2] git checkout main # list remote branches that have already been merged into main git branch -r --merged # list local branches that have already been merged into main git branch --merged References: [1]: #trim-branches-no-longer-on-origin [2]: #find-branches-already-merged

change_speed = (speed) => [...document.querySelectorAll('video')].map(v => v.playbackRate=v.playbackRate+speed) When a python module is called it is assigned the __name__ of __main__ otherwise if it’s imported it will be assigned the __name__ of the module. Concrete example # [1] Let’s create a module to play with __name__ a bit. We will call this module nodes.py. It is a module that we may want to run by it’self or import and use in other modules. #!python # nodes.py if __name__ == "nodes": import sys import __main__ print(f"you have imported me {__name__} from {sys.modules['__main__'].__file__}") if __name__ == "__main__": print("you are running me as main") I have set this module up to execute one of two if statements based on whether the module it’self is being ran or if the module is being imported. Note it is not common to have a if __name__ == "nodes": block, this is just for demnonstration purposes. running python nodes.py # [2] Running a python script with the...

Creating a directory that is a union of several directories can be achieved with a few symlinks at the command line. Creating a Virtual File Gallery # [1] Here is how I am creating a virtual directory of all my projects that is a combination of both work and not-work projects. I am creating symlinks for every directory under ~/work and ~/git. rm -rf ~/projects mkdir ~/projects ln -sf ~/work/* ~/projects ln -sf ~/git/* ~/projects ⚠ Notice that first I am recreating the directory each time. This will ensure that any project that is deleted from their actual directory is removed from the virtual gallery. Updating the gallery # [2] Since links are always kept up to date without any extra work, all the data is still in the same place it started. But as new directories are added to any project directory they will not be automatically added to the virtual gallery. - cron - bashrc/zshrc If you’re concerned about system resources, you can add it to a cron job to run at a regular sch...