I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity.

What is Kedro

Start practicing #

practice building pipelines with #kedro today

Go to your playground directory, and if you don’t have one, make one.

cd ~/playground

get pipx #

Install pipx in your system python. This is one of the very few, and possibly the only python library that deserves to be installed in your system directory, primarily because its used to sanbox clis in their own virtual environment automatically for you.

pip install pipx

make a new project #

From inside your playground directory, start your new kedro project. This is quite simple and painless. So much so that if you mess this one up doing something wild, it might be easier to make a new one that fixing the wild one.

pipx run kedro new
# answer the questions it asks

I use this quite often to try out new things in a safe place.

Make a virtual environment #

Using Conda #

Conda is a fine choice to manage your virtual environments. It used to make things so much easier on windows that it was almost required. Nowadays getting python running on windows has become so much easier that this is less so.

conda create -n my-project python=3.8 -y
conda activate my-project
python  -m pip install --upgrade pip
pip install -e src

one great benefit of conda is that it lets you choose the interpreter to go with your virtual environment.

Your new environment will be listed in your list of conda env here.

conda info --envs

Using venv #

venv is what I use now. Nothing against conda, it works great. venv just feels a bit lighter and more common. I’ve actually grown to appreciate that the venv is right where I put it, most often in the project directory.

python -m venv .venv
source ./.venv/bin/activate
python  -m pip install --upgrade pip
pip install -e src

using pipenv #

pipenv is another fine choice. I like how in one command it makes the environment and activates it for you. pipenv also puts virtual environments in the global directory.

pipx run pipenv shell
python  -m pip install --upgrade pip
pip install -e src

Make pipelines #

Now go make some pipelines with your new project, try something wild, break it, and make another.

One of the first things I noticed broken in my terminal based workflow moving from Windows wsl to ubuntu was that my clipboard was all messed up and not working with my terminal apps. Luckily setting tmux and neovim to work with the system clipboard was much easier than it was on windows.

First off you need to get xclip if you don’t already have it provided by your distro. I found it in the apt repositories. I have used it between Ubuntu 18.04 and 21.10 and they all work flawlessly for me.

I have tmux setup to automatically copy any selection I make to the clipboard by setting the following in my ~/.tmux.conf. While I have neovim open I need to be in insert mode for this to pick up.

# ~/tmux.conf
bind -T copy-mode-vi Enter send-keys -X copy-pipe-and-cancel "xclip -i -f -selection primary | xclip -i -selection clipboard"
bind-key -T copy-mode-vi MouseDragEnd1Pane send-keys -X copy-pipe-and-cancel "xclip -selection clipboard -i"

To get my yanks to go to the system clipboard in neovim, I just added unnamedplus to my existing clipboard variable.

# ~/.config/nvim/init.vim
set clipboard+=unnamedplus

If you need to copy something right from the terminal you can use xclip directly. I do this semi-often to send someone a message in chat.

cat file.txt | clip -sel copy

I set up some alias’s for doing this a bit more efficiently, but don’t find myself using them very often. This helps me grab commands from history and copy them.

alias hclip="history | tail -n1 | cut -c 8- | xclip -sel clip"
alias fclip="history -n 1000 | fzf | cut -c 8- | xclip -sel clip"
alias fclip="history -n 1000 | fzf | xclip -sel clip"

With the latest version of minecraft it requires a very new, possibly the latest, version of java. Lately we have been getting into modded minecraft and I maintain the server for us. It’s been tricky to say the least. One hurdle I recently hit involves having the wrong version of java.

I was getting this error trying to get a 1.12.2 forge server running.

Caused by: java.lang.ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and java.net.URLClassLoader are in module java.base of loader ‘bootstrap’)

In researching our errors, I found this on a forum.

Pre-1.13 Forge only works with Java 8.

I don’t write java, or really know how to manage different versions of java, but I have nixpkgs installed and it has a ton of odd stuff like this readily available, so searching nixpkgs landed me with this.

nix-env -iA nixpkgs.jdk8

once I had this installed I then just changed out java for the full path to my new nixpkgs.jdk8 java and it worked.

/home/walkers/.nix-profile/bin/java -server -Xms${MIN_RAM} -Xmx${MAX_RAM} ${JAVA_PARAMETERS} -jar ${SERVER_JAR} nogui

I don’t write java or do anything other than host minecraft servers wtih it. There is probably a better way of maintaining java versions than this, but this worked for me.

I have added a hotkey to my copier template setup to quickly access all my templates at any time from tmux. At any point I can hit <c-b><c-b>, thats holding control and hitting bb, and I will get a popup list of all of my templates directory names. Its an fzf list, which means that I can fuzzy search through it for the template I want, or arrow key to the one I want if I am feeling insane. I even setup it up so that the preview is a list of the files that come with the template in tree view.

bind-key c-b popup -E -w 80% -d '#{pane_current_path}' "\
    pipx run copier copy ~/.copier-templates/`ls ~/.copier-templates |\
    fzf --header $(pwd) --preview='tree ~/.copier-templates/{} |\
    lolcat'` . \
    "

I’ve had this on my systems for a few weeks now and I am constantly using it for my tils, blogs, and my .envrc file that goes into all of my projects to make sure that I have a virtual environment installed and running any time I open it.

I often pop into my blog from neovim with the intent to look at just a single series of posts, til, gratitude, or just see todays posts. Markata has a great way of mapping over posts and returning their path that is designe exactly for this use case.

To tie these into a Telescope picker you add the command as the find_command, and comma separate the words of the command, with no spaces. I did also --sort,date,--reverse in there so that the newest posts are closest to the cursor.

nnoremap geit <cmd>Telescope find_files find_command=markata,list,--map,path,--filter,date==today<cr>
nnoremap geil <cmd>Telescope find_files find_command=markata,list,--map,path,--filter,templateKey=='til',--sort,date,--reverse<cr>
nnoremap geig <cmd>Telescope find_files find_command=markata,list,--map,path,--filter,templateKey=='gratitude',--sort,date,--reverse<cr>

NOTE telescope treates each word as a string, do not wrap an extra layer of quotes around your words, it gets messy.

Copier allows you to run post render tasks, just like cookiecutter. These are defined as a list of tasks in your copier.yml. They are simply shell commands to run.

The example I have below runs an update-gratitude bash script after the copier template has been rendered.

# copier.yml
num: 128
_answers_file: .gratitude-copier-answers.yml
_tasks:
  - "update-gratitude"

I have put the script in ~/.local/bin so that I know it’s always on my $PATH. It will reach back into the copier.yml and update the default number.

#!/bin/bash
# ~/.local/bin/update-gratitude
current=`awk '{print $2}' ~/.copier-templates/gratitude/copier.yml | head -n 1`
new=`expr $current + 1`
echo $current
echo $new
sed -i "s/$current/$new/g" ~/.copier-templates/gratitude/copier.yml

I’ve referenced a video from Anthony Sotile in passing conversation several times. Walking through his gradual typing process has really helped me understand typing better, and has helped me make some projects better over time rather than getting slammed with typing errors.

https://youtu.be/Rk-Y71P_9KE

Step 1

Run Mypy as is, don’t get fancy yet. This will not reach into any functions unless they are alreay explicitly typed. It will not enforce you to type them either.

pip install mypy
mypy .
# or your specific project to avoid .venvs
mypy src
# or a single file
mypy my-script.py

Step 2 #

Next we will add check-untyped-defs, this will start checking inside functions that are not typed. To add this to your config create a setup.cfg with the following.

[mypy]
check_untyped_defs = True

Step 3 #

The final stage to this series is to add disallow_untyped_defs. This will start requiring all of your functions to be type hinted. This one is probably the toughest, because as you type functions mypy can uncover more issues for you to fix. Often times the list of errors grows before it shrinks.

[mypy]
check_untyped_defs = True
disallow_untyped_defs = True

Anthony’s video #

Make sure that you watch Anthony’s video, give him a sub, he deserves it for all the great things he is doing for the python community.

https://www.youtube.com/watch?v=Rk-Y71P_9KE

In order to make an auto title plugin for markata I needed to come up with a way to reverse the slug of a post to create a title for one that does not explicitly have a title.

 a slug is generally all lowercase and free of spaces, and is a way to

Here I have a path available that gives me the articles path, ex. python-reverse-sluggify.md. An easy way to get rid of the file extension, is to pass it into pathlib.Path and ask for the stem, which returns python-reverse-sluggify. Then from There I chose to replace - and _ with a space.

article["title"] = (
    Path(article["path"]).stem.replace("-", " ").replace("_", " ").title()
)

To turn this into a markata plugin I put it into a pre_render hook.

from pathlib import Path

from markata.hookspec import hook_impl, register_attr


@hook_impl
@register_attr("articles")
def pre_render(markata) -> None:
    for article in markata.filter('title==""'):
        article["title"] = (
            Path(article["path"]).stem.replace("-", " ").replace("_", " ").title()
        )

I really appreciate that in linux anything can be scripted, including setting the wallpaper. So everytime I disconnect a monitor I can just rerun my script and fix my wallpaper without digging deep into the ui and fussing through a bunch of settings.

feh --bg-scale ~/.config/awesome/wallpaper/my_wallpaper.png

I set my default wallpaper with feh using the command above.

Leaning in on feh, we can use fzf to pick a wallpaper from a directory full of wallpapers with very few keystrokes.

alias wallpaper='ls ~/.config/awesome/wallpaper | fzf --preview="feh --bg-scale ~/.config/awesome/wallpaper/{}" | xargs -I {} feh --bg-scale ~/.config/awesome/wallpaper/{}'

I have mine alias’d to wallpaper so that I can quickly run it from my terminal.

Getting docstrings from python’s ast is far simpler and more reliable than any method of regex or brute force searching. It’s also much less intimidating than I originally thought.

Parsing #

First you need to load in some python code as a string, and parse it with ast.parse. This gives you a tree like object, like an html dom.

py_file = Path("plugins/auto_publish.py")
raw_tree = py_file.read_text()
tree = ast.parse(raw_tree)

Getting the Docstring #

You can then use ast.get_docstring to get the docstring of the node you are currently looking at. In the case of freshly loading in a file, this will be the module level doctring that is at the very top of a file.

module_docstring = ast.get_docstring(tree)

Walking for all functions #

To get all of the functions docstrings we can use ast.walk to look for nodes that are an instance of ast.FunctionDef, then run get_docstring on those nodes.

functions = [f for f in ast.walk(tree) if isinstance(f, ast.FunctionDef)]
function_docs = [ast.get_docstring(f) for f in functions]

ast.walk docs: Recursively yield all descendant nodes in the tree starting at node (including node itself), in no specified order. This is useful if you only want to modify nodes in place and don’t care about the context.

Example #

Here is an image of me running this example through ipython.

Many tools such as ripgrep respect the .gitignore file in the directory it’s searching in. This helps make it incredibly faster and generally more intuitive for the user as it just searches files that are part of thier project and not things like their virtual environments, node modules, or compiled builds.

Editors like vscode often do not include files that are .gitignored in their search either.

pathspec is a pattern matching library that implements git’s wildmatch pattern so that you can ignore files included in your .gitignore patterns. You might want this to help make your libraries more performant, or more intuitive for you users.

import pathspec
from pathlib import Path

markdown_files = Path().glob('**/*.md')
if (Path(".gitignore").exists():
    lines = Path(".gitignore").read_text().splitlines()

    spec = pathspec.PathSpec.from_lines("gitwildmatch", lines)

    markdown_files = [
        file for file in markdown_files if not spec.match_file(str(file))
    ]

pathspec home page

I don’t use refactoring tools as much as I probably should. mostly because I work with small functions with unique names, but I recently had a case where a variable name m was everywhere and I wanted it named better. This was not possible with find and replace, because there were other m’s in this region.

I first tried the nvim lsp rename, and it failed, Then I pip installed rope, a refactoring tool for python, and it just worked!

pip install rope

Once you have rope installed you can call rename on the variable.

:lua vim.lsp.buf.rename()

When running a python process that requires a port it’s handy if there is an option for it to just run on the next avaialble port. To do this we can use the socket module to determine if the port is in use or not before starting our process.

import socket

def find_port(port=8000):
    """Find a port not in ues starting at given port"""
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        if s.connect_ex(("localhost", port)) == 0:
            return find_port(port=port + 1)
        else:
            return port

Adding a --pdb flag to your applications can make them much easier for those using it to debug your application, especially if your applicatoin is a cli application where the user has much fewer options to start this for themselves. To add a pdb flag --pdb to your applications you will need to wrap your function call in a try/except, and start a post_mortem debugger. I give credit to this stack overflow post for helping me figure this out.

import pdb, traceback, sys


def bombs():
    a = []
    print(a[0])


if __name__ == "__main__":
    if "--pdb" in sys.argv:
        try:
            bombs()
        except:
            extype, value, tb = sys.exc_info()
            traceback.print_exc()
            pdb.post_mortem(tb)
    else:
        bombs()

Using –pdb #

python yourfile.py --pdb

Converting markdown posts to pdf on ubuntu takes a few packages from the standard repos. I had to go through a few stack overflow posts, and nothing seemed to have all the fonts and packages that I needed to convert markdown, but this is what ended up working for me.

Installing all the packages #

sudo apt install \
  pandoc \
  texlive-latex-base \
  texlive-fonts-recommended \
  texlive-extra-utils \
  texlive-latex-extra \
  texlive-xetex

Using pandoc to convert markdown to a pdf #

# older versions of pandoc, I needed this one on ubuntu 18.04
pandoc pages/til/convert-markdown-pdf-linux.md -o convert-markdown-pdf.pdf --latex-engine=xelatex
# newer versions of pandoc, I needed this one on ubuntu 21.04
pandoc pages/til/convert-markdown-pdf-linux.md -o convert-markdown-pdf.pdf --pdf-engine=xelatex

Python comes with an enum module for creating enums. You can make your own enum by inheriting importing and inheriting from Enum.

from enum import Enum


class LifeCycle(Enum):
    configure = 1
    glob = 2
    pre_render = 3
    render = 4
    post_render = 5
    save = 6

auto incrementing #

Enum values can be auto incremented by importing auto, and calling auto() as their value.

from enum import Enum, auto


class LifeCycle(Enum):
    configure = auto()
    glob = auto()
    pre_render = auto()
    render = auto()
    post_render = auto()
    save = auto()

using the enum #

Enum’s are accessed directy under the class itself, and have primarily two methods underneath each thing you make, .name and .value.

Lifecycle.glob
Lifecycle.glob.value
Lifecycle.glob.name

I recently paired up with another dev running windows with Ubuntu running in wsl, and we had a bit of a stuggle to get our project off the ground because they were missing com system dependencies to get going.

Straight in the terminal #

Open up a terminal and get your required system dependencies using the apt package manager and the standard ubuntu repos.

sudo apt update
sudo apt upgrade
sudo apt install \
      python3-dev \
      python3-pip \
      python3-venv \
      python3-virtualenv
pip install pipx

Using an Ansible-Playbook #

I like running things like this through an ansible-playbook as it give me some extra control and repeatability next time I have a new machine to setup.

- hosts: localhost
  gather_facts: true
  become: true
  become_user: "{{ lookup('env', 'USER') }}"

  pre_tasks:
    - name: update repositories
      apt: update_cache=yes
      become_user: root
      changed_when: False
  vars:
    user: "{{ ansible_user_id }}"
  tasks:
    - name: Install System Packages 1 (terminal)
      become_user: root
      apt:
        name:
          - build-essential
          - python3-dev
          - python3-pip
          - python3-venv
          - python3-virtualenv
    - name: check is pipx installed
      shell: command -v pipx
      register: pipx_exists
      ignore_errors: yes

    - name: pipx
      when: pipx_exists is failed
      pip:
        name: pipx
      tags:
        - pipx

video clip #

Here is a clip of me getting pipx running on ubuntu 21.10, and running a few of my favorite pipx commands.

Stow is an incredible way to manage your dotfiles. It works by managing symlinks between your dotfiles directory and the rest of the system. You can then make your dotfiles directory a git repo and have it version controlled. In my honest opinion, when I was trying to get started the docs straight into deep detail of things I frankly don’t really care about and jumped right over how to use it.

When using stow its easiest to keep your dotfiles directory (you may name it what you want) in your home directory, with application directories inside of it.

Then each application directory should reflet the same diretory structure as you want in your home directory.

zsh #

Here is a simple example with my zshrc.

mkdir ~/dotfiles
cd ~/dotfiles
mkdir zsh
mv ~/.zshrc zsh
stow --simulate zsh

You can pass in the –simulate if you wish, it will tell you if there are going to be any more errors or not, but it wont give much more than that.

WARNING: in simulation mode so not modifying filesystem.

Once your ready you can stow your zsh application.

stow zsh

nvim #

A slightly more complicated example is neovim since its diretory structure does not put configuration files directly in your home directory, but rather at a deeper level.

mkdir ~/dotfiles/nvim/.config/nvim/ -p
cd ~/dotfiles
mv ~/.config/nvim/ ~/dotfiles/nvim/.config/nvim/
stow zsh

!notice how the nvim directory inside of dotfiles is structured like it would be in your $HOME directory.

The copier answers file is a key component to making your templates re-runnable. Let’s look at the example for my setup.py.

❯ tree ~/.copier-templates/setup.py
/home/walkers/.copier-templates/setup.py
├── [[ _copier_conf.answers_file ]].tmpl
├── copier.yml
├── setup.cfg
└── setup.py.tmpl

0 directories, 4 files

Inside of my [[ _copier_conf.answers_file ]].tmpl file is this, a message not to muck around with it, and the ansers in yaml form. The first line is just a helper for the blog post.

# ~/.copier-templates/setup.py/\[\[\ _copier_conf.answers_file\ \]\].tmpl
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
[[_copier_answers|to_nice_yaml]]

Inside my copier.yml I have setup my _answers_file to point to a special file. This is because this is not a whole projet template, but one just for a single file.

# copier.yml
# ...
_answers_file: .setup-py-copier-answers.yml

Once I change the _answers_file I was incredibly stuck

Run it #

I’m making a library of personal copier templates in my ~/.copier-templates directory and I am going to run it from there.

copier copy ~/.copier-templates/setup.py

Results #

After rendering the template we have the following content in our .setup.setup-py-copier-answers.yml file. This will allow us to update quick if we ever change our template.

# .setup-py-copier-answers.yml
# Changes here will be overwritten by Copier; NEVER EDIT MANUALLY
_src_path: /home/walkers/.copier-templates/setup.py
author_github: waylonwalker
author_name: Waylon Walker
description: awesomeness
framework: null
keywords: null
package_name: my-package

Update it #

This is where I was most stuck, primarily becuase -a <answers_file> must come exactly after the base command copier. This felt a bit odd to and not where I expected it so it.

copier -a .setup-py-copier-answers.yml update

Stop asking all these damn questions #

So the defaults are now changed to our previous results, but it keeps asking for them. To stop asking we can simply add a -f flag.

copier -fa .setup-py-copier-answers.yml update

Once you have made your sick looking cli apps with rich, eventually you are going to want to add some keybindings to them. Currently Textual, also written by @willmcgugan, does this extremely well. Fair Warning it is in super beta mode and expected to change a bunch. So take it easy with hopping on the train so fast.

Get the things #

Install them from the command line.

pip install textual
pip install rich

Import make a .py file and import them in it.

from textual.app import App
from textual.widget import Widget
from rich.panel import Panel

Make what you have a widget #

If you return your rich renderable out of class that inherits from textual.widget.Widget, you can then dock this inside of an app class inheriting from textual.app.App.

class MyWidget(Widget):
    def render(self):
        my_renderable = Panel("press q to quit")
        return my_renderable

class MyApp(App):
    async def on_mount(self) -> None:
        await self.view.dock(MyWidget(), edge="top")
        await self.bind("q", "quit")

run it #

You’ve made a TUI (text user interface). Run the classmethod run to display the it in its full screen glory.

MyApp.run(log="textual.log")

Final result #

At this point It probably does not look much different, but it can be interactive by binding keys to any method on your app that starts with the word action_, this includes the built-in actions such as action_quit.

from textual.app import App
from textual.widget import Widget
from rich.panel import Panel


class MyWidget(Widget):
    def render(self):
        my_renderable = Panel("press q to quit")
        return my_renderable


class MyApp(App):
    async def on_mount(self) -> None:
        await self.view.dock(MyWidget(), edge="top")
        await self.bind("q", "quit")


if __name__ == "__main__":
    MyApp.run(log="textual.log")