Drafts

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique. Slide One # [1] Lets use this section to show what it looks like as I change my styles. from markata import Markata Markata() markata.run() ☝ This is how my website is built - write markdown - build site - publish default # [2] This is what the above slide looks like in lookatme. [3] Set focus to the most important element # [4] The way I write my slides I want the most prominant element to be the slides title, not the presentation title. The slides title is generally the point I am trying to make, I will leave some supporting information if I want, but sometimes, I just have a title. styles: title: bg: default fg: '#e1af66' headings: '1': bg: default fg: '#ff66c4,bold,italics' prefix: ' ⇁ ' suffix: ' ↽ ' [5] by default he prefix/suffix was a full block that just went transparant into the slide. I thought the harpoons were fun and went with them on a whim The box characters bother me # [6] The box characters are fine really, but it really bothers me th...

I use a package eyeseast/python-frontmatter [1]{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml). Install # [2] It’s on pypi, so you can install it into your virtual environment [3] with pip. python -m pip install python-frontmatter 🙋 What’s Frontmatter # [4] Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging platform that also builds all of its posts with markdown and yaml frontmatter. Let’s see an example # [5] Here is the exact frontmatter for this post you are reading on my site. --- date: 2022-03-24 03:18:48.631729 templateKey: til title: How I load Markdown in Python tags: - linux - python --- This is where the markdown content for the post goes. So it’s yaml # [6] yaml is the most commmon, but python-frontmatter [7]{.hoverlink} also supports Handlers [8]{.hoverlink} for toml and json. If you want a good set of examples of yaml learnxi...

Today I was watching the python web conf 2022 and saw @davidbujic [1] use the new Dict Union Operator Live on stage during his Functional Programming [2] talk. This operator was first introduced into python 3.9 with pep584 [3]. Merge Dicts # [4] I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result. default_config = {'url': 'https://example.com', 'assets_dir': 'static' } user_config = {'url': 'https://waylonwalker.com'} # **unpacking goes back much further than 3.9 config = {**default_config, **user_config} print(config) # {'url': 'https://waylonwalker.com', 'assets_dir': 'static'} # the same can be achieved through the new to python 3.9 | operator config = default_config | user_config print(config) # {'url': 'https://waylonwalker.com', 'assets_dir': 'static'} understanding python *args and **kwargs [5] More on unpacking in this post. Update Dicts # [6] With the release there is also a new update syntax |= that you can use to update. I dont often mutate variables fo...

I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides. Present from the terminal # [1] I will most often just present right from the terminal using lookatme [2]. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly. reveal.js # [3] I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share. New Slides # [4] I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style. Installation # [5] lookatme is a python library that is available on pypi, you can install it with the pip command. python -m pip install lookatme Since it’s a command line application it works great with pipx. This prevents the need to manage virtual environments yourself or ending up with packages clashing in your system python e...

When I need to read contents from a plain text file in python I find the easiest way is to just use Pathlib. from pathlib import Path Path('path_to_file').read_text()

Let’s make a vim command to automatically collect all the links in these posts at the end of each article. Regex confuses the heck out of me… I don’t have my regex liscense, but regex can be so darn powerful especially in an editor. Step one # [1] Before you run someone’s regex from the internet that you don’t fully understand, check your git status and make sure you are all clear with git [2] before you wreck something Inspiration # [3] Something that I have always appreciated form Nick Janetakis [4] is his links section. I often try to gather up the links at the end of my posts, but often end up not doing it or forgetting. Making a Links section # [5] Searchng through the internet I was able to find an article from Vitaly Parnas called vim ref links [6] that did almost exactly what I needed, except it was more complicated and made them into ref liks. Here is my interpretation of the code I took from Vitaly’s post. It makes a Links section like the one at the bottom of this post. function! MdLinks() $norm o## Links $norm o g/\[[^\]]\+\]([^)]\+)/t$ silent! '^,$s/\v[^\[]*(\[[^\]]+\])\(([^)]+)\)[^\[]*/* \1(\2)/g nohl endfunction command! MdLinks call MdLinks() So far ...

If you ever end up on a linux machine that just does not have enough ram to suffice what you are doing and you just need to get the job done you can give it some more swap. You can look up reccomendations for how much swap you should have this is more about just trying to get your job done when you are almost there, but running out of memory on the hardware you have. make the /swap file # [1] You can put this where you wish, for this example I am going to pop it into /swap sudo fallocate -l 4G /swap sudo chmod 600 /swap sudo mkswap /swap sudo swapon /swap make sure that your swap is on # [2] You can make sure that your swap is working by using the free command, I like using the -h flag to get human readable numbers. ❯ free -h total used free shared buff/cache available Mem: 15Gi 5.5Gi 4.9Gi 458Mi 5.2Gi 9.3Gi Swap: 4.0Gi 0B 4.0Gi Reclaim memory usage in Jupyter [3] I also used this trick in this article to give my python process a bit more oompf and get it on home. References: [1]: #make-the-swap-file [2]: #make-sure-that-your-swap-is-on [3]: /reset-ipython/

A very common task for any script is to look for files on the system. My go to method when globbing for files in python is to use pathlib. Setup # [1] I setup a directory to make some examples about globbing. Here is what the directory looks like. ❯ tree . . ├── content │ ├── hello.md │ ├── hello.py │ ├── me.md │ └── you.md ├── readme.md ├── README.md ├── READMES.md └── setup.py 1 directory, 8 files Pathlib # [2] Pathlib is a standard library module available in all LTS versions of python at this point. ❯ from pathlib import Path Creating a Path instance. # current working directory Path() Path.cwd() # The users home directory Path.home() # Path to a directory by string Path('/path/to/directory') # The users ~/.config directory Path.home() / '.config' Globbing Examples # [3] The path object has a glob method that allows you to glob for files with a unix style glob pattern to search for files. Note that it gives you a generator. This is great for many use cases, but for examples its easier to turn them to a list to print them out. If you need some more detail on what globbing is there is a wikipedia [4] article discussing it. I am just showing how to glob with pathlib...

Setting up your git [1] pager to your liking can help you navigate diffs and logs much more efficiently. You can set it to whatever pager you like so that your keys feel nice and smooth and your fingers know exactly what to do. You might even gain a few extra features. Setting the pager # [2] You can set the pager right from your command line with the following command. git config --global core.pager 'more' You can also set your pager by editing your global .gitconfig file which by default is set to ~/.gitconfig. [core] pager = more Color # [3] In my experience you need to turn colors off with nvim. bat handles them and looks good either way, but nvim will be plain white and display the color codes as plain text if color is on. git config --global color.pager no Pagers I have tried # [4] Here are some various configs that I tried. For some reason line numbers in bat really bothered me, but when in nvim they felt ok. I am going to try running both of them for a few days and see which I like better. I think having some of my nvim config could be really handy for things like yanking a commit hash to the system clipboard without touching the mouse. # bat git config --global...

If you have ever mistyped a git [1] command very close to an existing one you have likely seen this message. ❯ git chekout dev git: 'chekout' is not a git command. See 'git --help'. The most similar command is checkout Automatically run the right one # [2] What you might not have known is that you can configure git to just run this command for you. # Gives you 0.1 seconds to respond git config --global help.autocorrect 1 # Gives you 1 seconds to respond git config --global help.autocorrect 10 # Gives you 5 seconds to respond git config --global help.autocorrect 50 Fat Fingers Gone # [3] Now when you typo a git command it will autmatically run after the configured number of tenths of a second. ❯ git chkout get-error WARNING: You called a Git command named 'chkout', which does not exist. Continuing in 1.0 seconds, assuming that you meant 'checkout'. M pages/blog/how-i-deploy-2021.md M pages/hot_tips/001.md M pages/templates/gratitude_card.html M plugins/index.py M plugins/publish_amp.py M plugins/render_template_variables.py M plugins/youtube.py M requirements.txt M static/index.html Switched to branch 'get-error' My config # [4] I’m rocking 10 for now just to see how I ...

yq is a command line utility for parsing and querying yaml, like jq does for json. This is for me # [1] I love that all of these modern tools built in go and rust, just give you a zipped up executable right from GitHub releases, but it’s not necessarily straight forward how to install them. yq does one of the best jobs I have seen, giving you instructions on how to get a specific version and install it. I use a bunch of these tools, and for what its worth I trust the devs behind them to make sure they don’t break. This so far has worked out well for me, but if it ever doesn’t I can always pick an older version. Just give me the latest # [2] Since I am all trusting of them I just want the latest version. I do not want to update a shell script with new versions, or even care about what then next version is, I just want it. Luckily you can script the release page for the latest version on all that I have came accross. What is the latest # [3] I wrote or stole, I think I wrote it, this line of bash quite awhile ago, and it has served me well for finding the latest release for any GitHub project using releases. Just update it with the name of the tool, org, and repo and it wor...

Last Thursday I learned about pytest-mock at a local python meetup. The presenter showed how he uses pytest-mock for his work, and it was kinda eye opening. I knew what mocking was, but I had not seen it in this context. Discovery # [1] Watching him use pytest-mock I realized that mocking was not as hard as I had made it out to be. You can install pytest-mock, use the mocker fixture, and patch objects methods with what you want them to be. install # [2] pytest-mock is out on pypi and can be installed with pip. python -m pip install pytest-mock What I actually did # [3] Sometimes I fall victim to making these posts nice and easy to follow. It takes more steps than just pip install, you need a place to practice in a nice sandbox. Here is how I make my sandboxes. mkdir ~/git/learn-pytest-mock cd ~/git/learn-pytest-mock # well actually open a new tmux session there echo pytest-mock > requirements.txt # I copied in my .envrc, and ran direnv allow, which actually just made me a virtual env as follows python3 -m venv .venv --prompt $(basename $PWD) source .venv/bin/activate # now install pytest-mock pip install -r requirements.txt # make some tests to mock mkdir tests nvim t...

Python 3.8 came out two and a half years ago and I have yet to really lean in on the walrus operator. Partly because it always seemed like something kinda silly (my use cases) to require a python version bump for, and partly because I really didn’t understand it the best. Primarily I have wanted to use it in comprehensions, but I did not really understand how. Now that Python 3.6 is end of life, and most folks are using at least 3.8 it seems time to learn and use it. What’s a Walrus # [1] := The assignment operator in python is more commonly referred to as the walrus operator due to how := looks like a walrus. It allows you to assign and use a variable in a single expression. This example from the docs avoids a second call to the len function. if (n := len(a)) > 10: print(f"List is too long ({n} elements, expected <= 10)") Let’s get some data # [2] without a walrus In this example we are going to do a dict comp to generate a map of content from urls, only if their status code is 200. When doing this in a dictionary comprehension we end up needing to hit the url twice for successful urls. Once for the filter and once for the data going into the dictionary. { url: reque...

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier. Install kedro rich # [1] There is no pypi package yet, but it’s on github. You can pip install it with the git [2] url. pip install git+https://github.com/datajoely/kedro-rich Kedro run # [3] You can run your pipeline just as you normally would, except you get progress bars and pretty prints. kedro run [4] Kedro catalog # [5] Listing out catalog entries from the command line now print out a nice pretty table. kedro catalog list [6] Give it a star # [7] Go to the GitHub repo [8] and give it a star, Joel deserves it. References: [1]: #install-kedro-rich [2]: /glossary/git/ [3]: #kedro-run [4]: https://images.waylonwalker.com/kedro-rich-run.png [5]: #kedro-catalog [6]: https://images.waylonwalker.com/kedro-rich-catalog-list.png [7]: #give-it-a-star [8]: https://github.com/datajoely/kedro-rich

So worktrees, I always thought they were a big scary things. Turns out they are much simpler than I thought. Myth #1 # [1] no special setup I thought you had to be all in or worktrees or normal git [2], but not both. When I see folks go all in on worktrees they start with a bare repo, while its true this is the way you go all in, its not true that this is required. Lets make a worktree # [3] Making a worktree is as easy as making a branch. It’s actually just a branch that lives in another place in your filesystem. # checkout a new worktree called compare based on main in /tmp/project git worktree add -b compare /tmp/project main # checkout a new worktree called compare based on HEAD in /tmp/project git worktree add -b compare /tmp/project # checkout a worktree from an existing feature branch in /tmp/project git worktree add /tmp/project my-existing-feature-branch The worktree that you create is considered a linked worktree, while the original worktree is called the main worktree Note that I put this in my tmp directory because I don’t expect it to live very long, my recent use case was to compare two files after a big formatting change. You put these where you want, bu...

Has no upstream branch errors in git [1] can be such a damn productivity killer. You gotta stop your flow and swap over the branch, there is a config so that you don’t have to do this. fatal has no upstream branch # [2] If you have not yet configured git to always push to the current branch, you will get a has no upstream branch error if you don’t explicitly set it. Let’s show an example git checkout -b feat/ingest-inventory-data git add conf/base/catalog.yml git commit -m "feat: ingest inventory data from abc-db" git push You will be presented with the following error. fatal: The current branch feat/ingest-inventory-data has no upstream branch. To push the current branch and set the remote as upstream, use git push --set-upstream origin feat/ingest-inventory-data Option 1: follow the instructions # [3] To resolve this fatal error your first option is simply to follow the instructions given. Just copy and paste it in. git push --set-upstream origin feat/ingest-inventory-data Option 2: push to the current branch without setting upstream # [4] Honestly, I am pretty aware of the branch I am on, and Very few times have I ever accidentally pushed to the wrong branch. The on...

I write many of these posts from a 10 year old desktop that sits in my office these days. It does a very fine job running all of the things I need it to for my side work, but sometimes I want a mobile setup. I don’t really want to spend the $$ on a new laptop just for the few times I want to be somewhere else in the house. What I do have though is a chromebook. I’ve tried to get the chromebook into my workflow in the past, but have failed. Much because by the time I got all of my tools up and running in the linux vm it was taking up quite a bit of space on the device and made it harder for others to use as a chromebook. Today I am giving it a second try, but this time with ssh. Checking for existing sshd # [1] Before doing anything I checked to see if sshd is already running. Using the following command. sudo service ssh status # or pgrep -l sshd Both returned nothing so I know that its not running. setting up sshd # [2] just apt install it Next install the openssh-client and openssh-server sudo apt install openssh-client -y sudo apt install openssh-server -y After this I can see that its now running by checking its status once again. sudo service ssh status Gives me t...

Mermaid gives us a way to style nodes through the use of css, but rather than using normal css selectors we need to use style <nodeid>. This also applies to subgraphs, and we can use the name of the subgraph in place of the nodeid. graph TD; a --> A A --> B B --> C style A fill:#f9f,stroke:#333,stroke-width:4px style B fill:#f9f,stroke:#333,stroke-width:4px subgraph one a end style one fill:#BADA55 produces the following graph graph TD; a --> A A --> B B --> C style A fill:#f9f,stroke:#333,stroke-width:4px style B fill:#f9f,stroke:#333,stroke-width:4px subgraph one a end style one fill:#BADA55 import mermaid from '/assets/vendor/mermaid/mermaid.esm.min.mjs'; const rootStyle = getComputedStyle(document.documentElement); const css = (name, fallback) => (rootStyle.getPropertyValue(name) || fallback).trim(); const isDark = window.matchMedia('(prefers-color-scheme: dark)').matches || document.documentElement.dataset.theme === 'dark'; const accent = css('--color-primary', '#ffcd11'); const flowchart = { nodeSpacing: 60, rankSpacing: 90, padding: 12, }; const themeCSS = ` .label foreignObject > div { padding: 14px 14px 10px; line-height: 1.2; } .nodeLabel ...

I recently found a really great plugin [1] by mhinz [2] to open files in neovim from a different tmux split, without touching neovim at all. Installation # [3] neovim-remote [1] is not a neovim plugin at all, it’s a python cli that you can install with pip. Unlike the repo suggests, I use pipx to install nvr. pipx install neovim-remote How I use it # [4] I have this added to my .envrc that is in every one of my projects. This will tie a neovim session to that directory, and all directories under it. export NVIM_LISTEN_ADDRESS=/tmp/nvim-$(basename $PWD) In my workflow I open a tmux session for each project, so this essentially ties a neovim session to a tmux session. Open neovim # [5] First open neovim, but with the nvr command. This will open neovim, and look pretty much the same as always. nvr If you try to run nvr again in another shell nothing will happen as its already runnin under that address, but if you give it a filename it will open the file in the first instance of neovim that you opened. nvr readme.md Links # [6] - GitHub [1] References: [1]: https://github.com/mhinz/neovim-remote [2]: https://github.com/mhinz [3]: #installation [4]: #how-i-use-it [5]: #op...

Mermaid provides some really great ways to group or fence in parts of your graphs through the use of subgraphs. Here we can model some sort of data ingest with some raw iot device and our warehouse in different groups. graph TD; subgraph raw_iot a end subgraph warehouse A --> B B --> C end graph TD; subgraph raw_iot a end subgraph warehouse A --> B B --> C end connecting subgroups # [1] If we want to connect them, we can make a connection between a and A outside of the subgraphs. graph TD; subgraph raw_iot a end a --> A subgraph warehouse A --> B B --> C end graph TD; subgraph raw_iot a end a --> A subgraph warehouse A --> B B --> C end separation of concerns # [2] It’s also possible to specify subgraphs separate from where you define your nodes. which allows for some different levels of grouping that would not be possible if you were to define all your nodes inside of a subgraph. graph TD; a --> A A --> B B --> C subgraph one A C end graph TD; a --> A A --> B B --> C subgraph warehouse A C end import mermaid from '/assets/vendor/mermaid/mermaid.esm.min.mjs'; const rootStyle = getComputedStyle(document.documentElement); co...