Draft and unpublished posts
From the same Author that brought us command line essentials like fd and bat written in rust comes pastel an incredible command-line tool to generate, analyze, convert and manipulate colors.
You can install from one of the releases, follow the instructions for your system from the repo. I chose to go the nix route. I have enjoyed the simplicity of the nix package manager being cross platform and have very up to date packages in it.
...
Dunk is a beautiful git diff tool built on top of rich.
Browsing through twitter the other day I discovered it through this tweet by _darrenburns.
https://twitter.com/_darrenburns/status/1510350016623394817
Before I dive in deep, I do want to mention that Dunk is super new and beta at this point. I am making it my default pager, because I know what I am doing and can quickly shift back...
...
I’m poking a bit into gamedev. Partly to better understand, partly because it’s stretching different parts of my brain/skillset than writing data pipelines does, but mostly for the experience of designing them with my 9yo Wyatt.
I’ve seen several pygame boilerplate templates, but they all seem to rely heavily on globl variables. That’s just not how I generally develop anything. I want a package that I can pip install, run, import, test, all the good stuff.
What currently have is a single module starter package that is on github so that I can install it and start building games with very little code.
Since it’s a package on GitHub you can install it with the git+ prefix.
...
This morning I was trying to install a modpack on my minecraft server after getting a zip file, and its quite painful when I unzip everything in the current directory rather than the directory it belongs in.
So I’ve been struggling to get mods installed on linux lately and the easiest way to download the entire pack rather than each mod one by one seems to be to use the overwolf application on windows. Once I have the modpack I can start myself a small mod-server by zipping it, putting it in a mod-server directory and running a python http.server
python -m http.server
Then I go back to my server and download the modpack with wget.
wget 10.0.0.171:8000/One%2BBlock%2BServer%2BPack-1.4.zip
Now I can unzip my mods into the minecraft-data directory.
...
My personal Site build went down last week, and I was unable to publish a new article. This is the process I went through to get it back up and running quickly.
Classic IT fix, rerun it and see if you get the same error. Everyone is busy and when you have your build go down you are probably busy doing something else. My first step is often to simply click rerun right from GitHub actions. Sometimes this will fix it, and sometimes it doesn’t. It’s an easy fix to run in the meantime you are not focused on fixing it.
Also worth a check to see if GitHub is having a hiccup or not. This error felt pretty obviously not GitHub’s fault, but it’s a good one to check when you run into a weird unexplainable error.
Check github status for any downtime issues with actions.
...
I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have.
Try running these examples in a .py file for yourself and try to kill them with control-c.
Since things such as KeyboardInterrupt are created as an exception that inherits from BaseException, if you except BaseException you can no longer KeyboardInterrupt.
from time import sleep while True: try: sleep(30) except BaseException: # ❌ pass
If you except from exception or something than inherits from it you will be better off, and avoid unintended side effects.
...
When I need a consistent key for a pythohn object I often reach for hashlib.md5 It works for me and the use cases I have.
Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object.
How I setup a sqlite cache in python
does not work
...
When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it.
Install diskcache into your virtual environement of choice using pip from your command line.
python -m pip install diskcache
There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs
from diskcache import Cache cache = FanoutCache('.mycache', statistics=True)
Adding to the cache only needs a key and value.
...
The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache.
The easiest and most common to setup in python is a builtin functools.lru_cache.
from functools import lru_cache @lru_cache def get_cars(): print('pulling cars data') return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})
Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache.
when same *args, **kwargs always return the same value
...
I keep a small cars.csv on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate.
I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request.
This breaks my go to example dataset.
pd.read_csv("https://waylonwalker.com/cars.csv") # HTTPError: HTTP Error 403: Forbidden
What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does.
...
Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.
Requests is on pypi and can be installed into your virtual environtment with pip.
python -m pip install requests
Requests makes getting content from a web url as easy as possible.
import requests r = requests.get('https://waylonwalker.com/til/htmx-get/') article = r.content
...
I recently attended python web conf 2022 and after seeing some incredible presentations on it I am excited to give htmx a try.
Start with some html boilerplate, pop in a script tag to add the htmx.org script, and a button that says click me. I added just a tish of style so that it does not sear your delicate developer your eyes.
<!DOCTYPE html> <html lang="en"> <head> <title></title> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <style> html {...
...
I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique.
Lets use this section to show what it looks like as I change my styles.
from markata import Markata Markata() markata.run()
☝ This is how my website is built
This is what the above slide looks like in lookatme.
...
I use a package eyeseast/python-frontmatter{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml).
It’s on pypi, so you can install it into your virtual environment with pip.
python -m pip install python-frontmatter
Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging...
...
Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584.
I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result.
default_config = {'url': 'https://example.com', 'assets_dir': 'static' } user_config = {'url':...
...
I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides.
I will most often just present right from the terminal using lookatme. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly.
I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share.
I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style.
...
When I need to read contents from a plain text file in python I find the easiest way is to just use Pathlib.
Let’s make a vim command to automatically collect all the links in these posts at the end of each article. Regex confuses the heck out of me… I don’t have my regex liscense, but regex can be so darn powerful especially in an editor.
Before you run someone’s regex from the internet that you don’t fully understand, check your git status and make sure you are all clear with git before you wreck something
Something that I have always appreciated form Nick Janetakis is his links section. I often try to gather up the links at the end of my posts, but often end up not doing it or forgetting.
Searchng through the internet I was able to find an article from Vitaly Parnas called ...
If you ever end up on a linux machine that just does not have enough ram to suffice what you are doing and you just need to get the job done you can give it some more swap. You can look up reccomendations for how much swap you should have this is more about just trying to get your job done when you are almost there, but running out of memory on the hardware you have.
You can put this where you wish, for this example I am going to pop it into /swap
sudo fallocate -l 4G /swap sudo chmod 600 /swap sudo mkswap /swap sudo swapon /swap
You can make sure that your swap is working by using the free command, I like using the -h flag to get human readable numbers.
❯ free -h total used free shared buff/cache available Mem: 15Gi 5.5Gi 4.9Gi 458Mi 5.2Gi 9.3Gi Swap: 4.0Gi 0B 4.0Gi