GitHub Stars

I’m poking a bit into gamedev. Partly to better understand, partly because it’s stretching different parts of my brain/skillset than writing data pipelines does, but mostly for the experience of designing them with my 9yo Wyatt. pygame boilerplates # [1] I’ve seen several pygame boilerplate templates, but they all seem to rely heavily on globl variables. That’s just not how I generally develop anything. I want a package that I can pip install, run, import, test, all the good stuff. My current starter # [2] What currently have is a single module starter package that is on github so that I can install it and start building games with very little code. Installation # [3] Since it’s a package on GitHub you can install it with the git [4]+ prefix. pip install git+https://github.com/WaylonWalker/pygame-starter Example Game # [5] You can make a quick game by inheriting from Game, and calling .run(). This example just fills the screen with an aqua color, but you can put all of your game logic in the game method. from pygame_starer import Game class MyGame(Game): def game(self): self.screen.fill((128, 255, 255)) if __name__ == "__main__": game = MyGame() game.run() The st...

Just starred dunk [1] by darrenburns [2]. It’s an exciting project with a lot to offer. Prettier git [3] diffs in the terminal 🎨 References: [1]: https://github.com/darrenburns/dunk [2]: https://github.com/darrenburns [3]: /glossary/git/

This morning I was trying to install a modpack on my minecraft server after getting a zip file, and its quite painful when I unzip everything in the current directory rather than the directory it belongs in. I had the files on a Windows Machine # [1] So I’ve been struggling to get mods installed on linux lately and the easiest way to download the entire pack rather than each mod one by one seems to be to use the overwolf application on windows. Once I have the modpack I can start myself a small mod-server by zipping it, putting it in a mod-server directory and running a python http.server python -m http.server Downoading on the server # [2] Then I go back to my server and download the modpack with wget. wget 10.0.0.171:8000/One%2BBlock%2BServer%2BPack-1.4.zip Unzip to the minecraft-data directory # [3] Now I can unzip my mods into the minecraft-data directory. unzip One+Block+Server+Pack-1.4.zip -d minecraft-data Running the server with docker # [4] I run the minecraft server with docker, which is setup to mount the minecraft-data directory. Running a Minecraft Server in Docker [5] A bit more on that in the other post, but when I download the whole modpack like this I ...

If you’re into interesting projects, don’t miss out on dagger [1], created by dagger [2]. An engine to run your pipelines in containers References: [1]: https://github.com/dagger/dagger [2]: https://github.com/dagger

My personal Site build went down last week, and I was unable to publish a new article. This is the process I went through to get it back up and running quickly. Is it a fluke? # [1] Classic IT fix, rerun it and see if you get the same error. Everyone is busy and when you have your build go down you are probably busy doing something else. My first step is often to simply click rerun right from GitHub actions. Sometimes this will fix it, and sometimes it doesn’t. It’s an easy fix to run in the meantime you are not focused on fixing it. Is GitHub having issues? # [2] Also worth a check to see if GitHub is having a hiccup or not. This error felt pretty obviously not GitHub’s fault, but it’s a good one to check when you run into a weird unexplainable error. Check github status [3] for any downtime issues with actions. Build Down # [4] Alright down to the error message I got. The error is pretty obvious that somewhere I am trying to import a non-existing module from click. Run markata build --no-pretty Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.8.12/x64/bin/markata", line 33, in <module> sys.exit(load_entry_point('markata==0.1.0', 'console_scripts...

Gramformer [1] by PrithivirajDamodaran [2] is a game-changer in its space. Excited to see how it evolves. A framework for detecting, highlighting and correcting grammatical errors on natural language text. Created by Prithiviraj Damodaran. Open to pull requests and other forms of collaboration. References: [1]: https://github.com/PrithivirajDamodaran/Gramformer [2]: https://github.com/PrithivirajDamodaran

I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have. Try running these examples in a .py file for yourself and try to kill them with control-c. You cannot Keybard interrupt # [1] Since things such as KeyboardInterrupt are created as an exception that inherits from BaseException, if you except BaseException you can no longer KeyboardInterrupt. from time import sleep while True: try: sleep(30) except BaseException: # ❌ pass except from Exception or higher # [2] If you except from exception or something than inherits from it you will be better off, and avoid unintended side effects. from time import sleep while True: try: sleep(30) except Exception: # ✅ pass This goes with Custom Exceptions as well # [3] When you make custom exceptions expect that users, or your team members will want to catch them and try to handle them if they can. If you inherit from BaseException you will put them in a similar situation when they use your custom Exception. class MyFancyException(BaseException): # ❌ ... class MyFancyException(Exception): # ✅ ... Ref...

When I need a consistent key for a pythohn object I often reach for hashlib.md5 It works for me and the use cases I have. diskcache # [1] Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object. How I setup a sqlite cache in python [2] hash # [3] does not work My first thought was to just hash the files, this will give me a unique key for each. This will work, and give you a consistant key for one and only one given python process. If you start a new interpreter you will get different keys. waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) ❯ ipython waylonwalker ↪main v3.8.0 ipython ❯ hash("waylonwalker") -3862245013515310359 waylonwalker ↪main v3.8.0 ipython ❯ hash("waylonwalker") -3862245013515310359 waylonwalker ↪main v3.8.0 ipython ❯ exit waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) ❯ ipython waylonwalker ↪main v3.8.0 ipython ❯ hash("waylonwalker") -83673051278873734 here is a snapshot of my terminal proving that you can get the same hash in one session, but it changes whe...

When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it. install diskcache # [1] Install diskcache into your virtual environement of choice using pip from your command line. python -m pip install diskcache setup the cache # [2] There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs [3] from diskcache import Cache cache = FanoutCache('.mycache', statistics=True) Adding to the cache # [4] Adding to the cache only needs a key and value. cache.add('me', 'waylonwalker' ) Set the expire time # [5] Optionally you can set the seconds before it expires. The cache invalidation tools like this is what really makes diskcache shine over using raw sqlite or any sort of static file. cache.add('me', 'waylonwalker', expire=60) tagging # [6] Diskcache supports tagging entries added to the cache. # add an item to the cache with a tag cache.add('me', 'waylonwalker', expire=60, tag='people') This seems to let you do a few new things like getting items from the cach...

The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache. lru_cache # [1] The easiest and most common to setup in python is a builtin functools.lru_cache. from functools import lru_cache @lru_cache def get_cars(): print('pulling cars data') return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'}) when to use lru_cache # [2] Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache. when same *args, **kwargs always return the same value lru_cache only works for one python process. If you are running multiple subprocesses, or running the same script over and over, lru_cache will not work. lru_cache only caches in a single python process max_size # [3] lru_cache can take an optional parameter maxsize to set the size of your cache. By default its set to 128, if you want to store more or less items in your cache you can adjust this value....

I keep a small cars.csv [1] on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate. Hosts switched # [2] I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request. Not my go to dataset 😭 # [3] This breaks my go to example dataset. pd.read_csv("https://waylonwalker.com/cars.csv") # HTTPError: HTTP Error 403: Forbidden But requests works??? # [4] What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does. requests.get("https://waylonwalker.com/cars.csv") <Response [200]> Setting the User Agent in pandas.read_csv # [5] this fixed the issue for me! After a bit of googling I realize that this is a common thing, and that setting the user-agent fixes it. This is the point I remember seeing in the cloudflare dashbard that they protect against a lot of different attacks, aparantly it treats pd.read_...

Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement. Install Requests # [1] Requests is on pypi and can be installed into your virtual environtment with pip. python -m pip install requests Getting the content of a request # [2] Requests makes getting content from a web url as easy as possible. import requests r = requests.get('https://waylonwalker.com/til/htmx-get/') article = r.content requests is not limited to html # [4] Requests can handle any web request and is not limited to only html. Here are some examples to get a markdown file, a csv, and a png image. htmx_get_md = requests.get('https://waylonwalker.com/til/htmx-get.md').content cars = requests.get('https://waylonwalker.com/cars.csv').content profile = requests.get('https://images.waylonwalker.com/8bitc.png').content RTFM # [5] There is way more to requests, this ju...

I recently attended python web conf 2022 [1] and after seeing some incredible presentations on it I am excited to give htmx [2] a try. The base page # [3] Start with some html [4] boilerplate, pop in a script tag to add the htmx [5].org script, and a button that says click me. I added just a tish of style so that it does not sear your delicate developer your eyes. <!DOCTYPE html> <html lang="en"> <head> <title></title> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <style> html { background: #1f2022; color: #eefbfe; font-size: 64px; } button {font-size: 64px;} body { height: 100vh; width: 100vw; display: flex; justify-content: center; align-items:center; } </style> <!-- Load from unpkg --> <script src="https://unpkg.com/[email protected]"></script> </head> <body> <!-- have a button POST a click via AJAX --> <button hx-get="/partial" hx-swap="outerHTML"> Click Me </button> </body> </html> Save this as index.html and fire up a webserver and you will be presented with this big beefcake of a button. [6] If you don’t have a development server preference I reccomend opening the terminal and running python -m http.serve...

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique. Slide One # [1] Lets use this section to show what it looks like as I change my styles. from markata import Markata Markata() markata.run() ☝ This is how my website is built - write markdown - build site - publish default # [2] This is what the above slide looks like in lookatme. [3] Set focus to the most important element # [4] The way I write my slides I want the most prominant element to be the slides title, not the presentation title. The slides title is generally the point I am trying to make, I will leave some supporting information if I want, but sometimes, I just have a title. styles: title: bg: default fg: '#e1af66' headings: '1': bg: default fg: '#ff66c4,bold,italics' prefix: ' ⇁ ' suffix: ' ↽ ' [5] by default he prefix/suffix was a full block that just went transparant into the slide. I thought the harpoons were fun and went with them on a whim The box characters bother me # [6] The box characters are fine really, but it really bothers me th...

I use a package eyeseast/python-frontmatter [1]{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml). Install # [2] It’s on pypi, so you can install it into your virtual environment [3] with pip. python -m pip install python-frontmatter 🙋 What’s Frontmatter # [4] Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging platform that also builds all of its posts with markdown and yaml frontmatter. Let’s see an example # [5] Here is the exact frontmatter for this post you are reading on my site. --- date: 2022-03-24 03:18:48.631729 templateKey: til title: How I load Markdown in Python tags: - linux - python --- This is where the markdown content for the post goes. So it’s yaml # [6] yaml is the most commmon, but python-frontmatter [7]{.hoverlink} also supports Handlers [8]{.hoverlink} for toml and json. If you want a good set of examples of yaml learnxi...

I like WaylonWalker’s [1] project markata [2]. A plugins all the way down static site generator written in python. Plugins all the way down means that you can completely change the behavior of how it works by swapping plugins, installing new ones, or creating your own all in python. References: [1]: https://github.com/WaylonWalker [2]: https://github.com/WaylonWalker/markata

Today I was watching the python web conf 2022 and saw @davidbujic [1] use the new Dict Union Operator Live on stage during his Functional Programming [2] talk. This operator was first introduced into python 3.9 with pep584 [3]. Merge Dicts # [4] I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result. default_config = {'url': 'https://example.com', 'assets_dir': 'static' } user_config = {'url': 'https://waylonwalker.com'} # **unpacking goes back much further than 3.9 config = {**default_config, **user_config} print(config) # {'url': 'https://waylonwalker.com', 'assets_dir': 'static'} # the same can be achieved through the new to python 3.9 | operator config = default_config | user_config print(config) # {'url': 'https://waylonwalker.com', 'assets_dir': 'static'} understanding python *args and **kwargs [5] More on unpacking in this post. Update Dicts # [6] With the release there is also a new update syntax |= that you can use to update. I dont often mutate variables fo...

I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides. Present from the terminal # [1] I will most often just present right from the terminal using lookatme [2]. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly. reveal.js # [3] I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share. New Slides # [4] I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style. Installation # [5] lookatme is a python library that is available on pypi, you can install it with the pip command. python -m pip install lookatme Since it’s a command line application it works great with pipx. This prevents the need to manage virtual environments yourself or ending up with packages clashing in your system python e...

Check out eyeseast [1] and their project python-frontmatter [2]. Parse and manage posts with YAML (or other) frontmatter References: [1]: https://github.com/eyeseast [2]: https://github.com/eyeseast/python-frontmatter

When I need to read contents from a plain text file in python I find the easiest way is to just use Pathlib. from pathlib import Path Path('path_to_file').read_text()