Drafts

I am getting ready to do some timeseries analysis on a git repo with python, my first step is to figure out a way to list all of the git commits so that I can analyze each one however I want. The GitPython library made this almost trivial once I realized how.

from git import Repo

repo = Repo('.')
commits = repo.iter_commits()

This returns a generator, if you are iterating over them this is likely what you want.

commits
# <generator object Commit._iter_from_process_or_stream at 0x7f3307584510>

The generator will return git.Commit objects with lots of information about each commit such as hexsha, author, commited_datetime, gpgsig, and message.

next(commits)
# <git.Commit "d125317892d0fab10a36638a2d23356ba25c5621">

I was editing some blog posts over ssh, when I ran into this error. gpg was failing to sign my commits. I realized that this was because I could not answer to the desktop keyring over ssh, but had no idea how to fix it.

Error #

This is the error message I was seeing.

gpg failed to sign the data ssh

The fix #

The fix ended up being pretty simple, but quite a ways down this stack overflow post. This environment variable tells gpg that we are not logged into a desktop and it does not try to use the desktop keyring, and asks to unlog the gpgkey right in the terminal.

export GPG_TTY=$(tty)

The log in menu #

This is what it looks like when it asks for the passphrase.

EDIT-another way #

So this did not fix the issue on Arch BTW, and I have seen it not work for wsl users either. This did work for me and reported to have worked by a wsl user on a github issue.

echo '' | gpg --clearsign

This will unlock the gpg key then let you commit.

Sometimes you get a PR on a project, but cannot review it without wrecking your current working setup. This might be because it needs to be compiled, or a new set of requirements. Git worktrees is a great way to chekout the remote branch in a completely separate directory to avoid changing any files in your current project.

# pattern
# git worktree add -b <branch-name> <PATH> <remote>/<branch-name>
git worktree add -b fix-aws-service-cnsn /tmp/project origin/fix-aws-service-cnsn

This will create a new directory /tmp/project that you can review the branch fix-aws-service-cnsn from the remote origin. If you have setup different remotes locally you can check for the name of it with git remote -v

GitPython is a python api for your git repos, it can be quite handy when you need to work with git from python.

Use Case #

I recently made myself a handy tool for making screenshots in python and it need to do a git commit and push from within the script. For this I reached for GitPython.

How I Quickly Capture Screenshots directly into My Blog

Installation #

GitPython is a python library hosted on pypi that we will want to install into our virtual environments using pip.

pip install GitPython

Create a Repo Object #

Import Repo from the git library and create an instance of the Repo object by giving it a path to the directory containing your .git directory.

from git import Repo
repo = Repo('~/git/waylonwalker.com/')

Two interfaces #

from the docs

It provides abstractions of git objects for easy access of repository data, and additionally allows you to access the git repository more directly using either a pure python implementation, or the faster, but more resource intensive git command implementation.

I only needed to use the more intensive but familar to me git command implementation to get me project off the ground. There is a good tutorial to get you started with their pure python implementation in their docs.

Status #

Requesting the git status can be done as follows.

note I have prefixed my commands with »> to distinguish between the command I entered and the output.

>>> print(repo.git.status())

On branch main
Your branch is ahead of 'origin/main' by 1 commit.
  (use "git push" to publish your local commits)

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        blog/

You can even pass in flags that you would pass into the cli.

>>> print(repo.git.status("-s"))

<!--markata-attribution-->
?? blog/

log #

Example of using the log.

print(repo.git.log('--oneline', '--graph'))

* 0d28bd8 fix broken image link
* 3573928 wip screenshot-to-blog
* fed9abc wip screenshot-to-blog
* d383780 update for wsl2
* ad72b14 wip screenshot-to-blog
* 144c2f3 gratitude-180

Find Deleted Files #

We can even do things like find all files that have been deleted and the hash they were deleted.

print(repo.git.log('--diff-filter', 'D', '--name-only', '--pretty=format:"%h"'))

git find deleted files

full post on finding deleted files

My Experience #

This library seemed pretty straightforward and predicatable once I realized there were two main implementations and that I would already be familar with the more intensive git command implementation.

Python, click install

Edit the System Environment Variables

Environment Variables button

Add the following path to your users Path Variable

C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\dotnet\;C:\Users\quadm\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts;

Sometimes you just want python to do something else when you hit an exception, maybe that’s fire a text, slack message, email, or system notification like I wanted.

I am working on a quick and dirty python script designed to take screenshots and land them on my website in a single hotkey. With it being designed to run with a hotkey, if it were to error I would not see it.

I could have gone down a logging route, but honestly this is meant to be quick, dirty, and work on my system for me. I just want to get it in my system notification.

sys.excepthook #

Python exposes sys.excepthook for just this case. Here is what I ended up doing to fire a system notification as well as printing the message. Yaya a log would be mroe appropriate, but this is designed to just get done quick and do the job I want it to do.

def notify_exception(type, value, tb):
    traceback_details = "\n".join(traceback.extract_tb(tb).format())

    msg = f"caller: {' '.join(sys.argv)}\n{type}: {value}\n{traceback_details}"
    print(msg)
    Popen(
        f'notify-send "screenshot.py hit an exception" "{msg}" -a screenshot.py',
        shell=True,
    )


sys.excepthook = notify_exception
0 / 0

I recently was unable to boot into my home Linux Desktop, it got stuck at diskcheck fsck. I found that I was able to get in to a tty through a hotkey.

https://twitter.com/_WaylonWalker/status/1512281106120384519

What’s a TTY? #

There’s probably more to it, but to me its a full screen terminal with zero gui, not even your gui fonts. It does log into your default shell so if you have a comfy command line setup it will be here for you even though it looks much different without fonts and full colorspace.

Normal setup #

Normally you have 6 TTY’s running, the first is dedicated to your desktop manager, which is your login screen it might be something like gdm or lightdm.

• ctrl+alt+F1: login screen
• ctrl+alt+F2: Desktop
• ctrl+alt+F3: TTY 3
• ctrl+alt+F4: TTY 4
• ctrl+alt+F5: TTY 5
• ctrl+alt+F6: TTY 6

In my case the desktop manager neverstarted, so ctrl+alt+F1 brought me into a tty.

What happened?? #

Well after getting back in and having some time to reflect, I think my Desktop manager was installed or just broken, possibly during a update I ran a few days prior.

I tried a bunch of things like switching to lightdm, and manually running startx.

Getting back in #

The final commands that ended up getting me back in were installing and starting gdm3.

sudo apt install gdm3
sudo systemctl start gdm3

pygame events are stored in a queue, by default the most suggested way shown in all tutorials “pumps” the queue, which removes all the messages.

start up pygame #

You don’t necessarily need a full boilerplate to start looking at events, you just just need to pygame.init() and to capture any keystrokes you need a window to capture them on, so you will need a display running.

import pygame
pygame.init()
pygame.display.set_mode((854, 480))

get some events #

Let’s use pygames normal event.get method to get events.

events = pygame.event.get()

printing the events reveal this

[
    <Event(1541-JoyDeviceAdded {'device_index': 0, 'guid': '030000005e0400008e02000010010000'})>,
    <Event(4352-AudioDeviceAdded {'which': 0, 'iscapture': 0})>,
    <Event(4352-AudioDeviceAdded {'which': 1, 'iscapture': 0})>,
    <Event(4352-AudioDeviceAdded {'which': 2, 'iscapture': 0})>,
    <Event(4352-AudioDeviceAdded {'which': 0, 'iscapture': 1})>,
    <Event(4352-AudioDeviceAdded {'which': 1, 'iscapture': 1})>,
    <Event(32774-WindowShown {'window': None})>,
    <Event(32777-WindowMoved {'x': 535, 'y': 302, 'window': None})>,
    <Event(32770-VideoExpose {})>,
    <Event(32776-WindowExposed {'window': None})>,
    <Event(32788-WindowTakeFocus {'window': None})>,
    <Event(32768-ActiveEvent {'gain': 1, 'state': 1})>,
    <Event(32785-WindowFocusGained {'window': None})>,
    <Event(768-KeyDown {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(771-TextInput {'text': 'a', 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(768-KeyDown {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(771-TextInput {'text': 'a', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(771-TextInput {'text': 'a', 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'a', 'key': 97, 'mod': 0, 'scancode': 4, 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(771-TextInput {'text': 'f', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'f', 'key': 102, 'mod': 0, 'scancode': 9, 'window': None})>,
    <Event(768-KeyDown {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(771-TextInput {'text': 's', 'window': None})>,
    <Event(769-KeyUp {'unicode': 's', 'key': 115, 'mod': 0, 'scancode': 22, 'window': None})>,
    <Event(768-KeyDown {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(771-TextInput {'text': 'd', 'window': None})>,
    <Event(769-KeyUp {'unicode': 'd', 'key': 100, 'mod': 0, 'scancode': 7, 'window': None})>,
    <Event(768-KeyDown {'unicode': '', 'key': 1073742051, 'mod': 1024, 'scancode': 227, 'window': None})>,
    <Event(772-Unknown {})>,
    <Event(769-KeyUp {'unicode': '', 'key': 1073742051, 'mod': 0, 'scancode': 227, 'window': None})>,
    <Event(32768-ActiveEvent {'gain': 0, 'state': 1})>,
    <Event(32786-WindowFocusLost {'window': None})>,
    <Event(772-Unknown {})>
]

Lets get some more events. #

Let’s say that we have multpile sprites all asking for the events from different places in our game. If we assume that our game loop runs very fastand we get events one after another the second one will have no events.

events_one = pygame.event.get()
events_two = pygame.event.get()

printing the events reveals that there are no events, well i

[]

Making things more maddening #

Even simple games don’t quite run infinitely fast there is some delay, with this delay most events will go to event_one, while any that occur in the short time between event_one and two will be in event_two’s queue.

import time
events_one = pygame.event.get()
time.sleep(.05) # simulating some delay that would naturally occur
events_two = pygame.event.get()

How to Resolve this #

Store events for each frame in memory.

Pump #

I thought pump=False would be the answer I was looking for, but I was proven wrong. It does not behave intuitivly to me.

events_one = pygame.event.get(pump=False) # all events since last pump
events_two = pygame.event.get(pump=False) # no events
events_three = pygame.event.get() # all events since last pump

events_one and events_three will have a list of events, while events_two will be empty. It seems that pump=False leaves the events there for the next event.get(), but appears cleared to any event.get(pump=False).

Keep a Game State #

If you want objects to do their own event handling, outside of the main game, you will need to give them some game state with the events in it. However you decide, you may only call event.get() once per game loop otherwise weird things will happen.

One of the most essential concepts of pygame to start making a game you will need to understand is loading images and blitting them to the screen.

blit stands for block image transfer, to me it feels a lot like layering up layers/images in photoshop or Gimp.

Loading an image #

I started by making a spotlight in Gimp, by opening a 64x64 pixel image and painting the center with a very soft brush.

Now we can load this into pygame.

import pygame
img = pygame.image.load("assets/spotlight.png")

Converting to the pygame colorspace #

To make pygame a bit more efficient we can convert the image to pygames colorspace once when we load it rather than every time we blit it onto another surface.

import pygame

# convert full opaque images
img = pygame.image.load("assets/spotlight.png").convert()

# convert pngs with transparancy
img = pygame.image.load("assets/spotlight.png").convert_alpha()

blitting #

To display the image onto the screen we need to use the blit method which needs at least two arguments, something to blit and a position to blit it at.

screen = pygame.display.set_mode(self.screen_size)
screen.blit( img, (0, 0),)

note blitting to the position (0, 0) will align the top left corners of the object we are blitting onto (screen) and the object we are blitting (img).

Starter #

Now we need an actual game running to be able to put on the screen. I am using my own starter/boilerplate, if you want to follow along you can install it from github into your own virtual environment.

pip install git+https://github.com/WaylonWalker/pygame-starter

Pygame Boilerplate Apr 2022

You can read more about my starter in this post

Let’s place this image right in the middle #

Now we can use the starter to create a new game, and with just a bit of offset we can put the spotlight directly in the middle.

import pygame
from pygame_starter import Game


class MyGame(Game):
    def __init__(self):
        super().__init__()
        # load in the image one time and store it inside the object instance
        self.img = pygame.image.load("assets/spotlight.png").convert_alpha()
    def game(self):
        # fill the screen with aqua
        self.screen.fill((128, 255, 255))
        # transfer the image to the middle of the screen
        self.screen.blit(
            self.img,
            (
                self.screen_size[0] / 2 - self.img.get_size()[0],
                self.screen_size[1] / 2 - self.img.get_size()[1],
            ),
        )


if __name__ == "__main__":
    game = MyGame()
    game.run()

If we save this as load_and_blit.py we can run it at the command like as so.

python load_and_blit.py

And we should get the following results.

the results of putting the image in the middle

convert a transparent png #

What happens when we accidently use .convert() rather than .convert_alpha()?

Making snow #

A common concept in pygame, that is built into my starter, is that you typically want to reset the screen each and every frame. Building on this with our new concept of blitting spotlights onto the screen we can make a random noise of snow by blitting a bunch of images to the screen.

import random

import pygame
from pygame_starter import Game


class MyGame(Game):

    def __init__(self):
        super().__init__()
        self.img = pygame.image.load("assets/spotlight.png").convert_alpha()

    def game(self):
        self.screen.fill((128, 255, 255))
        for  in range(100):
            self.screen.blit(
                self.img,
                (
                    random.randint(0, self.screen_size[0]) - self.img.get_size()[0],
                    random.randint(0, self.screen_size[1]) - self.img.get_size()[1],
                ),
            )


if __name__ == "__main__":
    game = MyGame()
    game.run()

the results #

From the same Author that brought us command line essentials like fd and bat written in rust comes pastel an incredible command-line tool to generate, analyze, convert and manipulate colors.

Install #

You can install from one of the releases, follow the instructions for your system from the repo. I chose to go the nix route. I have enjoyed the simplicity of the nix package manager being cross platform and have very up to date packages in it.

nix-env --install pastel

Mixing colors #

Something I often do to blend colors together is add a little alpha to something over top of a background. I can simulate this by mixing colors.

pastel color cornflowerblue | pastel mix goldenrod -f .1

Here is one from the docs that show how you can generate a color palette from random colors, mix in some red, lighten and format all in one pipe.

pastel random | pastel mix red | pastel lighten 0.2 | pastel format hex

color picker #

I am on Ubuntu 20.10 as I write this and it works flawlessly. When I call the command, a color picker gui pops up along with an rgb panel. I can pick from the panel or from anywhere on my screen.

pastel color-picker

Conversions #

I often will want to convert a color from rgb to hex or hsl vice versa. I open google and search. This is one part that I could really use adding to my workflow.

Check it #

Here I can mix up a dark grey with rgb, mix in 20% cornflowerblue, and grab the hex value.

pastel color 50,50,50 | pastel mix cornflowerblue -f .2

I really want to get this into my workflow. I saw it quite awhile ago but have not done much color work. Lately I have been doing a bit more front end, and have been getting into game development. This is the time to stop googling random color mixers and use this one.

Dunk is a beautiful git diff tool built on top of rich.

Browsing through twitter the other day I discovered it through this tweet by _darrenburns.

https://twitter.com/_darrenburns/status/1510350016623394817

Dunk is beta #

Before I dive in deep, I do want to mention that Dunk is super new and beta at this point. I am making it my default pager, because I know what I am doing and can quickly shift back if I need to, no sweat. If you are a little less comfortable with the command line, terminal, or reading any issues that might come up, it might be best if you just pipe into Dunk when you want to use it.

The author really cautions the use of it as your default pager this early, I’m just showing that it’s possible, and I’m trying it.

He notes that it might have some issues especially with partially staged files.

try it #

You can try it with pipx.

git diff | pipx run dunk

install it #

If you like it, you can install it with pip or pipx, I prefer pipx for cli applications like this.

pipx install dunk

set it as your default pager #

You can configure dunk as your default pager with the command line, or by editing your .gitconfig file.

git config --global pager.diff "dunk | less -R`

[pager]
    diff = dunk | less -R

As pointed out by _darrenburns dunk is not a pager and you can gain back all of the benefits of using a pager by piping into less with the -R flag.

reset it if you don’t like it #

You can --unset your pager configuration from the command line or edit your .gitconfig file by hand to remove the lines shown above.

git config --global --unset pager.diff

Comparison #

I have some edits to a game my son and I are working on unstaged so I ran git diff on that project with and without dunk to compare the differences.

Dunk just looks so good.

Always install #

If you follow along here often you know that I am a big fan of installing all my tools in an ansible playbook so that I don’t suffer configuring a new machine for months after getting it and wondering why its not exactly like the last.

# Dunk - prettier git diffs
# https://github.com/darrenburns/dunk
- name: check is dunk installed
  shell: command -v black
  register: dunk_exists
  ignore_errors: yes

- name: install dunk
  when: dunk_exists is failed
  shell: pipx install dunk

[[ ansible-install-if-not-callable ]]

More on conditionally installing tools with ansible in this post.

I’m poking a bit into gamedev. Partly to better understand, partly because it’s stretching different parts of my brain/skillset than writing data pipelines does, but mostly for the experience of designing them with my 9yo Wyatt.

pygame boilerplates #

I’ve seen several pygame boilerplate templates, but they all seem to rely heavily on globl variables. That’s just not how I generally develop anything. I want a package that I can pip install, run, import, test, all the good stuff.

My current starter #

What currently have is a single module starter package that is on github so that I can install it and start building games with very little code.

Installation #

Since it’s a package on GitHub you can install it with the git+ prefix.

pip install git+https://github.com/WaylonWalker/pygame-starter

Example Game #

You can make a quick game by inheriting from Game, and calling .run(). This example just fills the screen with an aqua color, but you can put all of your game logic in the game method.

from pygame_starer import Game

class MyGame(Game):
    def game(self):
        self.screen.fill((128, 255, 255))

if __name__ == "__main__":
    game = MyGame()
    game.run()

The starter #

Here is what the current game.py looks like. I will probably update it as time goes on and I learn more about the things I want to do with it.

from typing import Tuple

import pygame


class Game:
    def __init__(
        self,
        screen_size: Tuple[int, int] = (854, 480),
        caption: str = "pygame-starter",
        tick_speed: int = 60,
    ):
        """

        screen_size (Tuple[int, int]): The size of the screen you want to use, defaults to 480p.
        caption (str): the name of the game that will appear in the title of the window, defaults to `pygame-starter`.
        tick_speed (int): the ideal clock speed of the game, defaults to 60

        ## Example Game

        You can make a quick game by inheriting from Game, and calling
        `.run()`.  This example just fills the screen with an aqua color, but
        you can put all of your game logic in the `game` method.

        ``` python
        from pygame_starer import Game

        class MyGame(Game):
            def game(self):
                self.screen.fill((128, 255, 255))

        if __name__ == "__main__":
            game = MyGame()
            game.run()

        ```
        """
        pygame.init()
        pygame.display.set_caption(caption)

        self.screen_size = screen_size
        self.screen = pygame.display.set_mode(self.screen_size)
        self.clock = pygame.time.Clock()
        self.tick_speed = tick_speed

        self.running = True
        self.surfs = []

    def should_quit(self):
        """check for pygame.QUIT event and exit"""
        for event in pygame.event.get():
            if event.type == pygame.QUIT:
                self.running = False

    def game(self):
        """
        This is where you put your game logic.

        """
        ...

    def reset_screen(self):
        """
        fill the screen with black
        """
        self.screen.fill((0, 0, 0))

    def update(self):
        """
        run one update cycle
        """
        self.should_quit()
        self.reset_screen()
        self.game()
        for surf in self.surfs:
            self.screen.blit(surf, (0, 0))
        pygame.display.update()
        self.clock.tick(self.tick_speed)

    def run(self):
        """
        run update at the specified tick_speed, until exit.
        """
        while self.running:
            self.update()
        pygame.quit()


if __name__ == "__main__":
    game = Game()
    game.run()

This morning I was trying to install a modpack on my minecraft server after getting a zip file, and its quite painful when I unzip everything in the current directory rather than the directory it belongs in.

I had the files on a Windows Machine #

So I’ve been struggling to get mods installed on linux lately and the easiest way to download the entire pack rather than each mod one by one seems to be to use the overwolf application on windows. Once I have the modpack I can start myself a small mod-server by zipping it, putting it in a mod-server directory and running a python http.server

python -m http.server

Downoading on the server #

Then I go back to my server and download the modpack with wget.

wget 10.0.0.171:8000/One%2BBlock%2BServer%2BPack-1.4.zip

Unzip to the minecraft-data directory #

Now I can unzip my mods into the minecraft-data directory.

unzip One+Block+Server+Pack-1.4.zip -d minecraft-data

Running the server with docker #

I run the minecraft server with docker, which is setup to mount the minecraft-data directory.

Running a Minecraft Server in Docker

A bit more on that in the other post, but when I download the whole modpack like this I make these changes to my docker compose. (commented out lines)

version: "3.8"

services:
  mc:
    container_name: walkercraft
    image: itzg/minecraft-server:java8
    environment:
      EULA: "TRUE"
      TYPE: "FORGE"
      VERSION: 1.15.2
      # MODS_FILE: /extras/mods.txt
      # REMOVE_OLD_MODS: "true"
    tty: true
    stdin_open: true
    restart: unless-stopped
    ports:
      - 25565:25565
    volumes:
      - ./minecraft-data:/data
      # - ./mods.txt:/extras/mods.txt:ro

volumes:
  data:

My personal Site build went down last week, and I was unable to publish a new article. This is the process I went through to get it back up and running quickly.

Is it a fluke? #

Classic IT fix, rerun it and see if you get the same error. Everyone is busy and when you have your build go down you are probably busy doing something else. My first step is often to simply click rerun right from GitHub actions. Sometimes this will fix it, and sometimes it doesn’t. It’s an easy fix to run in the meantime you are not focused on fixing it.

Is GitHub having issues? #

Also worth a check to see if GitHub is having a hiccup or not. This error felt pretty obviously not GitHub’s fault, but it’s a good one to check when you run into a weird unexplainable error.

Check github status for any downtime issues with actions.

Build Down #

Alright down to the error message I got. The error is pretty obvious that somewhere I am trying to import a non-existing module from click.

Run markata build --no-pretty
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.12/x64/bin/markata", line 33, in <module>
    sys.exit(load_entry_point('markata==0.1.0', 'console_scripts', 'markata')())
  File "/opt/hostedtoolcache/Python/3.8.12/x64/bin/markata", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markata/__init__.py", line 25, in <module>
    from markata.cli.plugins import Plugins
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markata/cli/__init__.py", line 1, in <module>
    from .cli import app, cli, make_layout, run_until_keyboard_interrupt
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/markata/cli/cli.py", line 3, in <module>
    import typer
  File "/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/typer/__init__.py", line 12, in <module>
    from click.termui import get_terminal_size as get_terminal_size
ImportError: cannot import name 'get_terminal_size' from 'click.termui' (/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/click/termui.py)

Check pypi’s release date of click #

So the latest click was released just a few hours before this build. This feels like we are getting somewhere. Either click did a poor job of issuing deprecation warnings, or I was ignoring them in my build pipeline.

pin it and push #

let’s fix this build now

To get the build up and running today so that we don’t stop the flow of new posts I am going to open my requirements.txt file, and pin under the version that was just built.

click<8.1.0

Since I am still busy doing other things that fixing this, and am pretty confident that things were working before, I am just going to commit this and ship it.

watch ci #

Coming back to actions a few minutes later shows the site is building successfully without the same error as before. New posts will now be flowing to the site with the slightly older version of click.

looking for an issue #

Let’s make sure that the issue is going to be resolved. After not being busy and having time to investigate the issue, I can see that typer is the library making the import to get_terminal_size. Lets checkout its GitHub-repo and make sure someone is working on it.

By the time I go to the package that was having this issue there was already an issue up, and PR waiting approval. I gave the Issue a reaction 👍 to signal that I also care, and appreciate the issue author taking time to submit.

I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have.

Try running these examples in a .py file for yourself and try to kill them with control-c.

You cannot Keybard interrupt #

Since things such as KeyboardInterrupt are created as an exception that inherits from BaseException, if you except BaseException you can no longer KeyboardInterrupt.

from time import sleep

while True:
    try:
        sleep(30)
    except BaseException: # ❌
        pass

except from Exception or higher #

If you except from exception or something than inherits from it you will be better off, and avoid unintended side effects.

from time import sleep

while True:
    try:
        sleep(30)
    except Exception: # ✅
        pass

This goes with Custom Exceptions as well #

When you make custom exceptions expect that users, or your team members will want to catch them and try to handle them if they can. If you inherit from BaseException you will put them in a similar situation when they use your custom Exception.

class MyFancyException(BaseException): # ❌
    ...

class MyFancyException(Exception): # ✅
    ...

When I need a consistent key for a pythohn object I often reach for hashlib.md5 It works for me and the use cases I have.

diskcache #

Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object.

How I setup a sqlite cache in python

hash #

does not work

My first thought was to just hash the files, this will give me a unique key for each. This will work, and give you a consistant key for one and only one given python process. If you start a new interpreter you will get different keys.

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com)
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ hash("waylonwalker")
-3862245013515310359

waylonwalker ↪main v3.8.0 ipython
❯ hash("waylonwalker")
-3862245013515310359

waylonwalker ↪main v3.8.0 ipython
❯ exit

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com)
❯ ipython


waylonwalker ↪main v3.8.0 ipython
❯ hash("waylonwalker")
-83673051278873734

here is a snapshot of my terminal proving that you can get the same hash in one session, but it changes when you restart ipython.

hashlib.md5 #

Here is a quick couple ipython sessions showing that md5 cache is consistent accross multiple sessions.

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1)
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker")
[PYFLYBY] import hashlib
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ <ipython-input-1-1537c4473c74>:1 in <module>                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Unicode-objects must be encoded before hashing

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8"))
<md5 HASH object @ 0x7fe4ba6832d0>

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
'1c7c1073ca096ffdb324471770911fe2'

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
'1c7c1073ca096ffdb324471770911fe2'

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
'1c7c1073ca096ffdb324471770911fe2'

waylonwalker ↪main v3.8.0 ipython
❯ exit


waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1) took 47s
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ hashlib.md5("waylonwalker".encode("utf-8")).hexdigest()
[PYFLYBY] import hashlib
'1c7c1073ca096ffdb324471770911fe2'

key for diskcache #

Since it is consistent we can use it as a cache key for diskcache operations. I setup a little funciton that allows me to pass a bunch of differnt things in to cache. As long as the str method exists and is gives the data that you want to cache key on, this will work.

def make_hash(self, *keys: str) -> str:
    str_keys = [str(key) for key in keys]
    return hashlib.md5("".join(str_keys).encode("utf-8")).hexdigest()

understanding python *args and **kwargs

If the *args is confusing, I have a full article on *args and **kwargs.

See it in action #

Here you can see it in action. Anything passed into the function gets to be part of the key.

waylonwalker ↪main v3.8.0 ipython
❯ def make_hash(self, *keys: str) -> str:
...:     str_keys = [str(key) for key in keys]
...:     return hashlib.md5("".join(str_keys).encode("utf-8")).hexdigest()
...:

waylonwalker ↪main v3.8.0 ipython
❯ make_hash(1, "one", "1", 1.0)
'73901d019df012a1cdab826ce301217d'

waylonwalker ↪main v3.8.0 ipython
❯ exit


waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1) took 19m19s
❯

waylonwalker.com on  main [$✘!?] via  v5.1.5  v3.8.0 (waylonwalker.com) on  (us-east-1)
❯ ipython

waylonwalker ↪main v3.8.0 ipython
❯ def make_hash(self, *keys: str) -> str:
...:     str_keys = [str(key) for key in keys]
...:     return hashlib.md5("".join(str_keys).encode("utf-8")).hexdigest()
[PYFLYBY] import hashlib

waylonwalker ↪main v3.8.0 ipython
❯ make_hash(1, "one", "1", 1.0)
'73901d019df012a1cdab826ce301217d'

When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it.

install diskcache #

Install diskcache into your virtual environement of choice using pip from your command line.

python -m pip install diskcache

setup the cache #

There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs

from diskcache import Cache
cache = FanoutCache('.mycache', statistics=True)

Adding to the cache #

Adding to the cache only needs a key and value.

cache.add('me', 'waylonwalker' )

Set the expire time #

Optionally you can set the seconds before it expires. The cache invalidation tools like this is what really makes diskcache shine over using raw sqlite or any sort of static file.

cache.add('me', 'waylonwalker', expire=60)

tagging #

Diskcache supports tagging entries added to the cache.

# add an item to the cache with a tag
cache.add('me', 'waylonwalker', expire=60, tag='people')

This seems to let you do a few new things like getting items from the cache by both key and tag, or evict all tags from the cache.

# evict all items tagged as 'people' from the cache
cache.evict(tag='people')

Reading from the cache #

You can read from the cache by using the .get method and giving it the key you want to retrieve.

who = cache.get('me')
# who == 'waylonwalker'

Cache Misses #

Cache misses will return a None just like any dictionary .get miss.

missed = cache.get('missing')
# missed == None

⭐ #

Give Grant some love and give grantjenks/python-diskcache a ⭐.

The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache.

lru_cache #

The easiest and most common to setup in python is a builtin functools.lru_cache.

from functools import lru_cache

@lru_cache
def get_cars():
    print('pulling cars data')
    return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

when to use lru_cache #

Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache.

when same *args, **kwargs always return the same value

lru_cache only works for one python process. If you are running multiple subprocesses, or running the same script over and over, lru_cache will not work.

lru_cache only caches in a single python process

max_size #

lru_cache can take an optional parameter maxsize to set the size of your cache. By default its set to 128, if you want to store more or less items in your cache you can adjust this value.

The get_cars example is a bit of a unique one. As anthonywritescode points out this implementation is behaving like a singleton, and we can optimize the size of the cache by allocating exactly how many items we will ever have in it by setting its value to 1.

from functools import lru_cache

@lru_cache(maxsize=1)
def get_cars():
    print('pulling cars data')
    return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

My example stretches the rule a little bit #

The example above does a web request. As a Data Engineer I often write scripts that run for a short time then stop. I do not expect the output of this function to change during the runtime of this job, and if it did I may actually want them to match anyways.

web request do change their output

If I were building webapps, or some sort of process that was running for a long time. Something that starts and waits for work, this may not be a good application of lru_cache. If this process is running for days or months my assumption that the request does not change is no longer valid.

There’s also a typed kwarg for lru_cache #

This one is new to me but you can cache not only on the value, but the type of the value being passed into your function.

(from the docstring) If typed is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results.

I keep a small cars.csv on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate.

Hosts switched #

I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request.

Not my go to dataset 😭 #

This breaks my go to example dataset.

pd.read_csv("https://waylonwalker.com/cars.csv")

# HTTPError: HTTP Error 403: Forbidden

But requests works??? #

What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does.

requests.get("https://waylonwalker.com/cars.csv")

<Response [200]>

Setting the User Agent in pandas.read_csv #

this fixed the issue for me!

After a bit of googling I realize that this is a common thing, and that setting the user-agent fixes it. This is the point I remember seeing in the cloudflare dashbard that they protect against a lot of different attacks, aparantly it treats pd.read_csv as an attack on my cloudflare pages site.

pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

# success

Now my data is back #

Now this works again, but it feels like just a bit more effort than I want to do by hand. I might need to look into my cloudflare settings to see if I can allow this dataset to be accessed by pd.read_csv.