Drafts

Draft and unpublished posts

0 posts simple view

Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.

Install Requests #

Requests is on pypi and can be installed into your virtual environtment with pip.

python -m pip install requests

Getting the content of a request #

Requests makes getting content from a web url as easy as possible.

import requests

r = requests.get('https://waylonwalker.com/til/htmx-get/')
article = r.content

html">requests is not limited to html #

Requests can handle any web request and is not limited to only html. Here are some examples to get a markdown file, a csv, and a png image.

htmx_get_md = requests.get('https://waylonwalker.com/til/htmx-get.md').content
cars = requests.get('https://waylonwalker.com/cars.csv').content
profile = requests.get('https://waylonwalker.com/8bitc.png').content

RTFM #

There is way more to requests, this just scratches the surface while covering what you are going to need to get going. The requests docs have way more details.

I recently attended python web conf 2022 and after seeing some incredible presentations on it I am excited to give htmx a try.

The base page #

Start with some html boilerplate, pop in a script tag to add the htmx.org script, and a button that says click me. I added just a tish of style so that it does not sear your delicate developer your eyes.

<!DOCTYPE html>
<html lang="en">
  <head>
    <title></title>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style>
      html  { background: #1f2022; color: #eefbfe; font-size: 64px; }
      button {font-size: 64px;}
      body { height: 100vh; width: 100vw; display: flex; justify-content: center; align-items:center; }
    </style>
    <!-- Load from unpkg -->
    <script src="https://unpkg.com/[email protected]"></script>
  </head>
  <body>
  <!-- have a button POST a click via AJAX -->
  <button hx-get="/partial" hx-swap="outerHTML">
    Click Me
  </button>

  </body>
</html>

Save this as index.html and fire up a webserver and you will be presented with this big beefcake of a button.

big beefcake of a button

If you don’t have a development server preference I reccomend opening the terminal and running python -m http.server 8000 then opening your browser to localhost:8000

The Partial #

Now the page has a button that is ready to replace itself, notice the hx-swap="outerHTML">, with the contents of /partial. To create a static api of sorts we can simply host a partial page in a file at /partial/index.html with the following contents.

<p>
hello
</p>
the final results

Tree #

To make it a bit clearer here is what the file tree looks like after setting this up.

~/git/htmx  v3.9.7 (git)
❯ tree
.
├── clicked
│   └── index.html
└── index.html

Demo #

I added htmx to this page and setup a partial below, check out this easter egg.

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique.

Slide One #

Lets use this section to show what it looks like as I change my styles.

from markata import Markata
Markata()
markata.run()

☝ This is how my website is built

  • write markdown
  • build site
  • publish

default #

This is what the above slide looks like in lookatme.

default styles

Set focus to the most important element #

The way I write my slides I want the most prominant element to be the slides title, not the presentation title. The slides title is generally the point I am trying to make, I will leave some supporting information if I want, but sometimes, I just have a title.

styles:
    title:
        bg: default
        fg: '#e1af66'
    headings:
        '1':
            bg: default
            fg: '#ff66c4,bold,italics'
            prefix: ' ⇁ '
            suffix: ' ↽ '
set the focus on the slide title styles

by default he prefix/suffix was a full block that just went transparant into the slide. I thought the harpoons were fun and went with them on a whim

The box characters bother me #

The box characters are fine really, but it really bothers me that they are not conneted. The author is probably doing this because it looks ok on most systems, and many terminals dont have their fonts right and wont align anyways. I am not sure if I ever had a windows terminal other than their new Terminal that properly connected box characters.

    quote:
        side: '│'
        style:
            bg: default
            fg: '#aaa'
        top_corner: '╭'
        bottom_corner: '╰'

Add Author #

Adding author to the root of the frontmatter of the document will add it to the bottom left of the slides.

author: '@_waylonwalker'
lookatme slides with author defined

Style the author #

We can style the foreground and background of this text by adding something like this to the styles section of the frontmatter.

author:
    bg: default
    fg: '#368ce2'

While we are at it, lets style the rest of the footer to my own theme. Let’s pop this into the style and see what it looks like.

date:
    bg: default
    fg: '#368ce2'
slides:
    bg: default
    fg: '#368ce2'
lookatme slides with author styled

reduce the padding #

When I am presenting I am punched in as big as I can go, and which makes the padding massive. I want as much as the screen real estate devoted to making big readable text as I can.

padding:
    bottom: 0
    left: 0
    right: 0
    top: 0
lookatme slides with no more padding

final results #

Here is what the final frontmatter looks like to fully style my talk.

---
date: 2022-03-24
templateKey: til
title: Style Lookatme Slides a bit more Personal
tags:
  - python
  - cli
  - python
author: '@_waylonwalker'
styles:
    padding:
        bottom: 0
        left: 0
        right: 0
        top: 0
    title:
        bg: default
        fg: '#e1af66'
    date:
        bg: default
        fg: '#368ce2'
    slides:
        bg: default
        fg: '#368ce2'
    headings:
        '1':
            bg: default
            fg: '#ff66c4,bold,italics'
            prefix: ' ⇁ '
            suffix: ' ↽ '
    quote:
        side: '│'
        style:
            bg: default
            fg: '#aaa'
        top_corner: '╭'
        bottom_corner: '╰'
    author:
        bg: default
        fg: '#368ce2'
---

I use a package eyeseast/python-frontmatter{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml).

Install #

It’s on pypi, so you can install it into your virtual environment with pip.

python -m pip install python-frontmatter

🙋 What’s Frontmatter #

Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging platform that also builds all of its posts with markdown and yaml frontmatter.

Let’s see an example #

Here is the exact frontmatter for this post you are reading on my site.

---
date: 2022-03-24 03:18:48.631729
templateKey: til
title: How I load Markdown in Python
tags:
  - linux
  - python

---

This is where the markdown content for the post goes.

So it’s yaml #

yaml is the most commmon, but python-frontmatter{.hoverlink} also supports Handlers{.hoverlink} for toml and json.

If you want a good set of examples of yaml learnxinyminutes{.hoverlink} has a fantastic set of examples in one page.

How to load yaml frontmatter in python #

Here is how I would load this post into python using python-frontmatter{.hoverlink}.

import frontmatter
inspect(frontmatter.load("pages/til/python-frontmatter.md"))

We can use rich{.hoverlink} to inspect the Post object to see what all it contains.

❯ inspect(frontmatter.load("pages/til/python-frontmatter.md"))
╭────────────────────────────────────────────────────────── <class 'frontmatter.Post'> ───────────────────────────────────────────────────────────╮
│ A post contains content and metadata from Front Matter. This is what gets                                                                       │
│ returned by :py:func:`load <frontmatter.load>` and :py:func:`loads <frontmatter.loads>`.                                                        │
│ Passing this to :py:func:`dump <frontmatter.dump>` or :py:func:`dumps <frontmatter.dumps>`                                                      │
│ will turn it back into text.                                                                                                                    │
│                                                                                                                                                 │
│ ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │
│ │ <frontmatter.Post object at 0x7f03c4c23ca0>                                                                                                 │ │
│ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                                                                 │
│  content = "I use a package\n[eyeseast/python-frontmatter](https://github.com/eyeseast/python-frontmatter)\nto load files with frontmatter in   │
│            them.  Its a handy package that allows you to\nload files with structured frontmatter (yaml, json, or toml).\n\n## Install\n\nIt's   │
│            on pypi, so you can install it into your virtual environment with pip.\n\n```bash\npython -m pip install                             │
│            python-frontmatter\n```\n\n## 🙋 What's Frontmatter\n\nFrontmatter is a handy way to add metadata to your plain text files.          │
│            It's\nquite common to have yaml frontmatter in markdown.  All of my blog posts have\nyaml frontmatter to give the post metadata such │
│            as post date, tags, title, and\ntemplate.  dev.to is a popular developer blogging platform that also builds all\nof its posts with   │
│            markdown and yaml frontmatter.\n\n## Let's see an example\n\nHere is the exact frontmatter for this post you are reading on my       │
│            site.\n\n```markdown\n---\ndate: 2022-03-24 03:18:48.631729\ntemplateKey: til\ntitle: How I load Markdown in Python\ntags:\n  -      │
│            linux\n  - python\n\n---\n\nThis is where the markdown content for the post goes.\n```\n\n## So it's yaml\n\nyaml is the most        │
│            commmon, but\n[eyeseast/python-frontmatter](https://github.com/eyeseast/python-frontmatter)\nalso                                    │
│            supports\n[Handlers](https://python-frontmatter.readthedocs.io/en/latest/handlers.html?highlight=toml#module-frontmatter.default_ha… │
│            toml and json.\n\nIf you want a good set of examples of yaml\n[learnxinyminutes](https://learnxinyminutes.com/docs/yaml/) has a      │
│            fantastic set\nof examples in one page.\n\n## How to load yaml frontmatter in python"                                                │
│  handler = <frontmatter.default_handlers.YAMLHandler object at 0x7f03bffbd910>                                                                  │
│ metadata = {                                                                                                                                    │
│                'date': datetime.datetime(2022, 3, 24, 3, 18, 48, 631729),                                                                       │
│                'templateKey': 'til',                                                                                                            │
│                'title': 'How I load Markdown in Python',                                                                                        │
│                'tags': ['linux', 'python', 'python']                                                                                            │
│            }                                                                                                                                    │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Getting Metadata #

You can get items from the posts metadata just as you would from a dict.

post = frontmatter.load("pages/til/python-frontmatter.md")
post['date']
# datetime.datetime(2022, 3, 24, 3, 18, 48, 631729)

post.get('date')
# datetime.datetime(2022, 3, 24, 3, 18, 48, 631729)

python dict get

I have recently become fond of the .get method to give it an easy default value.

Content is content #

The content of the document is stored under .content

post.content

Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584.

Merge Dicts #

I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result.

default_config = {'url': 'https://example.com', 'assets_dir': 'static' }
user_config = {'url': 'https://waylonwalker.com'}

# **unpacking goes back much further than 3.9

config = {**default_config, **user_config}
print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}


# the same can be achieved through the new to python 3.9 | operator

config = default_config | user_config
print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}

understanding python *args and **kwargs

More on unpacking in this post.

Update Dicts #

With the release there is also a new update syntax |= that you can use to update. I dont often mutate variables for some reason, so I cant think of a better example for this from my personal use cases. So I will give a similar example to above, except creating a config, then updating it.

# old python <3.9 way
config = {'url': 'https://example.com', 'assets_dir': 'static' }
config.update({'url': 'https://waylonwalker.com'})

# new python 3.9+ way
config = {'url': 'https://example.com', 'assets_dir': 'static' }
config |= {'url': 'https://waylonwalker.com'}

print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}

Should you use it? #

Are you writing libraries/applications that are only going to be ran on 3.9? Then ya go for it there is nothing to loose. If there is any chance someone is going to run your code on 3.8 or older then just use **, or .update.

RTFM #

This is what comes first to my mind on how to use this new syntax, read pep584 for all the gritty details on it.

I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides.

Present from the terminal #

I will most often just present right from the terminal using lookatme. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly.

reveal.js #

I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share.

New Slides #

I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style.

Installation #

lookatme is a python library that is available on pypi, you can install it with the pip command.

python -m pip install lookatme

Since it’s a command line application it works great with pipx. This prevents the need to manage virtual environments yourself or ending up with packages clashing in your system python environment.

pipx install lookatme

From my terminal #

lookatme {filepath}

I just run it with pipx.

pipx run \
 --spec git+https://github.com/waylonwalker/lookatme \
 lookatme {filepath} \
 --live-reload \
 --style gruvbox-dark

Note, I use a custom fork of lookatme. It’s schema validation did not like the date format of my blog posts, so I have a one line fix built into my fork that is pretty specific to me.

From Neovim #

most often what I do

From Neovim I use a plugin I created for sending out commands to tmux called telegraph. This sends the above command to a new session that I can bounce between quickly.

nnoremap <leader><leader>s <cmd>lua require'telegraph'.telegraph({cmd='pipx run --spec git+https://github.com/waylonwalker/lookatme lookatme {filepath} --live-reload --style gruvbox-dark', how='tmux'})<CR>

When I need to read contents from a plain text file in python I find the easiest way is to just use Pathlib.

from pathlib import Path

Path('path_to_file').read_text()

Let’s make a vim command to automatically collect all the links in these posts at the end of each article. Regex confuses the heck out of me… I don’t have my regex liscense, but regex can be so darn powerful especially in an editor.

Step one #

Before you run someone’s regex from the internet that you don’t fully understand, check your git status and make sure you are all clear with git before you wreck something

Inspiration #

Something that I have always appreciated form Nick Janetakis is his links section. I often try to gather up the links at the end of my posts, but often end up not doing it or forgetting.

Searchng through the internet I was able to find an article from Vitaly Parnas called vim ref links that did almost exactly what I needed, except it was more complicated and made them into ref liks.

Here is my interpretation of the code I took from Vitaly’s post. It makes a Links section like the one at the bottom of this post.

function! MdLinks()
    $norm o## Links
    $norm o
    g/\[[^\]]\+\]([^)]\+)/t$
    silent! '^,$s/\v[^\[]*(\[[^\]]+\])\(([^)]+)\)[^\[]*/* \1(\2)/g
    nohl
endfunction
command! MdLinks call MdLinks()

So far it is working for me and saving me a few seconds off each post I make.

If you ever end up on a linux machine that just does not have enough ram to suffice what you are doing and you just need to get the job done you can give it some more swap. You can look up reccomendations for how much swap you should have this is more about just trying to get your job done when you are almost there, but running out of memory on the hardware you have.

make the /swap file #

You can put this where you wish, for this example I am going to pop it into /swap

sudo fallocate -l 4G /swap
sudo chmod 600 /swap
sudo mkswap /swap
sudo swapon /swap

make sure that your swap is on #

You can make sure that your swap is working by using the free command, I like using the -h flag to get human readable numbers.

❯ free -h
               total        used        free      shared  buff/cache   available
Mem:            15Gi       5.5Gi       4.9Gi       458Mi       5.2Gi       9.3Gi
Swap:          4.0Gi          0B       4.0Gi

Reclaim memory usage in Jupyter

I also used this trick in this article to give my python process a bit more oompf and get it on home.

A very common task for any script is to look for files on the system. My go to method when globbing for files in python is to use pathlib.

Setup #

I setup a directory to make some examples about globbing. Here is what the directory looks like.

❯ tree .
.
├── content
│   ├── hello.md
│   ├── hello.py
│   ├── me.md
│   └── you.md
├── readme.md
├── README.md
├── READMES.md
└── setup.py

1 directory, 8 files

Pathlib #

Pathlib is a standard library module available in all LTS versions of python at this point.

❯ from pathlib import Path

Creating a Path instance.

# current working directory
Path()
Path.cwd()

# The users home directory
Path.home()

# Path to a directory by string
Path('/path/to/directory')

# The users ~/.config directory
Path.home() / '.config'

Globbing Examples #

The path object has a glob method that allows you to glob for files with a unix style glob pattern to search for files. Note that it gives you a generator. This is great for many use cases, but for examples its easier to turn them to a list to print them out.

If you need some more detail on what globbing is there is a wikipedia article discussing it. I am just showing how to glob with pathlib.


❯ Path().glob("**/*.md")
<generator object Path.glob at 0x7fa35adc4f90>

❯ list(Path().glob("**/*.md"))

[
    PosixPath('readme.md'),
    PosixPath('READMES.md'),
    PosixPath('README.md'),
    PosixPath('content/you.md'),
    PosixPath('content/me.md'),
    PosixPath('content/hello.md')
]

❯ list(Path().glob("**/*.py"))
[PosixPath('setup.py'), PosixPath('content/hello.py')]

❯ list(Path().glob("*.md"))
[PosixPath('readme.md'), PosixPath('READMES.md'), PosixPath('README.md')]

❯ list(Path().glob("*.py"))
[PosixPath('setup.py')]

❯ list(Path().glob("**/*hello*"))
[PosixPath('content/hello.py'), PosixPath('content/hello.md')]

❯ list(Path().glob("**/REA?ME.md"))
[PosixPath('README.md')]

Setting up your git pager to your liking can help you navigate diffs and logs much more efficiently. You can set it to whatever pager you like so that your keys feel nice and smooth and your fingers know exactly what to do. You might even gain a few extra features.

Setting the pager #

You can set the pager right from your command line with the following command.

git config --global core.pager 'more'

You can also set your pager by editing your global .gitconfig file which by default is set to ~/.gitconfig.

[core]
    pager = more

Color #

In my experience you need to turn colors off with nvim. bat handles them and looks good either way, but nvim will be plain white and display the color codes as plain text if color is on.

git config --global color.pager no

Pagers I have tried #

Here are some various configs that I tried. For some reason line numbers in bat really bothered me, but when in nvim they felt ok. I am going to try running both of them for a few days and see which I like better. I think having some of my nvim config could be really handy for things like yanking a commit hash to the system clipboard without touching the mouse.

# bat
git config --global core.pager 'bat'

# nvim in read only mode
git config --global core.pager 'nvim -R'

# turn colors off
git config --global color.pager no

# bat with no line numbers
git config --global core.pager 'bat --style=plain'

# nvim with no line numbers and a specific rc file
git config --global core.pager "nvim -R +'set nonumber norelativenumber' -u ~/.config/nvim/init-git.vim"

reset back to the default #

If you are afraid to try one of these settings, don’t be you can always change it back. If you tried one and dont like it just --unset the config that you just tried.

git config --global --unset core.pager
git config --global --unset color.pager

The other option you have is to open your .gitconfig file and delete the lines of config that set your pager.

If you have ever mistyped a git command very close to an existing one you have likely seen this message.

❯ git chekout dev
git: 'chekout' is not a git command. See 'git --help'.

The most similar command is
        checkout

Automatically run the right one #

What you might not have known is that you can configure git to just run this command for you.

# Gives you 0.1 seconds to respond
git config --global help.autocorrect 1

# Gives you 1 seconds to respond
git config --global help.autocorrect 10

# Gives you 5 seconds to respond
git config --global help.autocorrect 50

Fat Fingers Gone #

Now when you typo a git command it will autmatically run after the configured number of tenths of a second.

❯ git chkout get-error
WARNING: You called a Git command named 'chkout', which does not exist.
Continuing in 1.0 seconds, assuming that you meant 'checkout'.
M       pages/blog/how-i-deploy-2021.md
M       pages/hot_tips/001.md
M       pages/templates/gratitude_card.html
M       plugins/index.py
M       plugins/publish_amp.py
M       plugins/render_template_variables.py
M       plugins/youtube.py
M       requirements.txt
M       static/index.html
Switched to branch 'get-error'

My config #

I’m rocking 10 for now just to see how I feel about it, but honestly I cannot think of a time that I have seen the original warning that was not what I wanted. This at least gives me some time to respond if I am unsure.

git config --global help.autocorrect 10

yq is a command line utility for parsing and querying yaml, like jq does for json.

This is for me #

I love that all of these modern tools built in go and rust, just give you a zipped up executable right from GitHub releases, but it’s not necessarily straight forward how to install them. yq does one of the best jobs I have seen, giving you instructions on how to get a specific version and install it.

I use a bunch of these tools, and for what its worth I trust the devs behind them to make sure they don’t break. This so far has worked out well for me, but if it ever doesn’t I can always pick an older version.

Just give me the latest #

Since I am all trusting of them I just want the latest version. I do not want to update a shell script with new versions, or even care about what then next version is, I just want it. Luckily you can script the release page for the latest version on all that I have came accross.

What is the latest #

I wrote or stole, I think I wrote it, this line of bash quite awhile ago, and it has served me well for finding the latest release for any GitHub project using releases. Just update it with the name of the tool, org, and repo and it works.

YQ_VERSION=$(curl --silent https://github.com/mikefarah/yq/releases/latest | tr -d '"' | sed 's/^.*tag\///g' | sed 's/>.*$//g' | sed 's/^v//')

Install with your shell #

Now that we know how to consistently get the right version, I generally right click the release in the releases page, replace the version with ${TOOL_VERSION} and put it in this wget call, then move the binary over to ~/.local/bin

local tmp=`mktemp -dt install-XXXXXX`
pushd $tmp
YQ_VERSION=$(curl --silent https://github.com/mikefarah/yq/releases/latest | tr -d '"' | sed 's/^.*tag\///g' | sed 's/>.*$//g' | sed 's/^v//')
wget https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz -O- -q | tar -zxf - -C /tmp
cp yq_linux_amd64 ~/.local/bin/yq
popd

Install with ansible #

Now I don’t want to worry about missing yq again, so I am added it to my ansible install script. This way it’s installed everyt time I setup a new system with all of my favorite cli’s.

- name: check is yq installed
  shell: command -v yq
  register: yq_exists
  ignore_errors: yes
  tags:
    - yq

- name: Install yq
  when: yq_exists is failed
  shell: |
    local tmp=`mktemp -dt install-XXXXXX`
    pushd $tmp
    YQ_VERSION=$(curl --silent https://github.com/mikefarah/yq/releases/latest | tr -d '"' | sed 's/^.*tag\///g' | sed 's/>.*$//g' | sed 's/^v//')
    wget https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz -O- -q | tar -zxf - -C /tmp
    cp yq_linux_amd64 {{ lookup('env', 'HOME') }}/.local/bin/yq
    popd
  tags:
    - yq

This is how I installed it, of course you can always follow Mike’s instructions from the repo.

Last Thursday I learned about pytest-mock at a local python meetup. The presenter showed how he uses pytest-mock for his work, and it was kinda eye opening. I knew what mocking was, but I had not seen it in this context.

Discovery #

Watching him use pytest-mock I realized that mocking was not as hard as I had made it out to be. You can install pytest-mock, use the mocker fixture, and patch objects methods with what you want them to be.

install #

pytest-mock is out on pypi and can be installed with pip.

python -m pip install pytest-mock

What I actually did #

Sometimes I fall victim to making these posts nice and easy to follow. It takes more steps than just pip install, you need a place to practice in a nice sandbox. Here is how I make my sandboxes.

mkdir ~/git/learn-pytest-mock
cd ~/git/learn-pytest-mock
# well actually open a new tmux session there
echo pytest-mock > requirements.txt

# I copied in my .envrc, and ran direnv allow, which actually just made me a virtual env as follows
python3 -m venv .venv --prompt $(basename $PWD)
source .venv/bin/activate

# now install pytest-mock
pip install -r requirements.txt

# make some tests to mock
mkdir tests
nvim tests/test_me.py

create a tests/test_me.py #

I just wanted to do something that was worth mocking, the first thing that came to mind was to do something that made a network call. Here I made a method that uses requests to go get the content on my homepage, but changes it’s return behavior based on the status_code of the request.

I want to mock out requests to ensure that GoGetter can handle both 200 (http success) and 404 (http not found) status codes.

# tests/test_me.py
import requests


class GoGetter:
    """
    The thing I am testing, this is usually imported into the test file, but
    defined here for simplicity.
    """
    def get(self):
        """
        Get the content of `https://waylonwalker.com` and return it as a string
        if successfull, or False if it's not found.
        """
        r = requests.get("https://waylonwalker.com")
        if r.status_code == 200:
            return r.content
        if r.status_code == 404:
            return False


class DummyRequester:
    def __init__(self, content, status_code):
        """
        mock out content and status_code
        """

        self.content = content
        self.status_code = status_code

    def __call__(self, url):
        """
        The way I set this up GoGetter is going to call an instance of this
        class, so the easiest way to make it work was to implement __call__.
        """
        self.url = url
        return self


def test_success_get(mocker):
    """
    Show that the GoGetter can handle successful calls.
    """
    go_getter = GoGetter()

    # Use the mocker fixture to change how requests.get works while inside of test_success_get
    mocker.patch.object(requests, "get", DummyRequester("waylonwalker", 200))
    assert "waylon" in go_getter.get()


def test_failed_get(mocker):
    """
    Show that the GoGetter can handle failed calls.
    """
    go_getter = GoGetter()

    # Use the mocker fixture to change how requests.get works while inside of test_failed_get
    mocker.patch.object(requests, "get", DummyRequester("waylonwalker", 404))
    assert go_getter.get() is False

Python 3.8 came out two and a half years ago and I have yet to really lean in on the walrus operator. Partly because it always seemed like something kinda silly (my use cases) to require a python version bump for, and partly because I really didn’t understand it the best. Primarily I have wanted to use it in comprehensions, but I did not really understand how.

Now that Python 3.6 is end of life, and most folks are using at least 3.8 it seems time to learn and use it.

What’s a Walrus #

:=

The assignment operator in python is more commonly referred to as the walrus operator due to how := looks like a walrus. It allows you to assign and use a variable in a single expression.

This example from the docs avoids a second call to the len function.

if (n := len(a)) > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

Let’s get some data #

without a walrus

In this example we are going to do a dict comp to generate a map of content from urls, only if their status code is 200. When doing this in a dictionary comprehension we end up needing to hit the url twice for successful urls. Once for the filter and once for the data going into the dictionary.

{
    url: requests.get(url).content
    for url in ["https://waylonwalker.com/", "https://waylonwalker.com/broken"]
    if requests.get(url).status_code == 200
}

Gimme some walrus #

using walrus in a dict comp

Using the walrus operator := list comp allows us to only put things into the dictionary that we want to keep, and not hit the url twice.

{
    url: r.content
    for url in ["https://waylonwalker.com/", "https://waylonwalker.com/broken"]
    if (r := requests.get(url)).status_code == 200
}

FIN #

The walrus is a nice to have option to save on extra function/network calls, and micro optimize your code without adding much extra.

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.

Install kedro rich #

There is no pypi package yet, but it’s on github. You can pip install it with the git url.

pip install git+https://github.com/datajoely/kedro-rich

Kedro run #

You can run your pipeline just as you normally would, except you get progress bars and pretty prints.

kedro run
kedro rich pretty run

Kedro catalog #

Listing out catalog entries from the command line now print out a nice pretty table.

kedro catalog list
kedro rich catalog list table output

Give it a star #

Go to the GitHub repo and give it a star, Joel deserves it.

So worktrees, I always thought they were a big scary things. Turns out they are much simpler than I thought.

Myth #1 #

no special setup

I thought you had to be all in or worktrees or normal git, but not both. When I see folks go all in on worktrees they start with a bare repo, while its true this is the way you go all in, its not true that this is required.

Lets make a worktree #

Making a worktree is as easy as making a branch. It’s actually just a branch that lives in another place in your filesystem.

# checkout a new worktree called compare based on main in /tmp/project
git worktree add -b compare /tmp/project main

# checkout a new worktree called compare based on HEAD in /tmp/project
git worktree add -b compare /tmp/project

# checkout a worktree from an existing feature branch in /tmp/project
git worktree add /tmp/project my-existing-feature-branch

The worktree that you create is considered a linked worktree, while the original worktree is called the main worktree

Note that I put this in my tmp directory because I don’t expect it to live very long, my recent use case was to compare two files after a big formatting change. You put these where you want, but dont come at me when your /tmp gets wiped and you loose work.

Myth #2 #

they are hidden mysterious creatures

Just like branches git has some nice commands to help us understand what worktrees we have on our system. Firstly we have something very specific to worktrees to list them out.

git worktree list

gives the output

/home/u_walkews/git/git-work-play  b202442 [main]
/tmp/another                       d9b2cf1 [another]

Even the branch command gives a bit different output for a worktree.

git branch

gives this output, notice the + denotes an actively linked worktree, and the * gives the active branch. If you cd over to the worktree directory, these will switch roles.

+ another
  just-a-branch
* main

You can only checkout a branch in one place #

If you try to checkout a branch that is checked out in a linked worktree, you will be presented with an error, and it will not let you check out a second copy of that branch.

❯ git checkout another
fatal: 'another' is already checked out at '/tmp/another'

Myth #3 #

once you go worktree, you worktree

Once you have worktrees on your system, you have a few ways to get rid of them. Using git’s way feels much superior, but if your a doof like me and didn’t read the manual before you rm /tmp/another -rf you will notice that the worktree is still active. If you run git worktree prune it will clean that right up.

git worktree remove another

rm /tmp/another
git worktree prune

It won’t let you remove if you have changes #

This makes me think that remove is a much safer option. If you have uncommitted changes, git worktree remove will throw an error, and make you commit or use --force to remove the worktree.

❯ git worktree remove another
fatal: 'another' contains modified or untracked files, use --force to delete it

RTFM #

read the friendly manual

There is a ton more information in the man page for worktrees, these are just the parts that seemed really useful to me out of the gate.

man git worktree

Has no upstream branch errors in git can be such a damn productivity killer. You gotta stop your flow and swap over the branch, there is a config so that you don’t have to do this.

fatal has no upstream branch #

If you have not yet configured git to always push to the current branch, you will get a has no upstream branch error if you don’t explicitly set it.

Let’s show an example

git checkout -b feat/ingest-inventory-data
git add conf/base/catalog.yml
git commit -m "feat: ingest inventory data from abc-db"
git push

You will be presented with the following error.

fatal: The current branch feat/ingest-inventory-data has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin feat/ingest-inventory-data

Option 1: follow the instructions #

To resolve this fatal error your first option is simply to follow the instructions given. Just copy and paste it in.

git push --set-upstream origin feat/ingest-inventory-data

Option 2: push to the current branch without setting upstream #

Honestly, I am pretty aware of the branch I am on, and Very few times have I ever accidentally pushed to the wrong branch. The one that you might have a bigger chance with a more detrimental effect is main, which I will argue you should have blocked to require a passing ci, and potential reviewers to merge in. Therefore you can’t even push to main anyway.

To just push to the branch you are currently on each and every time and never see this error again, you can run this to configure git to always push to your current branch.

git config --global push.default current

I write many of these posts from a 10 year old desktop that sits in my office these days. It does a very fine job running all of the things I need it to for my side work, but sometimes I want a mobile setup. I don’t really want to spend the $$ on a new laptop just for the few times I want to be somewhere else in the house. What I do have though is a chromebook.

I’ve tried to get the chromebook into my workflow in the past, but have failed. Much because by the time I got all of my tools up and running in the linux vm it was taking up quite a bit of space on the device and made it harder for others to use as a chromebook.

Today I am giving it a second try, but this time with ssh.

Checking for existing sshd #

Before doing anything I checked to see if sshd is already running. Using the following command.

sudo service ssh status
# or
pgrep -l sshd

Both returned nothing so I know that its not running.

setting up sshd #

just apt install it

Next install the openssh-client and openssh-server

sudo apt install openssh-client -y
sudo apt install openssh-server -y

After this I can see that its now running by checking its status once again.

sudo service ssh status

Gives me the result.

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-03-08 08:17:05 CST; 12min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 181185 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 181189 (sshd)
      Tasks: 1 (limit: 19119)
     Memory: 2.8M
        CPU: 96ms
     CGroup: /system.slice/ssh.service
             └─181189 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups

Accessing the desktop #

I have already enabled the Linux terminal on my chromebook, so I just opened the terminal, and ran the following.

ssh <username>@<ip-address>

It prompted for my password and I was in. I had all of my vim, tmux, and zsh comforts that I enjoy without installing anything. It worked so well that this whole post was written from my chromebook.

Limitations #

This does limit me to being on the same network as my desktop, which these days is almost always true.

ssh keys #

Out of the box I am just using passwords to get in, but if this were public I would lock down to requiring an ssh key to enter. I’ll likey do this in a future post.

Mermaid gives us a way to style nodes through the use of css, but rather than using normal css selectors we need to use style <nodeid>. This also applies to subgraphs, and we can use the name of the subgraph in place of the nodeid.

graph TD;
    a --> A
    A --> B
    B --> C

    style A fill:#f9f,stroke:#333,stroke-width:4px
    style B fill:#f9f,stroke:#333,stroke-width:4px

    subgraph one
        a
    end

    style one fill:#BADA55

produces the following graph

graph TD; a --> A A --> B B --> C style A fill:#f9f,stroke:#333,stroke-width:4px style B fill:#f9f,stroke:#333,stroke-width:4px subgraph one a end

style one fill:#BADA55