Posts tagged: python

All posts with the tag "python"

310 posts latest post 2026-05-06
Publishing rhythm
Jan 2026 | 3 posts

When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is diskcache. It’s built on sqlite with just enough cacheing niceties that make it very worth it.

install diskcache #

Install diskcache into your virtual environement of choice using pip from your command line.

python -m pip install diskcache

setup the cache #

There are a couple of different types of cache, Cache, FanoutCache, and DjangoCache, you can read more about those in the docs

from diskcache import Cache
cache = FanoutCache('.mycache', statistics=True)

Adding to the cache #

Adding to the cache only needs a key and value.

cache.add('me', 'waylonwalker' )

Set the expire time #

Optionally you can set the seconds before it expires. The cache invalidation tools like this is what really makes diskcache shine over using raw sqlite or any sort of static file.

cache.add('me', 'waylonwalker', expire=60)

tagging #

Diskcache supports tagging entries added to the cache.

# add an item to the cache with a tag
cache.add('me', 'waylonwalker', expire=60, tag='people')

This seems to let you do a few new things like getting items from the cache by both key and tag, or evict all tags from the cache.

# evict all items tagged as 'people' from the cache
cache.evict(tag='people')

Reading from the cache #

You can read from the cache by using the .get method and giving it the key you want to retrieve.

who = cache.get('me')
# who == 'waylonwalker'

Cache Misses #

Cache misses will return a None just like any dictionary .get miss.

missed = cache.get('missing')
# missed == None

#

Give Grant some love and give grantjenks/python-diskcache a ⭐.

The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don’t need to recompute anything you can simply retrieve it from a cache.

lru_cache #

The easiest and most common to setup in python is a builtin functools.lru_cache.

from functools import lru_cache

@lru_cache
def get_cars():
    print('pulling cars data')
    return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

when to use lru_cache #

Any time you have a function where you expect the same results each time a function is called with the same inputs, you can use lru_cache.

when same *args, **kwargs always return the same value

lru_cache only works for one python process. If you are running multiple subprocesses, or running the same script over and over, lru_cache will not work.

lru_cache only caches in a single python process

max_size #

lru_cache can take an optional parameter maxsize to set the size of your cache. By default its set to 128, if you want to store more or less items in your cache you can adjust this value.

The get_cars example is a bit of a unique one. As anthonywritescode points out this implementation is behaving like a singleton, and we can optimize the size of the cache by allocating exactly how many items we will ever have in it by setting its value to 1.

from functools import lru_cache

@lru_cache(maxsize=1)
def get_cars():
    print('pulling cars data')
    return pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

My example stretches the rule a little bit #

The example above does a web request. As a Data Engineer I often write scripts that run for a short time then stop. I do not expect the output of this function to change during the runtime of this job, and if it did I may actually want them to match anyways.

web request do change their output

If I were building webapps, or some sort of process that was running for a long time. Something that starts and waits for work, this may not be a good application of lru_cache. If this process is running for days or months my assumption that the request does not change is no longer valid.

There’s also a typed kwarg for lru_cache #

This one is new to me but you can cache not only on the value, but the type of the value being passed into your function.

(from the docstring) If typed is True, arguments of different types will be cached separately. For example, f(3.0) and f(3) will be treated as distinct calls with distinct results.

I keep a small cars.csv on my website for quickly trying out different pandas operations. It’s very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate.

Hosts switched #

I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request.

Not my go to dataset 😭 #

This breaks my go to example dataset.

pd.read_csv("https://waylonwalker.com/cars.csv")

# HTTPError: HTTP Error 403: Forbidden

But requests works??? #

What’s weird is, requests still works just fine! Not sure why using urllib the way pandas does breaks the request, but it does.

requests.get("https://waylonwalker.com/cars.csv")

<Response [200]>

Setting the User Agent in pandas.read_csv #

this fixed the issue for me!

After a bit of googling I realize that this is a common thing, and that setting the user-agent fixes it. This is the point I remember seeing in the cloudflare dashbard that they protect against a lot of different attacks, aparantly it treats pd.read_csv as an attack on my cloudflare pages site.

pd.read_csv("https://waylonwalker.com/cars.csv", storage_options = {'User-Agent': 'Mozilla/5.0'})

# success

Now my data is back #

Now this works again, but it feels like just a bit more effort than I want to do by hand. I might need to look into my cloudflare settings to see if I can allow this dataset to be accessed by pd.read_csv.

Python’s requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement.

Install Requests #

Requests is on pypi and can be installed into your virtual environtment with pip.

python -m pip install requests

Getting the content of a request #

Requests makes getting content from a web url as easy as possible.

import requests

r = requests.get('https://waylonwalker.com/til/htmx-get/')
article = r.content

html">requests is not limited to html #

Requests can handle any web request and is not limited to only html. Here are some examples to get a markdown file, a csv, and a png image.

htmx_get_md = requests.get('https://waylonwalker.com/til/htmx-get.md').content
cars = requests.get('https://waylonwalker.com/cars.csv').content
profile = requests.get('https://waylonwalker.com/8bitc.png').content

RTFM #

There is way more to requests, this just scratches the surface while covering what you are going to need to get going. The requests docs have way more details.

I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked… classic procrastination technique.

Slide One #

Lets use this section to show what it looks like as I change my styles.

from markata import Markata
Markata()
markata.run()

☝ This is how my website is built

  • write markdown
  • build site
  • publish

default #

This is what the above slide looks like in lookatme.

default styles

Set focus to the most important element #

The way I write my slides I want the most prominant element to be the slides title, not the presentation title. The slides title is generally the point I am trying to make, I will leave some supporting information if I want, but sometimes, I just have a title.

styles:
    title:
        bg: default
        fg: '#e1af66'
    headings:
        '1':
            bg: default
            fg: '#ff66c4,bold,italics'
            prefix: ' ⇁ '
            suffix: ' ↽ '
set the focus on the slide title styles

by default he prefix/suffix was a full block that just went transparant into the slide. I thought the harpoons were fun and went with them on a whim

The box characters bother me #

The box characters are fine really, but it really bothers me that they are not conneted. The author is probably doing this because it looks ok on most systems, and many terminals dont have their fonts right and wont align anyways. I am not sure if I ever had a windows terminal other than their new Terminal that properly connected box characters.

    quote:
        side: '│'
        style:
            bg: default
            fg: '#aaa'
        top_corner: '╭'
        bottom_corner: '╰'

Add Author #

Adding author to the root of the frontmatter of the document will add it to the bottom left of the slides.

author: '@_waylonwalker'
lookatme slides with author defined

Style the author #

We can style the foreground and background of this text by adding something like this to the styles section of the frontmatter.

author:
    bg: default
    fg: '#368ce2'

While we are at it, lets style the rest of the footer to my own theme. Let’s pop this into the style and see what it looks like.

date:
    bg: default
    fg: '#368ce2'
slides:
    bg: default
    fg: '#368ce2'
lookatme slides with author styled

reduce the padding #

When I am presenting I am punched in as big as I can go, and which makes the padding massive. I want as much as the screen real estate devoted to making big readable text as I can.

padding:
    bottom: 0
    left: 0
    right: 0
    top: 0
lookatme slides with no more padding

final results #

Here is what the final frontmatter looks like to fully style my talk.

---
date: 2022-03-24
templateKey: til
title: Style Lookatme Slides a bit more Personal
tags:
  - python
  - cli
  - python
author: '@_waylonwalker'
styles:
    padding:
        bottom: 0
        left: 0
        right: 0
        top: 0
    title:
        bg: default
        fg: '#e1af66'
    date:
        bg: default
        fg: '#368ce2'
    slides:
        bg: default
        fg: '#368ce2'
    headings:
        '1':
            bg: default
            fg: '#ff66c4,bold,italics'
            prefix: ' ⇁ '
            suffix: ' ↽ '
    quote:
        side: '│'
        style:
            bg: default
            fg: '#aaa'
        top_corner: '╭'
        bottom_corner: '╰'
    author:
        bg: default
        fg: '#368ce2'
---

I use a package eyeseast/python-frontmatter{.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml).

Install #

It’s on pypi, so you can install it into your virtual environment with pip.

python -m pip install python-frontmatter

🙋 What’s Frontmatter #

Frontmatter is a handy way to add metadata to your plain text files. It’s quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata such as post date, tags, title, and template. dev.to is a popular developer blogging platform that also builds all of its posts with markdown and yaml frontmatter.

Let’s see an example #

Here is the exact frontmatter for this post you are reading on my site.

---
date: 2022-03-24 03:18:48.631729
templateKey: til
title: How I load Markdown in Python
tags:
  - linux
  - python

---

This is where the markdown content for the post goes.

So it’s yaml #

yaml is the most commmon, but python-frontmatter{.hoverlink} also supports Handlers{.hoverlink} for toml and json.

If you want a good set of examples of yaml learnxinyminutes{.hoverlink} has a fantastic set of examples in one page.

How to load yaml frontmatter in python #

Here is how I would load this post into python using python-frontmatter{.hoverlink}.

import frontmatter
inspect(frontmatter.load("pages/til/python-frontmatter.md"))

We can use rich{.hoverlink} to inspect the Post object to see what all it contains.

 inspect(frontmatter.load("pages/til/python-frontmatter.md"))
╭────────────────────────────────────────────────────────── <class 'frontmatter.Post'> ───────────────────────────────────────────────────────────╮
 A post contains content and metadata from Front Matter. This is what gets                                                                       
 returned by :py:func:`load <frontmatter.load>` and :py:func:`loads <frontmatter.loads>`.                                                        
 Passing this to :py:func:`dump <frontmatter.dump>` or :py:func:`dumps <frontmatter.dumps>`                                                      
 will turn it back into text.                                                                                                                    
                                                                                                                                                 
 ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ 
  <frontmatter.Post object at 0x7f03c4c23ca0>                                                                                                  
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ 
                                                                                                                                                 
  content = "I use a package\n[eyeseast/python-frontmatter](https://github.com/eyeseast/python-frontmatter)\nto load files with frontmatter in   │
            them.  Its a handy package that allows you to\nload files with structured frontmatter (yaml, json, or toml).\n\n## Install\n\nIt's   │
            on pypi, so you can install it into your virtual environment with pip.\n\n```bash\npython -m pip install                             
            python-frontmatter\n```\n\n## 🙋 What's Frontmatter\n\nFrontmatter is a handy way to add metadata to your plain text files.          │
            It's\nquite common to have yaml frontmatter in markdown.  All of my blog posts have\nyaml frontmatter to give the post metadata such │
            as post date, tags, title, and\ntemplate.  dev.to is a popular developer blogging platform that also builds all\nof its posts with   
            markdown and yaml frontmatter.\n\n## Let's see an example\n\nHere is the exact frontmatter for this post you are reading on my       │
            site.\n\n```markdown\n---\ndate: 2022-03-24 03:18:48.631729\ntemplateKey: til\ntitle: How I load Markdown in Python\ntags:\n  -      
            linux\n  - python\n\n---\n\nThis is where the markdown content for the post goes.\n```\n\n## So it's yaml\n\nyaml is the most        │
            commmon, but\n[eyeseast/python-frontmatter](https://github.com/eyeseast/python-frontmatter)\nalso                                    
            supports\n[Handlers](https://python-frontmatter.readthedocs.io/en/latest/handlers.html?highlight=toml#module-frontmatter.default_ha… │
            toml and json.\n\nIf you want a good set of examples of yaml\n[learnxinyminutes](https://learnxinyminutes.com/docs/yaml/) has a      
            fantastic set\nof examples in one page.\n\n## How to load yaml frontmatter in python"                                                │
  handler = <frontmatter.default_handlers.YAMLHandler object at 0x7f03bffbd910>                                                                  
 metadata = {                                                                                                                                    
                'date': datetime.datetime(2022, 3, 24, 3, 18, 48, 631729),                                                                       
                'templateKey': 'til',                                                                                                            
                'title': 'How I load Markdown in Python',                                                                                        
                'tags': ['linux', 'python', 'python']                                                                                            
            }                                                                                                                                    
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Getting Metadata #

You can get items from the posts metadata just as you would from a dict.

post = frontmatter.load("pages/til/python-frontmatter.md")
post['date']
# datetime.datetime(2022, 3, 24, 3, 18, 48, 631729)

post.get('date')
# datetime.datetime(2022, 3, 24, 3, 18, 48, 631729)

python dict get

I have recently become fond of the .get method to give it an easy default value.

Content is content #

The content of the document is stored under .content

post.content

Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584.

Merge Dicts #

I’ve long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the | operator to achieve the same result.

default_config = {'url': 'https://example.com', 'assets_dir': 'static' }
user_config = {'url': 'https://waylonwalker.com'}

# **unpacking goes back much further than 3.9

config = {**default_config, **user_config}
print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}


# the same can be achieved through the new to python 3.9 | operator

config = default_config | user_config
print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}

understanding python *args and **kwargs

More on unpacking in this post.

Update Dicts #

With the release there is also a new update syntax |= that you can use to update. I dont often mutate variables for some reason, so I cant think of a better example for this from my personal use cases. So I will give a similar example to above, except creating a config, then updating it.

# old python <3.9 way
config = {'url': 'https://example.com', 'assets_dir': 'static' }
config.update({'url': 'https://waylonwalker.com'})

# new python 3.9+ way
config = {'url': 'https://example.com', 'assets_dir': 'static' }
config |= {'url': 'https://waylonwalker.com'}

print(config)
# {'url': 'https://waylonwalker.com', 'assets_dir': 'static'}

Should you use it? #

Are you writing libraries/applications that are only going to be ran on 3.9? Then ya go for it there is nothing to loose. If there is any chance someone is going to run your code on 3.8 or older then just use **, or .update.

RTFM #

This is what comes first to my mind on how to use this new syntax, read pep584 for all the gritty details on it.

I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides.

Present from the terminal #

I will most often just present right from the terminal using lookatme. Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly.

reveal.js #

I sometimes also use reveal.js, but that’s for another post. It is handy that it lives in the browser and is easier to share.

New Slides #

I leverage auto slides when I write my slides in markdown. The largest heading, usually an h2 for me, becomes the new slide marker. Otherwise my process is not much different, It just becomes a shorter writing style.

Installation #

lookatme is a python library that is available on pypi, you can install it with the pip command.

python -m pip install lookatme

Since it’s a command line application it works great with pipx. This prevents the need to manage virtual environments yourself or ending up with packages clashing in your system python environment.

pipx install lookatme

From my terminal #

lookatme {filepath}

I just run it with pipx.

pipx run \
 --spec git+https://github.com/waylonwalker/lookatme \
 lookatme {filepath} \
 --live-reload \
 --style gruvbox-dark

Note, I use a custom fork of lookatme. It’s schema validation did not like the date format of my blog posts, so I have a one line fix built into my fork that is pretty specific to me.

From Neovim #

most often what I do

From Neovim I use a plugin I created for sending out commands to tmux called telegraph. This sends the above command to a new session that I can bounce between quickly.

nnoremap <leader><leader>s <cmd>lua require'telegraph'.telegraph({cmd='pipx run --spec git+https://github.com/waylonwalker/lookatme lookatme {filepath} --live-reload --style gruvbox-dark', how='tmux'})<CR>

When I need to read contents from a plain text file in python I find the easiest way is to just use Pathlib.

from pathlib import Path

Path('path_to_file').read_text()

A very common task for any script is to look for files on the system. My go to method when globbing for files in python is to use pathlib.

Setup #

I setup a directory to make some examples about globbing. Here is what the directory looks like.

❯ tree .
.
├── content
│   ├── hello.md
│   ├── hello.py
│   ├── me.md
│   └── you.md
├── readme.md
├── README.md
├── READMES.md
└── setup.py

1 directory, 8 files

Pathlib #

Pathlib is a standard library module available in all LTS versions of python at this point.

 from pathlib import Path

Creating a Path instance.

# current working directory
Path()
Path.cwd()

# The users home directory
Path.home()

# Path to a directory by string
Path('/path/to/directory')

# The users ~/.config directory
Path.home() / '.config'

Globbing Examples #

The path object has a glob method that allows you to glob for files with a unix style glob pattern to search for files. Note that it gives you a generator. This is great for many use cases, but for examples its easier to turn them to a list to print them out.

If you need some more detail on what globbing is there is a wikipedia article discussing it. I am just showing how to glob with pathlib.


 Path().glob("**/*.md")
<generator object Path.glob at 0x7fa35adc4f90>

 list(Path().glob("**/*.md"))

[
    PosixPath('readme.md'),
    PosixPath('READMES.md'),
    PosixPath('README.md'),
    PosixPath('content/you.md'),
    PosixPath('content/me.md'),
    PosixPath('content/hello.md')
]

 list(Path().glob("**/*.py"))
[PosixPath('setup.py'), PosixPath('content/hello.py')]

 list(Path().glob("*.md"))
[PosixPath('readme.md'), PosixPath('READMES.md'), PosixPath('README.md')]

 list(Path().glob("*.py"))
[PosixPath('setup.py')]

 list(Path().glob("**/*hello*"))
[PosixPath('content/hello.py'), PosixPath('content/hello.md')]

 list(Path().glob("**/REA?ME.md"))
[PosixPath('README.md')]

Last Thursday I learned about pytest-mock at a local python meetup. The presenter showed how he uses pytest-mock for his work, and it was kinda eye opening. I knew what mocking was, but I had not seen it in this context.

Discovery #

Watching him use pytest-mock I realized that mocking was not as hard as I had made it out to be. You can install pytest-mock, use the mocker fixture, and patch objects methods with what you want them to be.

install #

pytest-mock is out on pypi and can be installed with pip.

python -m pip install pytest-mock

What I actually did #

Sometimes I fall victim to making these posts nice and easy to follow. It takes more steps than just pip install, you need a place to practice in a nice sandbox. Here is how I make my sandboxes.

mkdir ~/git/learn-pytest-mock
cd ~/git/learn-pytest-mock
# well actually open a new tmux session there
echo pytest-mock > requirements.txt

# I copied in my .envrc, and ran direnv allow, which actually just made me a virtual env as follows
python3 -m venv .venv --prompt $(basename $PWD)
source .venv/bin/activate

# now install pytest-mock
pip install -r requirements.txt

# make some tests to mock
mkdir tests
nvim tests/test_me.py

create a tests/test_me.py #

I just wanted to do something that was worth mocking, the first thing that came to mind was to do something that made a network call. Here I made a method that uses requests to go get the content on my homepage, but changes it’s return behavior based on the status_code of the request.

I want to mock out requests to ensure that GoGetter can handle both 200 (http success) and 404 (http not found) status codes.

# tests/test_me.py
import requests


class GoGetter:
    """
    The thing I am testing, this is usually imported into the test file, but
    defined here for simplicity.
    """
    def get(self):
        """
        Get the content of `https://waylonwalker.com` and return it as a string
        if successfull, or False if it's not found.
        """
        r = requests.get("https://waylonwalker.com")
        if r.status_code == 200:
            return r.content
        if r.status_code == 404:
            return False


class DummyRequester:
    def __init__(self, content, status_code):
        """
        mock out content and status_code
        """

        self.content = content
        self.status_code = status_code

    def __call__(self, url):
        """
        The way I set this up GoGetter is going to call an instance of this
        class, so the easiest way to make it work was to implement __call__.
        """
        self.url = url
        return self


def test_success_get(mocker):
    """
    Show that the GoGetter can handle successful calls.
    """
    go_getter = GoGetter()

    # Use the mocker fixture to change how requests.get works while inside of test_success_get
    mocker.patch.object(requests, "get", DummyRequester("waylonwalker", 200))
    assert "waylon" in go_getter.get()


def test_failed_get(mocker):
    """
    Show that the GoGetter can handle failed calls.
    """
    go_getter = GoGetter()

    # Use the mocker fixture to change how requests.get works while inside of test_failed_get
    mocker.patch.object(requests, "get", DummyRequester("waylonwalker", 404))
    assert go_getter.get() is False

Python 3.8 came out two and a half years ago and I have yet to really lean in on the walrus operator. Partly because it always seemed like something kinda silly (my use cases) to require a python version bump for, and partly because I really didn’t understand it the best. Primarily I have wanted to use it in comprehensions, but I did not really understand how.

Now that Python 3.6 is end of life, and most folks are using at least 3.8 it seems time to learn and use it.

What’s a Walrus #

:=

The assignment operator in python is more commonly referred to as the walrus operator due to how := looks like a walrus. It allows you to assign and use a variable in a single expression.

This example from the docs avoids a second call to the len function.

if (n := len(a)) > 10:
    print(f"List is too long ({n} elements, expected <= 10)")

Let’s get some data #

without a walrus

In this example we are going to do a dict comp to generate a map of content from urls, only if their status code is 200. When doing this in a dictionary comprehension we end up needing to hit the url twice for successful urls. Once for the filter and once for the data going into the dictionary.

{
    url: requests.get(url).content
    for url in ["https://waylonwalker.com/", "https://waylonwalker.com/broken"]
    if requests.get(url).status_code == 200
}

Gimme some walrus #

using walrus in a dict comp

Using the walrus operator := list comp allows us to only put things into the dictionary that we want to keep, and not hit the url twice.

{
    url: r.content
    for url in ["https://waylonwalker.com/", "https://waylonwalker.com/broken"]
    if (r := requests.get(url)).status_code == 200
}

FIN #

The walrus is a nice to have option to save on extra function/network calls, and micro optimize your code without adding much extra.

Kedro rich is a very new and unstable (it’s good, just not ready) plugin for kedro to make the command line prettier.

Install kedro rich #

There is no pypi package yet, but it’s on github. You can pip install it with the git url.

pip install git+https://github.com/datajoely/kedro-rich

Kedro run #

You can run your pipeline just as you normally would, except you get progress bars and pretty prints.

kedro run
kedro rich pretty run

Kedro catalog #

Listing out catalog entries from the command line now print out a nice pretty table.

kedro catalog list
kedro rich catalog list table output

Give it a star #

Go to the GitHub repo and give it a star, Joel deserves it.

I recently found a really great plugin by mhinz to open files in neovim from a different tmux split, without touching neovim at all.

Installation #

neovim-remote is not a neovim plugin at all, it’s a python cli that you can install with pip. Unlike the repo suggests, I use pipx to install nvr.

pipx install neovim-remote

How I use it #

I have this added to my .envrc that is in every one of my projects. This will tie a neovim session to that directory, and all directories under it.

export NVIM_LISTEN_ADDRESS=/tmp/nvim-$(basename $PWD)

In my workflow I open a tmux session for each project, so this essentially ties a neovim session to a tmux session.

Open neovim #

First open neovim, but with the nvr command. This will open neovim, and look pretty much the same as always.

nvr

If you try to run nvr again in another shell nothing will happen as its already runnin under that address, but if you give it a filename it will open the file in the first instance of neovim that you opened.

nvr readme.md

As I am toying around with textual, I am wanting some popup user input to take over. Textual is still pretty new and likely to change quite significantly, so I don’t want to overdo the work I put into it, So for now on my personal tuis I am going to shell out to tmux.

The Problem #

The main issue is that when you are in a textual app, it kinda owns the input. So if you try to run another python function that calls for input it just cant get there. There is a textual-inputs library that covers this, and it might work really well for some use cases, but many of my use cases have been for things that are pre-built like copier, and I am trying to throw something together quick.

textual is still very beta

Part of this comes down to the fact that textual is still very beta and likely to change a lot, so all of the work I have done with it is for quick and dirty, or fun side projects.

The Solution #

So the solution that was easiest for me… shell out to a tmux popup. The application I am working on wants to create new documents using copier templates. copier has a fantastic cli that walks throught he template variables and asks the user to fill them in, so I just shell out to that with Popen. Make sure that you wait for this process to finish otherwise there will be bit of jank in your textual app.

async def action_new_post(self) -> None:
    proc = subprocess.Popen(
        'tmux popup "copier copy plugins/todo-template tasks"', shell=True
    )
    proc.wait()

example #

Here is what the running todo application looks like with the copier popup over it.

example of the popup running over textual

tmux popups

a bit more on tmux-popus [here] https://waylonwalker.com/tmux-popups/)

Mermaid diagrams provide a way to display graphs defined as plain text. Some markdown renderers support this as a plugin. GitHub now supports it.

example #

You can define nodes like this in mermaid, and GitHub will now render them as a pretty graph diagram. Its rendered in svg, so its searchable with control f and everything.

graph TD;
      A-->B;
      A-->C;
      B-->D;
      C-->D-->OUT;
      E-->F-->G-->OUT
Here is what the example looks like on GitHub

Glances is a system monitor with a ton of features, including docker processes.

I have started using portainer to look at running docker processes, its a great heavy-weight docker process monitor. glances works as a great lightweight monitor to just give you the essentials, ( Name, Status, CPU%, MEM, /MAX, IOR/s, IOW/s, Rx/s, Tx/s, Command)

install #

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances
pipx inject glances "glances[docker]"

You will be presented with this success message.

  injected package glances into venv glances
done! ✨ 🌟 ✨

results #

Now running glances will also show information about your running docker containers.

running glances with docker installed will show your docker processes

Glances has a pretty incredible webui to view system processes and information like htop, or task manager for windows.

The nice thing about the webui is that it can be accessed from a remote system. This would be super nice on something like a raspberry pi, or a vm running in the cloud. Its also less intimidating and easier to search if you are not a terminal junky.

install #

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances
pipx inject glances "glances[web]"

You will be presented with this success message.

  injected package glances into venv glances
done! ✨ 🌟 ✨

running the webui #

Now that you have glances installed you can run it with the -w flag to run the webui.

glances -w

This will present you with the following success message.

Glances Web User Interface started on http://0.0.0.0:61208/

Open it in your browser #

Now that its running you can open your web browser to localhost:61208 and be presented with the glances webui.

running the glances webui on my system

Glances is a fully featured system monitoring tool written in python. Out of the box it’s quite similar to htop, but has quite a few more features, and can be ran without installing anything other than pipx, which you should already have installed if you do anything with python.

pipx run glances

Once you run this you will be in a tui application similar to htop. You can kill processes with k, use left and right arrows to change the sorting column, and up and down to select different processes.

running pipx run glances on my ubuntu 21.10 machine inside the kitty terminal

python requirements text files can in fact depend on each other due to the fact that you can pass pip install arguments right into your requirements.txt file. The trick is to just prefix the file with a -r flag, just like you would if you were installing it with pip install

try it out #

Lets create two requirements files in a new directory to play with.

mkdir requirements-nest
cd requirements-nest
touch requirements.txt requirements_dev.txt

Then add the following to each requirements file.

# requirements.txt
kedro[pandas.ParquetDataSet]
# requirements_dev.txt
-r requirements.txt
ipython

Installing #

Installing requirements_dev.txt will install both ipython and pandas since it includes the base requirements file.

# this will install only pandas
pip install -r requirements.txt

# this will install both ipython and pandas
pip install -r requirements_dev.txt

This is covered in the pip user guide, but it is not obvious that this can be done in a requirements.txt file.