Drafts

Draft and unpublished posts

0 posts simple view

I recently found a really great plugin by mhinz to open files in neovim from a different tmux split, without touching neovim at all.

Installation #

neovim-remote is not a neovim plugin at all, it’s a python cli that you can install with pip. Unlike the repo suggests, I use pipx to install nvr.

pipx install neovim-remote

How I use it #

I have this added to my .envrc that is in every one of my projects. This will tie a neovim session to that directory, and all directories under it.

export NVIM_LISTEN_ADDRESS=/tmp/nvim-$(basename $PWD)

In my workflow I open a tmux session for each project, so this essentially ties a neovim session to a tmux session.

Open neovim #

First open neovim, but with the nvr command. This will open neovim, and look pretty much the same as always.

nvr

If you try to run nvr again in another shell nothing will happen as its already runnin under that address, but if you give it a filename it will open the file in the first instance of neovim that you opened.

nvr readme.md

Mermaid provides some really great ways to group or fence in parts of your graphs through the use of subgraphs.

Here we can model some sort of data ingest with some raw iot device and our warehouse in different groups.

graph TD;

    subgraph raw_iot
        a
    end

    subgraph warehouse
        A --> B
        B --> C
    end
graph TD;
subgraph raw_iot
    a
end

subgraph warehouse
    A --> B
    B --> C
end

connecting subgroups #

If we want to connect them, we can make a connection between a and A outside of the subgraphs.

graph TD;

    subgraph raw_iot
        a
    end

    a --> A

    subgraph warehouse
        A --> B
        B --> C
    end
graph TD;
subgraph raw_iot
    a
end

a --> A

subgraph warehouse
    A --> B
    B --> C
end

separation of concerns #

It’s also possible to specify subgraphs separate from where you define your nodes. which allows for some different levels of grouping that would not be possible if you were to define all your nodes inside of a subgraph.

graph TD;
    a --> A
    A --> B
    B --> C

    subgraph one
        A
        C
    end
graph TD; a --> A A --> B B --> C
subgraph warehouse
    A
    C
end

If you have ever ran which <command> and see duplicate entries it’s likely that you have duplicate entries in your $PATH. You can clean this up with a one liner at the end of your bashrc or zshrc.

eval "typeset -U path"

Since GitHub started supporting mermaid in their markdown I wanted to take another look at how to implement it on my site, I think it has some very nice opportunities in teaching, documenting, and explaining things.

The docs kinda just jumped right into their mermaid language and really went through that in a lot of depth, and skipped over how to implement it yourself, turns out its pretty simple. You just write mermaid syntax in a div with a class of mermaid on it!

<script src='https://unpkg.com/[email protected]/dist/mermaid.min.js'></script>
<div class='mermaid'>
graph TD;
a --> A
A --> B
B --> C
</div>

You just write mermaid syntax in a div with a class of mermaid on it!

The above gets me this diagram.

graph TD; a --> A A --> B B --> C

This feels so quick and easy to start getting some graphs up and running, but does lead to layout shift and extra bytes down the pipe. The best solution in my opionion would be to forgo the js and ship svg. That said, this is do dang convenient I will be using it for some things.

There is GNU coreutils command called mktemp that is super handy in shell scripts to make temporary landing spots for files so that they never clash with another instance, and will automatically get cleaned up when you restart, or whenever /tmp gets wiped. I’m not sure when that is, but I don’t expect it to be long.

Making temp directories #

Here are some examples of making temp directories in different places, my favorite is mktemp -dt mytemp-XXXXXX.

# makes a temporary directory in /tmp/ with the defaul template tmp.XXXXXXXXXX
mktemp

# makes a temporary directory in your current directory
mktemp --directory mytemp-XXXXXX
# shorter version
mktemp -d mytemp-XXXXXX

# same thing, but makes a file
mktemp mytemp-XXXXXX

# makes a temporary directory in your /tmp/ directory (or what ever you have configured as your TMPDIR)
mktemp --directory --tmpdir mytemp-XXXXXX
# shorter version
mktemp -dt mytemp-XXXXXX

# same thing, but makes a file
mktemp --tmpdir mytemp-XXXXXX
# shorter version
mktemp -t mytemp-XXXXXX

Use Case #

Here is a sample script that shows how to capture the tempdir as a variable and reuse it. Here is an example of curling my bootstrap file into a temp directory and running it from that directory.

local tmp=`mktemp -dt bootstrap-XXXXXX`
pushd $tmp
curl https://raw.githubusercontent.com/WaylonWalker/devtainer/main/bootstrap > bootstrap
bash bootstrap
popd

Templates #

You must have at least 3 trailing X’s that mktemp will replace with random characters. I played with it for a bit, it kinda allows for some trailing characters, and will not fill groups of X’s earlier in your template, only the last consecutive string.

My randomm samples I played with.

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com) took 2m24s
❯ mktemp myXtemp-XaXbXXXX -dt
/tmp/myXtemp-XaXbx9hn

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXXXs -dt
/tmp/myXtemp-XaXb2tpGs

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXcXXs -dt
mktemp: too few X's in template ‘myXtemp-XaXbXXcXXs’

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXcXXs -dt

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXXXt -dt
/tmp/myXtemp-XaXbe8PWt

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XXX-you-XXX -dt
/tmp/myXtemp-XXX-you-48l

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XXX-you-XX -dt
mktemp: too few X's in template ‘myXtemp-XXX-you-XX’

RTFM #

The man page has good stuff on all the flags that you might need.

man mktemp

Once you give a branch the big D (git branch -D mybranch) its gone, its lost from your history. It’s completely removed from your log. There will be no reference to these commits, or will there?

TLDR #

Checkout is your savior, all you need is the commit hash.

Immediate Regret #

your terminal is still open

We have all done this, you give branch the big D only to realize it was the wrong one. Don’t worry, not all is lost, this is the easiest to recover from. When you run the delete command you will see something like this.

❯ git branch -D new
Deleted branch new (was bc02a64).

Notice the hash is right there is the hash of your commit. You can use that to get your content back.

git checkout -b bc02a64
git branch new

# or in one swoop checkout your new branch at the `start-point` you want
git checkout -b new bc02a64

Delayed reaction #

you have closed your terminal

If you have closed your terminal, or have deleted with a gui or something that does not tell you the hash as you run it, don’t fret, all your work is still there (as long as you have commited). You just have to dig it out. The reflog contains a list of all git operations that have occurred on your git repo, and can be incredibly helpful with this.

Kinda Recent #

If your botched delete operation was recent just diving right into the reflog will show it.

❯ git reflog
03a3338 (main) HEAD@{0}: checkout: moving from new to main
bc02a64 (HEAD -> another, new) HEAD@{4}: commit: newfile
03a3338 (main) HEAD@{2}: checkout: moving from main to new

In this example, I checked out a branch called new, commited a new file, then switched back to main and deleted new.

Now That I have the commit hash I can use the same solution to get my branch back.

git checkout -b bc02a64
git branch new

# or in one swoop checkout your new branch at the `start-point` you want
git checkout -b new bc02a64

A lot has happened since then #

If a lot has happened since then, you are going to need to pull out some more tool to sift through that reflog, especially if its a big one. The first suggestion that I have is to pipe into grep and look for commit messages, or the name of the branch.

❯ git reflog | grep "moving from"
03a3338 HEAD@{1}: checkout: moving from main to branch/oops
03a3338 HEAD@{2}: checkout: moving from oops to main
03a3338 HEAD@{4}: checkout: moving from main to oops
03a3338 HEAD@{5}: checkout: moving from another to main
bc02a64 HEAD@{6}: checkout: moving from main to another
03a3338 HEAD@{7}: checkout: moving from another to main
bc02a64 HEAD@{8}: checkout: moving from new to another
bc02a64 HEAD@{9}: checkout: moving from bc02a64bbe5683d905e333e8dfcbbb91a5e77549 to new
bc02a64 HEAD@{10}: checkout: moving from main to bc02a64bbe56
03a3338 HEAD@{11}: checkout: moving from new to main
03a3338 HEAD@{13}: checkout: moving from main to new
03a3338 HEAD@{14}: checkout: moving from other to main
03a3338 HEAD@{18}: checkout: moving from main to other

git has a built in --grep flag, but I don’t think there is a way to filter by branch name, regardless it still is helpful.

❯ git reflog --grep new
bc02a64 (HEAD -> another, new) HEAD@{4}: commit: newfile

Maybe if you can remember a filename you can pass in -- <filename>.

git reflog -- readme.md

RTFM #

There are many other ways to slice up a git log, and reflog alike. check out man git log for some more flags.

It’s nearly impossible to completely loose a file if it is commited to git. It’s likely harder to fully remove the file than it is to recover it, but how do we go about recovering those precious files that we have lost.

Listing all the deleted files in all of git history can be done by combining git log with --diff-filter. The log gives you lots of options to show different bits of information about the commit that happened at that point. It’s even possible to get a completely clean list of files that are in your git history but have been deleted.

git log –diff-filter #

These various commands will show all files that were ever deleted on your current branch.

# This one includes the date, commit hash, and Author
git log --diff-filter D

# this one could be a git alias, but includes empty lines
git log --diff-filter D --pretty="format:" --name-only

# this one has the empty lines cleaned up
git log --diff-filter D --pretty="format:" --name-only | sed '/^$/d'

git diff-filter

git reflog –diff-filter #

The reflog can be super powerful in finding lost files here, as it only cares about git operations, not just the current branch. It will search accross all branches for deleted files and report them.

# This one includes the commit hash, branch, tag, and commit message
git reflog --diff-filter D

# You might want to at least add the filename
git reflog --diff-filter D --name-only

# this one could be a git alias, but includes empty lines
git reflog --diff-filter D --pretty="format:" --name-only

# this one has the empty lines cleaned up
git reflog --diff-filter D --pretty="format:" --name-only | sed '/^$/d'

get the last commit from a file #

git log -n 1 --pretty=format:%H -- file

If you want dont like how the output looks or you want your default pager to be different you can configure the default pager see Set Your Git Pager Config.

Git commands such as diff, log, whatchanged all take a flag called --diff-filter. This can filter for only certain types of diffs, such as added (A), modified (M), or deleted (D).

Man page #

You can find the full description by searching for --diff-filter in the man git diff page.

--diff-filter=[(A|C|D|M|R|T|U|X|B)...[*]]
    Select only files that are Added (A), Copied (C), Deleted (D), Modified (M), Renamed (R), have their type (i.e. regular file, symlink, submodule, ...)
    changed (T), are Unmerged (U), are Unknown (X), or have had their pairing Broken (B). Any combination of the filter characters (including none) can be used.
    When * (All-or-none) is added to the combination, all paths are selected if there is any file that matches other criteria in the comparison; if there is no
    file that matches other criteria, nothing is selected.

    Also, these upper-case letters can be downcased to exclude. E.g.  --diff-filter=ad excludes added and deleted paths.

    Note that not all diffs can feature all types. For instance, diffs from the index to the working tree can never have Added entries (because the set of paths
    included in the diff is limited by what is in the index). Similarly, copied and renamed entries cannot appear if detection for those types is disabled.

Try it out #

Open up a git repo and play around with this, here are some example that I played with that seemed useful to me.

# find when any files were deleted
git log --diff-filter D

# find when all files were added
git log --diff-filter A

# only one specific file
git log --diff-filter A -- readme.md

# partial match to a single file
git log --diff-filter A -- read*

# Find when all python files were added
git log --diff-filter A -- *.py

As I am toying around with textual, I am wanting some popup user input to take over. Textual is still pretty new and likely to change quite significantly, so I don’t want to overdo the work I put into it, So for now on my personal tuis I am going to shell out to tmux.

The Problem #

The main issue is that when you are in a textual app, it kinda owns the input. So if you try to run another python function that calls for input it just cant get there. There is a textual-inputs library that covers this, and it might work really well for some use cases, but many of my use cases have been for things that are pre-built like copier, and I am trying to throw something together quick.

textual is still very beta

Part of this comes down to the fact that textual is still very beta and likely to change a lot, so all of the work I have done with it is for quick and dirty, or fun side projects.

The Solution #

So the solution that was easiest for me… shell out to a tmux popup. The application I am working on wants to create new documents using copier templates. copier has a fantastic cli that walks throught he template variables and asks the user to fill them in, so I just shell out to that with Popen. Make sure that you wait for this process to finish otherwise there will be bit of jank in your textual app.

async def action_new_post(self) -> None:
    proc = subprocess.Popen(
        'tmux popup "copier copy plugins/todo-template tasks"', shell=True
    )
    proc.wait()

example #

Here is what the running todo application looks like with the copier popup over it.

example of the popup running over textual

tmux popups

a bit more on tmux-popus [here] https://waylonwalker.com/tmux-popups/)

Big announcement recently that obs studio now builds out to a flatpak, hopefully making it easier for all of us to install, especially us near normies that don’t regularly compile anything from source.

install flatpak #

I did not have flatpak installed so the first thing I had to do was get the flatpak command installed, and add their default repo.

sudo apt install flatpak
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

Once I had flatpak, I was able to get obs installed with the following command.

flatpak install flathub com.obsproject.Studio

Once Installed it fired right up for me with the next command they suggested.

flatpak run com.obsproject.Studio

It Works #

Pretty straightforward, following the instructions given it all worked for me, but it was missing a lot of the plugins that the current snap package I am using gives me (namely virtual webcam). So I am not ready to jump onto it until I figure out how to manage my own obs plugins. For now I think the snap is working just well enough.

Mermaid diagrams provide a way to display graphs defined as plain text. Some markdown renderers support this as a plugin. GitHub now supports it.

example #

You can define nodes like this in mermaid, and GitHub will now render them as a pretty graph diagram. Its rendered in svg, so its searchable with control f and everything.

graph TD;
      A-->B;
      A-->C;
      B-->D;
      C-->D-->OUT;
      E-->F-->G-->OUT
Here is what the example looks like on GitHub

Git has a built in way to rebase all the way back to the beginning of time. There is no need to scroll through the log to find the first hash, or find the total number of commits. Just use --root.

git rebase --root

Glances is a system monitor with a ton of features, including docker processes.

I have started using portainer to look at running docker processes, its a great heavy-weight docker process monitor. glances works as a great lightweight monitor to just give you the essentials, ( Name, Status, CPU%, MEM, /MAX, IOR/s, IOW/s, Rx/s, Tx/s, Command)

install #

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances
pipx inject glances "glances[docker]"

You will be presented with this success message.

  injected package glances into venv glances
done! ✨ 🌟 ✨

results #

Now running glances will also show information about your running docker containers.

running glances with docker installed will show your docker processes

Git reflog can perform some serious magic in reviving your hard work from the dead if you happen to loose it.

caveat #

You must git commit! If you never commit the file, git cannot help you. You might look into your trashcan, filesystem versions, onedrive, box, dropbox. If you have none of this, then you are probably hosed.

practice #

I really like to practice these techniques before I need to use them so that I understand how they work in a low stakes fashion. This helps me understand what I can and cannot do, and how to do it in a place that does not matter in any way at all.

This is what I did to revive a dropped docker-compose.yml file. The idea is that if I can find the commit hash, I can cherry-pick it.

git init
touch readme.md
git add readme.md
git commit -m "add readme"
touch docker-compose.yml
git add docker-compose.yml
git commit -m "add docker-compose"
git reset 3cfc --hard
git reflog
# copy the hash of the commit with my docker-compose commit
git cherry-pick fd74df3

reflog #

Here was the final reflog that shows all of my git actions. note I did reset twice.

❯ git reflog --name-only
0404b6a (HEAD -> main) HEAD@{0}: cherry-pick: add docker-compose
docker-compose.yml
3cfcab9 HEAD@{1}: reset: moving to 3cfc
readme.md
9175695 HEAD@{2}: cherry-pick: add docker-compose
docker-compose.yml
3cfcab9 HEAD@{3}: reset: moving to 3cfc
readme.md
fd74df3 HEAD@{4}: commit: add docker-compose
docker-compose.yml
3cfcab9 HEAD@{5}: reset: moving to HEAD
readme.md
3cfcab9 HEAD@{6}: commit (initial): add readme
readme.md

Glances has a pretty incredible webui to view system processes and information like htop, or task manager for windows.

The nice thing about the webui is that it can be accessed from a remote system. This would be super nice on something like a raspberry pi, or a vm running in the cloud. Its also less intimidating and easier to search if you are not a terminal junky.

install #

You will need to install glances to use the glances webui. We can still use pipx to manage our virtual environment for us so that we do not need to do so manually or run the risk of globally installed package dependency hell.

pipx install glances
pipx inject glances "glances[web]"

You will be presented with this success message.

  injected package glances into venv glances
done! ✨ 🌟 ✨

running the webui #

Now that you have glances installed you can run it with the -w flag to run the webui.

glances -w

This will present you with the following success message.

Glances Web User Interface started on http://0.0.0.0:61208/

Open it in your browser #

Now that its running you can open your web browser to localhost:61208 and be presented with the glances webui.

running the glances webui on my system

Right inside the git docs, is states that the git reflog command runs git reflog show by default which is an alias for git log -g --abbrev-commit --pretty=oneline

This epiphany deepens my understanding of git, and lets me understand that most git log flags might also work with git log -g.

full or short format #

Here are some git commands for you to try out on your own that are all pretty similar, but vary in how much information they show.

# These show only first line of the commit message subject, the hash, and index
git reflog
git log -g --abbrev-commit --pretty=oneline

# similar to git log, this is a fully featured log with author, date, and full
# commit message
git log -g

add files #

If I am looking for a missing file, I might want to leverage --name-only or --stat, to see where I might have hard reset that file, or deleted it.

git reflog --stat
git log -g --stat --abbrev-commit --pretty=oneline

git reflog --name-only
git log -g --name-only --abbrev-commit --pretty=oneline

example #

Here is an example where I lost my docker-compose.yml file in a git reset, and got it back by finding the commit hash with git reflog and cherry picked it back.

❯ git reflog --name-only
0404b6a (HEAD -> main) HEAD@{0}: cherry-pick: add docker-compose
docker-compose.yml
3cfcab9 HEAD@{1}: reset: moving to 3cfc
readme.md
9175695 HEAD@{2}: cherry-pick: add docker-compose
docker-compose.yml
3cfcab9 HEAD@{3}: reset: moving to 3cfc
readme.md
fd74df3 HEAD@{4}: commit: add docker-compose
docker-compose.yml
3cfcab9 HEAD@{5}: reset: moving to HEAD
readme.md
3cfcab9 HEAD@{6}: commit (initial): add readme
readme.md

This just proves that its harder to remove something from git, than it is to get it back. It can feel impossible to get something back, but once its in, it feels even more impossible to get it out.

Glances is a fully featured system monitoring tool written in python. Out of the box it’s quite similar to htop, but has quite a few more features, and can be ran without installing anything other than pipx, which you should already have installed if you do anything with python.

pipx run glances

Once you run this you will be in a tui application similar to htop. You can kill processes with k, use left and right arrows to change the sorting column, and up and down to select different processes.

running pipx run glances on my ubuntu 21.10 machine inside the kitty terminal

python requirements text files can in fact depend on each other due to the fact that you can pass pip install arguments right into your requirements.txt file. The trick is to just prefix the file with a -r flag, just like you would if you were installing it with pip install

try it out #

Lets create two requirements files in a new directory to play with.

mkdir requirements-nest
cd requirements-nest
touch requirements.txt requirements_dev.txt

Then add the following to each requirements file.

# requirements.txt
kedro[pandas.ParquetDataSet]
# requirements_dev.txt
-r requirements.txt
ipython

Installing #

Installing requirements_dev.txt will install both ipython and pandas since it includes the base requirements file.

# this will install only pandas
pip install -r requirements.txt

# this will install both ipython and pandas
pip install -r requirements_dev.txt

This is covered in the pip user guide, but it is not obvious that this can be done in a requirements.txt file.

In my adventure to put more homelab in docker, I moved our modded minecraft setup to docker.

Getting Mods #

So far I have found all of our mods from curse forge. modpacks make getting multiple mods working together much easier, someone else has already vetted a pack of often times 100+ mods that all play well together. I have yet to get these working in docker, I will, but for not I just have individual mods.

download file #

under the hood docker is using wget to get the mod. The link you click on from curseforge will block wget. What I do is pop open the devtools (f12 in chrome), click on the network tab, click the download link on the web page, and watch the real link show up.

minecraft mod in netwrok tab

Docker-compose #

I am using docker compose, it makes the command much easier to start, and all the things needed stored in a file. I am not using compose to run multiple things, just for the simple start command.

Create a directory for your server and add the following to a docker-compose.yml file.

version: "3.8"

services:
  mc:
    container_name: walkercraft
    image: itzg/minecraft-server
    ports:
      - 25565:25565
    environment:
      EULA: "TRUE"
      TYPE: "FORGE"
      VERSION: 1.16.5
      MODS_FILE: /extras/mods.txt
      REMOVE_OLD_MODS: "true"
    tty: true
    stdin_open: true
    restart: unless-stopped
    ports:
      - 25565:25565
    volumes:
      - ./minecraft-data:/data
      - ./mods.txt:/extras/mods.txt:ro

volumes:
  data:

mods.txt #

Once you have your mod file link from the network tab add them to a mods.txt file next to your docker-compose file.

https://media.forgecdn.net/files/3620/189/engineersdecor-1.16.5-1.1.16.jar

start your server #

Once you have made it this far starting the server is pretty simple.

docker compose up -d

kill your server #

If your still in the same directory, taking down the server should be pretty easy as well.

docker compose down

# if that does not work you can kill it
docker ps
# copy the id of your container
docker kill <id>

Reading eventbridge rules from the command line can be a total drag, pipe it into visidata to make it a breeze.

I just love when I start thinking through how to parse a bunch of json at the command line, maybe building out my own custom cli, then the solution is as simple as piping it into visidata. Which is a fantastic tui application that had a ton of vim-like keybindings and data features.

alias awsevents = aws events list-rules | visidata -f json