Posts tagged: bash Archive

Vaulted Secrets Without Git Churn

2026-02-26T09:59:57Z

Ansible Vault keeps secrets out of sight, but the ciphertext changes on every encrypt. That turns Git diffs into noise and makes it hard to tell if anything actually changed. Decrypting, editing, and re-encrypting often leaves uncertainty about whether any plaintext changed. This is amplified when secret repos are tightly coupled to dependent repositories. A typical cycle includes decrypting, adding a key, updating a value, applying changes, and returning later with little clarity about what changed while secrets were in plaintext.

Today a new workflow was created with @gpt-5.2-codex to keep diffs clean and avoid re-encrypting when the plaintext is identical.

Chat-reply

This repo has ansible vaulted secrets and an encrypt/decrypt process, but no way to compare. Please research compare options. The goal is to avoid changing files on encrypt/decrypt when plaintext is unchanged, ideally by comparing decrypted content and reusing the remote encrypted file.

@gpt-5.2-codex

The repo workflow will be reviewed, compare options identified, and a low-churn GitOps approach proposed to avoid re-encrypt noise.

The problem #

Re-encrypting a file with the same content produces different ciphertext. A simple decrypt/encrypt cycle can look like a full file change, even when the plaintext is identical. That creates uncertainty and makes GitOps workflows feel brittle.

Chat-reply

Minimal changes needed to keep GitOps clear?

@gpt-5.2-codex

Add compare-before-encrypt, decrypted diffs, and ignore cache artifacts.

The solution at a glance #

Three pieces make it work:

Compare-before-encrypt
- just decrypt stores a plaintext hash and caches the current encrypted file.
- just encrypt compares the plaintext hash and, if unchanged, restores the cached ciphertext instead of re-encrypting.
Readable diffs for vaulted files
- .gitattributes sets private/* diff=ansible-vault.
- A small textconv script decrypts for git diff.
Ignored cache artifacts
- .vault-hashes/ and .vault-cache/ are ignored so they never hit the repo.

How it works #

Chat-reply

How does compare-before-encrypt operate?

@gpt-5.2-codex

Hash plaintext on decrypt, reuse cached ciphertext on encrypt if the hash matches.

Compare-before-encrypt #

When a file is decrypted, a SHA-256 hash of the plaintext is created and stored in .vault-hashes/. A copy of the encrypted file is stored in .vault-cache/.

When encrypting, the current plaintext file is hashed. If the hash matches the cached hash, the cached encrypted file is restored instead of generating new ciphertext. Result: no churn when nothing actually changed.

Decrypted diffs #

Git is configured to use ansible-vault view as a text conversion step for files under private/. As a result, git diff shows decrypted content instead of ciphertext.

Enable the diff driver locally:

git config diff.ansible-vault.textconv "scripts/ansible-vault-diff"

What changes will be seen #

If a secret changes: the diff shows the plaintext change.
If decrypt and re-encrypt occurs without changes: the diff stays clean.
Large base64 blobs may still look like full-line diffs because they are single-line values.

Chat-reply

What is the day-to-day workflow?

@gpt-5.2-codex

`just decrypt`, edit, then `just encrypt`. Clean diffs, no churn.

Recommended usage #

just decrypt
# apply edits
just encrypt

That is sufficient to keep Git history clean and reduce uncertainty about whether a secret actually changed.

COLUMNS env var

2025-11-26T13:24:38Z

setting COLUMNS env var to a number greater than 0 will make the terminal resize to that number of columns.

COLUMNS=80 uvx --from rich-cli rich myscript.py

Note

Not all programs respct the COLUMNS env var, but rich does, and a lot of stuff I’m building uses rich.

I discovered this when I was trying to make a low effort readme generated from the code, but did not depend on the size of terminal it was ran on.

# justfile
readme:
    echo "# Workspaces" > README.md
    echo "" >> README.md
    echo '``` bash' >> README.md
    COLUMNS=80 ./workspaces.py --help >> README.md
    echo '```' >> README.md

tar over ssh

2025-09-13T10:50:42Z

Today I learned how to use tar over ssh to save hours in file transfers. I keep all of my projects in ~/git (very creative I know, I’ve done it for years and haven’t changed). I just swapped out my main desktop from bazzite to hyprland, and wanted to get all of my projects back. Before killing my bazzite install I moved everything over (16GB of many small files), it took over 14 hours, maybe longer. I had started in the morning and just let it churn.

This was not going to happen for re-seeding all of my projects on my new system, I knew there had to be a better way, I looked at rsync, but for seeding I ran into this tar over ssh technique and it only took me 6m51s to pull all of my projects off of my remote server.

ssh user@192.168.1.100 'tar -C /tank/git -cpf - .' \
  | tar -C "$HOME/git" -xpf -

💭 2025-07-10 Notes

2025-07-18T15:26:28Z

2025-07-09 Notes from yesterday I have temporal stuff kind of going with postiz in a windsurf session working on [[thoughts-to-nostr]] Been cleaning up my z" loading="lazy">

2025-07-10 Notes | Nic Payne

2025-07-09 Notes from yesterday I have temporal stuff kind of going with postiz in a windsurf session working on [[thoughts-to-nostr]] Been cleaning up my z

pype.dev

big fan of eza and dust, I like these aliases to have some common commands at my fingertips. I often use the tree command and yes it sometimes goes too deep to actually be useful.

    alias lt='eza -T --level=2'             # Tree view, 2 levels deep
    alias ltt='eza -T --level=3'            # Tree view, 3 levels deep

    alias du1='dust -d 1'                   # Show only 1 level deep
    alias du2='dust -d 2'                   # Show 2 levels deep

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

fc

2025-07-18T07:52:53Z

I am a linux user through and through. Desktop, server, vms, containers, everything except my phone is linux. With this I spend a lot of time in the terminal, and have been a long time user of !! to rerun the last command, but with the ability to tack something on at the beginning or end.

TIL about fc, which opens the last command in your shell history in your $EDITOR or pass in your editor -e nvim.

man fc

Rcap of how !! works #

!! pronounces bang bang and will run the last command in your history.

ls -l

!! | wc -l
# ls -l | wc -l

sudo !!
# sudo ls -l | wc -l

!!:s/-l/-l \/tmp
# sudo ls -l /tmp | wc -l

`fc` enters the chat #

Now making complex edits in your shell can be a bit of a chore, so fc moves this work to your $EDITOR.

fc

This pops open your $EDITOR with the last command in your history.

sudo ls -l | wc -l

Shell History #

fc shows up in shell history, but !! does not, !! gets replaced by the command that it becomes.

Up Arrow #

yaya yaya, I know you can also up-arrow c-e, but what fun is that, it’s barely a flex. fc just looks big brained and like you really know what you are doing.

unset multiple environment variables

2025-06-03T11:21:11Z

You can unset multiple environment variables at once. I did not know this was a thing, its something that ended up happening organically on a call and asking someone to run unset. They had never done it before and did not know how it works, but did exactly as I said instead of what I meant. I like this handy shortcut doing it in one line rather than each one individually, I will be using this in the future. You might need this for something like running aws cli commands with localstack.

unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION

💭 bic | Static blog generator, in bash

2025-01-09T14:22:47Z

bic

Static blog generator, in bash

bic · bic.sh

Intereresting someone built a blog generator in bash. it comes with normal markdown to html, static content, robots.txt, sitemap, rss, and tags. It uses pandoc to take markdown to html and mustache for page templates.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 casey/just: 🤖 Just a command runner

2024-12-14T17:04:06Z

GitHub - casey/just: 🤖 Just a command runner

🤖 Just a command runner. Contribute to casey/just development by creating an account on GitHub.

GitHub · github.com

new versions of just now come with color variables already set.

[group('manage')]
version:
    #!/usr/bin/env bash
    version=$(cat version)
    echo current version {{BOLD}}{{GREEN}}$version{{NORMAL}}

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 pipedream/justfile at main · thechangelog/pipedream

2024-12-14T16:54:34Z

pipely/justfile at main · thechangelog/pipely

I like the idea of having like this 20-line Varnish config that we deploy around the world, and it’s like: Look at our CDN! - thechangelog/pipely

GitHub · github.com

I found this nugget in thechangelogs justfile, it lets you add color to your justfile with variables quite easily.

# https://linux.101hacks.com/ps1-examples/prompt-color-using-tput/

_BOLD := "$(tput bold)"
_RESET := "$(tput sgr0)"
_BLACK := "$(tput bold)$(tput setaf 0)"
_RED := "$(tput bold)$(tput setaf 1)"
_GREEN := "$(tput bold)$(tput setaf 2)"
_YELLOW := "$(tput bold)$(tput setaf 3)"
_BLUE := "$(tput bold)$(tput setaf 4)"
_MAGENTA := "$(tput bold)$(tput setaf 5)"
_CYAN := "$(tput bold)$(tput setaf 6)"
_WHITE := "$(tput bold)$(tput setaf 7)"
_BLACKB := "$(tput bold)$(tput setab 0)"
_REDB := "$(tput setab 1)$(tput setaf 0)"
_GREENB := "$(tput setab 2)$(tput setaf 0)"
_YELLOWB := "$(tput setab 3)$(tput setaf 0)"
_BLUEB := "$(tput setab 4)$(tput setaf 0)"
_MAGENTAB := "$(tput setab 5)$(tput setaf 0)"
_CYANB := "$(tput setab 6)$(tput setaf 0)"
_WHITEB := "$(tput setab 7)$(tput setaf 0)"

Usage

echo:
    echo {{_BOLD}}{{_GREEN}}hello there{{_RESET}}

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

urllink

2024-12-14T11:11:52Z

Today I discovered the Urllink function in bash from the ujust tool from ublue.it. Seems like a cool trick, but might not work everywhere.

########
### Special text formating
########
## Function to generate a clickable link, you can call this using
# url=$(Urllink "https://ublue.it" "Visit the ublue website")
# echo "${url}"
function Urllink (){
    URL=$1
    TEXT=$2

    # Generate a clickable hyperlink
    printf "\e]8;;%s\e\\%s\e]8;;\e\\" "$URL" "$TEXT${n}"
}
```j

💭 hostnamectl to easily change hostname | Pype.dev

2024-12-06T14:30:41Z

hostnamectl to easily change hostname | Nic Payne

hostnamectl is apparently a linux utility for easily changing your hostname in a variety of ways I learned there's transient and static hostnames, so that&

pype.dev

For some reason the ublue ecosystem does not prompt you to set your hostname on install and you get a hostname like bazzite showing up. Looks like this is the fix.

hostnamectl –static hostname babyblue-aurora

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

convert mp4 for twitter with ffmpeg

2024-07-25T12:51:29Z

I’ve had a couple of uploads to twitter fail recently and has been a pain. I tried some online converters for convenience, but none of them worked. I reached out to chatgpt and found succeess with this ffmpeg command.

ffmpeg -i input.mp4 \
  -vf "scale=trunc(oh*a/2)*2:min(720\,trunc(ih*a/2)*2)" \
  -c:v libx264 -profile:v high -level:v 4.1 \
  -b:v 3500k -maxrate 3500k -bufsize 7000k \
  -pix_fmt yuv420p \
  -c:a aac -b:a 128k -ar 44100 \
  -movflags +faststart \
  output.mp4

cli auth

2023-12-17T20:09:03Z

Authentication from cli tools can be a bit of a bear, and I have to look it up every time. This is my reference guide for future me to remember how to easily do it.

I set up a fastapi server running on port 8000, it uses a basic auth with waylonwalker as the username and asdf as the password. The server follows along with what comes out of the docs. I have it setup to take basic auth, form username and password, or a bearer token for authentication.

curl #

The og of command line url tools.

# basic auth
curl -u 'waylonwalker:asdf' -X POST localhost:8000/token
# basic auth with password prompt
curl -u 'waylonwalker' -X POST localhost:8000/token
# token
curl -H 'Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg' -X POST localhost:8000/token

wget #

My go to if I want the result to go into a file.

# basic auth
wget -q -O - --auth-no-challenge --http-user=waylonwalker --http-password=asdf --post-data '' localhost:8000/token

# token
wget -q -O - --header="Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg" -O - --post-data '' localhost:8000/token

httpx #

An http client written in python, primarilty used with the python api, but has a nice cli.

# install
python3 -m pip install httpx

# basic auth
httpx -m POST --auth waylonwalker asdf http://localhost:8000/token

# basic auth with password prompt
httpx -m POST --auth waylonwalker - http://localhost:8000/token

# token
httpx -m POST --headers="Authorization" "bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg" http://localhost:8000/token

httpie #

A modern http client written in python.

# install
python3 -m pip install httpie

# basic auth
http POST localhost:8000/token -a waylonwalker:asdf

# basic auth with password prompt
http POST localhost:8000/token -a waylonwalker

# token
http POST localhost:8000/token -A bearer -a eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg

httpie with plugin #

# install
python3 -m pip install httpie-credential-store
# usage
http POST localhost:8000/token -A creds

httpie prompt #

http-prompt comes from the httpie org, and has an interactive cli interface into apis. You can even specify a spec file to autocomplete on api methods.

http-prompt localhost:8000 --auth waylonwalker:asdf --spec openapi.json

jpillora-installer

2023-10-04T19:28:33Z

jpillora/installer is the install script generator I have been looking for. It downloads binaries for your machine from GitHub releases and unzips them for you. It grabs the latest release, so you can easily update them. I have tried scripting these installs in the past and struggled to consistently get the latest version for every package and unpack it correctly.

Also these pre-compiled binaries install rediculously fast compared to building them from source.

Check out some example links.

opening in a browser will show metadata

https://i.jpillora.com/serve

If you pass in script=true it will instead return the install script as it would by default through curl.

https://i.jpillora.com/serve?script=true

Use it to install neovim #

All you need to do to generate an install script is to pass in the GitHub repo slug with the org.

curl https://i.jpillora.com/neovim/neovim | bash

The shell script that it generates for neovim looks like this.

#!/bin/bash
if [ "$DEBUG" == "1" ]; then
    set -x
fi
TMP_DIR=$(mktemp -d -t jpillora-installer-XXXXXXXXXX)
function cleanup {
    rm -rf $TMP_DIR > /dev/null
}
function fail {
    cleanup
    msg=$1
    echo "============"
    echo "Error: $msg" 1>&2
    exit 1
}
function install {
    #settings
    USER="neovim"
    PROG="neovim"
    ASPROG=""
    MOVE="false"
    RELEASE="stable"
    INSECURE="false"
    OUT_DIR="$(pwd)"
    GH="https://github.com"
    #bash check
    [ ! "$BASH_VERSION" ] && fail "Please use bash instead"
    [ ! -d $OUT_DIR ] && fail "output directory missing: $OUT_DIR"
    #dependency check, assume we are a standard POISX machine
    which find > /dev/null || fail "find not installed"
    which xargs > /dev/null || fail "xargs not installed"
    which sort > /dev/null || fail "sort not installed"
    which tail > /dev/null || fail "tail not installed"
    which cut > /dev/null || fail "cut not installed"
    which du > /dev/null || fail "du not installed"
    #choose an HTTP client
    GET=""
    if which curl > /dev/null; then
        GET="curl"
        if [[ $INSECURE = "true" ]]; then GET="$GET --insecure"; fi
        GET="$GET --fail -# -L"
    elif which wget > /dev/null; then
        GET="wget"
        if [[ $INSECURE = "true" ]]; then GET="$GET --no-check-certificate"; fi
        GET="$GET -qO-"
    else
        fail "neither wget/curl are installed"
    fi
    #debug HTTP
    if [ "$DEBUG" == "1" ]; then
        GET="$GET -v"
    fi
    #optional auth to install from private repos
    #NOTE: this also needs to be set on your instance of installer
    AUTH="${GITHUB_TOKEN}"
    if [ ! -z "$AUTH" ]; then
        GET="$GET -H 'Authorization: $AUTH'"
    fi
    #find OS #TODO BSDs and other posixs
    case `uname -s` in
    Darwin) OS="darwin";;
    Linux) OS="linux";;
    *) fail "unknown os: $(uname -s)";;
    esac
    #find ARCH
    if uname -m | grep -E '(arm|arch)64' > /dev/null; then
        ARCH="arm64"

        # no m1 assets. if on mac arm64, rosetta allows fallback to amd64
        if [[ $OS = "darwin" ]]; then
            ARCH="amd64"
        fi

    elif uname -m | grep 64 > /dev/null; then
        ARCH="amd64"
    elif uname -m | grep arm > /dev/null; then
        ARCH="arm" #TODO armv6/v7
    elif uname -m | grep 386 > /dev/null; then
        ARCH="386"
    else
        fail "unknown arch: $(uname -m)"
    fi
    #choose from asset list
    URL=""
    FTYPE=""
    case "${OS}_${ARCH}" in
    "linux_amd64")
        URL="https://github.com/neovim/neovim/releases/download/stable/nvim-linux64.tar.gz"
        FTYPE=".tar.gz"
        ;;
    "darwin_amd64")
        URL="https://github.com/neovim/neovim/releases/download/stable/nvim-macos.tar.gz"
        FTYPE=".tar.gz"
        ;;
    *) fail "No asset for platform ${OS}-${ARCH}";;
    esac
    #got URL! download it...
    echo -n "Downloading"
    echo -n " $USER/$PROG"
    if [ ! -z "$RELEASE" ]; then
        echo -n " $RELEASE"
    fi
    if [ ! -z "$ASPROG" ]; then
        echo -n " as $ASPROG"
    fi
    echo -n " (${OS}/${ARCH})"

    echo "....."

    #enter tempdir
    mkdir -p $TMP_DIR
    cd $TMP_DIR
    if [[ $FTYPE = ".gz" ]]; then
        which gzip > /dev/null || fail "gzip is not installed"
        bash -c "$GET $URL" | gzip -d - > $PROG || fail "download failed"
    elif [[ $FTYPE = ".tar.bz" ]] || [[ $FTYPE = ".tar.bz2" ]]; then
        which tar > /dev/null || fail "tar is not installed"
        which bzip2 > /dev/null || fail "bzip2 is not installed"
        bash -c "$GET $URL" | tar jxf - || fail "download failed"
    elif [[ $FTYPE = ".tar.gz" ]] || [[ $FTYPE = ".tgz" ]]; then
        which tar > /dev/null || fail "tar is not installed"
        which gzip > /dev/null || fail "gzip is not installed"
        bash -c "$GET $URL" | tar zxf - || fail "download failed"
    elif [[ $FTYPE = ".zip" ]]; then
        which unzip > /dev/null || fail "unzip is not installed"
        bash -c "$GET $URL" > tmp.zip || fail "download failed"
        unzip -o -qq tmp.zip || fail "unzip failed"
        rm tmp.zip || fail "cleanup failed"
    elif [[ $FTYPE = ".bin" ]]; then
        bash -c "$GET $URL" > "neovim_${OS}_${ARCH}" || fail "download failed"
    else
        fail "unknown file type: $FTYPE"
    fi
    #search subtree largest file (bin)
    TMP_BIN=$(find . -type f | xargs du | sort -n | tail -n 1 | cut -f 2)
    if [ ! -f "$TMP_BIN" ]; then
        fail "could not find find binary (largest file)"
    fi
    #ensure its larger than 1MB
    #TODO linux=elf/darwin=macho file detection?
    if [[ $(du -m $TMP_BIN | cut -f1) -lt 1 ]]; then
        fail "no binary found ($TMP_BIN is not larger than 1MB)"
    fi
    #move into PATH or cwd
    chmod +x $TMP_BIN || fail "chmod +x failed"
    DEST="$OUT_DIR/$PROG"
    if [ ! -z "$ASPROG" ]; then
        DEST="$OUT_DIR/$ASPROG"
    fi
    #move without sudo
    OUT=$(mv $TMP_BIN $DEST 2>&1)
    STATUS=$?
    # failed and string contains "Permission denied"
    if [ $STATUS -ne 0 ]; then
        if [[ $OUT =~ "Permission denied" ]]; then
            echo "mv with sudo..."
            sudo mv $TMP_BIN $DEST || fail "sudo mv failed"
        else
            fail "mv failed ($OUT)"
        fi
    fi
    echo "Downloaded to $DEST"
    #done
    cleanup
}
install

Self Host Your Own #

I’d reccomend self hosting your own. This way you know that it’s consistent and unlikely to change in a way that breaks your use.

curl -s https://i.jpillora.com/installer | bash

Repos I am using installer for #

Here are the repos I am using installer for.

atuinsh/atuin
benbjohnson/litestream
bootandy/dust
BurntSushi/ripgrep
chmln/sd
cjbassi/ytop
dalance/procs
dbrgn/tealdeer
ducaale/xh
go-task/task
imsnif/bandwhich
imsnif/diskonaut
kovidgoyal/kitty
mgdm/htmlq
neovim/neovim
ogham/dog
ogham/exa
pemistahl/grex
sharkdp/bat
sharkdp/fd
sharkdp/pastel
sirwart/ripsecrets
starship/starship
topgrade-rs/topgrade
zellij-org/zellij

💭 Deleting Specific Lines in a File with sed or yq

2023-07-28T14:59:37Z

Deleting Specific Lines in a File with sed or yq — Nick Janetakis

Nick Janetakis · nickjanetakis.com

sed can be a tricky beast, I often stumble when trying to pipe into it. Next time I need to use sed, I should reference this article by Nick Janetakis. He makes it looks much easier than my experience has been, and it appears to behave like a vim :%s/ substitution does, or a g/ g command.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

dash e your github docker actions

2022-12-07T13:09:37Z

Give github actions the -e flag in the shebang #! so they fail on any one command failure. Otherwise each line will set the exit status, but only the last one will be passed to ci.

#!/bin/bash -e

What is -e #

The -e flag to the bash command allows your script to exit immediately if any command within the script returns a non-zero exit status. This can be useful for ensuring that your script exits with an error if any of the commands it runs fail, which can help you identify and debug issues in your script. For example, if you have a script that runs several commands and one of those commands fails, the script will continue running without the -e flag, but will exit immediately if the -e flag is present. This can make it easier to troubleshoot your script and ensure that it runs correctly.

Solution for Windows #

In windows the solution is not quite as simple. You can define a function in a Windows batch script that wraps an if statement to check the exit status of a command and handle any errors that may have occurred. Here is an example of how you might define a function called “check_error” that does this:

:check_error
if errorlevel 1 (
  echo An error occurred!
  exit /b 1
)

To use this function in your script, you would simply call it after running a command, like this:

some_command
call :check_error

This would run the “some_command” and then call the “check_error” function to check the exit status and handle any errors that may have occurred. This approach allows you to reuse the error-checking logic in your script, which can make it easier to write and maintain.

Bash function to edit scripts faster

2022-05-10T14:57:35Z

I am often editing my own scripts as I develop them. I want to make a better workflow for working with scripts like this.

Currently #

Currently I am combining nvim with a which subshell to etit these files like this.

for now lets use my todo command as an example

nvim `which todo`

First pass #

On first pass I made a bash function to do exactly what I have been doing.

ewhich () {$EDITOR `which "$1"`}

The $1 will pass the first input to the which subshell. Now we can edit our todo script like this.

ewich todo

Note, I use bash functions instead of aliases for things that require input.

Final State #

This works fine for commands that are files, but not aliases or shell functions. Next I jumped to looking at the output of command -V $1.

if the command is not found, search for a file
if its a builtin, exit
if its an alias, open my ~/.alias file to that line
if its a function, open my ~/.alias file to that line

ewhich () {
case `command -V $1` in
    "$1 not found")
        FILE=`fzf --prompt "$1 not found searching ..." --query $1`
        [ -z "$FILE" ] && echo "closing" || $EDITOR $FILE;;
    *"is a shell builtin"*)
        echo "$1 is a builtin";;
    *"is an alias"*)
        $EDITOR ~/.alias +/alias\ $1;;
    *"is a shell function"*)
        $EDITOR ~/.alias +/^$1;;
    *)
        $EDITOR `which "$1"`;;
esac

a bit more ergo, and less readable #

To make it easier to type, at the sacrifice of readability for anyone watching I added a single character e alias to ewhich. So when I want to edit anything I just use e.

alias e=ewhich

Results #

Here is a quick screencast of how it works.

Sorry, your browser doesn't support embedded videos.

Unzip minecraft mods to their directory from the command line

2022-04-02T16:20:34Z

This morning I was trying to install a modpack on my minecraft server after getting a zip file, and its quite painful when I unzip everything in the current directory rather than the directory it belongs in.

I had the files on a Windows Machine #

So I’ve been struggling to get mods installed on linux lately and the easiest way to download the entire pack rather than each mod one by one seems to be to use the overwolf application on windows. Once I have the modpack I can start myself a small mod-server by zipping it, putting it in a mod-server directory and running a python http.server

python -m http.server

Downoading on the server #

Then I go back to my server and download the modpack with wget.

wget 10.0.0.171:8000/One%2BBlock%2BServer%2BPack-1.4.zip

Unzip to the minecraft-data directory #

Now I can unzip my mods into the minecraft-data directory.

unzip One+Block+Server+Pack-1.4.zip -d minecraft-data

Running the server with docker #

I run the minecraft server with docker, which is setup to mount the minecraft-data directory.

Running a Minecraft Server in Docker

A bit more on that in the other post, but when I download the whole modpack like this I make these changes to my docker compose. (commented out lines)

version: "3.8"

services:
  mc:
    container_name: walkercraft
    image: itzg/minecraft-server:java8
    environment:
      EULA: "TRUE"
      TYPE: "FORGE"
      VERSION: 1.15.2
      # MODS_FILE: /extras/mods.txt
      # REMOVE_OLD_MODS: "true"
    tty: true
    stdin_open: true
    restart: unless-stopped
    ports:
      - 25565:25565
    volumes:
      - ./minecraft-data:/data
      # - ./mods.txt:/extras/mods.txt:ro

volumes:
  data:

Bash mktemp

2022-03-02T14:52:08Z

There is GNU coreutils command called mktemp that is super handy in shell scripts to make temporary landing spots for files so that they never clash with another instance, and will automatically get cleaned up when you restart, or whenever /tmp gets wiped. I’m not sure when that is, but I don’t expect it to be long.

Making temp directories #

Here are some examples of making temp directories in different places, my favorite is mktemp -dt mytemp-XXXXXX.

# makes a temporary directory in /tmp/ with the defaul template tmp.XXXXXXXXXX
mktemp

# makes a temporary directory in your current directory
mktemp --directory mytemp-XXXXXX
# shorter version
mktemp -d mytemp-XXXXXX

# same thing, but makes a file
mktemp mytemp-XXXXXX

# makes a temporary directory in your /tmp/ directory (or what ever you have configured as your TMPDIR)
mktemp --directory --tmpdir mytemp-XXXXXX
# shorter version
mktemp -dt mytemp-XXXXXX

# same thing, but makes a file
mktemp --tmpdir mytemp-XXXXXX
# shorter version
mktemp -t mytemp-XXXXXX

Use Case #

Here is a sample script that shows how to capture the tempdir as a variable and reuse it. Here is an example of curling my bootstrap file into a temp directory and running it from that directory.

local tmp=`mktemp -dt bootstrap-XXXXXX`
pushd $tmp
curl https://raw.githubusercontent.com/WaylonWalker/devtainer/main/bootstrap > bootstrap
bash bootstrap
popd

Templates #

You must have at least 3 trailing X’s that mktemp will replace with random characters. I played with it for a bit, it kinda allows for some trailing characters, and will not fill groups of X’s earlier in your template, only the last consecutive string.

My randomm samples I played with.

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com) took 2m24s
❯ mktemp myXtemp-XaXbXXXX -dt
/tmp/myXtemp-XaXbx9hn

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXXXs -dt
/tmp/myXtemp-XaXb2tpGs

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXcXXs -dt
mktemp: too few X's in template ‘myXtemp-XaXbXXcXXs’

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXcXXs -dt

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXXXt -dt
/tmp/myXtemp-XaXbe8PWt

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XXX-you-XXX -dt
/tmp/myXtemp-XXX-you-48l

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XXX-you-XX -dt
mktemp: too few X's in template ‘myXtemp-XXX-you-XX’

RTFM #

The man page has good stuff on all the flags that you might need.

man mktemp

View AWS event bridge rules with visidata

2022-02-15T03:04:26Z

Reading eventbridge rules from the command line can be a total drag, pipe it into visidata to make it a breeze.

I just love when I start thinking through how to parse a bunch of json at the command line, maybe building out my own custom cli, then the solution is as simple as piping it into visidata. Which is a fantastic tui application that had a ton of vim-like keybindings and data features.

alias awsevents = aws events list-rules | visidata -f json

ag silver searcher look ahead and look behind

2022-02-08T13:54:08Z

A super useful tool when doing PR’s or checking your own work during a big refactor is the silver searcher. Its a super fast command line based searching tool. You just run ag "<search term>" to search for your search term. This will list out every line of every file in any directory under your current working directory that contains a match.

Ahead/Behind #

It’s often useful to need some extra context around the change. I recently reviewed a bunch of PR’s that moved schema from save_args to the root of the dataset in all yaml configs. To ensure they all made it to the top level DataSet configuraion, and not underneath save_args. I can do a search for all the schemas, and ensure that none of them are under save_args anymore.

ag "schema: " -A 12 -B 12

Neovim Config for Git

2022-02-04T14:23:57Z

Creating a minimal config specifically for git commits has made running git commit much more pleasant. It starts up Much faster, and has all of the parts of my config that I use while making a git commit. The one thing that I often use is autocomplete, for things coming from elsewhere in the tmux session. For this cmpe-tmux specifically is super helpful.

The other thing that is engrained into my muscle memory is jj for escape. For that I went agead and added my settings and keymap with no noticable performance hit.

Here is the config that has taken

~/.config/nvim/init-git.vim

source ~/.config/nvim/settings.vim
source ~/.config/nvim/keymap.vim
source ~/.config/nvim/git-plugins.vim
lua require'waylonwalker.cmp'

~/.config/nvim/git-plugins.vim

call plug#begin('~/.local/share/nvim/plugged')

" cmp
Plug 'hrsh7th/nvim-cmp'
Plug 'hrsh7th/cmp-nvim-lsp'
Plug 'hrsh7th/cmp-buffer'
Plug 'hrsh7th/cmp-path'
Plug 'hrsh7th/cmp-calc'
Plug 'andersevenrud/compe-tmux', { 'branch': 'cmp' }


call plug#end()

~/.gitconfig

[core]
    editor = nvim -u ~/.config/nvim/init-git.vim

fuzzy wallpaper with fzf

2022-01-19T01:17:26Z

I really appreciate that in linux anything can be scripted, including setting the wallpaper. So everytime I disconnect a monitor I can just rerun my script and fix my wallpaper without digging deep into the ui and fussing through a bunch of settings.

feh --bg-scale ~/.config/awesome/wallpaper/my_wallpaper.png

I set my default wallpaper with feh using the command above.

Leaning in on feh, we can use fzf to pick a wallpaper from a directory full of wallpapers with very few keystrokes.

alias wallpaper='ls ~/.config/awesome/wallpaper | fzf --preview="feh --bg-scale ~/.config/awesome/wallpaper/{}" | xargs -I {} feh --bg-scale ~/.config/awesome/wallpaper/{}'

I have mine alias’d to wallpaper so that I can quickly run it from my terminal.

2 minutes to stow

2022-01-09T00:33:58Z

Stow is an incredible way to manage your dotfiles. It works by managing symlinks between your dotfiles directory and the rest of the system. You can then make your dotfiles directory a git repo and have it version controlled. In my honest opinion, when I was trying to get started the docs straight into deep detail of things I frankly don’t really care about and jumped right over how to use it.

When using stow its easiest to keep your dotfiles directory (you may name it what you want) in your home directory, with application directories inside of it.

Then each application directory should reflet the same diretory structure as you want in your home directory.

zsh #

Here is a simple example with my zshrc.

mkdir ~/dotfiles
cd ~/dotfiles
mkdir zsh
mv ~/.zshrc zsh
stow --simulate zsh

You can pass in the –simulate if you wish, it will tell you if there are going to be any more errors or not, but it wont give much more than that.

WARNING: in simulation mode so not modifying filesystem.

Once your ready you can stow your zsh application.

stow zsh

nvim #

A slightly more complicated example is neovim since its diretory structure does not put configuration files directly in your home directory, but rather at a deeper level.

mkdir ~/dotfiles/nvim/.config/nvim/ -p
cd ~/dotfiles
mv ~/.config/nvim/ ~/dotfiles/nvim/.config/nvim/
stow zsh

!notice how the nvim directory inside of dotfiles is structured like it would be in your $HOME directory.

Tmux Pop size

2022-01-05T17:37:26Z

tmux popups can be sized how you like based on the % width of the terminal on creation by using the flags (h, w, x, y) for height, width, and position.

# normal popup
tmux popup figlet "Hello"
# fullscreen popup
tmux popup -h 100% -w 100% figlet "Hello"
# 75% centered popup
tmux popup -h 100% -w 75% figlet "Hello"
# 75% popup on left side
tmux popup -h 100% -w 75% -x 0% figlet "Hello"

Sorry, your browser doesn't support embedded videos.

example running these commands

Changing copier template strings (_endops)

2022-01-04T10:55:12Z

I was completely stuck for awhile. copier was not replacing my template variables. I found out that adding all these _endops fixed it. Now It will support all of these types of variable wrappers

# copier.yml
_templates_suffix: .jinja
_envops:
  block_end_string: "%}"
  block_start_string: "{%"
  comment_end_string: "#}"
  comment_start_string: "{#"
  keep_trailing_newline: true
  variable_end_string: "}}"
  variable_start_string: "{{"

!RTFM: Later I read the docs and realized that copier defaults to using [[ and ]] for its templates unlike other tools like cookiecutter.

copier template variables

2022-01-03T12:37:32Z

I’ve been looking for a templating tool for awhile that works well with single files. My go to templating tool cookiecutter does not work for single files, it needs to put files into a directory underneath of it.

template variables #

By default copier uses double square brackets for its variables. variables in files, directory_names, or file_names will be substituted for their value once you render them.

# hello-py/hello.py.tmpl
print('hello-[[name]]')

note! by default copier will not inject variables into your template-strings unless you use a .tmpl suffix.

Before running copier we need to tell copier what variables to ask for, we do this with a copier.yml file.

# copier.yml
name:
  default: my_name
  type: str
  help: What is your name

installing copier #

I prefer to install cli tools that I need globally with pipx, this always gives me access to the tool without worrying about dependency conflicts, bloating my system site-packages, or managing a separate virtual environment for it myself.

pipx install copier

running copier #

When running copier copy we pass in the directory of the template, and the directory that we want to render the template into.

copier copy hello-py .

note! the directory ‘.’ is often referred to in cli programs to represent the current working directory that we are calling the command from.

results #

The resulting files will have your variables injected into them if you have setup your template and copier.yml up correctly.

print('hello-you')

List all the files containing a phrase

2021-12-31T20:24:48Z

One of the most useful skills you can acquire to make you faster at almost any job that uses a computer is getting good at finding text in your current working diretory and identifying the files that its in. I often use the silver searcher ag or ripgrep rg to find files in large directories quickly. Both have a sane set of defaults that ignore hidden and gitignored files, but getting them to list only the filenames and not the matched was not trivial to me.

I’ve searched throught he help/man pages many times looking for these flags and they always seem to evade me.

ag #

Passing the flag -l to ag will get it to list only the filepath, and not the match. Here I gave it a --md as well to only return markdown filetypes. ag supports a number of filetypes in a very similar way.

ag nvim --md -l

rg #

Giving rg the --files-with-matches flag will yield you a similar set of results, giving only the filepaths themselves and not the match statement. Also passing in the -g "*.md" will similarly yield only results from markdown files.

rg --files-with-matches you -g "*.md"

My first impressions with pyenv

2021-12-30T15:26:01Z

pyenv provides an easy way to install almost any version of python from a large list of distributions. I have simply been using the version of python from the os package manager for awhile, but recently I bumped my home system to Ubuntu 21.10 impish, and it is only 3.9+ while the libraries I needed were only compatable with up to 3.8.

I needed to install an older version of python on ubuntu

I’ve been wanting to check out pyenv for awhile now, but without a burning need to do so.

installing #

Based on the Readme it looked like I needed to install using homebrew,so this is what I did, but I later realized that there is a pyenv-installer repo that may have saved me this need.

Installing Homebrew on Linux

List out install candidates #

You can list all of the available versions to install with pyenv install --list. It does reccomend updating pyenv if you suspect that it is missing one. At the time of writing this comes out to 532 different versions!

pyenv install --list

Let’s install the latest 3.8 patch #

Installing a version is as easy as pyenv install 3.8.12. This will install it, but not make it active anywhere.

pyenv install 3.8.12

let’s use python 3.8.12 while in this directory #

Running pyenv local will set the version of python that we wish to use while in this directory and any directory underneath of it while using the pyenv command.

pyenv local python3.8.12

.python-version file #

This creates a .python-version files in the directory I ran it in, that contains simply the version number.

3.8.12

using with pipx #

I immediately ran into the same issue I was having before when trying to run pipx, as pipx was running my system python. I had to install pipx in the python3.8 environment to get it to use it.

pyenv exec pip install pipx
pyenv exec pipx run kedro new

python is still the system python #

When I open a terminal and call python its still my system python that I installed and set with update-alternatives. I am not sure if this is expected or based on how I had installed the system python previously, but it’s what happened on my system.

update-alternatives --query python

Name: python
Link: /home/walkers/.local/bin/python
Status: auto
Best: /usr/bin/python3
Value: /usr/bin/python3

making a virtual environment #

To make a virtual environment, I simply ran pyenv exec python in place of where I would normally run python and it worked for me. There is a whole package to get pyenv and venv to play nicely together, so I suspect that there is more to it, but this worked well for me and I was happy.

pyenv exec python -m venv .venv --prompt $(basename $PWD)

Now when my virtual environment is active it points to the python in that virtual environment, and is the version of python that was used to create the environment.

Links #

https://github.com/pyenv/pyenv#installation

Installing Homebrew on Linux

2021-12-29T20:24:48Z

Installing brew on linux proved quite easy and got pyenv running for me within 4 commands.

I had never used homebrew before, honestly I thought it was a mac only thing for years. Today I wanted to try out pyenv, and the reccommended way to install was using homebrew. I am not yet sure if I want either in my normal workflow, so for now I am just going to pop open a new terminal and install homebrew and see how it goes.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"' >> /home/walkers/.zprofile
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

That was it, now homebrew is working. Starting a new shell and running the command to install pyenv worked.

brew install pyenv

Links #

homebrew

Serve html from your command line

2021-12-28T20:24:48Z

When I first moved to vim from and ide like vscode or sublime text one of my very first issues was trying to preview my website at localhost:8000. There had always just been a button there to do it in all of my other editors, not vim. There are not many buttons for anything in vim. While there is probably a plugin that can run a webserver for me in vim, it’s not necessary, we just need the command line we are already in.

running a separate process #

You will need a way to run another process alongside vim, here are a couple ideas to get you going that are not the focus here.style

use background jobs
- c-z to send a job to the background
- fg to bring it back
use a second terminal
use a second tab
use tmux and run it in a separate split/window
use an embeded nvim terminal

running a development webserver from the command line #

Python already exists on most linux systems by default, and most are now on python3. If you are on windows typing python will take you directly to the windows store to install it, or you can also use wsl.

# python3
python -m http.server

# running on port 5000
python -m http.server --directory markout 5000

# for the low chance you are on python2
python -m SimpleHTTPServer

# running on port 5000
python -m SimpleHTTPServer 5000
python -m SimpleHTTPServer --directory markout 5000

using nodejs #

If you are a web developer it’s likely that you need nodejs and npm on your system anyways and may want to use one of the servers from npm. I’ll admit with these not being tied to the long term support of a language they are much more feature rich with things like compression out of the box. In my opinion they are nice things that you would want out of a production server, but may not be necessary for development.

installing npx #

# if you don't alredy have npx
npm i -g npx

npx is a handy tool that lets you run command line applications straight from npm without installing them. It pulls the latest version every time you want to run, then executes it without it being installed.

running the http-server with npx #

npx http-server

# running on port 5000
npx http-server -p 5000
npx http-server markout -p 5000

Code Review from the comfort of vim | Diffurcate

2021-12-04T11:34:47Z

I often review Pull requests from the browser as it just makes it so easy to see the diffs and navigate through them, but there comes a time when the diffs get really big and hard to follow. That’s when its time to bring in the comforts of vim.

https://youtu.be/5NKaZFavM0E

Plugins needed #

This all stems from the great plugin by AndrewRadev. It breaks a down into a project. So rather than poping into a pager from git diff, you can pipe to diffurcate and it will setup a project in a tmp directory for you and you can browse this project just like any other except it’s just a diff.

Plug 'AndrewRadev/diffurcate.vim'

My aliases #

First to quickly checkout PR’s from azure devops I have setup an alias to fuzzy select a pr and let the az command do the checkout.

alias azcheckout='az repos pr checkout --id $(az repos pr list --output table | tail -n -2 | fzf | cut -d " " -f1)'

Next I have a few aliases setup for checking diffs. The first one checks what is staged vs the current branch, the others check the current branch vs main or master.

alias diffstaged="git diff --staged | nvim - +Diffurcate '+Telescope find_files'"
alias diffmain="git diff main.. | nvim - +Diffurcate '+Telescope find_files'"
alias diffmaster="git diff master.. | nvim - +Diffurcate '+Telescope find_files'"

diffcommit() {
    git diff $1 | nvim - +Diffurcate '+Telescope find_files'
}

Links #

diffurcte.vim

Open files FAST from zsh | or bash if thats your thing

2021-12-02T11:34:47Z

https://youtu.be/PQw_is7rQSw

I am often in a set of tmux splits flying back and forth, accidentally close my editor, so when I come back to that split and hit my keybinds to edit files I enter them into zsh rather than into nvim like I intended. Today I am going to sand off that rough edge and get as similar behavior to nvim as I can with a couple of aliases.

Make sure you check out the YouTube video to see all of my improvements.

what’s an alias #

If you have never heard of an alias before it’s essentially a shortcut to a given command. You can pass additional flags to the underlying command and they will get passed in. Most of the time they are just shorter versions of commands that you run often or even like in this case a common muscle memory typo that occurs for you.

My new alias’s for fuzzy editing files from zsh #

Here are the new aliases that I came up with to smooth out my workflow. These give me a similar feel to how these keys work in neovim but from zsh.

# fuzzy select file to edit
alias p='nvim `fzf --preview="bat --color always {}"`'

# give me the same syntax as edit while in neovim
alias :e='nvim '

Follow the YouTube channel or the rss feed to stay up to date.

playlist for my dotfiles challenge

30 days dotfile ricing

2021-11-29T11:34:47Z

https://youtu.be/Jq1Y48F_rOU

I am challenging myself to 30 days of dotfile ricing. I have been on linux desktop for a few months now and have a pretty good workflow going, I have the coarse edits done to my workflow, but it has some rough edges that need sanded down. It’s time to squash some of those little annoyances that still exist in my setup.

This is primarily going to be focused on productivity, but may have a few things to just look better. This will comprise heavily of aliases, zsh, and nvim config.

Follow the YouTube channel or the rss feed to stay up to date.

Update Alternatives in Linux

2021-11-20T10:38:00Z

update-alternatives --query python

update-alternatives: error: no alternatives for python

sudo update-alternatives --install /usr/local/bin/python python `which python3.8` 2
# update-alternatives: using /usr/bin/python3.8 to provide /usr/local/bin/python (python) in auto mode

sudo update-alternatives --install /usr/local/bin/python python `which python2.7` 5
# update-alternatives: using /usr/bin/python2.7 to provide /usr/local/bin/python (python) in auto mode

update-alternatives --query python
# Name: python
# Link: /usr/local/bin/python
# Status: auto
# Best: /usr/bin/python2.7
# Value: /usr/bin/python2.7
#
# Alternative: /usr/bin/python2.7
# Priority: 5
#
# Alternative: /usr/bin/python3.8
# Priority: 2

sudo update-alternatives --install /usr/local/bin/python python `which python3.8` 20
# update-alternatives: using /usr/bin/python3.8 to provide /usr/local/bin/python (python) in auto mode

JUT | Read Notebooks in the Terminal

2021-11-20T10:38:00Z

Trying to read a .ipynb file without starting a jupyter server? jut has you covered.

https://youtu.be/t8AvImnwor0

watch the video version of this post on YouTube

install #

jut is packaged and available on pypi so installing is as easy as pip installing it.

pip install jut

! This is my first time including snippets of the video in the article like this, let me know what you think!

examples #

jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb
jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --head 3
jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --tail 2

what are all the commands available for jut? #

Take a look at the help of the jut cli to explore all the options that it offers.

jut --help

There is some good information on the projects readme as well.

without installing #

using pipx

Don’t want jut cluttering up your venv, or want to save yourself from making a new one, pipx can manage a separate virual environment for you. This is one of the biggest selling points for me.

pipx run jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --head 3

nbconvert #

jut is the lightweight option that I think will fit the bill often for me, but it just doesn’t always cut it. Mostly if there are images in the notebook or large output that is hard to read, its time to pull out the medium guns that sit between fulling running jupyter and jut.

pip install nbconvert

nbconvert does not have its own cli, instead it sits under the jupyter command.

html">converting to html #

Need to see images, go here!

wget https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb
jupyter nbconvert flame_temperature.ipynb --to html
python -m http.server

Note, nb convert does not work with a url, you will need to have the notebook locally.

what other options does nbconvert offer? #

nbconvert also offers a standard help flag that you can access by passing in the --help flag

jupyter nbconvert --help

converting to markdown #

nbconvert also supports converting to many other formats, one option that is quite interesting for use in the terminal is markdown. We could simply convert the notebook to markdown and cat it out.

jupyter nbconvert flame_temperature.ipynb --to maarkdown
cat flameflame_temperature.md

viewing markdown with glow #

Glow is a terminal markdown viewer that looks really good. These days I use bat as cat so I don’t get quite as much benefit from glow, but it still looks pretty good.

glow flameflame_temperature.md

viewing markdown as slides with lookatme #

lookatme

Lookatme is my slideshow tool of choice. Creating slides in markdown is such a fantasic user experience, It realy lets you go from outline to finished slide deck fluidly. Refactoring the whole thing is also so much easier than with gui tools. Once you have your idea fleshed out it does make the process of making slides in powerpoint much easier if thats what you need.

On to nbconvert, without even changing the notebook we can look at the notebook as a slideshow in the terminal. The only thing that it needs is some markdown headers to start new slides from.

lookatme flameflame_temperature.md

viewing markdown with rich #

Bringing this full circle, lets take a look at the converted markdown with rich. Here you will notice a surprising similarity to what we saw with jut.

pip install rich
python -m rich.markdown flame_temperature.md

Rich still cannot pull directly from a url or display markdown with out being installed and managed by yourself. Unlike how jut can leverage pipx to manage the installation sandbox for you.

Links #

jut - View notebooks in the terminal
nbconvert - convert notebooks to other formats
flame-temperature notebook - The sample notebook I used
glow - Terminal Markdown viewer
lookatme - Terminal Markdown slideshow tool
pipx - Run python cli’s without maintianing a virtual environment
rich - Beautiful python terminal formatter

Trim unused git branches

2021-05-07T09:47:24Z

Trim branches no longer on origin #

git remote prune origin --dry-run
git remote prune origin

Find branches already merged #

git checkout main
# list remote branches that have already been merged into main
git branch -r --merged
# list local branches that have already been merged into main
git branch --merged

Create a Virtual File Gallery with Symlinks

2021-05-05T08:23:33Z

Creating a directory that is a union of several directories can be achieved with a few symlinks at the command line.

Creating a Virtual File Gallery #

Here is how I am creating a virtual directory of all my projects that is a combination of both work and not-work projects. I am creating symlinks for every directory under ~/work and ~/git.

rm -rf ~/projects
mkdir ~/projects
ln -sf ~/work/* ~/projects
ln -sf ~/git/* ~/projects

⚠ Notice that first I am recreating the directory each time. This will ensure that any project that is deleted from their actual directory is removed from the virtual gallery.

Updating the gallery #

Since links are always kept up to date without any extra work, all the data is still in the same place it started. But as new directories are added to any project directory they will not be automatically added to the virtual gallery.

cron
bashrc/zshrc

If you’re concerned about system resources, you can add it to a cron job to run at a regular schedule that makes sense to you. For me, I just popped those 4 lines right in my ~/.zshrc. It’s a bit overkill, maybe bloat, but it runs in an impercieveable amount of time.

Automatically CD to the real directory #

When you cd into a ~/projects/my-proj directory, your $PWD will still be ~/projects/my-proj. I did not want this for my use case. I wanted to follow the symlink to the real directory. I found two options that worked for me.

alias cd='cd -P'
set -o physical

Add either of these to your .bashrc/.zshrc to follow symlinks to the actual directory.

How to Install micromamba on linux (from the comamnd line only)

2021-04-21T13:32:27Z

I really like using conda (miniconda) as my python virtual environment manager of choice. It’s simple and it includes its own python interpreter using the version that I specify at creation.

Mamba #

from their readme

Mamba is a reimplementation of the conda package manager in C++.

parallel downloading of repository data and package files using multi-threading
libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE
core parts of mamba are implemented in C++ for maximum efficiency

At the same time, mamba utilize the same command line parser, package installation and deinstallation code and transaction verification routines as conda to stay as compatible as possible.

Installing Micromamba #

Similar to miniconda micromamba can be installed with a few lines of bash

wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
./bin/micromamba shell init -s bash -p ~/micromamba
source ~/.bashrc

Creating Environments with Micromamba #

Creating new environments with micromamba is pretty similar to using conda.

micromamba create -n mamba-new python=3.9 -y -c conda-forge

-c is required #

I was unable to figure out how to configure channels to micromamba, so I needed to add -c conda-forge to my commands.

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

WARNING No 'channels' specified
Encountered problems while solving:
  - nothing provides requested python 3.9**

ERROR   Could not solve for environment specs

⚠ micromamba thows this error when -c conda-forge is missing from the create command.

Speed #

micromamba is built for speed. I tried it out in a replit.com session, while it felt quite snappy creating a new environment was still within a few seconds of conda on subsequent environment creations. Their marketing says it should be faster, but for what I use conda for I didn’t see it.

pip #

I used conda install years ago while on windows machines struggling to compile c-extensions and install certain troublesome packages, but I haven’t used a conda install in years, pip works just fine for my use.

Useful Links #

GitHub: https://github.com/mamba-org/mamba
Gitter: https://gitter.im/QuantStack/Lobby
Documentation: https://mamba.readthedocs.io/en/latest/

Vim Wsl Clipboard

2021-04-17T00:00:00Z

I’ve long used neovim from within windows wsl, and for far too long, I went without a proper way to get text out of it and into windows.

wsl has access to cmd applications #

wsl can access clip.exe. You can do some cool things with it, such as cat a file into the clipboard, sending output from a command to the clipboard, or set an autocmd group in vim to send yank to the windows clipboard.

using clip.exe #

Let’s say you want to send a teammate the tail of a log file over chat. You can tail the file into clip.exe.

tail -n 1 info.log | clip.exe

pipe streams of text into clip.exe

make it a bit more natural #

I recently made mine feel a bit more natural by aliasing it to clip.

alias clip=clip.exe

pop this in your ~/.bashrc or ~/.zshrc

yanking to windows clipboard from vim #

I use neovim as my daily text editor and its a pain to share code with a teammate over chat, stack overflow, into a gist, or whatever you need. The following snippet has been quite useful and flawless for me.

if system('uname -r') =~ "Microsoft"
    augroup Yank
        autocmd!
        autocmd TextYankPost * :call system('/mnt/c/windows/system32/clip.exe ',@")
        augroup END
endif

add this to your ~/.vimrc or your ~/.config/nvim/init.vim

Wsl2 #

Based on some feedback from l-sannin the ‘uname -r’ command now returns uname -r command returns '5.10.16.3-microsoft-standard-WSL2' So you will need an all lowercase microsoft.

if system('uname -r') =~ "microsoft"
  augroup Yank
  autocmd!
  autocmd TextYankPost * :call system('/mnt/c/windows/system32/clip.exe ',@")
  augroup END
endif

Quickly Edit Posts

2021-01-18T00:00:00Z

Recently I automated starting new posts with a python script. Today I want to work on the next part that is editing those posts quickly.

Automating my Post Starter

Check out this post about setting up my posts with python 🐍

Enter Bash #

For the process of editing a post I just need to open the file in vim quickly. I dont need much in the way of parsing and setting up the frontmatter. I think this is a simple job for a bash script and fzf.

change to the root of my blog
fuzzy find the post
open it with vim
change back to the directory I was in

bash function #

For this I am going to go with a bash function. This is partly due to being able to track where I was and get back. Also the line with nvim will run fzf everytime you source your ~/.alias file which is not what we want.

Lets setup the boilerplate. Its going to create a function called ep "edit post", track our current directory, create a sub function _ep. Then call that function and cd back to where we were no matter if _ep fails or succeeds.

boilerplate

ep () {
    _dir=$(pwd)
    _ep () {
        # open file here
    }
    _ep && cd $_dir || cd $_dir
}

Creating Reusable Bash Scripts

check out this post for more information about writing reusable bash scripts.

FZF #

Let’s focus in on that _ep function here that is going to do the bulk of the work to edit the post.

cd to where I want to edit from

cd ~/git/waylonwalkerv2/

Next I need to find all markdown pages within my posts directory. There is probably a better way to filter with the find command directly, but I am not worried about perf here and I knew how to do it without google.

find all markdown

find ~/git/waylonwalkerv2/src/pages/ | grep .md$

Now that we can list all potential posts, sending the selected post back to neovim is as easy as piping those files into fzf inside of a command substitution that is called with neovim.

putting together the edit command

$EDITOR $(find ~/git/waylonwalkerv2/src/pages/ | grep .md$ | fzf)

Final Script #

final ep function

ep () {
    _dir=$(pwd)
    _ep () {
        cd ~/git/waylonwalkerv2/
        $EDITOR $(find ~/git/waylonwalkerv2/src/pages/ | grep .md$ | fzf)
    }
    _ep && cd $_dir || cd $_dir
}

If Tmux

2021-01-09T00:00:00Z

I do much of my work from tmux, I love it so much that I want to setup some functionality that puts me in tmux even if I didn’t ask for it.

Bash Function #

Bash function to check if the shell is in a tmux session.

in_tmux () {
  if [ -n "$TMUX" ]; then
    return 0
  else
    return 1
  fi
  }

Using the bash function #

I often open up vim to do some quite edits, but before I know it I have several splits open and I need access to another shell utility, but I forgot to start in tmux. This function makes sure tht I start in tmux everytime.

Using if_tmux to ensure vim is opened in tmux.

vim () {
  in_tmux \
    && nvim \
    || bash -c "\
    tmux new-session -d;\
    tmux send-keys nvim Space +GFiles C-m;\
    tmux -2 attach-session -d;
    "
  }

I am not quite sure if this is proper use of the && and ||, let me know if you have a better way to do one thing if in_tmux returns true and another if it returns faslse.

Newsboat

2021-01-02T00:00:00Z

Web browsers are a black hole of productivity. I try to use them as little as possible when it is time to focus. I try to use help, ?, or ?? with ipython, or –help at the command line as much as possible. What about that time I am trying to see what my online friends are posting on their sites? I used to used google reader quite heavily before that was taken down.

Newsboat #

I am going to give a terminal rss reader a try for a bit and see how that goes for me. I have really struggled to get into an rss reader since google reader died.

installation #

I installed with the reccomended snap for Ubuntu.

sudo snap install newsboat

Adding feeds #

super simple

Running help for newsboat directed me towards their config files at the bottom.

❯ newsboat --help
newsboat r2.22
usage: /snap/newsboat/3849/usr/local/bin/newsboat [-i <file>|-e] [-u <urlfile>] [-c <cachefile>] [-x <command> ...] [-h]
    -e, --export-to-opml        export OPML feed to stdout
    -r, --refresh-on-start      refresh feeds on start
    -i, --import-from-opml=<file>   import OPML file
    -u, --url-file=<urlfile>    read RSS feed URLs from <urlfile>
    -c, --cache-file=<cachefile>    use <cachefile> as cache file
    -C, --config-file=<configfile>  read configuration from <configfile>
    -X, --vacuum            compact the cache
    -x, --execute=<command>...  execute list of commands
    -q, --quiet         quiet startup
    -v, --version           get version information
    -l, --log-level=<loglevel>  write a log with a certain loglevel (valid values: 1 to 6)
    -d, --log-file=<logfile>    use <logfile> as output log file
    -E, --export-to-file=<file> export list of read articles to <file>
    -I, --import-from-file=<file>   import list of read articles from <file>
    -h, --help          this help
        --cleanup           remove unreferenced items from cache

Files:
    - configuration:  /home/nic/snap/newsboat/3849/.newsboat/config
    - feed URLs:      /home/nic/snap/newsboat/3849/.newsboat/urls
    - cache:          /home/nic/snap/newsboat/3849/.newsboat/cache.db

Support at #newsboat at https://freenode.net or on our mailing list https://groups.google.com/g/newsboat
For more information, check out https://newsboat.org/

I just need to edit its urls file.

nvim ~/snap/newsboat/3849/.newsboat/urls

The Urls file is just a list of urls to rss feeds. Adding mine in allowed me to see all of my posts.

https://waylonwalker.com/rss.xml

Config #

I took most of my config from a blog post that I found by Evan Travers. It set some sane defaults to the reading width and vim keys.

# https://evantravers.com/articles/2020/04/15/reworking-my-rss-reading/
# Hide feeds where all the items are read.
show-read-feeds no

# Make the text width readable
text-width 50

# Use multiple threads to download all the news faster.
reload-threads 11

# browser ~/bin/newsboat-browser.sh
# browser "/usr/bin/brave-browser %u"

# unbind keys
unbind-key ENTER
unbind-key j
unbind-key k
unbind-key J
unbind-key K
unbind-key ^D
unbind-key ^U
unbind-key o
unbind-key g
unbind-key G

# bind keys - vim style
bind-key j down
bind-key k up
bind-key l open
bind-key h quit
bind-key ^D pagedown
bind-key ^U pageup
bind-key b toggle-source-view
bind-key U toggle-show-read-feeds
bind-key u show-urls
bind-key g home
bind-key G end
bind-key b open-in-browser-and-mark-read
bind-key B open-in-browser
bind-key i sort
bind-key I rev-sort

GUI Browser #

No matter how many different guides I tried I keedp getting error code 127 when trying to open-in-browser. Please let me know if you know how to fix this. For now I am just going to roll with it.

Here’s how it looks #

browsing a feed in newsboat

reading an article in newsboat

Ipython Ninjitsu

2020-12-14T00:00:00Z

?docstring
??sourcecode
%run
%debug
%autoreload
%history
autoformat
%reset
!shell commands

?docstring #

Stop going to google everytime your stuck and stay in your workflow. The ipython ? is a superhero for productivity and staying on task.

from kedro.pipeline import Pipeline
Pipeline?

Init signature:
Pipeline(
    nodes: Iterable[Union[kedro.pipeline.node.Node, ForwardRef('Pipeline')]],
    *,
    tags: Union[str, Iterable[str]] = None,
)
Docstring:
A ``Pipeline`` defined as a collection of ``Node`` objects. This class
treats nodes as part of a graph representation and provides inputs,
outputs and execution order.
Init docstring:
Initialise ``Pipeline`` with a list of ``Node`` instances.

Args:
    nodes: The iterable of nodes the ``Pipeline`` will be made of. If you
        provide pipelines among the list of nodes, those pipelines will
        be expanded and all their nodes will become part of this
        new pipeline.
    tags: Optional set of tags to be applied to all the pipeline nodes.

Raises:
    ValueError:
        When an empty list of nodes is provided, or when not all
        nodes have unique names.
    CircularDependencyError:
        When visiting all the nodes is not
        possible due to the existence of a circular dependency.
:

Note This does jump you into a pager, a j,k or up, down to navigate, q to quit.

??sourcecode #

Docstring not enough for you use case. I often run into cases where the docstring is not clear enough and I need to see the implementation for myself to see what a function does.

%run #

I turned my nose up at this one, prior to seeing the famous I don’t like notebooks by Joel Grus. My first snobby reaction was that developing modules and using autoreload was superior. I have since realized there is a place for %run, and it can cut down on some keystrokes to import, setup, and run even when developing in modules.

%debug #

ipython comes with a post-mortem debugger, and it can be a lifesaver. If we have a long running function that runs into an error it can be a complete buzzkill.

def long_func():
   import time
   time.sleep(12)
   n = 12
   df = pd.Data({'a': range(n)})
   return df

long_func()

%reset #

Reclaim memory usage in Jupyter

%autoreload #

Autoreload in Ipython

c.InteractiveShellApp.extensions = ["autoreload"]
c.InteractiveShellApp.exec_lines = ["%autoreload 2"]
c.InteractiveShellApp.exec_lines.append(
    'print("Warning: disable autoreload in ipython_config.py to improve performance.")'
)

place this in your ~/.ipython/profile_default/ipython_config.py to auto reload without needing to run the magic every time

autoformat #

This is a relatively new feature to ipython. I really enjoy it, as the time that I need the most help autoformatting my code is riffing on an ad hoc analysis at the command line.

c.TerminalInteractiveShell.autoformatter = "black"

place this in your ~/.ipython/profile_default/ipython_config.py to autoformat with black by default

new prompt #

reverse history search #

Control R

Want automatic imports?? #

Smoother Python with automatic imports | pyflyby

This article covers how I setup automatic imports in ipython

Scripting Tmux Layouts

2020-12-13T00:00:00Z

This is how I script a tmux layout

 bash -c "tmux new-session -t 'editor' -d;\
    tmux split-window -v 'zsh';
    tmux send-keys nvim Space /src/ Space +GFiles C-m; \
    tmux rotate-window; \
    tmux select-pane -U; \
    tmux -2 attach-session -d
    "

Compare Directories In Bash

2020-12-11T00:00:00Z

Today I needed to check for articles that used the same slug from two directories, bash made it super simple.

diff -rq src/pages/blog src/pages/notes

Find and Replace in the Terminal.

2020-11-12T05:00:00Z

grepr #

grepr() {grep -iRl "$1" | xargs sed -i "s/$1/$2/g"}

```bash
grepr() {grep -iRl "$1" | xargs sed -i "s/$1/$2/g"}

grepd #

grepd() {grep -iRl "$1" | xargs sed -i "/^$1/d"}

CocSearch #

:CocSearch published: false -g *.md

Creating Reusable Bash Scripts

2020-08-13T05:00:00Z

Bash is a language that is quite useful for automation no matter what language you write in. Bash can do so many powerful system-level tasks. Even if you are on windows these days you are likely to come across bash inside a cloud VM, Continuous Integration, or even inside of docker.

I have three techniques that help me write more composable bash scripts.

functions

Functions #

Break scripts down into reusable components

Functions in bash are quite simple. They are something that I wish I would have started using long ago. They make your code much more reusable. I often use them in my aliases as well since they can simplify the process and allow more flexibility.

syntax

#!/bin/sh
# hello_world
hello_world () {
    echo "hello world"
}

Source the file to load the function and run it from the terminal.

run it

source hello_world
hello_world

outputs

hello world

Arguments #

Make functions a little more flexible

Arguments and options are quite a bit more complex in bash. For now, we will focus on the basics which are not all that bad.

positional arguments #

easiest and most common to use

Positional arguments can be pulled out quite easily using $1 for the first one, $2 for the second, and so on.

note $0 is the command that was called. You will see this often used to find the command called to open up your current shell.

syntax

#!/bin/sh
# hello
hello () {
    echo "hello $1"

run it

source hello
hello Waylon

Now we have a function that accepts positional arguments and we can call it by passing things into it.

outputs

hello Waylon

More than one argument would be ignored since we are only looking at $1.

run it

source hello
hello Waylon Walker

outputs

hello Waylon

Just the same as before since we do not use the second argument.

All Arguments #

Bash has another special variable $@ that stores all arguments in one.

syntax

#!/bin/sh
# hello
hello () {
    echo "hello $@"
}

Just the same as before.

run it

source hello
hello Waylon Walker

Now the function will output all arguments that are passed into it since we are using the $@ variable.

outputs

hello Waylon Walker

Error Handling #

Super powers in a single line

The easiest and most common way to handle an error in bash is through the use of the logical operators && (and) and || (or).

Here I have a concrete example from earlier today. I was creating a bash script to run a python script from cron. The bash script is there to make sure that we have the python environment, activate it, and run. If it doesn’t have it, it should create it.

# creates the conda environment
create_env() {
    conda create -n "$1" python=3.8
    conda activate "$1"
    pip install -r requirements.txt
}


# checks if the conda environment exists
env_exists() {
conda info --envs | awk '{print $1}' | tail -n +3 | grep -w "$1" > /dev/null
}


# creates the conda environment if it doesn't exist
create_if () {
env_exists "$1" && echo "environment exists" || create_env "$1"
}

create_if my_env

If we look at the create_if function, it will check if the environment exists if there is a passing status code 0, then it will run echo "environment exists" otherwise it will run create_env $1.

note Inside of env_exists grep will look for whole words if there is a match it will give a status code 0 if it finds a match and not 0 if there is no match.

main script #

_if __name__ == " __main__"_

The last thing I want to discuss is making a bash script both runnable and sourceable. This makes it so that you can source filename.sh and run each function individually, or bash filename.sh to run the script. This is a similar concept to if __name__ == " __main__" from python.

I did try this from bash and zsh with success. The following is an example that would pass all arguments into a main function.

syntax

if [["${BASH_SOURCE[0]}" == "${0}" ]]; then
    main "$@"
fi

Extending the example above that created a python example if necessary we can create the env if necessary, activate the environment, and run the script.

syntax

if [["${BASH_SOURCE[0]}" == "${0}" ]]; then
create_if my_env
conda activate my_env
python script.py
fi

Using this syntax to run our “main” functions will allow us to both runs the script or source the script to utilize the functions that we created.

How to Install miniconda on linux (from the command line only)

2020-08-10T05:00:00Z

miniconda is a python distribution from continuum. It’s a slimmed-down version of their very popular anaconda distribution. It comes with its own environment manager and has eased the install process for many that do not have a way to compile c-extensions. It made it much easier to install the data science stack on windows a few years ago. These days windows are much better than it was back then at compiling c-extensions. I still like its environment manager, which installs to a global directory rather than a local directory for your project.

Installing miniconda on Linux #

Installing miniconda on Linux can be a bit tricky the first time you do it completely from the terminal. The following snippet will create a directory to install miniconda into, download the latest python 3 based install script for Linux 64 bit, run the install script, delete the install script, then add a conda initialize to your bash or zsh shell. After doing this you can restart your shell and conda will be ready to go.

mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

Options #

The miniconda.sh script comes with a few basic options. Most notably we used -b to be able to run unattended, which means that all of the agreements are automatically accepted without user prompt. -u updates any existing installation in the directory of install if there is one. -p is the directory to install into.

usage: /root/miniconda3/miniconda.sh [options]

Installs Miniconda3 4.6.14

-b           run install in batch mode (without manual intervention),
             it is expected the license terms are agreed upon
-f           no error if install prefix already exists
-h           print this help message and exit
-p PREFIX    install prefix, defaults to /root/miniconda3, must not contain spaces.
-s           skip running pre/post-link/install scripts
-u           update an existing installation
-t           run package tests after installation (may install conda-build)

Silent/Logged #

A quick and easy way to silence everything or to log it to a file during an automated install is to wrap the script into a bash function, or save it to its own file, and call the file. I like the function method since I can still copy it right into a terminal, or keep my install script as one single file.

install_miniconda () {
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh
}

normal

install_miniconda

silent

install_miniconda > /dev/null 2>&1

logged

install_miniconda > miniconda_install.log 2>&1

Creating Reusable Bash Scripts

This posts covers more ways to write reuable bash scripts

Cross posted to DEV #

Check out this post and many more on dev.to/waylonwalker

025.md

2020-07-29T02:16:24Z

Convert Markdown to reveal.js slides

setup

wget https://github.com/hakimel/reveal.js/archive/master.tar.gz
tar -xzvf master.tar.gz
mv reveal.js-master reveal.js

convert

pandoc -t revealjs -s -o myslides.html myslides.md -V revealjs-url=https://unpkg.com/reveal.js@3.9.2/

026.md

2020-07-28T05:00:00Z

Convert Markdown to reveal.js slides

setup

wget https://github.com/hakimel/reveal.js/archive/master.tar.gz
tar -xzvf master.tar.gz
mv reveal.js-master reveal.js

convert

pandoc -t revealjs -s\
   -o myslides.html myslides.md \
   -V revealjs-url=https://unpkg.com/reveal.js@3.9.2/

inline YAML or path to separate file (e.g.: .github/filters.yaml)

2020-06-21T02:16:24Z

Conditionally run GitHub Actions Steps

- uses: dorny/paths-filter@v2.2.0
  id: filter
  with:
      # inline YAML or path to separate file (e.g.: .github/filters.yaml)
      filters: |
      backend:
          - 'backend/**/*'
      frontend:
          - 'frontend/**/*'

# run only if 'backend' files were changed
- name: backend unit tests
  if: steps.filter.outputs.backend == 'true'
  run: ...

023

2020-06-21T02:16:24Z

Find and replace Groups in VSCode $1 referrs to the second group

(filepath: top)(.*)
filepath: s3://bucket/top$1.parquet

- filepath: top/raw/scooters
+ filepath: s3://bucket/top/raw/scooters.parquet

022

2020-06-21T02:16:24Z

File System Full 🤔 #

Move files then symlink them

with Bash #

mkdir /mnt/mounted_drive
mv ~/bigdir /mnt/mounted_drive
ln -s /mnt/mounted_drive/bigdir ~/bigdir

021

2020-06-21T02:16:24Z

File System Full 🤔 #

show largest files in current directory

with Bash #

du . -h --max-depth=1

020

2020-06-21T02:16:24Z

File System Full 🤔 #

Show Remaining Space on Drives

with Bash #

df -h

019

2020-06-21T02:16:24Z

batch rename files #

with bash #

for f in *.jpeg; do
    mv -- "$f" "${f%.jpeg}.jpg"
done

Supercharge Zsh Startup

2020-01-03T06:00:00Z

I have been using oh-my-zsh successfully for about 2 years now. But lately my startup time has been really bothersome. It has grown to the point where it was taking about 5.5s to startup a shell! This is ok if I am going to spend some time in here for awhile and do some work that benefits from all of the autocompletions, plugins, and shortcuts that oh-my-zsh brings. But to only jump in to run a handful of commands is infuriating.

📑 My Setup #

I believe the real issue is io speed on wsl. I have some remote servers with similar configs that are 10x faster or more, loading in 100s of milliseconds rather than seconds. Sourcing all of the individual plugin files are just too much for it.

💨 How Fast can it be #

Quick side note: your zsh config is controled by your ~/.zshrc file. This file can source other files, load plugins, or run literally anything.

Time the initial time

time zsh -c exit

Move your ~/.zshrc config file.

mv ~/.zshrc ~/.zshrc-back

Time the fastest startup possible with nothing in your ~/.zshrc config file.

time zsh -c exit

Move your ~/.zshrc back

mv ~/.zshrc-back ~/.zshrc

🕵️‍♂️Profile your startup time #

It is possible to profile your zsh startup time by adding zmodload zsh/zprof to the start of your ~/.zshrc file and zprof at the end. This was unsuccessfull for me. I ended up just backing up ~/.zshrc file, then deleting half of it to see where the hot spots were. I found that two places that were really hot for me. One I was inadvertantly setting git and npm settings everytime that didn’t need to be set everytime. That was an easy 2s gain. Another easy 3s gain was removing oh-my-zsh.

# ~/.zshrc
zmodload zsh/zprof
...
..
.
zprof

😭But I really like oh-my-zsh #

without all the bells and whistle that oh-my-zsh provided zsh became lightning fast to load, but incredibly boring. It was also very painful to manually type out everything that it autocompleted or aliased all the time. Next I headed down a path to get all of that functionality back without sacrificing load time.

Without oh-my-zsh, zsh became incredibly boring.

🔌 Plugin Managers #

Photo by Steve Johnson on Unsplash

oh-my-zsh
zplugin
zgen

There are a number of plugin managers for zsh, I tried each of the ones listed above, but found that as I approached a nice setup that I liked the load time would creep up above the 2s mark each time. I would turn certain plugins on and off, try different plugin managers, before realizing that I had spent enough time on this problem and it was going to be time to settle on fast startup or functionality.

⚖ Finding Balance #

Semi-lazy loading

Photo by Jeppe Hove Jensen on Unsplash

After struggling to get all of the features I wanted with a fast load time, I decided to only load what I needed upon startup. Next I created a simple alias that loads in zgen and all of the plugins I want. By doing this I get two main benefits. Obviously I get a faster starup time by loading less. I got my startup time down to about 0.25s.

# ~/.zshrc
p () {
zgen load zsh-users/zsh-autosuggestions
....
..
.
}

⚡ Fast Loading #

I really like the fast startup time, because sometimes I am only loading up zsh to run a handful of commands that dont need much in the way of plugins

simple commands that need 💨 blazing start speed

any single easy to type command, these are my common commands that I will open a terminal in my editor and just need to run quick.

vim
git add . && git commit && git push
sh my_script.sh
make build
bake build
pytest
gatsby develop
npm i
npm update
pip install
ipython

The second benefit was that I can continue typing while plugins are loading.

>Prompt #

After removing oh-my-zsh the first thing that I missed was the themes that it provided. I went through a number of them and the one that seemed to have the smallest effect on performance and everything I needed was starship. It’s a really fast prompt written in rust. The biggest thing that I needed to have that other prompts were misssing was conda environments. I live much of my work life running python from various conda environments and it is crutial that I can see what environment I am in at all times.

💰 Bonus #

I applied the same logic to neovim and achieved similar results. Again it just had too many plugins loading on startup for simple tasks. I ended up taking a shortcut and load any heavy plugins upon NerdTreeToggle. I dont really use NERDTree that much, but when I do its for more than just a quick edit.

# ~/.config/nvim/init.vim
Plug 'valloric/youcompleteme', {'do': './install.py', 'on': 'NERDTreeToggle'}

📝 Bash Notes

2019-09-08T05:00:00Z

Bash is super powerful.

File System Full #

Show Remaining Space on Drives

df -h

show largest files in current directory

du . -h --max-depth=1

Move files then symlink them

mkdir /mnt/mounted_drive
mv ~/bigdir /mnt/mounted_drive
ln -s /mnt/mounted_drive/bigdir ~/bigdir

Fuzzy One Liners #

a() {source activate "$(conda info --envs | fzf | awk '{print $

edit in vim

vf() { fzf | xargs -r -I % $EDITOR % ;}

cat a file

vf() { fzf | xargs -r -I % $EDITOR % ;}

bash execute

bf() { bash "$(fzf)" }

git add

gadd() { git status -s | fzf -m | awk '{print $2}' | xargs git add && git status -s}

git reset

greset() { git status -s |  fzf -m | awk '{print $2}' |xargs git reset && git status -s}

Kill a process

fkill() {kill $(ps aux | fzf | awk '{print($2)}')}

Finding things #

Files #

fd-find is amazing for finding files, it even respects your .gitignore file 😲. Install with apt install fd-find.

fd md

ag -g python

find . -n "*.md"

++Vanilla Bonus

Content #

** show matching text **

ag python

grep -iR Python

++Vanilla Bonus

** show file names only **

ag -l python

grep -iRl python

++Vanilla Bonus

Recursively Replace text #

agr() {ag -l "$1" | xargs sed -i "s/$1/$2/g"}

++Vanilla Bonus

grepr() {grep -iRl "$1" | xargs sed -i "s/$1/$2/g"}

Extending **agr** or **grepr**

There are so many options inside of grep, ag, and sed that you could many an enormous amount of these if you really wanted to, but I like to keep it simple. These cover 90% of my usage. If I wanted to change something in the second half I would just paste in this command and edit it. More often though I want to limit the input, say only replace word1 to word2 inside of markdown files.

Limited Scope

fd md | xargs argr python python3

find . -n "*.md" | xargs grepr python python3

++Vanilla Bonus

Large Refactor At The Command Line

I use these replace commands heavily when doing large refactorings.

conditionally configure #

I like this one when there is not a good cli into config files and I need to replace something like a true to false if the value is in the config and append to the config if its not.

grepr() {
    # replaces first string with second string inside file from third argument
    # example:
    #   grepr "allow_conda_downgrades:.*" "allow_conda_downgrades: true" ~/.condarc
    if grep -xq $1 $3
    then
        sed -i "s|$1|$2|g" $3
    else
        echo "$2" >> $3
    fi
}

Watch the time #

watch -n 1 date

++Vanilla Bonus

with figlet

watch -n 1 bash -c "date | figlet"

watch a function #

run () {
date
aws s3 sync $BUCKET .
}
export -f run
watch -n 10 run

if conda environment does not exist create it #

conda info --envs | grep my_env && echo "my_env environment is installed" || conda create -n my_env python=3.8 -y
source activate my_env

Rename multiple files

more info from linuxize

for f in *.png; do
   mv ${f} prefix-${f}
done

using the rename command #

sudo apt install rename

rename "s/.GIF/.gif/" *.GIF

convert all files in a directory to unix #

dos2unix **/*

recursively remove all whitespace from .py files #

find **/*.py -type f -exec sed -i 's/ *$//' '{}' ';'

recursively autopep8 #

find . -name '*.py' -exec autopep8 --in-place '{}' \;

make bash script a runnable command #

include a shebang

#! /bin/bash

chmod

chmod +x /usr/local/bin/my_script

accept positional input

#! /bin/bash
input=$1
echo input

Using pyp #

pipx install pyp

replacement for cut #

❯ python -m http.server 5000 &
[1] 8574

✦ ❯ Serving HTTP on 0.0.0.0 port 5000 (http://0.0.0.0:5000/) ...

✦ ❯ ps aux | grep "python -m http.server" | grep -v grep | pyp 'line.split()[1]' | xargs kill
[1]  + terminated  python -m http.server 5000

replacement for wc #

conda info --envs | pyp 'len(lines) - 3 # account for header and base'

print contents of shell function #

declare -f <function-name>

batch rename files #

for f in *.jpeg; do
    mv -- "$f" "${f%.jpeg}.jpg"
done

convert markdown files to reveal.js #

https://github.com/jgm/pandoc/wiki/Using-pandoc-to-produce-reveal.js-slides{.hoverlink} install pandoc

apt install pandoc

setup

wget https://github.com/hakimel/reveal.js/archive/master.tar.gz
tar -xzvf master.tar.gz
mv reveal.js-master reveal.js

convert

pandoc -t revealjs -s -o myslides.html myslides.md -V revealjs-url=https://unpkg.com/reveal.js@3.9.2/

Render Markdown at the command line #

Glow is a terminal markdown renderer written in go. There iis a prebuilt binary that can simply be unzipped and executed to render markdow.

wget https://github.com/charmbracelet/glow/releases/download/v0.2.0/glow_0.2.0_linux_x86_64.tar.gz
tar -xzf glow_0.2.0_linux_x86_64.tar.gz
chmod +x glow
sudo mv glow /usr/bin

glow <filename>

Autocomplete for click applications #

see the docs for more details

Autocomplete for non click python cli’s #

shtab https://github.com/iterative/shtab

Ensure functions reset context #

project_log() {
   _dir=$(pwd)
   _project_log() {
      cd ~/projects/project
      git log
   }
   _project_log $@ && cd $_dir || cd $_dir
}

Finding Files #

rg --files . | grep bash
rg --files . | rg bash
rg --files . | ag bash
rg --files . | fzf
rg -e 'hook' -g '*.md'
rg -e 'hook' -g '*.py'

rg --files-with-matches
rg --files-without-match

mu-repo #

# installation
pip install mu-repo

## register repos
mu register --recursive
mu list

# run git commands
mu status --short
mu diff -U0 --color | bat

# run shell commands
mu sh $(grep -iRl "KEDRO_GID=0" | xargs sed -i "s/KEDRO_GID=0/KEDRO_GID=5/g")

Maintianing multiple git repos with mu-repo

See the full post for mu-repo for more

mu unregister all #

I ran into some issues with mu unregister * before, but this seems to work everywhere.

mu list | tail -n +3 | xargs -I {} mu unregister {}

mu register all repos that have an rg match #

Add all repos that have a versionspec of 3.7 in them.

rg --hidden=true 'versionSpec:.*.3.7' -l | xargs -I {} mu register {}

xrandr #

xrandr -s 1920x1080

Posts tagged: bash Archive

Vaulted Secrets Without Git Churn

The problem #

The solution at a glance #

How it works #

Compare-before-encrypt #

Decrypted diffs #

What changes will be seen #

Recommended usage #

COLUMNS env var

tar over ssh

💭 2025-07-10 Notes

fc

Rcap of how !! works #

fc enters the chat #

Shell History #

Up Arrow #

unset multiple environment variables

💭 bic | Static blog generator, in bash

💭 casey/just: 🤖 Just a command runner

💭 pipedream/justfile at main · thechangelog/pipedream

urllink

💭 hostnamectl to easily change hostname | Pype.dev

convert mp4 for twitter with ffmpeg

cli auth

curl #

wget #

httpx #

httpie #

httpie with plugin #

httpie prompt #

jpillora-installer

Use it to install neovim #

Self Host Your Own #

Repos I am using installer for #

💭 Deleting Specific Lines in a File with sed or yq

dash e your github docker actions

What is -e #

Solution for Windows #

Bash function to edit scripts faster

Currently #

First pass #

Final State #

a bit more ergo, and less readable #

Results #

Unzip minecraft mods to their directory from the command line

I had the files on a Windows Machine #

Downoading on the server #

Unzip to the minecraft-data directory #

Running the server with docker #

Bash mktemp

Making temp directories #

Use Case #

Templates #

RTFM #

View AWS event bridge rules with visidata

ag silver searcher look ahead and look behind

Ahead/Behind #

Neovim Config for Git

fuzzy wallpaper with fzf

2 minutes to stow

zsh #

nvim #

Tmux Pop size

Changing copier template strings (_endops)

copier template variables

template variables #

installing copier #

running copier #

results #

List all the files containing a phrase

ag #

rg #

My first impressions with pyenv

installing #

List out install candidates #

Let’s install the latest 3.8 patch #

let’s use python 3.8.12 while in this directory #

.python-version file #

using with pipx #

`fc` enters the chat #