Posts tagged: bash

I am often editing my own scripts as I develop them. I want to make a better workflow for working with scripts like this.

Currently #

Currently I am combining nvim with a which subshell to etit these files like this.

for now lets use my todo command as an example

nvim `which todo`

First pass #

On first pass I made a bash function to do exactly what I have been doing.

ewhich () {$EDITOR `which "$1"`}

The $1 will pass the first input to the which subshell. Now we can edit our todo script like this.

ewich todo

Note, I use bash functions instead of aliases for things that require input.

Final State #

This works fine for commands that are files, but not aliases or shell functions. Next I jumped to looking at the output of command -V $1.

• if the command is not found, search for a file
• if its a builtin, exit
• if its an alias, open my ~/.alias file to that line
• if its a function, open my ~/.alias file to that line

ewhich () {
case `command -V $1` in
    "$1 not found")
        FILE=`fzf --prompt "$1 not found searching ..." --query $1`
        [ -z "$FILE" ] && echo "closing" || $EDITOR $FILE;;
    *"is a shell builtin"*)
        echo "$1 is a builtin";;
    *"is an alias"*)
        $EDITOR ~/.alias +/alias\ $1;;
    *"is a shell function"*)
        $EDITOR ~/.alias +/^$1;;
    *)
        $EDITOR `which "$1"`;;
esac

a bit more ergo, and less readable #

To make it easier to type, at the sacrifice of readability for anyone watching I added a single character e alias to ewhich. So when I want to edit anything I just use e.

alias e=ewhich

Results #

Here is a quick screencast of how it works.

This morning I was trying to install a modpack on my minecraft server after getting a zip file, and its quite painful when I unzip everything in the current directory rather than the directory it belongs in.

I had the files on a Windows Machine #

So I’ve been struggling to get mods installed on linux lately and the easiest way to download the entire pack rather than each mod one by one seems to be to use the overwolf application on windows. Once I have the modpack I can start myself a small mod-server by zipping it, putting it in a mod-server directory and running a python http.server

python -m http.server

Downoading on the server #

Then I go back to my server and download the modpack with wget.

wget 10.0.0.171:8000/One%2BBlock%2BServer%2BPack-1.4.zip

Unzip to the minecraft-data directory #

Now I can unzip my mods into the minecraft-data directory.

unzip One+Block+Server+Pack-1.4.zip -d minecraft-data

Running the server with docker #

I run the minecraft server with docker, which is setup to mount the minecraft-data directory.

Running a Minecraft Server in Docker

A bit more on that in the other post, but when I download the whole modpack like this I make these changes to my docker compose. (commented out lines)

version: "3.8"

services:
  mc:
    container_name: walkercraft
    image: itzg/minecraft-server:java8
    environment:
      EULA: "TRUE"
      TYPE: "FORGE"
      VERSION: 1.15.2
      # MODS_FILE: /extras/mods.txt
      # REMOVE_OLD_MODS: "true"
    tty: true
    stdin_open: true
    restart: unless-stopped
    ports:
      - 25565:25565
    volumes:
      - ./minecraft-data:/data
      # - ./mods.txt:/extras/mods.txt:ro

volumes:
  data:

There is GNU coreutils command called mktemp that is super handy in shell scripts to make temporary landing spots for files so that they never clash with another instance, and will automatically get cleaned up when you restart, or whenever /tmp gets wiped. I’m not sure when that is, but I don’t expect it to be long.

Making temp directories #

Here are some examples of making temp directories in different places, my favorite is mktemp -dt mytemp-XXXXXX.

# makes a temporary directory in /tmp/ with the defaul template tmp.XXXXXXXXXX
mktemp

# makes a temporary directory in your current directory
mktemp --directory mytemp-XXXXXX
# shorter version
mktemp -d mytemp-XXXXXX

# same thing, but makes a file
mktemp mytemp-XXXXXX

# makes a temporary directory in your /tmp/ directory (or what ever you have configured as your TMPDIR)
mktemp --directory --tmpdir mytemp-XXXXXX
# shorter version
mktemp -dt mytemp-XXXXXX

# same thing, but makes a file
mktemp --tmpdir mytemp-XXXXXX
# shorter version
mktemp -t mytemp-XXXXXX

Use Case #

Here is a sample script that shows how to capture the tempdir as a variable and reuse it. Here is an example of curling my bootstrap file into a temp directory and running it from that directory.

local tmp=`mktemp -dt bootstrap-XXXXXX`
pushd $tmp
curl https://raw.githubusercontent.com/WaylonWalker/devtainer/main/bootstrap > bootstrap
bash bootstrap
popd

Templates #

You must have at least 3 trailing X’s that mktemp will replace with random characters. I played with it for a bit, it kinda allows for some trailing characters, and will not fill groups of X’s earlier in your template, only the last consecutive string.

My randomm samples I played with.

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com) took 2m24s
❯ mktemp myXtemp-XaXbXXXX -dt
/tmp/myXtemp-XaXbx9hn

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXXXs -dt
/tmp/myXtemp-XaXb2tpGs

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXcXXs -dt
mktemp: too few X's in template ‘myXtemp-XaXbXXcXXs’

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXcXXs -dt

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XaXbXXXXt -dt
/tmp/myXtemp-XaXbe8PWt

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XXX-you-XXX -dt
/tmp/myXtemp-XXX-you-48l

waylonwalker.com on  main [!?]  v3.9.7 (waylonwalker.com)
❯ mktemp myXtemp-XXX-you-XX -dt
mktemp: too few X's in template ‘myXtemp-XXX-you-XX’

RTFM #

The man page has good stuff on all the flags that you might need.

man mktemp

Reading eventbridge rules from the command line can be a total drag, pipe it into visidata to make it a breeze.

I just love when I start thinking through how to parse a bunch of json at the command line, maybe building out my own custom cli, then the solution is as simple as piping it into visidata. Which is a fantastic tui application that had a ton of vim-like keybindings and data features.

alias awsevents = aws events list-rules | visidata -f json

A super useful tool when doing PR’s or checking your own work during a big refactor is the silver searcher. Its a super fast command line based searching tool. You just run ag "<search term>" to search for your search term. This will list out every line of every file in any directory under your current working directory that contains a match.

Ahead/Behind #

It’s often useful to need some extra context around the change. I recently reviewed a bunch of PR’s that moved schema from save_args to the root of the dataset in all yaml configs. To ensure they all made it to the top level DataSet configuraion, and not underneath save_args. I can do a search for all the schemas, and ensure that none of them are under save_args anymore.

ag "schema: " -A 12 -B 12

Creating a minimal config specifically for git commits has made running git commit much more pleasant. It starts up Much faster, and has all of the parts of my config that I use while making a git commit. The one thing that I often use is autocomplete, for things coming from elsewhere in the tmux session. For this cmpe-tmux specifically is super helpful.

The other thing that is engrained into my muscle memory is jj for escape. For that I went agead and added my settings and keymap with no noticable performance hit.

Here is the config that has taken

~/.config/nvim/init-git.vim

source ~/.config/nvim/settings.vim
source ~/.config/nvim/keymap.vim
source ~/.config/nvim/git-plugins.vim
lua require'waylonwalker.cmp'

~/.config/nvim/git-plugins.vim

call plug#begin('~/.local/share/nvim/plugged')

" cmp
Plug 'hrsh7th/nvim-cmp'
Plug 'hrsh7th/cmp-nvim-lsp'
Plug 'hrsh7th/cmp-buffer'
Plug 'hrsh7th/cmp-path'
Plug 'hrsh7th/cmp-calc'
Plug 'andersevenrud/compe-tmux', { 'branch': 'cmp' }


call plug#end()

~/.gitconfig

[core]
    editor = nvim -u ~/.config/nvim/init-git.vim

I really appreciate that in linux anything can be scripted, including setting the wallpaper. So everytime I disconnect a monitor I can just rerun my script and fix my wallpaper without digging deep into the ui and fussing through a bunch of settings.

feh --bg-scale ~/.config/awesome/wallpaper/my_wallpaper.png

I set my default wallpaper with feh using the command above.

Leaning in on feh, we can use fzf to pick a wallpaper from a directory full of wallpapers with very few keystrokes.

alias wallpaper='ls ~/.config/awesome/wallpaper | fzf --preview="feh --bg-scale ~/.config/awesome/wallpaper/{}" | xargs -I {} feh --bg-scale ~/.config/awesome/wallpaper/{}'

I have mine alias’d to wallpaper so that I can quickly run it from my terminal.

Stow is an incredible way to manage your dotfiles. It works by managing symlinks between your dotfiles directory and the rest of the system. You can then make your dotfiles directory a git repo and have it version controlled. In my honest opinion, when I was trying to get started the docs straight into deep detail of things I frankly don’t really care about and jumped right over how to use it.

When using stow its easiest to keep your dotfiles directory (you may name it what you want) in your home directory, with application directories inside of it.

Then each application directory should reflet the same diretory structure as you want in your home directory.

zsh #

Here is a simple example with my zshrc.

mkdir ~/dotfiles
cd ~/dotfiles
mkdir zsh
mv ~/.zshrc zsh
stow --simulate zsh

You can pass in the –simulate if you wish, it will tell you if there are going to be any more errors or not, but it wont give much more than that.

WARNING: in simulation mode so not modifying filesystem.

Once your ready you can stow your zsh application.

stow zsh

nvim #

A slightly more complicated example is neovim since its diretory structure does not put configuration files directly in your home directory, but rather at a deeper level.

mkdir ~/dotfiles/nvim/.config/nvim/ -p
cd ~/dotfiles
mv ~/.config/nvim/ ~/dotfiles/nvim/.config/nvim/
stow zsh

!notice how the nvim directory inside of dotfiles is structured like it would be in your $HOME directory.

tmux popups can be sized how you like based on the % width of the terminal on creation by using the flags (h, w, x, y) for height, width, and position.

# normal popup
tmux popup figlet "Hello"
# fullscreen popup
tmux popup -h 100% -w 100% figlet "Hello"
# 75% centered popup
tmux popup -h 100% -w 75% figlet "Hello"
# 75% popup on left side
tmux popup -h 100% -w 75% -x 0% figlet "Hello"

example running these commands

I was completely stuck for awhile. copier was not replacing my template variables. I found out that adding all these _endops fixed it. Now It will support all of these types of variable wrappers

# copier.yml
_templates_suffix: .jinja
_envops:
  block_end_string: "%}"
  block_start_string: "{%"
  comment_end_string: "#}"
  comment_start_string: "{#"
  keep_trailing_newline: true
  variable_end_string: "}}"
  variable_start_string: "{{"

!RTFM: Later I read the docs and realized that copier defaults to using [[ and ]] for its templates unlike other tools like cookiecutter.

I’ve been looking for a templating tool for awhile that works well with single files. My go to templating tool cookiecutter does not work for single files, it needs to put files into a directory underneath of it.

template variables #

By default copier uses double square brackets for its variables. variables in files, directory_names, or file_names will be substituted for their value once you render them.

# hello-py/hello.py.tmpl
print('hello-[[name]]')

note! by default copier will not inject variables into your template-strings unless you use a .tmpl suffix.

Before running copier we need to tell copier what variables to ask for, we do this with a copier.yml file.

# copier.yml
name:
  default: my_name
  type: str
  help: What is your name

installing copier #

I prefer to install cli tools that I need globally with pipx, this always gives me access to the tool without worrying about dependency conflicts, bloating my system site-packages, or managing a separate virtual environment for it myself.

pipx install copier

running copier #

When running copier copy we pass in the directory of the template, and the directory that we want to render the template into.

copier copy hello-py .

note! the directory ‘.’ is often referred to in cli programs to represent the current working directory that we are calling the command from.

results #

The resulting files will have your variables injected into them if you have setup your template and copier.yml up correctly.

print('hello-you')

One of the most useful skills you can acquire to make you faster at almost any job that uses a computer is getting good at finding text in your current working diretory and identifying the files that its in. I often use the silver searcher ag or ripgrep rg to find files in large directories quickly. Both have a sane set of defaults that ignore hidden and gitignored files, but getting them to list only the filenames and not the matched was not trivial to me.

I’ve searched throught he help/man pages many times looking for these flags and they always seem to evade me.

ag #

Passing the flag -l to ag will get it to list only the filepath, and not the match. Here I gave it a --md as well to only return markdown filetypes. ag supports a number of filetypes in a very similar way.

ag nvim --md -l

rg #

Giving rg the --files-with-matches flag will yield you a similar set of results, giving only the filepaths themselves and not the match statement. Also passing in the -g "*.md" will similarly yield only results from markdown files.

rg --files-with-matches you -g "*.md"

pyenv provides an easy way to install almost any version of python from a large list of distributions. I have simply been using the version of python from the os package manager for awhile, but recently I bumped my home system to Ubuntu 21.10 impish, and it is only 3.9+ while the libraries I needed were only compatable with up to 3.8.

I needed to install an older version of python on ubuntu

I’ve been wanting to check out pyenv for awhile now, but without a burning need to do so.

installing #

Based on the Readme it looked like I needed to install using homebrew,so this is what I did, but I later realized that there is a pyenv-installer repo that may have saved me this need.

Installing Homebrew on Linux

List out install candidates #

You can list all of the available versions to install with pyenv install --list. It does reccomend updating pyenv if you suspect that it is missing one. At the time of writing this comes out to 532 different versions!

pyenv install --list

Let’s install the latest 3.8 patch #

Installing a version is as easy as pyenv install 3.8.12. This will install it, but not make it active anywhere.

pyenv install 3.8.12

let’s use python 3.8.12 while in this directory #

Running pyenv local will set the version of python that we wish to use while in this directory and any directory underneath of it while using the pyenv command.

pyenv local python3.8.12

.python-version file #

This creates a .python-version files in the directory I ran it in, that contains simply the version number.

3.8.12

using with pipx #

I immediately ran into the same issue I was having before when trying to run pipx, as pipx was running my system python. I had to install pipx in the python3.8 environment to get it to use it.

pyenv exec pip install pipx
pyenv exec pipx run kedro new

python is still the system python #

When I open a terminal and call python its still my system python that I installed and set with update-alternatives. I am not sure if this is expected or based on how I had installed the system python previously, but it’s what happened on my system.

update-alternatives --query python

Name: python
Link: /home/walkers/.local/bin/python
Status: auto
Best: /usr/bin/python3
Value: /usr/bin/python3

making a virtual environment #

To make a virtual environment, I simply ran pyenv exec python in place of where I would normally run python and it worked for me. There is a whole package to get pyenv and venv to play nicely together, so I suspect that there is more to it, but this worked well for me and I was happy.

pyenv exec python -m venv .venv --prompt $(basename $PWD)

Now when my virtual environment is active it points to the python in that virtual environment, and is the version of python that was used to create the environment.

Links #

https://github.com/pyenv/pyenv#installation

Installing brew on linux proved quite easy and got pyenv running for me within 4 commands.

I had never used homebrew before, honestly I thought it was a mac only thing for years. Today I wanted to try out pyenv, and the reccommended way to install was using homebrew. I am not yet sure if I want either in my normal workflow, so for now I am just going to pop open a new terminal and install homebrew and see how it goes.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"' >> /home/walkers/.zprofile
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

That was it, now homebrew is working. Starting a new shell and running the command to install pyenv worked.

brew install pyenv

Links #

• homebrew

When I first moved to vim from and ide like vscode or sublime text one of my very first issues was trying to preview my website at localhost:8000. There had always just been a button there to do it in all of my other editors, not vim. There are not many buttons for anything in vim. While there is probably a plugin that can run a webserver for me in vim, it’s not necessary, we just need the command line we are already in.

running a separate process #

You will need a way to run another process alongside vim, here are a couple ideas to get you going that are not the focus here.style

• use background jobs
  • c-z to send a job to the background
  • fg to bring it back
• use a second terminal
• use a second tab
• use tmux and run it in a separate split/window
• use an embeded nvim terminal

running a development webserver from the command line #

Python already exists on most linux systems by default, and most are now on python3. If you are on windows typing python will take you directly to the windows store to install it, or you can also use wsl.

# python3
python -m http.server

# running on port 5000
python -m http.server --directory markout 5000

# for the low chance you are on python2
python -m SimpleHTTPServer

# running on port 5000
python -m SimpleHTTPServer 5000
python -m SimpleHTTPServer --directory markout 5000

using nodejs #

If you are a web developer it’s likely that you need nodejs and npm on your system anyways and may want to use one of the servers from npm. I’ll admit with these not being tied to the long term support of a language they are much more feature rich with things like compression out of the box. In my opinion they are nice things that you would want out of a production server, but may not be necessary for development.

installing npx #

# if you don't alredy have npx
npm i -g npx

npx is a handy tool that lets you run command line applications straight from npm without installing them. It pulls the latest version every time you want to run, then executes it without it being installed.

running the http-server with npx #

npx http-server

# running on port 5000
npx http-server -p 5000
npx http-server markout -p 5000