All posts with the tag "bash"
This morning I had a machine crash on me and came back to an error.
Error
zsh: corrupt history file /home/u_walkews/.zsh_history
Dammit I don’t want to redo my shell history, I checked with a clanker and they
came up with this solution using strings that only prints printable
characters.
cp ~/.zsh_history ~/.zsh_history.bak
mv ~/.zsh_history ~/.zsh_history.corrupt
touch ~/.zsh_history
chmod 600 ~/.zsh_history
strings ~/.zsh_history.corrupt > ~/.zsh_history
chmod 600 ~/.zsh_history
I’ve been deploying my site old school for most of this year, rsync to a volume mounted to nginx. I ran into an issue today where I updated my site and all of the pages updated first, followed by upload. The issue this created was that the new cache busted css files were not up yet and the site had no styles for a brief period during upload.
I found that delaying updates and delaying deletes until the new content exists first solves this problem pretty well. Theres still possiblility of jank while uploading to a live directory and not doing some sort of hot swap, but I’m good with this low budget option for now.
sync:
rsync -rlt --delete --omit-dir-times \
--info=progress2 \
--delay-updates \
--delete-delay \
./output/ \
server:/mnt/mysite
To ignore commands that start with a space character, use the
HIST_IGNORE_SPACE option in bash or zsh.
setopt HIST_IGNORE_SPACE
setting COLUMNS env var to a number greater than 0 will make the terminal resize to that number of columns.
COLUMNS=80 uvx --from rich-cli rich myscript.py
Note
Not all programs respct the COLUMNS env var, but rich does, and a lot of
stuff I’m building uses rich.
I discovered this when I was trying to make a low effort readme generated from the code, but did not depend on the size of terminal it was ran on.
# justfile
readme:
echo "# Workspaces" > README.md
echo "" >> README.md
echo '``` bash' >> README.md
COLUMNS=80 ./workspaces.py --help >> README.md
echo '```' >> README.md
Today I learned how to use tar over ssh to save hours in file transfers. I keep all of my projects in ~/git (very creative I know, I’ve done it for years and haven’t changed). I just swapped out my main desktop from bazzite to hyprland, and wanted to get all of my projects back. Before killing my bazzite install I moved everything over (16GB of many small files), it took over 14 hours, maybe longer. I had started in the morning and just let it churn.
This was not going to happen for re-seeding all of my projects on my new system, I knew there had to be a better way, I looked at rsync, but for seeding I ran into this tar over ssh technique and it only took me 6m51s to pull all of my projects off of my remote server.
ssh [email protected] 'tar -C /tank/git -cpf - .' \
| tar -C "$HOME/git" -xpf -
I am a linux user through and through. Desktop, server, vms, containers,
everything except my phone is linux. With this I spend a lot of time in the
terminal, and have been a long time user of !! to rerun the last command, but
with the ability to tack something on at the beginning or end.
TIL about fc, which opens the last command in your shell history in your
$EDITOR or pass in your editor -e nvim.
!! pronounces bang bang and will run the last command in your history.
ls -l
!! | wc -l
# ls -l | wc -l
sudo !!
# sudo ls -l | wc -l
!!:s/-l/-l \/tmp
# sudo ls -l /tmp | wc -l
fc enters the chat #Now making complex edits in your shell can be a bit of a chore, so fc moves
this work to your $EDITOR.
fc
This pops open your $EDITOR with the last command in your history.
sudo ls -l | wc -l
fc shows up in shell history, but !! does not, !! gets replaced by the
command that it becomes.
yaya yaya, I know you can also up-arrow c-e, but what fun is that, it’s barely a
flex. fc just looks big brained and like you really know what you are doing.
You can unset multiple environment variables at once. I did not know this was a
thing, its something that ended up happening organically on a call and asking
someone to run unset. They had never done it before and did not know how it
works, but did exactly as I said instead of what I meant. I like this handy
shortcut doing it in one line rather than each one individually, I will be
using this in the future. You might need this for something like
running aws cli commands with localstack.
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION
Today I discovered the Urllink function in bash from the ujust tool from
ublue.it. Seems like a cool trick, but might not work
everywhere.
########
### Special text formating
########
## Function to generate a clickable link, you can call this using
# url=$(Urllink "https://ublue.it" "Visit the ublue website")
# echo "${url}"
function Urllink (){
URL=$1
TEXT=$2
# Generate a clickable hyperlink
printf "\e]8;;%s\e\\%s\e]8;;\e\\" "$URL" "$TEXT${n}"
}
```j
I’ve had a couple of uploads to twitter fail recently and has been a pain. I tried some online converters for convenience, but none of them worked. I reached out to chatgpt and found succeess with this ffmpeg command.
ffmpeg -i input.mp4 \
-vf "scale=trunc(oh*a/2)*2:min(720\,trunc(ih*a/2)*2)" \
-c:v libx264 -profile:v high -level:v 4.1 \
-b:v 3500k -maxrate 3500k -bufsize 7000k \
-pix_fmt yuv420p \
-c:a aac -b:a 128k -ar 44100 \
-movflags +faststart \
output.mp4
Authentication from cli tools can be a bit of a bear, and I have to look it up every time. This is my reference guide for future me to remember how to easily do it.
I set up a fastapi server running on port 8000, it uses a basic auth with
waylonwalker as the username and asdf as the password. The server follows
along with what comes out of the docs. I have it setup to take basic auth,
form username and password, or a bearer token for authentication.
The og of command line url tools.
# basic auth
curl -u 'waylonwalker:asdf' -X POST localhost:8000/token
# basic auth with password prompt
curl -u 'waylonwalker' -X POST localhost:8000/token
# token
curl -H 'Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg' -X POST localhost:8000/token
My go to if I want the result to go into a file.
# basic auth
wget -q -O - --auth-no-challenge --http-user=waylonwalker --http-password=asdf --post-data '' localhost:8000/token
# token
wget -q -O - --header="Authorization: bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg" -O - --post-data '' localhost:8000/token
An http client written in python, primarilty used with the python api, but has a nice cli.
# install
python3 -m pip install httpx
# basic auth
httpx -m POST --auth waylonwalker asdf http://localhost:8000/token
# basic auth with password prompt
httpx -m POST --auth waylonwalker - http://localhost:8000/token
# token
httpx -m POST --headers="Authorization" "bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg" http://localhost:8000/token
A modern http client written in python.
# install
python3 -m pip install httpie
# basic auth
http POST localhost:8000/token -a waylonwalker:asdf
# basic auth with password prompt
http POST localhost:8000/token -a waylonwalker
# token
http POST localhost:8000/token -A bearer -a eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ3YXlsb253YWxrZXIiLCJleHAiOjE3MDI5NTI2MDJ9.GeYNt7DNal6LTiPoavJnqypaMt4vYeriXdq5lqu1ILg
# install
python3 -m pip install httpie-credential-store
# usage
http POST localhost:8000/token -A creds
http-prompt comes from the httpie org, and has an interactive cli interface
into apis. You can even specify a spec file to autocomplete on api methods.
http-prompt localhost:8000 --auth waylonwalker:asdf --spec openapi.json
jpillora/installer is the install script generator I have been looking for. It downloads binaries for your machine from GitHub releases and unzips them for you. It grabs the latest release, so you can easily update them. I have tried scripting these installs in the past and struggled to consistently get the latest version for every package and unpack it correctly.
Also these pre-compiled binaries install rediculously fast compared to building them from source.
Check out some example links.
opening in a browser will show metadata
If you pass in script=true it will instead return the install script as it would by default through curl.
https://i.jpillora.com/serve?script=true
All you need to do to generate an install script is to pass in the GitHub repo slug with the org.
curl https://i.jpillora.com/neovim/neovim | bash
The shell script that it generates for neovim looks like this.
#!/bin/bash
if [ "$DEBUG" == "1" ]; then
set -x
fi
TMP_DIR=$(mktemp -d -t jpillora-installer-XXXXXXXXXX)
function cleanup {
rm -rf $TMP_DIR > /dev/null
}
function fail {
cleanup
msg=$1
echo "============"
echo "Error: $msg" 1>&2
exit 1
}
function install {
#settings
USER="neovim"
PROG="neovim"
ASPROG=""
MOVE="false"
RELEASE="stable"
INSECURE="false"
OUT_DIR="$(pwd)"
GH="https://github.com"
#bash check
[ ! "$BASH_VERSION" ] && fail "Please use bash instead"
[ ! -d $OUT_DIR ] && fail "output directory missing: $OUT_DIR"
#dependency check, assume we are a standard POISX machine
which find > /dev/null || fail "find not installed"
which xargs > /dev/null || fail "xargs not installed"
which sort > /dev/null || fail "sort not installed"
which tail > /dev/null || fail "tail not installed"
which cut > /dev/null || fail "cut not installed"
which du > /dev/null || fail "du not installed"
#choose an HTTP client
GET=""
if which curl > /dev/null; then
GET="curl"
if [[ $INSECURE = "true" ]]; then GET="$GET --insecure"; fi
GET="$GET --fail -# -L"
elif which wget > /dev/null; then
GET="wget"
if [[ $INSECURE = "true" ]]; then GET="$GET --no-check-certificate"; fi
GET="$GET -qO-"
else
fail "neither wget/curl are installed"
fi
#debug HTTP
if [ "$DEBUG" == "1" ]; then
GET="$GET -v"
fi
#optional auth to install from private repos
#NOTE: this also needs to be set on your instance of installer
AUTH="${GITHUB_TOKEN}"
if [ ! -z "$AUTH" ]; then
GET="$GET -H 'Authorization: $AUTH'"
fi
#find OS #TODO BSDs and other posixs
case `uname -s` in
Darwin) OS="darwin";;
Linux) OS="linux";;
*) fail "unknown os: $(uname -s)";;
esac
#find ARCH
if uname -m | grep -E '(arm|arch)64' > /dev/null; then
ARCH="arm64"
# no m1 assets. if on mac arm64, rosetta allows fallback to amd64
if [[ $OS = "darwin" ]]; then
ARCH="amd64"
fi
elif uname -m | grep 64 > /dev/null; then
ARCH="amd64"
elif uname -m | grep arm > /dev/null; then
ARCH="arm" #TODO armv6/v7
elif uname -m | grep 386 > /dev/null; then
ARCH="386"
else
fail "unknown arch: $(uname -m)"
fi
#choose from asset list
URL=""
FTYPE=""
case "${OS}_${ARCH}" in
"linux_amd64")
URL="https://github.com/neovim/neovim/releases/download/stable/nvim-linux64.tar.gz"
FTYPE=".tar.gz"
;;
"darwin_amd64")
URL="https://github.com/neovim/neovim/releases/download/stable/nvim-macos.tar.gz"
FTYPE=".tar.gz"
;;
*) fail "No asset for platform ${OS}-${ARCH}";;
esac
#got URL! download it...
echo -n "Downloading"
echo -n " $USER/$PROG"
if [ ! -z "$RELEASE" ]; then
echo -n " $RELEASE"
fi
if [ ! -z "$ASPROG" ]; then
echo -n " as $ASPROG"
fi
echo -n " (${OS}/${ARCH})"
echo "....."
#enter tempdir
mkdir -p $TMP_DIR
cd $TMP_DIR
if [[ $FTYPE = ".gz" ]]; then
which gzip > /dev/null || fail "gzip is not installed"
bash -c "$GET $URL" | gzip -d - > $PROG || fail "download failed"
elif [[ $FTYPE = ".tar.bz" ]] || [[ $FTYPE = ".tar.bz2" ]]; then
which tar > /dev/null || fail "tar is not installed"
which bzip2 > /dev/null || fail "bzip2 is not installed"
bash -c "$GET $URL" | tar jxf - || fail "download failed"
elif [[ $FTYPE = ".tar.gz" ]] || [[ $FTYPE = ".tgz" ]]; then
which tar > /dev/null || fail "tar is not installed"
which gzip > /dev/null || fail "gzip is not installed"
bash -c "$GET $URL" | tar zxf - || fail "download failed"
elif [[ $FTYPE = ".zip" ]]; then
which unzip > /dev/null || fail "unzip is not installed"
bash -c "$GET $URL" > tmp.zip || fail "download failed"
unzip -o -qq tmp.zip || fail "unzip failed"
rm tmp.zip || fail "cleanup failed"
elif [[ $FTYPE = ".bin" ]]; then
bash -c "$GET $URL" > "neovim_${OS}_${ARCH}" || fail "download failed"
else
fail "unknown file type: $FTYPE"
fi
#search subtree largest file (bin)
TMP_BIN=$(find . -type f | xargs du | sort -n | tail -n 1 | cut -f 2)
if [ ! -f "$TMP_BIN" ]; then
fail "could not find find binary (largest file)"
fi
#ensure its larger than 1MB
#TODO linux=elf/darwin=macho file detection?
if [[ $(du -m $TMP_BIN | cut -f1) -lt 1 ]]; then
fail "no binary found ($TMP_BIN is not larger than 1MB)"
fi
#move into PATH or cwd
chmod +x $TMP_BIN || fail "chmod +x failed"
DEST="$OUT_DIR/$PROG"
if [ ! -z "$ASPROG" ]; then
DEST="$OUT_DIR/$ASPROG"
fi
#move without sudo
OUT=$(mv $TMP_BIN $DEST 2>&1)
STATUS=$?
# failed and string contains "Permission denied"
if [ $STATUS -ne 0 ]; then
if [[ $OUT =~ "Permission denied" ]]; then
echo "mv with sudo..."
sudo mv $TMP_BIN $DEST || fail "sudo mv failed"
else
fail "mv failed ($OUT)"
fi
fi
echo "Downloaded to $DEST"
#done
cleanup
}
install
I’d reccomend self hosting your own. This way you know that it’s consistent and unlikely to change in a way that breaks your use.
curl -s https://i.jpillora.com/installer | bash
Here are the repos I am using installer for.
atuinsh/atuin
benbjohnson/litestream
bootandy/dust
BurntSushi/ripgrep
chmln/sd
cjbassi/ytop
dalance/procs
dbrgn/tealdeer
ducaale/xh
go-task/task
imsnif/bandwhich
imsnif/diskonaut
kovidgoyal/kitty
mgdm/htmlq
neovim/neovim
ogham/dog
ogham/exa
pemistahl/grex
sharkdp/bat
sharkdp/fd
sharkdp/pastel
sirwart/ripsecrets
starship/starship
topgrade-rs/topgrade
zellij-org/zellij
Give github actions the -e flag in the shebang #! so they fail on any one
command failure. Otherwise each line will set the exit status, but only the
last one will be passed to ci.
#!/bin/bash -e
The -e flag to the bash command allows your script to exit immediately if any command within the script returns a non-zero exit status. This can be useful for ensuring that your script exits with an error if any of the commands it runs fail, which can help you identify and debug issues in your script. For example, if you have a script that runs several commands and one of those commands fails, the script will continue running without the -e flag, but will exit immediately if the -e flag is present. This can make it easier to troubleshoot your script and ensure that it runs correctly.
In windows the solution is not quite as simple. You can define a function in a Windows batch script that wraps an if statement to check the exit status of a command and handle any errors that may have occurred. Here is an example of how you might define a function called “check_error” that does this:
:check_error
if errorlevel 1 (
echo An error occurred!
exit /b 1
)
To use this function in your script, you would simply call it after running a command, like this:
some_command
call :check_error
This would run the “some_command” and then call the “check_error” function to check the exit status and handle any errors that may have occurred. This approach allows you to reuse the error-checking logic in your script, which can make it easier to write and maintain.