This morning I logged into my machine and was nearly out of space
- 64GB miniconda3!
- 5GB conda cache
- 4GM pip cache
- 34GB docker
These are the commands that I often use to reclaim space. Its so easy to fill up small vm's in the cloud, or in my case today let your dev machine go way too long without a good cleanup.
Show Remaining Space on Drives
This shows us where to start and gives a baseline of how much space we have reclaimed.
show largest files in current directory
Next keep drilling into directories that are big and running this command to see whats big inside of it. When you find somethign that you are willing to part with
rm -rf <directory> it and check
df -h to see if you have enough reclaimed yet.
du . -h --max-depth=1
Honestly I rarely bother unless the directory is in the GB's of space. A super simple filter for that is to just grep for G.
du . -h --max-depth=1 | grep G
As a first baseline lets see how many enviroments we are starting with. I started with 71. Yeah I have had this machine for 2 years, and dont regularly remove them.
conda info --envs | tail -n +2 | wc -l
Lets batch it out!
We are devs here surely we can automate this issue! The following four lines will generate a list of existing conda environments, edit them with vim, remove the remaining ones, then remove the text file we created to remove from.
Make sure that you only keep names of environments that you want to remove in
conda_envs_to_remove.txt and delete the environment names you want to keep.
conda info --envs | tail -n +2 | cut -d ' ' -f1 > conda_envs_to_remove.txt vim conda_envs_to_remove.txt cat ~/.conda_envs_remove | tr '\n' '\0' | xargs -l -0 conda remove --all -y -n rm conda_envs_to_remove.txt
📝 Side note
When I am creating one of these complicated bash pipelines including xargs I generally print out the command first and make sure that it does what I want. The following command will test the above script before doing dangerous things!
cat ~/.conda_envs_remove | tr '\n' '\0' | xargs -l -0 echo "conda remove --all -y -n "
If your feeling really strained for space, you can
rm -rf ~/.cache. Personally I like the improved speed of installing everything... obviously I install a lot of new environments.
For more information read this article, https://docs.docker.com/config/pruning/. I have all of the images that I want pushed remotely so I just dumped everything with the following command.
docker system prune docker system prune --volumes
Running these two sets of commands cleared up about 70GB of space for me with very little effort on my behalf. I hope others find the first command helpful to batch remove many conda environments at once.