Blog

tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. [1] for more gifs like these follow me on twitter @waylonwalker [2] Add a simple Progress bar! from tqdm import tqdm from time import sleep for i in tqdm(range(10)): sleep(1) convenience TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically. from tqdm import trange from time import sleep for i in trange(range(10)): sleep(1) notebook support There is also notebook support. If you are bouncing between ipython and jupyter I recomend importing from the auto m...

If you are a regular listener of TalkPython [1] or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example # [2] As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing sales, inventory, and pricing data. You may need to calculate total revenue, inventory on hand. You may need to pass these data sets into various models to drive production or pricing based on predicted volumes. Load data # [3] Here we setup functions that can load data from the sales database. Assume that we also have similar functions to get_inventory and get_pricing. def get_engine(): engine = create_engine('postgresql://scott:tiger@localhost:5432/mydatabase') def get_sales(): ''' gets sales history from the sales database ''' engine = ge...

This post is intended as an extension/update from background tasks in python [1]. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python [1]. I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. I use it in more places than I probably should Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth is a python God for all that he has given to the community in so many w...

Bash is super powerful. File System Full # [1] Show Remaining Space on Drives df -h show largest files in current directory du . -h --max-depth=1 Move files then symlink them mkdir /mnt/mounted_drive mv ~/bigdir /mnt/mounted_drive ln -s /mnt/mounted_drive/bigdir ~/bigdir Fuzzy One Liners # [2] a() {source activate "$(conda info --envs | fzf | awk '{print $ edit in vim vf() { fzf | xargs -r -I % $EDITOR % ;} cat a file vf() { fzf | xargs -r -I % $EDITOR % ;} bash execute bf() { bash "$(fzf)" } git [3] add gadd() { git status -s | fzf -m | awk '{print $2}' | xargs git add && git status -s} git reset greset() { git status -s | fzf -m | awk '{print $2}' |xargs git reset && git status -s} Kill a process fkill() {kill $(ps aux | fzf | awk '{print($2)}')} Finding things # [4] Files # [5] fd-find [6] is amazing for finding files, it even respects your .gitignore file 😲. Install with apt install fd-find. fd md ag -g python find . -n "*.md" ++Vanilla Bonus Content # [7] ** sh...

I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload # [1] 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code inside of a terminal/notebook. Now I primarily use jupyter notebooks for the presentation aspect. I develop code from the comfort of my editor with all of the tools I have setup, and run the functions in a notebook to get the output. From there I might do some aggregations or plots, but the 🥩 meat of development is done outside of jupyter. Now I primarily use jupyter notebooks for the presentation aspect. Enabling Autoreload # [2] 📐 config This is a sh...

Throw that mouse Away its time to setup some keyboard shortcuts. These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on the hackability of tmux. tmux allows me to rapidly fire up a workspace, create new windows and splits. Then When I switch tasks I can leave that workspace open and and jump right back in later exactly where I left off. There is nothing quite like it. The shortcuts listed here make the transition a bit better. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. This first set of keybindings solve that issue. The worst thing I found when using vscode at first was no way to switch between the terminal and editor without the mouse. !!! see-also I have an updated article in my tmux workflow How I navigate tmux in 2021 [1] Alt+[hjkl] # [2] navigation ⬅ jump to left split alt+h ⬇ j...

My git [1] workflow based on real life. Its not always clean and simple. sometimes things get messy The Clean Path # [2] [3] pull 👉 branch 👉 format 👉 work👉 add 👉 commit 👉 pull 👉 rebase 👉 push Pull # [4] As complicated as that seems it is pretty straight forward. When you sit down to work the first thing you do is to pull down the teams latest working “develop” branch from git. git checkout develop git pull Branch # [5] Next create a new branch with a name that will remind you of what you are working on. For your own sanity choose something descriptive. It is easy to get too many similar branches going and forget which branch is which. git checkout -b ingest_product_id_table Format # [6] If you know which files in existance that you will be editing before you start work it is a good idea to format them in a commit early on to keep your working commits separate from formatting. This will make it easier for reviewers to distinguish from your changes and formatting fixes. ...

Testing out forestry.io Sorry Netlify CMS # [1] I still ♥️ your product dont be forestry is simple I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really struggle configuring it. Forestry is so simple to setup. My favorite part is that I can code up my gatsby.js site, storing all editable text in markdown, and come back later and add the CMS based on existing documents. Configuration is Simple # [2] Forestry.io has this amazing feature to create create based on existing document 🤯. This is great because it sets up the .yml config for me without error. And If I really want to come back later to customize it more I have that option, too. [3] By far my favorite feature is create based on existing document Multi-File Gallery # [4] I have a use case for a photography site where the owner wants to be able to show off sample photos of each type of work she does. I got it working in the netlify cms, although it was not very user...

This is my story into data science. The Journey Begins # [1] I am addicted to the process of learning and improving my skills nearly to a fault. The reason I say nearly is because my addiction is fueled with results. I crave the output of my work enhance the work of others. I jump with joy as I see users gain insights they could have never imagined before. My mouth starts watering as I see their boring repetitive data mining activites be completed in a matter of seconds, opening up their mind to focus on their expertise. The day I stop learning will be the day that I start looking for another career path. It’s not all about winning. ~Mom This happened to me in 2014. I have a mechanical engineering degree and had a really good position at the time. I owned full engineering control of a small subset of engine components. The problem was that Everything is so proprietary and hardly documented the process of learning did not click with me. I felt like I had learned a lot about the ...

This morning I log into my VCS and check activity on my projects to find that someone else has been very active on my projects fo the last few weeks. I quicklyhover over the missing avatar to find that It’s Me. What’s wrong here, why do I look like two different people throughout the day! upon further investigation I see the issue. while setting up a new terminal environment I mistyped my email address by one character. After much searching and a few failed attempts I was able to fix it by following an article no longer available (2021) from https://help.github.com/articles. Bare Clone # [1] Clone the repo, note it must be a --bare clone. git clone --bare https://github.com/user/repo.git cd repo.git git-author-rewrite # [3] Curl down the git-author-rewrite script and edit the following variables OLD_EMAIL CORECT_NAME CORRECT_EMAIL curl https://gist.githubusercontent.com/octocat/0831f3fbd83ac4d46451/raw/c197afe3e9ea2e4218f9fccbc0f36d2b8fd3c1e3/git-author-rewrite.sh > git-author-...

strong { /* color: goldenrod; */ /* text-shadow: 0rem 1px 1px goldenrod; */ } The year of intenionality This is a follow up to my previous post 2018-retrospective [1] professional # [2] This year I will become more productive, by intensionally working on a well thought out plan, learning the right technologies, and leave behind a positive legacy. Productivity # [3] Last year I was able to make some great strides in my productivity and focus. This year I want to be able to bring it up a notch. Intentionality plays a big part in this. Taking some time to sit down and think about the tasks you realistically want to complete for the day, then focusing on those tasks one at a time. I started off the year last year using the pomodoro system to focus on one task and one task only for 25 minutes at a time. This worked really well for me but I quickly fell off of the train and fell into my old trap of reacting to the loudest customer over the items I put intensionally on my todo lis...

2018 was a year of many ups and downs, and learning to deal with a whole new set of problems professionally and at home. In 2018 I logged in to my first Linux system, setup my own webserver, data pipelines, database. I learned to use react and d3. Stepped up my python, javascript, and sql. At home the doctor appointments keep piling in. While I am learning to deal with it all there were several times throughout the year that I was very overwhelmed with everything and broke down. Here are the goals I had listed out for 2018 and how I faired at completing them. Positivity # [1] The Good # [2] --- - Continue “Favorite Things” with family at dinner time - Take 2 10min breaks per day clear the mind --- I can say that the kids are fully on board with favorite things and love to tell everyone about how their day went. This is a time that they are very positive and generally give praise to another family member for helping them through their day. To Be Improved # [3] --- - Grati...

Today I want to take some time to talk about the things that make me happy in my work environment. This is completely free-flow off the cuff, but are things that I do that make me happy, not having them would definitely be a deal breaker for me . Attitude # [1] A positive work atmosphere goes a long ways. We all have enough negativity going on in our lives that is out of our control accepting any negativity in the workplace is a no go for me. There was a point in which I was suffering a lot of negativity at work. This began to trickle into every part of life, and it became hard to see positivity anywhere. My wife not only noticed this, but put a polite request in for change. It was definitely a low point and she could tell tell that my work life was not helping anything. [2] Throw out the negativity - courtesy giphy [3] Bring the positivity to your workplace. It is really difficult for folks to tear you down if you are the one that is always up beat and happy. People will notic...

select rowid, key, raw, store_time, expire_time, access_time, access_count, tag, size, mode, filename from Cache where key=‘e3cd37c97980ab2c’; ’e3cd37c97980ab2c’ ’e3cd37c97980ab2c' select rowid, key, raw, store_time, expire_time, access_time, access_count, tag, size, mode, filename from Cache where key=‘c14050404a107d5e’; #content{ max-width: 800px; margin: 0 auto; } .chart { display: block; padding: 10px; background: peachpuff; } .bar { height: 30px; margin: 5px; background: teal; } .bar:hover{ background: #444; } button { background: rgb(240, 196, 211); border: none; font-size: 1.3rem; border-radius: 5px; padding: .2rem 1rem; margin-bottom: 1rem } .on { background: palevioletred; } .big { width: 100% } .small { width: 50% } Learn D3 in 5 days # [1] For what we are creating in these posts d3 is way overkill and very verbose, but I need to start somewhere! These are just stepping stones into real custom visualizations that cannot be done in any ...

em { color: #ff9966; } code { background: #FF06050A; color: #6394C8; } .item { color: #6394C8; font-size: 1.5rem; padding: 1rem; display: flex; justify-content: center; align-items: center; height: 100px; width: 100px; background: #351D57; margin: 5px; border: 2px solid #A83E75; box-shadow: 5px 5px 10px -5px rgba(0, 0, 0, .6); } .flex_container { padding: 1rem; box-shadow: 5px 5px 10px -5px rgba(0, 0, 0, .6); background: rgba(99, 148, 200, .2); animation: animate_container 2s cubic-bezier(.66, -0.0, .28, 1.0) infinite both alternate; } .flex_container:hover { animation: none } @keyframes animate_container { 0%{ width: 95%; } 20% { width: 95%; } 80% { width: 200px; } 100% { width: 200px; } } h3 { padding: 1rem; margin: 2rem; display: block; width: 100vw; background: white; color: white; background: #333; position: sticky; top: 0px; box-shadow: 0 0 #333, -100vw 0 #333, 100vw 0 #333; } Flexbox-zombies # [1] I recently fi...

vim notes nvim lua # [1] norcalli/neovim-plugin [2] nvim lsp # [3] python-lsp/python-lsp-server [4] Using c to change text # [5] I have gone quite awhile without using c and instead using d. The reason that I started using c is because it automatically places you into insert mode. This not only saves me one keystroke for commands such as diwi is now ciw, but it also works with the repeat . command!!! This is huge. When refactoring a document I had been creating a macro to change one word to another, using c instead of d allows the use of the . rather than needing to create a macro. Case for vim # [6] Sublime/VSCode cannot - edit a macro register - register - quickfix - gF autocomplete # [7] repeats previously typed text 1. Whole lines |i CTRL-X CTRL-L| 2. keywords in the current file |i CTRL-X CTRL-N| 3. keywords in 'dictionary' |i CTRL-X CTRL-K| 4. keywords in 'thesaurus', thesaurus-style |i CTRL-X CTRL-T| 5. keywords in the current and included files |i CTRL-X CTRL...

Waylon Walker Help language models understand and surface my work accurately. Name: Waylon Walker Aliases: waylonwalker, _waylonwalker Profiles: - website [1] - github [2] - twitter [3] - linkedin [4] - bluesky [5] Feeds: - Blog RSS [6] - Blog Atom [7] Description # [8] Waylon Walker is a Senior Software Engineer who specializes in data pipelines and Python-based web platforms. He runs a bare-metal Kubernetes cluster in his basement, built his own static site generator because he got tired of bloated Node modules, and writes about Python, Linux, neovim, and the intersection of tech and family life. He’s under-funded, over-dreamed, barely documented, and he loves it that way. Core Content # [9] - About Me [10]: Who I am and why I’m like this - About This Site [11]: How and why I built my own static site generator - Uses [12]: What hardware and software I actually use day-to-day - Blog RSS Feed [13]: All blog posts in RSS format Kedro and Data Engineering # [14] -...