Blog

Markata # [1] I open sourced the static site framework that I use to build my-blog [2] among other side projects. It’s a plugins all the way down static site generator, that makes me happy to use. {% gh_repo_list_topic “waylonwalker”, “markata” %} Repos used to build this blog # [3] my-blog [2] is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don’t have to worry about carrying around large images with my lightweight text files just to make some posts. {% gh_repo_list_topic “waylonwalker”, “personal-website” %} Kedro # [4] I am a heavy user of the kedro [5] framework, and a big advocate for using some sort of DAG framework for your data pipelines. kedro is built all in python which makes it easy for a python dev like me to extend, run, maintain, and deploy. {% gh_repo_list_topic “waylonwalker”, “kedro” %} Neovim Plugins # [6] I use vim for all of my text editing needs. It brings me joy to make any part of it just a...

After years of listening to talkpython.fm [1] I had the honor to be part of episode-337 [2] to talk about Kedro for maintainable data science. I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on. During the episode I tried my best to let Yetu and Ivan take the spotlight as the maintainer and chime in with my experience as a user of kedro. Video Version # [3] https://youtu.be/WTcjvwkXoY0 Michael made the call available on youtube as well as the audio only podcast [2] References: [1]: https://talkpython.fm/ [2]: https://talkpython.fm/episodes/show/337/kedro-for-maintainable-data-science [3]: #video-version

The community, that’s it, end of post, roll the credits. I’m a tinkerer # [1] I am a tinkerer, I am not going to run a stock desktop manager, mostly becuase that’s just not how my brain works. I need to tweak everything to fit my needs. Grantid I have not spent much time in many full fledged linux desktop environments. They are far more customizable than windows ever will be, I absolutely love that about them. Inevitibly I end up in a situation where I hit a wall, it just won’t do what I want it to do, or my lack of understanding what came wtih it holds me back. minimal # [2] I love minimal installs. I love just building up my system from the bottom up with things that I like, I understand, and that I can script. I’m a noob # [3] I spend a lot of my time in the terminal. I’d like to think I know how to use a linux command line for software development really well, but there are a lot of things that I still dont know all that well, mostly because I don’t need to. The AUR # [5]...

[1] I love using pipx for automatic virtual environment [2] management of my globally installed python cli applications, but sometimes the application is not compatible with your globally installed pipx Which version of python is pipx using?? # [3] This one took me a minute to figure out at first, please let me know if there is a better way. I am pretty certain that this is not the ideal way, but it works. My first technique was to make a package that printed out sys.version. # what version of python does the global pipx use? pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers # what version of python does the local pipx use? python -m pipx run --spec git+https://github.com/waylonwalker/pyvers pyvers Let’s setup some other versions of python with pyenv # [4] If you don’t already have pyenv [5] installed, you can follow their install instructions [6] to get it. pyenv install 3.8.13 pyenv install 3.10.5 I usually require a virtual environment # [7] I set the PIP...

https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch – Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow. During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off. - dotfiles: https://github.com/WaylonWalker/devtainer - today’s project: https://github.com/WaylonWalker/markata-todoui - website: https://waylonwalker.com/

When I am creating blog posts it’s often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content. Making Screenshots # [1] When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog. - take screenshot - optimize - conversion - publish - create img tag Created in 🐍Python # [2] I created this tool for myself in python because that is what I am most familiar with, but realistically most of what I am calling are shell scripts that I could do in just about any language. Install my screenshot tool # [3] My screenshot tool is quite hacky and not configurable, but works wonderfully for me. If you like it and want to use it, make it configurable or fork it. For now it lives on GitHub...

Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories. !!UPDATE # [1] As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update. This is a PSA # [2] I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like I do so that they know the dangers of using copier inside an existing directory. The issue # [3] The fix # [4] https://github.com/copier-org/copier/pull/273 As of the time of writing this version is still in beta, if you still want to use copier with existing directtories, I’d strongly encourage you to install the --pre release. pipx install copier --pip-args='--pre' confirm # [5] copier --version # copier 6.0.0b0 My update commit # [6] https://github.com/WaylonWalker/de...

Here are some CFP’s that I used for PyOhio 2022. https://pretalx.com/pyohio-2022/cfp Idea to blog post in minutes - Shorter # [1] Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it’s plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle. Lets build a whole blog site in 5 minutes. Add Kedro to your Pandas Workflow - Short # [2] Sometimes python scripts/notebooks take a long time to run, let kedro automatically save your datasets so that you can maintain your production code with ease. Lets take a pipeline with an issue 30 minutes in and solve the issue in 5 mintues. References: [1]: #idea-to-blog-post-in-minutes---shorter [2]: #add-kedro-to-your-pandas-workflow---short

This is not a flaky works half the time kind of plugin, it’s a seriously smooth editing experience. I’ve just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython. I can’t tell you how pumped I am for this, and how good its felt to use over the past few weeks. It’s glorious. YouTube video # [1] Listen to me rant on how great pyflyby is https://youtu.be/2QW5DJiEJH4 Give the video a watch, I did not have noise-cancelling on in obs. My apologies for the background hum and the mic stand bumps. I did my best to fix them up. Installation # [2] How to install pyflyby for automatic python imports pyflyby is hosted on pypi, so you can get it with pip. I have had no issues installing it on 3.8+ so far. pip install pyflyby Configuration setup with stow # [3] always stow your dotfiles If you’re going to configure any of your tools the first thing you should do is set it up w...

If you are running vim autocmd’s without a group, you’re killing your performance. Granted your probably not sourcing your vimscript files with autocmd’s too often, but every time you source that vimscript you are adding another command that needs to run redundantly. https://youtu.be/2ITTn4Dl0lc This is what I had # [1] Not silky smooth For WAAY too long I have had something like this in my vimrc or init.vim. It formats my python for me on every save, works great except if I source my dotfiles more than once I start adding how many times black runs. autocmd bufwritepre *.py execute 'Black' Why is a bare autocmd bad # [2] let me demonstrate Lets create a new file called format.vim and give it the :so %. Works great, it starts telling me that its formatting. autocmd bufwritepre *.py :echo("formatting with black") [3] BUT as every time I give it the :so % it formats an extra time on every single save. Setting up an augroup # [4] I’ve been told I need an augroup to prevent ...

I often review Pull requests from the browser as it just makes it so easy to see the diffs and navigate through them, but there comes a time when the diffs get really big and hard to follow. That’s when its time to bring in the comforts of vim. https://youtu.be/5NKaZFavM0E Plugins needed # [1] This all stems from the great plugin by AndrewRadev [2]. It breaks a down into a project. So rather than poping into a pager from git [3] diff, you can pipe to diffurcate and it will setup a project in a tmp directory for you and you can browse this project just like any other except it’s just a diff. Plug 'AndrewRadev/diffurcate.vim' My aliases # [4] First to quickly checkout PR’s from azure devops I have setup an alias to fuzzy select a pr and let the az command do the checkout. alias azcheckout='az repos pr checkout --id $(az repos pr list --output table | tail -n -2 | fzf | cut -d " " -f1)' Next I have a few aliases setup for checking diffs. The first one checks what is staged vs the...

I’ve gone far too long without a good setup for editing yaml files, I am missing out on autocomplete and proper diagnostics. This ends today as I setup yaml-language-server in neovim. https://youtu.be/xo4HrFoKF4c The video for this one is part of a challenge-playlist [1] I put out for myself to constantly improve my dotfiles for all of December. init.vim # [2] I have my init.vim setup to only source other modules, if you want everything in a single config, feel free to do as you wish. I broke mine up earlier this year as I doubled into nvim and am not going back. source ~/.config/nvim/plugins.vim lua require'waylonwalker.cmp' lua require'waylonwalker.lsp-config' Plugin setup # [3] You will need the following plugins. I use plug, if you don’t you will have to convert the syntax over to the plugin manager you use. neovim/nvim-lspconfig [4] is for configuring the lsp. It comes with a bunch of sane defaults for most servers, so you pretty much just have to call setup on that serv...

https://youtu.be/PQw_is7rQSw I am often in a set of tmux splits flying back and forth, accidentally close my editor, so when I come back to that split and hit my keybinds to edit files I enter them into zsh rather than into nvim like I intended. Today I am going to sand off that rough edge and get as similar behavior to nvim as I can with a couple of aliases. Make sure you check out the YouTube video to see all of my improvements. what’s an alias # [1] If you have never heard of an alias before it’s essentially a shortcut to a given command. You can pass additional flags to the underlying command and they will get passed in. Most of the time they are just shorter versions of commands that you run often or even like in this case a common muscle memory typo that occurs for you. My new alias’s for fuzzy editing files from zsh # [2] Here are the new aliases that I came up with to smooth out my workflow. These give me a similar feel to how these keys work in neovim but from zsh. #...

In honor of the neovim 0.6.0 release, I decided to do a funny skit installing neovim, and fix up my install script in the process as part of my challenge to fix up my dotfiles. I ran into one snag where I was not updating the repo that I cloned. I moved it to the directory I now keep third-party git [1] repos and set it to update with ansible. https://youtu.be/64oKLphhBuo The thing that took me the longest to realize was…. I had a path issue pointing me to an old install of the appimage over the fresh build, fixed that up and now we are on 0.7.0 nightly. Related Links # [2] https://neovim.io/ https://github.com/neovim/neovim https://github.com/neovim/neovim/releases/tag/v0.6.0 References: [1]: /glossary/git/ [2]: #related-links

https://youtu.be/Jq1Y48F_rOU I am challenging myself to 30 days of dotfile ricing. I have been on linux desktop for a few months now and have a pretty good workflow going, I have the coarse edits done to my workflow, but it has some rough edges that need sanded down. It’s time to squash some of those little annoyances that still exist in my setup. This is primarily going to be focused on productivity, but may have a few things to just look better. This will comprise heavily of aliases, zsh, and nvim config. Follow the YouTube channel [1] or the rss feed [2] to stay up to date. References: [1]: https://youtube.com/waylonwalker [2]: https://waylonwalker/rss/

Trying to read a .ipynb file without starting a jupyter server? jut has you covered. https://youtu.be/t8AvImnwor0 watch the video version of this post on YouTube [1] install # [2] jut is packaged and available on pypi so installing is as easy as pip installing it. pip install jut [3] ! This is my first time including snippets of the video in the article like this, let me know what you think! examples # [4] jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --head 3 jut https://cantera.org/examples/jupyter/thermo/flame_temperature.ipynb --tail 2 [5] what are all the commands available for jut? # [6] Take a look at the help of the jut cli to explore all the options that it offers. jut --help There is some good information on the projects readme [7] as well. [8] without installing # [9] using pipx Don’t want jut cluttering up your venv, or want to save yourself from making a ...

Hi, Hello, I’m Waylon # [1] Husband, dad of two, and hobbyist builder of things on the internet. When I’m not wrangling data pipeline platforms or building web platforms, you’ll find me gaming [2] with my kids, making art, or skating around the neighborhood. Reliving my mechanical engineering days with my 3d printer. Winding down at the end of the day binge-watching Big Bang Theory with my wife. What I Do # [3] I’m a Senior Software Engineer who specializes in data pipelines and Python-based web platforms. I help teams turn messy data into reliable systems that actually work. Why I Built This Site # [4] from scratch I got tired of: - Build times that took forever - Node modules folders that became black holes - Bloated pages that took ages to load - SEO tools that felt like an afterthought So I built my own platform from scratch using pluggy and diskcache. It’s under-funded, over-dreamed, barely documented, and I love it. This site is my sandbox for learning, teaching, a...

I use kedro catalog create to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key. https://youtu.be/_22ELT4kja4 What is Kedro [1] 👆 Unsure what kedro is? Check out this post. Running Kedro Catalog Create # [2] The command to ensure there are catalog entries for every dataset in the passed in pipeline. kedro catalog create --pipeline history_nodes - Create’s new yaml file, if needed - Fills in new dataset entries with the default dataset - Keeps existing datasets untouched - it will reformat your yaml file a bit - default sorting will be applied - empty newlines will be removed CONF_ROOT # [3] Kedro will respect your CONF_ROOT settings when it creates a new catalog file, or looks for existing catalog files. You can change the location of your configuration f...

https://youtu.be/E18m4KkJUnI --- Slides 👇 # [1] welcome # [2] Other possible titles # [3] - Using Vim as a Team Lead - I 💜 Tmux - Why I stopped using @code - Get there fast - How I vim It’s ok # [4] Use a graphical IDE if it works for you. Trick it out # [5] vim is so well integrated into the terminal, take advantage It wasn’t working for me anymore # [6] dozens of instances # [7] As a team lead I bounce betweeen a dozen projects a per day https://pbs.twimg.com/media/FAEmRjYUcAUk2eR?format=jpg&name=large [8] Move With Intent # [9] Running vim inside tmux lets me move swiftly between the exact project I need. https://twitter.com/_WaylonWalker/status/1438849269407047686/photo/1// [10]: <> (__) Hub and Spoke # [11] - direct link to specific projects - fuzzy into all projects - fuzzy into open projects How I navigate tmux in 2021 [12]#hub-and-spoke Other Things That Make this Possible # [13] - tmux - direnv vim adjacent things yes, vim is ugly, make it your...

It happens to the best of us # [1] We can’t all remember every single function signature out there, it’s just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal. https://youtu.be/TZrRAP-9UMk Docstrings # [2] In any python repl you can access the docstring of a function by calling for help. help(df.rolling) In Ipython we can even get some syntax highlighting with the ?. df.rolling? Source Code # [3] Sometimes the docstrings are not good enough, and don’t give us the content we need, and we just need to look at the source. Without leaving your terminal there are two ways I often use to get to the source of a function I am trying to use. import inspect inspect.getsource(df.rolling) The more common way I do it is with the ipython ??. df.rolling?? Bonus rich.inspect # [4] You thought the syntax highlighting was good with ipython, check out w...