In 2021 I changed the way I navigate between tmux sessions big time. Now I can create, kill, switch with ease, and generally keep work separated into logical groups. I took Chris's tmux course in December 2020 and i
2021-5-11
When a python module is called it is assigned the __name__ of __main__ otherwise if it's imported it will be assigned the __name__ of the module. Let's create a module to
2021-5-7
Creating a directory that is a union of several directories can be achieved with a few symlinks at the command line. Here is how I am creating a virtual directory of all my projects that is a combination of both work and not-work projects.
2021-5-5
I really like using conda ( miniconda ) as my python virtual environment manager of choice. It's simple and it includes its own python interpreter using the version that I specify at creation. from their
2021-4-21
I've long used neovim from within windows wsl, and for far too long, I went without a proper way to get text out of it and into windows. wsl can access clip.exe. You can do some cool things with it, such as cat a file into the clipboard,
2021-4-17
I am trying to see what an embeded replit
2021-4-15
Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive
2021-4-5
Draft Post I've really been digging @willmcgugan's rich library for creating TUI like interfaces in python. I've only recently started to take full advantage of it. I am working on a projec
2021-3-27
With the realease of kedro==0.17.2 came a new module in the project template pipeline_registry.py . Here are some notes that I learned while playing with this new module. You should now have s
2021-3-20
People who are quick to toss team members under a bus are not well trusted or highly thought of and it will lead to some toxic team dynamics. While collaborating on any project there are going to be decisions made that aren't necessarily y
2021-1-22
I create this blog with one person in mind, me. This is not completely selfish, as there are likely many others out there that think similarly to me. Everyone comes from different backgrounds and varying levels of experience. In no way d
2021-1-21
How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in setup.py and 27 in mini_kedro_pipeline.py .
2021-1-20
This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown.
2021-1-20
I am making another push in 2021 to get my content out in the world and meeting users where they are. See how I plan to execute. My content is written in markdown, all markdown. I find that markdown does a really great job at getting out
2021-1-19
Recently I automated starting new posts with a python script. Today I want to work on the next part that is editing those posts quickly. https://waylonwalker.com/automating-my-post-starter Check out this post about setting
2021-1-18
Gitui is a terminal-based git user interface (TUI) that will change the way that you work with git. I have been a long-time user of the git cli, and it's been hard to beat, mostly because there is nothing that keeps my fingers on the keybo
2021-1-17
Gitui is a terminal-based git user interface (TUI) that will change the way that you work with git. I have been a long-time user of the git cli, and it's been hard to beat, mostly because there is nothing that keeps my fingers on the keyboard quite l
2021-1-17
In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. These containers for data contain many convenient methods to manipulate table like
2021-1-14
Changing conda environments is a bit verbose, I use a function with fzf that both lists environments and selects the one I want in one go. I have used conda as a virtual environment tool for years now. I started using conda for its simpli
2021-1-11
Replacing text based on whats in the current search register is a quite handy tool that I use often. I believe I picked this tip up from Nick Janetakis, check out his YouTube channel for some amazing vim tips. https://www.youtube.com/wa
2021-1-10
What does it take to create an installable python package that can be hosted on pypi? This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in on
2021-1-10
I do much of my work from tmux, I love it so much that I want to setup some functionality that puts me in tmux even if I didnt ask for it. Bash function to check if the shell is in a tmux session. I often open up vim to do some
2021-1-9
If you are like me, you have created a macro or two that is pure glory, and you forget how you made it after a day or so, or you immediately want to store it away as a custom keybinding. As with most things with vim, it's easy to do once you understa
2021-1-4
Replacing text in vim can be quite frustrating especially since it doesn't have live feedback to what is changing. Today I was watching Josh Branchaud's Vim-Unalphabet series on Youtuve and realized that his vim was doing this and I had to have it.
2021-1-4
Web browsers are a black hole of productivity. I try to use them as little as possible when it is time to focus. I try to use help , ? , or ?? with ipython, or --help at the comma
2021-1-2
As projects grow patterns that worked early on break and we need to change things to make the project easier to work with, and more welcoming to new developers. Before you start mucking up a project with wild commands at the terminal check
2020-12-30
I use my ipython terminal daily. It's my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration. Activate your virtual environment of choice and pip
2020-12-20
I've grown tired of the standard ipython prompt as it doesn't do much to give me any useful information. The default one gives out a line number that only seems to add anxiety as I am working on a simple problem and see that number grow to several h
2020-12-20
One thing we all dread is mundane work of getting started, and all the hoops it takes to get going. This year I want to post more often and I am taking some steps towards making it easier for myself to just get started. When I start a new
2020-12-11
In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools and more-itertools` are full o
2020-12-10
This is episode 1 of the Waylon Walker Audio experience, posts from waylonwalker.com in audio form. So I have had this idea for awhile to add audio to my blog posts. The idea partly comes from the aws blog, if you h
2020-11-25
Inspired by discourse's link expansion I am rolling out expansions for one line links on the blog waylonwalker . I was able to find a gatsby plugin gatsby-remark-embedder that expands one line links for
2020-11-18
I wanted a super simple way to cross-link blog posts that require as little effort as possible, yet still looks good in vanilla markdown in GitHub. I have been using a snippet that puts HTML into the markdown. While this works, it's more manual/dif
2020-11-18
2020-11-12
2020-11-6
use this profile image Please share your professional role as a data scientist? [Also feel free to share about your personal projects, publications, etc.] I graduated with a Mechanical Engineering Degree 8 years ago. Much of
2020-11-2
There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefi
2020-11-1
2020-10-31
STOP LEAVING Browser Tabs open and save them here!
2020-10-28
Kedro 0.16.6 is out! Let's take a look through the release notes This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of the
2020-10-25
I started making stories as kind of a brain dump a few times per day and posting them to [LinkedIn]( https://www.linkedin.com/in/waylonwalker/(https://www.linkedin.com/in/waylonwalker/ ). Here are the last 11 days of stories.
2020-10-21
I was 20 commits into a hackoberfest PR when I suddenly realized they they all had my work email on them instead of my personal email 😱. This is the story of how I corrected my email address on 19 individual commits after already submitting for a PR
2020-10-17
I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called
2020-10-8
Today we ran into an issue where we had a one-off script that just needed to work, but it was just chewing threw memory like nothing. There are a number of ways that you can check the amount of memory on your system. The easiest is not ne
2020-10-1
A common linting error thrown by various linters is for trailing whitespace. I most often use flake8. I generally have [pre-commit]( https://waylonwalker.com/pre-commit-is-awesome ) hooks setup to strip this, but sometimes I
2020-9-30
There are many useful chrome extensions out there. I probably have way too many installed, here are four that I am currently using. This post was inspired from Chris over at daily-dev-tips Love it or hate it passw
2020-9-13
Bash is a language that is quite useful for automation no matter what language you write in. Bash can do so many powerful system-level tasks. Even if you are on windows these days you are likely to come across bash inside a cloud VM, Continuous Integ
2020-8-13
Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an altern
2020-8-11
miniconda is a python distribution from continuum. It's a slimmed-down version of their very popular anaconda distribution. It comes with its own environment manager and has eased the install process for many that do not have a way to compile c-exten
2020-8-10
Learn Kedro in 5 days
2020-8-8
This post was inspired by a comment I left on @dsteenman's post. {% post dsteenman/how-long-should-a-blogpost-be-2k6n %} Most of the time I prefer short as I am more likely to read the whole thing. If its setup as a series I a
2020-8-7
Check out the repo for git-auto-commit-action . It's a really cool GitHub action that will automatically commit files changed during the action. I was using this to render a new readme based on a template.
2020-8-3
I was particularly inspired by @chantastic episode 103 of the react podcast with @ParissAthena. They spoke about the black tech pipeline as well as Diversity, Equity, and Inclusion. Pariss is quite an inspiration. She has done so much work to create
2020-8-3
If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks. This one comes a bit surprising as it was just casually mentioned in
2020-8-1
As I continue to build out waylonwalker.com I sometimes run into some errors that are not caught because I do not have good testing implemented. I want to explore some integration testing options using GitHub's actions.
2020-7-27
Today I setup a new machine on Digital Ocean to use with TestProject.io, Here are my installation notes. envsubst < .github/ci/docker-compose.yml > docker-compose.yml
2020-7-25
When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these you can try out somethi
2020-7-25
I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. try pandas method -> try spark -> try dict/list -> none Is there an easy way to create a nosql database in memory from a a list of dic
2020-7-24
Mypy's config parser seems to be one of the most complex. This is likely in part to it having the largest backwards compatability of all projects that I looked at. mypy/config_parser options/config.py black
2020-7-21
I recently added a button to my blog, and subsequently my posts on DEV.to . It's the best thing that I have done for it in a while. It makes it so easy to do quick edits. I refer back to my old posts quite a bit, s
2020-7-18
When first learning to code its very common to hard code everything right into the code. This happens with most folks in just about any language. Whether its HTML or markdown for front end content, or even hardcoding parameters in our backend languag
2020-7-14
I am looking into a way to replace my google reader experience that I had back in 2013 before google took it from us. I am starting by learning how to parse feeds with python, and without much previous knowledge, it proved to be much easier than ant
2020-7-13
The input will be a yaml file containing a list of Items you want to stay up to date with. Inside each item will be a url, and weight. Each url will be pulled in and parsed into a standard data scructure. Some item
2020-7-12
I ran this post on dev.to and got a great response of great examples, check it out . The GitHub profile feature just went live for a subset of users. Simply creating a repo named after your username, and clicki
2020-7-11
Can someone explain how or why twitter cards render differently from device to device? I do understand that twitter cards a built from meta tags, the full list can be found in their docs Mobile Looks fine. On D
2020-7-11
I ran a discussion on dev that collected quite a list of examples in the comment section. So many great calls to action, animations, memes, and weird tricks. Upload all of your icons to the repo in a directory such as
2020-7-10
Python *args and **kwargs are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packe
2020-7-2
While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont e
2020-6-29
kedro 0.16.2 just dropped last week with a long-awaited feature... catalog search ! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Be
2020-6-22
Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing. Check out this po
2020-6-19
This day 8 years ago I started my first day as a Mechanical Engineer. I am so grateful for this journey that I have been able to have. There is no way that I could have planned this journey from the beginning. My initial career plans were
2020-6-19
I just did a quick refactoring of my JAMStack blog urls. Some didn't fit with my style, some had _ that I wanted to switch to - , and others were rediculously long. I've been using forestry as my CMS, I
2020-6-11
It's been a long time coming. We use some very harsh language within tech so much sometimes that we become numb to it. It's time to do my very small part in this movement and purge this language from my active repos starting with this blog right he
2020-6-11
Python *args and **kwargs are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packe
2020-6-10
Follow along the Journey as I build out kedro.dev . I have really enjoyed my own personal journey as I have started to build all of my data pipeline projects with the kedro framework. I want to start building a plac
2020-6-5
I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it's easy. It will automatically run checks for you. In some cases, it will even automati
2020-6-5
kedro-static-viz is out with support for the newly released hooks feature. This means that you can have kedro-static-viz automatically deploy a full gatsby site before_pipeline_run keeping you
2020-5-28
There are two main ways to create kedro hooks, with modules and classes. Each one still uses the same verbiage as the function/method names. Class hooks seem a bit special as they give you a way to configure them so that they are a bit mo
2020-5-23
This might be useful https://github.com/nfl/react-helmet/issues/146 https://github.com/gatsbyjs/gatsby/issues/13136
2020-5-22
This post is a 🧠 branstorming work in progress. If you are completely unsure what kedro is be sure to check out my what is kedro post hooks are executed in reverse order of the hooks list.
2020-5-22
I want to incorporate some of the wonderful comments, \U0001F495, \U0001F984, and \U0001F516's that I have been getting on dev.to on my website. I have dabbled once or twice with no avail this time I am taking notes on my journey, so follow along an
2020-5-20
GitHub's actions are a new GitHub feature that will trigger GitHub to spin up a virtual machine and run some tasks with some special access to your repo. It can interact with comments/issues, it can clone your repo, You can explicitly pass in secrets
2020-5-18
stack overflow link
2020-5-17
As usual while listening to python bytes 181 I heard of a tool that I had to try out right away! This thing is 🔥 hot off the press folks, we're talking the first release only 3 weeks ago. Its something that the pytho
2020-5-15
Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When run
2020-5-15
When creating cli's I often want some nice full-width character. I find it tough to find them, and when I do half the time it is an image or something that cannot be copied 👿. I rarely get very complex with my semi-manual ASCII art. I ca
2020-5-12
Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro 0.16.0 . They allow you to hook into catalog_created , pipeline_run , and node_run
2020-5-10
This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources.
2020-5-9
2020-5-7
find-kedro is a small library to enhance your kedro experience. It looks through your modules to find kedro pipelines, nodes, and iterables (lists, sets, tuples) of nodes. It then assembles them into a dictionary of pipelines, each modul
2020-5-4
Often when reading through javascript examples you will find some arrow functions use parentheses () while others use braces {} . This key difference is that parentheses will implicitly return the last s
2020-5-3
Inspired by Chris Achard Email course inspired by learn d3 in 5 days
2020-5-1
Is it possible to deep watch a single python function for changes? keeping track of a python functions hash is quite simple. There is a __hash__ method attached to every python function. Calling it will return a ha
2020-4-27
This past week I had a really weird bug in my kedro pipeline. For some reason data running through my pipeline was coming out completely made no sense, but if I manually request raw data outside of the pipeline it matched expe
2020-4-27
If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simple cli I have opted to use the cli for most of these, that way I know
2020-4-13
So often I see a variables type() inside of its name and it hurts me a little inside. Tell me I'm right or prove me wrong below. Pandas DataFrames are probably the worst offender that I see
2020-4-8
Here is one useful thing that you can do with GitHub actions no matter what language you use, send email. You might want to know right away when your ci passes. You might want to give your team a nice pat on the back when a new release is deployed.
2020-3-31
Github actions are written in configuration files using the YAML syntax. YAML is a superset of JSON. Most YAML can be expressed inline with JSON syntax. Similar to python YAML is whitespace driven by whitespace rather than brackets tags. The argu
2020-3-16
GitHub actions are an amazing tool that allows us to run code based on triggers inside of our repo. Their is a large and growing community of actions inside the marketplace to use with very little effort. Best of all they are free for public reposi
2020-3-16
Today I learned how to diff between two branches. Sometimes we get a little git add . && git commit -m "WIP" happy and mistakenly commit something that we just can't figure out. This is a good way to figure out what
2020-3-3
This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro . You can learn more about this Hello W
2020-3-2
You dont have to start out as a git wizard with the cleanest possible commit history. At first dont let yourself get too wrapped up in it, the most important part is that you make commits. You will find needs for more advanced stuff later.
2020-2-26
I am working through Wes Bos's beginnerjavascript.com/ I just hit module 18 on hoisting. It's something that I always knew was there, Its not something I typically see used or use myself. Do you have any use cases t
2020-2-25
Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate
2020-2-24
Inspired by Wes Bos's new uses.tech I wanted a custom scrollbar on my personal site. I had tried to do it in the past, but gave up after it was not working. Since uses.tech is open source I ju
2020-2-21
"Don’t waste your time learning everything." Inspired by this linkedIn post I felt that this comment was very powerful. Here are my 2 cents. Stick to what you know, and learn a little bit of some
2020-2-19
Moving into 2020 I have been really leaning on using purple as my theme color everywhere more and more. Its time for an update to my personal site, not just because it feels plain, not just because the cover art I am using for dev.to doesn't fit my
2020-2-17
It was so easy to get a professional looking navbar with just 3 lines of code. This package seriously is so usable on mobile it is rediculous. I found this package from day 4 of the 100 days of gatsby challenge. It is by the
2020-2-11
* https://getform.io Free to get started https://formkeep.com/?#pricing Free 7 day trial $4.99 cheapest after
2020-2-10
I am starting a redesign of my website. I have started cross posting to dev.to more regularly. With that I have been making more detailed cover images at the recommended 1000x420 . These images get cut
2020-2-8
I have happily had my personal site waylonwalker.com hosted on netlify for nearly 2 years now. In fact I have hosted about a dozen different toy projects to play with on there, 4 of which have gone far enough to get a custom d
2020-2-6
Finally after years of hand typing out a full git push --upstream my\_really\_long\_and\_descriptive\_branch\_name I foudn there is a setting to automatcally push to the current branch. More realisitically I just did a
2020-2-4
This morning I logged into my machine and was nearly out of space Photo by Simon Migaj on Unsplash These are the commands that I often use to reclaim space. Its so easy to fill
2020-2-1
Photo by Ozark Drones on Unsplash I love using URL shorteners to easily share links without hitting character limits, but they loose their meaning. Services like bit.ly will save my links for me so that
2020-1-29
cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Things a cli can do to enhance your library.
2020-1-29
I spend a lot of my time at the terminal for my daily work, mostly in Linux or wsl. One big reason for using wsl over cmd.exe is the ease of walking through history that fzf provides. This week we had a windows bug in a cli and I was stuck in vanil
2020-1-23
Mine is the python debugger. I was a long holdout thinking that print statements were sufficient. That was untill I started having errors crop up in functions that took minutes to run. The thing that I most notably wish I would have known about is po
2020-1-10
I have been using oh-my-zsh successfully for about 2 years now. But lately my startup time has been really bothersome. It has grown to the point where it was taking about 5.5s to startup a shell! This is ok if I am going to sp
2020-1-3
Vim's (neovim in my case) location list can provide some very useful information while developing. Mine gives me information about linting and type checking errors with fairly little config. Generally, it sits nicely at the bottom of the screen and
2020-1-1
2019-12-21
Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. **From the Docs To get the Python argument name, the chosen na
2019-11-11
I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips . Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. E
2019-11-2
|-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+.
2019-10-14
2019-10-1
Pathlib is an amazing cross-platform path tool. Current Directory Users Home Directory module directory Others Let's create a path relative to our current module. recur
2019-9-26
2019-9-25
Good for method chaining, i.e. adding more methods or filters without assigning a new variable. general purpose, this is probably the most common method you see in training/examples capable of including multiple strings to inclu
2019-9-24
I love digital ocean for it's simplicity and its commitment to open source.
2019-9-22
I started using aws in March 2019. Here are some of my notes.
2019-9-22
tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of perf
2019-9-18
If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. As you graduate your s
2019-9-11
This post is intended as an extension/update from background tasks in python . I started using background the week that Kenneth Reitz released it. It takes away so much boilerplate from running backgrou
2019-9-10
Bash is super powerful. Show Remaining Space on Drives show largest files in current directory Move files then symlink them edit in vim cat a file bash execute
2019-9-8
I have used %autoreload for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class module
2019-9-8
Throw that mouse Away its time to setup some keyboard shortcuts. These sortcuts were the baseline for switching from tmux/vim to vscode. Most folks posts I was able to find gave great tips on replacing vim, but very few have focused on t
2019-7-7
sometimes things get messy pull 👉 branch 👉 format 👉 work👉 add 👉 commit 👉 pull 👉 rebase 👉 push
2019-5-27
Testing out forestry.io I still ♥️ your product dont be forestry is simple I have been playing with the netlify cms for a while now, and it has been a decent experience, but I really str
2019-4-23
rage unstage to wipte out history of staged commit git checkout HEAD~n -- removes history --SOFT locally before push after push after push
2019-2-5
rage unstage to wipte out history of staged commit git checkout HEAD~n -- removes history --SOFT locally before push after push after push
2019-2-5
This is my story into data science. I am addicted to the process of learning and improving my skills nearly to a fault. The reason I say nearly is because my addiction is fueled with results. I crave the output of my work enhance the wor
2019-2-2
This morning I log into my VCS and check activity on my projects to find that someone else has been very active on my projects fo the last few weeks. I quicklyhover over the missing avatar to find that
2019-1-21
The year of intenionality This is a follow up to my previous post 2018-retrospective This year I will become more productive, by intensionally working on a well thought out plan, learning the right technologies,
2019-1-12
2018 was a year of many ups and downs, and learning to deal with a whole new set of problems professionally and at home. In 2018 I logged in to my first Linux system, setup my own webserver, data pipelines, database. I learned to use react and d3. S
2019-1-5
Today I want to take some time to talk about the things that make me happy in my work environment. This is completely free-flow off the cuff, but are things that I do that make me happy, not having them would definitely be a deal breaker for me .
2019-1-1
I recently finished up the flexbox-zombies course to learn more about flexbox, and to become proficient with it. I can truly say that this course has changed the way that I create layouts. Flexbox is very intuitive now. What this course does reall
2018-4-22