I am working through the textual tutorial, and I want to put it in a proper cli I setup a new project running https://waylonwalker.com/hatch-new-cli/ If you are using https://waylonwalker.com/minimal-python-package/ Now to get devtools through a cl
2022-10-24
I really like having global cli command installed with pipx. Since textual You can pipx install textual. But if you try to run any textual cli commands you will run into a In order to install optional dependencies with
2022-10-24
11ty https://www.rockyourcode.com/how-to-deploy-eleventy-to-github-pages-with-github-actions/ hugo puts it in the base url https://gohugo.io/getting-started/configuration/#baseurl mkdocs uses a special cli build command https://squidfunk.github.io/mk
2022-10-19
Markata now allows you to create jinja extensions that will be loaded right in The entry for 0.5.0.dev2 from markata Created entrypoint hook allowing for users to extend marka with jinja The first example that you can use right now is Get it with a
2022-10-11
In my adventure to learn django, I want to be able to setup REST api To get started lets open up a Now we need to declare Next I will create all the files that I need to get the api running. I already have the following model from last time I was p
2022-10-9
2022-10-9
Markata now uses hatch as its build backend, and version bumping tool. Markata 0.5.0 is now out, and it Over the last month I made a big push to get 0.5.0 out, which adds a whole Here Moved to PEP 517 build #59 0.5.0.dev1 Before cutting all of my per
2022-10-8
I am continuing my journey into django, but today I am not at my workstation. I I found this in
2022-9-26
2022-9-26
My next step into django made me realize that I do not have access to the admin panel, turns out that I need to create a cuper user first. Right away when trying to setup the superuser I ran into this issue Back to the yes I am still running remote
2022-9-26
I have no experience in django, and in my exploration to become a better python If you found this at all helpful make sure you check out the The first thing I need to do is render out a template to start the project. Once I have the project I need a
2022-9-25
While updating my site to use Markata Jinja comes with a handy utility for escaping strings. I definitly tried to The issue I ran into was when trying to setup meta tags with the new Using my article titles like this ended up causing this syntax err
2022-9-24
When I am developing python code I often have a repl open alongside of it So in the past the way I have setup a few extensions for myself is to add I missed the fact that some of these tools like The reccomended way is to add them to your The issue
2022-9-20
A long needed feature of markata has been the ability to really configure out Now you can add some extra style to your site with the existing built-in Each text entry in If this does not take you far enough yet, you can still eject out and use your
2022-9-11
I creating new versions created by myself with stable diffusion Note Running After running and naming the project You can also fill in the project name ahead of time, and it will run without Note hatch new Let I can setup packaging with hatch by ru
2022-9-2
Hatch has an amazing versioning cli for python packages that just works. It creating new versions created by myself with stable diffusion For trying out the The main hero of this post is the It is possible to set the version number inside the Sta
2022-9-1
Markata is a great python framework that allows you to go from markdown to a Full transparancy... I built markata. The talk is live on YouTube. Make sure you check out the other videos from the https://youtu.be/Wq9YBamSgs0
2022-8-31
I open sourced the static site framework that I use to build {% gh {% gh I am a heavy user of the {% gh I use vim for all of my text editing needs. It brings me joy to make any part {% gh
2022-8-30
I spoke at python webconf in March 2022 about how I deploy this blog on a Building this blog has brought me a lot of benefits. I have I also bump into people from time to time that recognize me from the blog, its The talk recently released on Youtub
2022-8-30
I got all the pypi packages that I own behind 2 factor authentication. 💪 Recently this really made it I feel like this caused a bit of confusion as turning on 2FA does not mean that You might wonder what this means for my projects. It means that to e
2022-8-29
After years of listening to I was quite nervous to talk on a show that I helped shape my career in such a During the episode I tried my best to let Yetu and Ivan take the spotlight as https://youtu.be/WTcjvwkXoY0 Michael made the call available on y
2022-8-27
I just love how some features of vim are so discoverable and memorable once you I recently used this to dedupe my autogenerated links section for Here is what the markdown looks like. Then typing
2022-8-25
Today I https://twitter.com/ While waiting for that reply let Lets import all the classes that we need from Now we need some code to highlight. I am going to rip my We could simply print out the code we have as a variable, but thats a bit hard prin
2022-8-24
a sprinter edging out his opponent by Dall-e It Markata started off partly as a python developer frustrated with using nodejs Over time this side project has grown some warts and some of them were now I like to use my tils articles for examples and t
2022-8-21
I love using pipx for automatic virtual environment management of my globally This one was not obvious to me at first, please let me know if there is a My first technique was to make a package that printed out If you don
2022-8-9
2022-8-3
A common meta thing that I need in python is to find the version of a package. In searching the internet for an answer nearly every one of them pointed me to While its not required its super handy and easy for anyone to remember off the Special thank
2022-7-29
xrandr is a great cli to manage your windows in a linux distro using x11, which Check out the xrander-manager is a python cli application that is simply a nice interface As with any python cli that is indended to be used as a global/system level cli
2022-7-16
So many terminal applications bind After digging way too deep into how IPython implements its It was that simple. This is not a game changer by any means, but I will now
2022-6-13
It Yes the solution comes from a competing templating framework. I install copier with pipx, so I need to inject cookiecutter in to my copier If you are using a normal virtual environment you can just pip install it. Now to enable the extension you n
2022-6-11
Textual has devtools in the upcoming css branch, and its pretty awesome Textual is still very early and not really ready for prime time, but it As https://twitter.com/willmcgugan/status/1531294412696956930 Textual is a tui application framework. Unl
2022-6-9
Whenever you are installing python packages, you should always use a virtual Pip is the pacakage tool for python. It installs third-party packages and is python packages often require many different dependencies, sometimes packages My one exception
2022-6-1
I I talked about why and how to use pyenv along with my first impressions in According to libsqlite3-dev When I make a fresh env and install ipython I still get the same error and I am After having this issue for awhile an coming back to I think I h
2022-5-31
https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love
2022-5-12
I really like the super clean look of no status menus, no url bar, no bookmarks
2022-5-11
When you first start qutebrowser It will create some config files in As far as I know qutebrowser will create this default config out of the You might want to confvert if you are more comfortable with the python
2022-5-11
I am getting ready to do some timeseries analysis on a git repo with python, my This returns a generator, if you are iterating over them this is likely what The generator will return
2022-5-9
When I am creating blog posts it When I have something to take a screenshot of, I need to take the shot, take screenshot optimize conversion publish create img tag I created this tool for myself in python because that is what I am most My screenshot
2022-4-30
Copier is a fantastic templating library written in python, but older versions As of May 15, 2022, the stable release of copier now includes these changes, if I Use copier several times per day and get fantastic benefit from this project, https://git
2022-4-30
GitPython I recently made myself a handy tool for making screenshots in python and it https://waylonwalker.com/screenshot-to-blog/ GitPython Import Repo from the git library and create an instance of the from the docs It provides abstractions of git
2022-4-30
Python, click install Edit the System Environment Variables Environment Variables button Add the following path to your users Path Variable
2022-4-18
https://pretalx.com/pyohio-2022/cfp Markata is a plugins all the way down static site generator, that covers all Lets build a whole blog site in 5 minutes. Sometimes python scripts/notebooks take a long time to run, let kedro
2022-4-12
Sometimes you just want python to do something else when you hit an exception, I am working on a quick and dirty python script designed to take screenshots I could have gone down a logging route, but honestly this is meant to be quick, Python exposes
2022-4-10
pygame events are stored in a queue, by default the most suggested way You don Let printing the events reveal this Let printing the events reveals that there are no events, well i Even simple games don Store events for each frame in memory. I thought
2022-4-7
One of the most essential concepts of pygame to start making a game you will I started by making a spotlight in Gimp, by opening a 64x64 pixel image and This is what it looks like Now we can load this into pygame. To make pygame a bit more efficient
2022-4-6
Browsing through twitter the other day I discovered it through this https://twitter.com/ Before I dive in deep, I do want to mention that Dunk is super new and beta at The author really cautions the use of it as your default pager this early, I He no
2022-4-4
I I What currently have is a single module starter package that is on github Since it You can make a quick game by inheriting from Game, and calling Here is what the current
2022-4-3
My personal Site build went down last week, and I was unable to publish a new Classic IT fix, rerun it and see if you get the same error. Everyone is busy Also worth a check to see if GitHub is having a hiccup or not. This error felt Check Alright
2022-4-1
I ran into a PR this week where the author was inheriting what BaseException Try running these examples in a Since things such as If you except from exception or something than inherits from it you will be When you make custom exceptions expect tha
2022-3-31
When I need a consistent key for a pythohn object I often reach for Yesterday we talked about setting up a persistant cache with python diskcache. https://waylonwalker.com/til/python-diskcache/ My first thought was to just hash the files, this will g
2022-3-30
When I need to cache some data between runs or share a cache accross multiple Install diskcache into your virtual environement of choice using pip from your command line. There are a couple of different types of cache, Adding to the cache only needs
2022-3-29
The easiest way to speed up any code is to run less code. A common technique The easiest and most common to setup in python is a builtin functools.lru Any time you have a function where you expect the same results each time a when same lru lru lru
2022-3-28
I keep a small I recently switched hosting from netlify over to cloudflare. Well cloudflare This breaks my go to example dataset. What After a bit of googling I realize that this is a common thing, and that setting Now this works again, but it feel
2022-3-27
Python Requests is on pypi and can be installed into your virtual environtment with pip. Requests makes getting content from a web url as easy as possible. Requests can handle any web request and is not limited to only html. Here are There is way mo
2022-3-26
I recently gave a talk at python web conf 2022, and one of the things I did Lets use this section to show what it ☝ This is how my write markdown build site publish This is what the above slide looks like in lookatme. The way I write my slides I wa
2022-3-24
I use a package It Frontmatter is a handy way to add metadata to your plain text files. It Here is the exact frontmatter for this post you are reading on my site. yaml is the most commmon, but If you want a good set of examples of yaml Here is how I
2022-3-23
Today I was watching the python web conf 2022 and saw I https://waylonwalker.com/python-args-kwargs/ More on unpacking in this post. With the release there is also a new update syntax Are you writing libraries/applications that are only going to be
2022-3-22
I love the freedom of writing in markdown. It allows me to write content from I will most often just present right from the terminal using I sometimes also use reveal.js, but that I leverage auto slides when I write my slides in markdown. The large
2022-3-21
When I need to read contents from a plain text file in python I find the
2022-3-20
A very common task for any script is to look for files on the system. My go to I setup a directory to make some examples about globbing. Here is what the 1 directory, 8 files Pathlib is a standard library module available in all LTS versions of pyt
2022-3-17
Last Thursday I learned about Watching him use pytest-mock is out on pypi and can be installed with pip. Sometimes I fall victim to making these posts nice and easy to follow. It I just wanted to do something that was worth mocking, the first thin
2022-3-14
Python 3.8 came out two and a half years ago and I have yet to really lean in Now that Python 3.6 is end of life, and most folks are using at least The assignment operator in python is more commonly referred to as the walrus This example from the do
2022-3-12
Kedro rich is a very new and unstable (it There is no pypi package yet, but it You can run your pipeline just as you normally would, except you get progress Listing out catalog entries from the command line now print out a nice pretty Go to the
2022-3-11
I recently found a really great I have this added to my In my workflow I open a tmux session for each project, so this First open neovim, but with the If you try to run
2022-3-6
As I am toying around with textual, I am wanting some popup user input The main issue is that when you are in a textual app, it kinda owns the textual is still very beta Part of this comes down to the fact that textual is still very beta and So the s
2022-2-26
Mermaid diagrams provide a way to display graphs defined as plain text. You can define nodes like this in mermaid, and GitHub will now render
2022-2-24
Glances is a system monitor with a ton of features, including docker processes. I have started using portainer to look at running docker processes, its a great You will need to install glances to use the glances webui. We can still use You will be p
2022-2-22
Glances has a pretty incredible webui to view system processes and information The nice thing about the webui is that it can be accessed from a remote system. You will need to install glances to use the glances webui. We can still use You will be pr
2022-2-20
Glances is a fully featured system monitoring tool written in python. Out of Once you run this you will be in a tui application similar to htop. You can
2022-2-18
python requirements text files can in fact depend on each other due to Lets create two requirements files in a new directory to play with. Then add the following to each requirements file. Installing requirements This is covered in the
2022-2-17
Reading eventbridge rules from the command line can be a total drag, pipe it I just love when I start thinking through how to parse a bunch of json at the
2022-2-15
In python, a string is a string until you add special characters. In browsing twitter this morning I came accross this tweet, that showed that https://twitter.com/bascodes/status/1492147596688871424 I popped open ipython to play with this. I could c
2022-2-12
I often run shell commands from python with Popen, but not often enough To get the stderr we must get it from the proc, read it, and decode the Now that we can read the
2022-2-10
In looking for a way to automatically generate descriptions for pages I It When I originally wrote this post, I did not realize at the time that
2022-2-5
For an embarassingly long time, til today, I have been wrapping my dict Lets consider this example for prices of supplies. Here we set a variable of What I would always do is try to get the key, and if it failed on KeyError, I What I noticed Nic doe
2022-2-3
BeautifulSoup is a DOM like library for python. It Lets make a sample.html file with the following contents. It mainly has Lets import our packages, read in our And what we get is a list of I recently added a heading
2022-2-1
I keep my nodes short and sweet. They do one thing and do it well. I Here are two examples, the first one Many times I just want to get the data in as fast as possible, learn Note: try not to take the idea of a one liner too far. If your
2022-1-31
As you work on your kedro projects you are bound to need to add more Before you start mucking around with any changes to dependencies make sure that New requirements get added to a requirements.in file. If you need to specify Here I added the popula
2022-1-28
I am a huge believer in practicing your craft. Professional athletes https://waylonwalker.com/what-is-kedro/ Go to your playground directory, and if you don Install pipx in your system python. This is one of the very few, and From inside your I us
2022-1-27
I have added a hotkey to my copier template setup to quickly access all my I
2022-1-24
I often pop into my blog from neovim with the intent to look at just a To tie these into a Telescope picker you add the command as the NOTE telescope treates each word as a string, do not wrap an extra
2022-1-23
Copier allows you to run post render tasks, just like cookiecutter. These are The example I have below runs an I have put the script in
2022-1-22
I https://youtu.be/Rk-Y71P Run Mypy as is, don Next we will add The final stage to this series is to add Make sure that you watch Anthony https://www.youtube.com/watch?v=Rk-Y71P
2022-1-21
In order to make an auto title plugin for markata I needed to come up ! Here I have a To turn this into a markata plugin I put it into a pre
2022-1-20
Getting docstrings from python First you need to load in some python code as a string, and parse it with You can then use To get all of the functions docstrings we can use ast.walk docs: Recursively yield all descendant nodes in the tree starting a
2022-1-18
Many tools such as ripgrep respect the Editors like vscode often do not include files that are .gitignored in pathspec pathspec
2022-1-17
I don I first tried the nvim lsp rename, and it failed, Then I pip installed Once you have rope installed you can call rename on the variable.
2022-1-16
When running a python process that requires a port it
2022-1-15
functools.total From the Docs: The class must define one of Here is an example using the Enum I was working on the other day.
2022-1-14
Adding a
2022-1-13
Python comes with an enum module for creating enums. You can make your Enum values can be auto incremented by importing auto, and calling Enum
2022-1-11
I recently paired up with another dev running windows with Ubuntu running in Open up a terminal and get your required system dependencies using the apt I like running things like this through an ansible-playbook as it give me some Here is a clip of m
2022-1-10
The copier answers file is a key component to making your templates Inside of my Inside my copier.yml I have setup my Once I change the I After rendering the template we have the following content in our This is where I was most stuck, primarily b
2022-1-8
2022-1-7
Once you have made your sick looking cli apps with rich, eventually you are Install them from the command line. Import make a .py file and import them in it. If you return your rich renderable out of class that inherits from You At this point It prob
2022-1-7
%%include til/copier %%include til/copier-template-variables %%include til/copier-answers
2022-1-5
I was completely stuck for awhile. copier was not replacing my template !
2022-1-4
I By default copier uses double square brackets for its variables. note Before running copier we need to tell copier what variables to ask for, I prefer to install cli tools that I need globally with pipx, this When running note The resulting files
2022-1-3
I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a But what I got back was not what I expected This is weird, why cant I run kedro new with pipx? Lets try pip. Same issue. https://waylonwalker.com/what-is-kedro/ Curious what kedro
2022-1-2
Pluggy makes it so easy to allow users to modify the behavior of a framework I As long as the framework document the hooks that are available and what it Installing pluggy is just like most python applications, install python, make At the time I star
2022-1-1
pyenv provides an easy way to install almost any version of python from I needed to install an older version of python on ubuntu I Based on the Readme it looked like I needed to install using homebrew,so this https://waylonwalker.com/til/installing-h
2021-12-30
Part of my neovim setup requires having the re-installing a bunch of things that are already installed can be quite check if the command is installed with register that step ignore if that step fails add a https://www.youtube.com/watch?v=MCFg6-W5S
2021-12-24
Adding a
2021-12-23
This is not a flaky works half the time kind of plugin, it I can https://youtu.be/2QW5DJiEJH4 Give the video a watch, I did not have noise-cancelling on in obs. My pyflyby is hosted on pypi, so you can get it with pip. I have had no issues If you Se
2021-12-20
find all nodes with raw in the name use parameters make and use a logger use find-kedro in spaceflights slide in a new node vizualize your pipeline find slow nodes move the configuration directory build kedro into docker and run it pyinstrument pdb m
2021-12-4
I use https://youtu.be/ https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. The command to ensure there are catalog entries for every dataset in the passed Create Fills in new dataset entries with the default datase
2021-11-15
I recently started playing with nix. automatically select correct python version per project activating one doesn https://search.nixos.org/
2021-11-14
We can https://youtu.be/TZrRAP-9UMk In any python repl you can access the docstring of a function by calling for In Ipython we can even get some syntax highlighting with the Sometimes the docstrings are not good enough, and don The more common way
2021-10-10
Parameters are a place for you to store variables for your pipeline that can be https://youtu.be/Jj5cQ5bqcjg https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. You can have multiple parameters files and choose which
2021-9-19
https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let You will need to import node from kedro.pipeline to start creating nodes. The Inputs and outputs can be None, a single catalog entry as a string, mutiple Sometimes in our pipeli
2021-9-13
Running your kedro pipeline from the command line could not be any easier to https://youtu.be/ZmccpLy-OEI https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. To run the whole darn project all we need to do is fire up
2021-8-24
Avoid serious version conflict issues, and use a virtual environment anytime https://youtu.be/ZSxc5VVCBhM conda venv pipenv I prefer to use conda as my virtual environment manager of choice as it give me stores environment in a root directory i.e. c
2021-8-23
Kedro pipeline create is a command that makes creating new https://youtu.be/HtyIKqlEoNw The kedro cli comes with the following command to scaffold out The directory structure that it creates looks like this.
2021-8-22
Kedro comes with an https://youtu.be/IWimEs-hHQg You must start by having your kedro project either cloned down https://waylonwalker.com/kedro-new/ this post covers kedro new https://waylonwalker.com/kedro-environment/ This post covers creating your
2021-8-21
Immediately after https://youtu.be/IGba3ytf Its as simple as these three commands to get started. I don
2021-8-20
https://youtu.be/uqiv5LAiJe0 Kedro new is simply a wrapper around the cookiecutter templating library. The https://waylonwalker.com/what-is-kedro/ Unsure what kedro is, Check out yesterdays post on What is Kedro. I reccomend using The kedro team do
2021-8-18
Kedro is an unopinionated Data Engineering framework that comes with a somewhat https://youtu.be/Wf4rnFsaFFU https://waylonwalker.com/what-is-kedro-1/ This is an updated version of my original what-is-kedro article If you are doing a series of operat
2021-8-17
Kedro versioned datasets can be mixed with incremental and partitioned datasets https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. This was a question presented to me at work. We had some plots being produces To en
2021-7-8
mu-repo
2021-6-13
I recently started streaming on python kedro Data Science Data Engineering webdev digital gardening It all started with My introduction to twitch came from After taking a jump in with the spaceflights tutorial I realized that it really I started stor
2021-6-12
https://stackoverflow.com/questions/16720541/python-string-replace-regular-expression I am starting to stream 3 days per week, before I start work in the morning. On Monday kedro-diff test kedro-diff on piplines with history setup deploy pipeline dep
2021-6-8
This was my seconf time ever streaming on
2021-6-7
https://youtu.be/I4VenHqIEng Doing some Digital Gardening on stream Ahrefs Errors ahrefs large images Automatic gif to mp4 After this stream all gifs on my site are converted to mp4/webm if they exist.
2021-6-6
This was my first time ever streaming on https://youtu.be/Y07UBr9Ccjs It all started with
2021-6-4
The Kedro node is an essential part of the pipeline. It defines what catalog https://waylonwalker.com/what-is-kedro/ 👆 Unsure what kedro is? Check out this post. The node function is the most common and reccomended way to define kedro nodes. The a
2021-6-3
With the latest releases of kedro https://waylonwalker.com/what-is-kedro/ If your just learning about kedro check out this post walking through it I To get kedro running, you will need a pipeline, catalog, and 👆 Above is the minimal setup to get a k
2021-5-23
I had an idea come to me via twitter. Short one word name packages are becoming I started with the simple index. Pypi provides a single page listing to every
2021-5-23
Kedro can have a chatty logger. While this is super nice in production First, how does one silence a python log? Python loggers can be retrieved by | Level | Numeric value | Getting a python logger is straightforward if we know the name of the l
2021-5-20
Setting up python with the native nvim>0.5 lsp was mr https://github.com/neovim/nvim-lspconfig https://github.com/palantir/python-language-server/issues/190 Getting mypy working with lsp was tricky for me. I had some issues trying to
2021-5-17
Running multiple processes using the same diskcache object can cause issues Minimum reporducible error is one of my superpowers in development. I do this Markata has a lot going on. It 📝 Here I have my editor showing the file on the top split and
2021-5-14
:on This one has nothing to do with fugitive, but is a native vim feature that C-i jump The jumplist is sorted Oldest to newest When navigating the jumplist with :Telescope jumplist adds to the jumplist Unlike In the file you want to stage hunks of
2021-5-8
When a python module is called it is assigned the Let I have set this module up to execute one of two if statements based on whether Note it is not common to have a Running a python script with the command This will print out https://waylonwalker
2021-5-7
DRAFT -
2021-5-2
I really like using conda ( Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used i
2021-4-21
I am trying to see what an embeded replit
2021-4-15
Zev Averbach, Q: Tell me about your journey as a spreadsheet jockey into Data Engineering? A: First of all, it I Now that I As the business grew I was spending more and more time copying entries from Google Sheets to the Freshbooks web app for invoic
2021-4-9
Testing print/log statements in pytest can be a bit tricky, capsys makes it capsys is a builtin pytest fixture that can be passed into any test to capture Simply create a test function that accepts capsys as an argument and pytest
2021-4-5
I I am working on a project in which I want to have a dev automatic port number auto-restart display I am very novice at best when it comes to sockets, the following function came I am going super basic here and just running Optionally if you wante
2021-3-27
I have automated my process to create virtual environments in my python I https://waylonwalker.com/tmux-nav-2021/ You can learn more about direnv is in many package repos, or can be installed from binary builds with a direnv works by executing the c
2021-3-22
2021-3-22
I have started doing more regular PR https://waylonwalker.com/what-is-kedro passing ci Variable Names Antipatterns No commented out code Docsttrings generally make sense Phase1 is typically a quick scan over the PR right within the PR window in my br
2021-3-21
With the realease of create a register_pipelines register hook_impl class You should now have something that looks like this in your pipeline I was not able to find any official documentation on how conflict resolution kedro automatically merges pi
2021-3-20
pytest has open sourced their amazing plugin framework My experience so far as a plugin user, and plugin author has been great. A hook is a single function that has a specific place that it is ran by the PluginManager. A Plugin is a collection of on
2021-1-23
There are so many tutorials out there for creating OG images with
2021-1-22
There are various ways to configure python tools, config files, code, or This will not include how they are implemented, I The motivation of this article is to serve as a bit of a reference guide for User settings can exist in the users ~/.config/fl
2021-1-21
This is a post that may be a work in progress for awhile, Its a collections of posts tags draft posts frontmatter filepath content template html Markdown.Markdown support extentsions provide ways to hook in or clean up the frontmatter load render sav
2021-1-20
How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. I have everything for this po
2021-1-20
Generating an api for a blog is much simpler than one might expect with python. fix missing data
2021-1-18
In python data science/engineering most of our data is in the form of some sort These containers for data contain many convenient methods to manipulate table https://waylonwalker.com/what-is-kedro/ unfamiliar with kedro, check out this post There are
2021-1-14
Changing conda environments is a bit verbose, I use a function with fzf that I have used conda as a virtual environment tool for years now. I started using What environment manager do you use for python? Conda environments are stored in a central lo
2021-1-11
What does it take to create an installable python package that can be hosted on pypi? setup.py my This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in o
2021-1-10
I If you already have an ipython config you can move on otherwise check out this https://waylonwalker.com/ipython-config/ I want something similar to the starship prompt I am using in the python version active environment git branch This is my zsh pr
2020-12-20
I use my ipython terminal daily. It Activate your virtual environment of choice and pip install it. Any time you You are using a virtual environment right? Virtual environments like venv or When you install ipython you start out with no config at a
2020-12-20
?docstring ??sourcecode %run %debug %autoreload %history autoformat %reset ! Stop going to google everytime your stuck and stay in your workflow. The Docstring not enough for you use case. I often run into cases where the I turned my nose up at thi
2020-12-14
One thing we all dread is mundane work of getting started, and all the hoops it When I start a new post I need to cd into my blog directory, start neovim in a frontmatter template Title slug tags date cover description create markdown file open in ne
2020-12-11
In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools This post is a walkthrough of me solving a problem wi
2020-12-10
black flake8 interrogate mypy pipeline constructs pipeline as expected nodes pipeline has minimum nodes test minimum tags test alternate tags test catalog follows naming structure test function does the correct operations on test data
2020-11-16
There are many reasons that you should be using kedro. If you are on a team of https://waylonwalker.com/what-is-kedro Kedro makes it super easy to get started with their cli that utilizes https://waylonwalker.com/create-new-kedro-project read more a
2020-11-1
Kedro This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Most of them hinge on a sweet combination of the ke
2020-10-25
I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called One thing
2020-10-8
Today I ran into an issue where we had a one-off script that just needed to It started with a colleague asking me How do I clear the memory in a Jupyter How do I clear the memory in a Jupyter notebook? There are a number of ways that you can check th
2020-10-1
Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an altern
2020-8-11
miniconda is a python distribution from continuum. It Installing miniconda on Linux can be a bit tricky the first time you do it completely from the terminal. The following snippet will create a directory to install miniconda into, download the lates
2020-8-10
If we take a look at the This one comes a bit surprising as it was just casually mentioned in As mentioned in It feels a bit crazy that simply installing a package will change the way that your pipeline gets executed. I do like that it requires ju
2020-8-1
As I continue to build out Running integration tests will not prevent bugs from happening completely, but it will allow me to quickly spot them and rollback. The very first thing that comes to my mind is anything that is loaded or ran client-side.
2020-7-27
When learning a new skill it When learning a new skill it Here are three different options that I have used in the past to try out something at some various levels. I am sure there are plenty more, but these are three that I have tried. I am not cove
2020-7-25
Mypy only uses pyproject.toml only uses pyproject.toml
2020-7-21
I am looking into a way to replace my google reader experience that I had back This is how I used python to parse rss and setup my own custom feed. Install the feedparser library. The feed is a feedparser.FeedParserDict. For all intents and purposes
2020-7-13
Python I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog See the full article
2020-7-2
While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont e
2020-6-29
kedro 0.16.2 just dropped last week with a long-awaited feature... The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections
2020-6-22
Passing inputs into kedro is a key concept. Understanding how it accepts a Check out this post for a review of how https://waylonwalker.com/python-args-kwargs/ When kedro runs your pipeline it uses the catalog to imperatively load your data, meanin
2020-6-19
2020-6-14
2020-6-14
2020-6-14
2020-6-14
Python Python * When recieving variables as a Never add Generally I find If your Here Here the function signature makes it clear what Inversely we can send a list of things as individual arguments by Just like Here is a function accepting
2020-6-10
👆 add this to your
2020-6-7
Load
2020-6-7
Combine a directory of
2020-6-7
load remote
2020-6-7
2020-6-6
Setup
2020-6-6
I recently discovered the ✨ awesomeness that is pre-commit. I steered away I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it It comes wi
2020-6-5
Even though it is a static site there is no functionality lost. The only thing that Even though it Kedro viz is a fantastic kedro plugin that allows you to visualize your data pipeline. Kedro allows you to quickly build production-ready pipelines w
2020-5-28
2020-5-26
Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When run
2020-5-15
As usual while listening to This thing is 🔥 hot off the press folks, we Nothing I have tried has come close to being this good It runs documentation coverage for your python project. It allows you to set the minimum amount of docstring coverage for
2020-5-15
Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro kedro hooks are an exciting upcoming feature of kedro https://waylonwalker.com/what-is-kedro/ If you are completely unsure what kedro is be sure to check out my what is kedro
2020-5-10
This past week I had a really weird bug in my After a few days of looking at it off and on, I pinpointed that it was all the way down in the raw layer. Right as data is coming off of the database. For this I already had existing I have dynamically
2020-4-27
If you are developing python packages and using GitHub here are four actions If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have suc
2020-4-13
So often I see a variables Pandas Sometimes vanilla structures too It Always name your containers plural, so that naming while iterating is simple. Before I start fights 🥊 in code review, am I inline here or just being pedantic?
2020-4-8
This is a quickstart to getting a new 🧹 Install 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization I use conda to control my virtual environments and will create a new environment called Options I try to keep my base
2020-3-2
https://waylonwalker.com/what-is-kedro This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set Currently, 💎 📉 🏗 Inside this core library is a data catalo
2020-2-24
2020-2-3
cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Things a cli can do to enhance your library. 🆚 print versio
2020-1-29
2019-12-21
Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. typically aliased with a shorthand ( ** To get the Python argument name, the chosen name is conver
2019-11-11
I am tweeting out most of these snippets as I add them, you can find them all here Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro
2019-11-2
Pathlib is an amazing cross-platform path tool.
2019-9-26
Good for method chaining, i.e. adding more methods or filters without assigning a new variable. general purpose, this is probably the most common method you see in training/examples capable of including multiple strings to include Good For partial ma
2019-9-24
I just started using portray and it is amazingly simple to use
2019-9-22
tqdm is one of my favorite general purpose utility libraries in python. It for more gifs like these follow me on twitter TQDM also has a convenience function called trange that wraps the range function with a tqdm progress bar automatically. There i
2019-9-18
If you are a regular listener of As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis
2019-9-11
This post is intended as an extension/update from This post is intended as an extension/update from I use it in more places than I probably should Before we get into it, I want to make a shout out to Kenneth Reitz for making this so easy. Kenneth
2019-9-10
I have used 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library code and test code
2019-9-8
Bash is super powerful. * * There are so many options inside of https://waylonwalker.com/refactor-in-cli/ I use these replace commands heavily when doing large refactorings. I like this one when there is not a good cli into config files and I need t
2019-9-8
I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without o
2018-5-16
These are my notes from pycon 2018 videos. I love the python community and especially the conference talks. This year I am going to take some notes from my favorite talks and post them here. This is an Vectorize with Numpy Looping in python can be
2018-5-12
I have tried most of the different methods in the past and found that copying and pasting the Recently a new python library was released to make running tasks in the background very simple. The I have pulled the latest version of the project in Sep
2017-9-16
Good afternoon fellow Data Geeks. Last week http://www.youtube.com/watch?v=u http://www.youtube.com/watch?v=abrcJ9MpF60 http://www.youtube.com/watch?v=knMg6G9 This afternoon we have a cup of from one of my favorite roasters Thirty Thiry Coffee. Th
2017-5-30
I
2000-1-15