-
fix feed descriptions
Today I fixed a bug in markata that has been occurring for a few months where the description for posts come out as None if coming from cache, the issue was a pretty simple check and pull properly from cache. This fixes all the descriptions in feeds and metadata on the post. Better description While in there we went ahead and improved our get_description to more accurately return plain text without escaped characters, remove cutoff words, and add an elipsis if the description cuts off the text.
-
ruff single line import
I've been using ruff to lint my python code for quite awhile now, I was pretty early to jump on it after release. Some of my projects have had a nice force-single-line setting and some have not. I dug into the docs and it was not clear what I needed to make it work. Turns out I was missing I sort in the select list.
-
💭 Red-Knot: Ruff's new Type Checker - Talk Python ...
Here's my thought on 💭 Red-Knot: Ruff's new Type Checker - Talk Python Live Stream - ... Astral is doing great things in the python industry. They are…
-
💭 Playground | Red Knot
Here's my thought on 💭 Playground | Red Knot ty, has a playground running at types.ruff.rs. You can edit code in there and see what the type checker results…
-
Markata list posts by year
I was looking back at my [[analytics]] page today and wondered what were my posts about back at the beginning. My blog is managed by [[markata]] so I looked at a few ways you could pull those posts up. Turns out it's pretty simple to do, use the markata map with a filter. !!! Note the filter is python eval that should evaluate to a boolean, all of the attributes of the post are available to filter on. Result You could use the command as well right within your shell and the same map and filte
-
💭 pypi package ty
Here's my thought on 💭 pypi package ty Astral is working on some great things around python, they have created a high standard for python tooling built on…
-
portal-platformer-devlog-1
Here is the current state of my platformer yet to really be named, I want to make something in between hollow knight and portal. Starting I made one once in make code arcace on a pybadge. It was quick and dirty, but fun to work on. It had the basic of blocks that I could move, blocks i could put a portal onto, and a goal for each level. Some levels you can just walk through and some levels required you to really think about where to place the portal. History So this version of the game is a l
-
💭 PyApp
Here's my thought on 💭 PyApp I think I'm getting really close to having a good workflow setup for using pyapp. Such an amazing project to allow developers…
-
pre-commit exclude
I run tailwind for my personal blog, whenever I update it, pre-commit goes in and fixes end of file. I'm sick of these things fighting each other, since it is a generated app it is going to et ignored from pre-commit from now on.
-
💭 valkey · PyPI
Here's my thought on 💭 valkey · PyPI python bindings for valkey, forked from redis. one notable difference I see from redis is that you can install with…
-
markata search
A side effect of [[ markata-didyoumean ]] is that we are able to implement some rudimentary search based on slug, title, tags, and description. screenshot-2025-01-31T14-50-00-094Z.png {.more-cinematic} I made a video of the early work on using this side effect to search for content on markata.dev. Replay markata-search-1.mp4 This was first implemented to solve 404 pages giving users a way to try to find the page that they tried to access. But it turned out to be a decent way to search through t
-
Markata DidYouMean
Coming in Markata 0.9.1 is far better documentation. i.e. Documentation that actually exists for everything. As part of poking around I realized that I often go to look up the docs for a plugin and forget that the path is , sometimes I might try or or or I might even forget the plugin name exactly and try something like and get a 404. So I added a plugin to markata that takes care of this. screenshot-2025-01-31T14-53-31-264Z.png {.more-cinematic} I made a quick recording of this earl
-
💭 Models - Pydantic
Here's my thought on 💭 Models - Pydantic I came accross from_attributes today it allows creation of pydantic models from objects such as a sqlalchemy…
-
💭 Top Python libraries of 2024 | Tryolabs
Here's my thought on 💭 Top Python libraries of 2024 | Tryolabs Really good listicle of new modern top python libraries from 2024. Very well done article…
-
💭 Choosing color palettes — seaborn 0.13.2 documen...
Here's my thought on 💭 Choosing color palettes — seaborn 0.13.2 documentation Good overview of seaborn color palettes. They have all sorts of different…
-
I fixed lists in my blog
A small improvement to my blog today. I was checking out my [[ now ]] post and noticed that lists have left padding but no discs or numbers. I popped open my added these lines and rebuilt. And rebuild, which I actually do with [[ casey-just ]], but showing the command here for clarity. WHY the applies {.bg-blue-900 .border-r-8 .border-black .rounded-xl .max-w-xl .p-6 .font-bold} Dude, you know that @apply is an anti-pattern in tailwind right? {.bg-green-900 .border-l-8 .border-black .rounded-x
-
one-shot-s3-cleanup
So I made a mistake in an app I am working on and ended up creating thumbnails of thumbnails, and thumbnails of thumbnails of thumbnails... etc. I was able to delete them all, but I wanted to make a one-shot script to do it. screenshot-2025-01-31T21-20-56-934Z.png One Shot App I got the idea of the one shot app from Simon Willison and replicated his setup in a chatgpt project screenshot-2025-01-12T18-16-57-443Z.png Initial Prompt s3_cleanup Here is the result of the s3_cleanup script. It ende
-
nvim-manager
I recently built a cli application as a nearly-one-shot-app called nvim-manager . It manages your nvim dotfiles install. screenshot-2025-01-31T21-21-40-707Z.png {.more-cinematic} Why {.chat-left} Don't we have stow? {.chat-right} Ya, thats not enough. {.chat-left} Why not?? {.chat-right} Inevitably shit goes sideways and I break my vim install. How is nvim manager any better nvim-manager allows you to install pinned versions of your dotfiles, your friends dotfiles, and distros in ~/.config. Thi
-
💭 cyclotruc/gitingest: Replace 'hub' with 'ingest'...
Here's my thought on 💭 cyclotruc/gitingest: Replace 'hub' with 'ingest' in any github... Gitingest has a python package on pypi that you can run with uvx,…
-
💭 kitze 🚀 on X: "holy fk I thought package.json wa...
Here's my thought on 💭 kitze 🚀 on X: "holy fk I thought package.json was bad but TIL ... kitze hating on python packaging in new ways. Python packaging has…
-
💭 4 Tips for Building a Production-Ready FastAPI B...
Here's my thought on 💭 4 Tips for Building a Production-Ready FastAPI Backend - YouTube Great list of 4 tips for running fastapi applications. Keep routes…
-
💭 Building Python tools with a one-shot prompt usi...
Here's my thought on 💭 Building Python tools with a one-shot prompt using uv run and ... This is really interesting, the lazy uv scripts are really becoming…
-
💭 PEP 723 – Inline script metadata | peps.python.o...
Here's my thought on 💭 PEP 723 – Inline script metadata | peps.python.org PEP 723 is what is inspiring all of these lazy self installing python scripts,…
-
💭 Switching from virtualenvwrapper to direnv, Star...
Here's my thought on 💭 Switching from virtualenvwrapper to direnv, Starship, and uv I've kinda fallen out of using direnv now that a lot of my projects use…
-
💭 Lazy self-installing Python scripts with uv
Here's my thought on 💭 Lazy self-installing Python scripts with uv I really like Trey's steps to making an executable python script with uv his old process…
-
thoughts 0.0.4
This is such an improvement to the backend of my website it warrants a blog post of celebration. For far too long I've been dealing with a tiny ass edit form on thoughts. I tend to not edit them, and try to get them right in one go. This is kinda the point of a thought, its a quick post meant to be the size of a tweet, but sometimes I'm leaving thoughts on a video or long post and want to make sure I have a good save point, but I just keep the thing in draft and hope I don't loose if for far
-
💭 Add a healthcheck to your FastAPI app | Pype.dev
Here's my thought on 💭 Add a healthcheck to your FastAPI app | Pype.dev Nice example of adding a healthcheck to fastapi, and integrating it with docker.…
-
💭 watchfiles
Here's my thought on 💭 watchfiles Simple, modern and high performance file watching and code reload in python. !!! note This post is a thought . It's a…
-
price an stl print on slant3d
I've been playing with 3d printing some items through the slant3d api. I've been pricing out different prints by running a slice request through their api. make a project I've been using uv for project management. It's been working well for quick projects like this while making it reproducible, I'm still all in on hatch for libraries. Get an api key You will need an api key from the slant api, which currently requires a google account and a credit card to create. slicing an stl with teh slant a
-
💭 Production-ready Python Docker Containers with u...
Here's my thought on 💭 Production-ready Python Docker Containers with uv Hynek has a sick dockerfile setup for using uv in python projects. !!! note This…
-
💭 Proper handling of None in WHERE condition · Iss...
Here's my thought on 💭 Proper handling of None in WHERE condition · Issue #109 · fast... SQLModel models ship with an , and that you can use to compare to…
-
💭 Document how to provide a negative number as an ...
Here's my thought on 💭 Document how to provide a negative number as an argument · fas... Today I learned that you cannot pass negative integers as values to…
-
💭 Will McGugan on X: "New Version of #Textual rele...
Here's my thought on 💭 Will McGugan on X: "New Version of #Textual released! 😎🤓🦾 The ... This new demo of textual is wildly awesome, so many widgets and…
-
💭 Pin versions of dependencies · Issue #2200 · Koz...
Here's my thought on 💭 Pin versions of dependencies · Issue #2200 · Kozea/WeasyPrint weazyprint was throwing me some errors, turns out that it's currently…
-
k3s config after first install
-
💭 Installing Python - uv
Here's my thought on 💭 Installing Python - uv now can install python for you. is inspired by rust's cargo, an all in one fast package and installation…
-
💭 python is simultaneously at the top and the bott...
Here's my thought on 💭 python is simultaneously at the top and the bottom of the tier... Hard to argue this take, happy to see that its at the top. With it…
-
💭 FastHX
Here's my thought on 💭 FastHX Very interesting approach to htmx and fast api. It uses separate decorators for returning template partials and json that can…
-
💭 FastHX
Here's my thought on 💭 FastHX Very interesting approach to htmx and fast api. It uses separate decorators for returning template partials and json that can…
-
diskcache as debounce
I've been using fastapi more and more lately and one feature I just started using is background tasks [[ thoughts-333 ]]. !!! seealso basic diskcache example [[ python-diskcache ]] One Background Task per db entry I am using it for longer running tasks and I don't want to give users the ability to spam these long running tasks with many duplicates running at the same time. And each fastapi worker will be running in a different process so I cannot keep track of work in memory, I have to do it in
-
💭 Background Tasks - FastAPI
Here's my thought on 💭 Background Tasks - FastAPI fastapi comes with a concept of background tasks which are functions that can be ran in the background…
-
💭 markdown-it-pyrs · PyPI
Here's my thought on 💭 markdown-it-pyrs · PyPI markdown it py running in rust claims to be 20x faster. I'll definitely look into this if markdown it py is…
-
💭 DiskCache API Reference — DiskCache 5.6.1 docume...
Here's my thought on 💭 DiskCache API Reference — DiskCache 5.6.1 documentation diskcache has a peekitem method that allows you to lookup the expire_time of a…
-
💭 PyAnnotate: Auto-generate PEP-484 annotations
Here's my thought on 💭 PyAnnotate: Auto-generate PEP-484 annotations pyannotate is a tool that uses runtime types to suggest type annotations to use. Test…
-
💭 tusharsadhwani/yen: Create virtual environments ...
Here's my thought on 💭 tusharsadhwani/yen: Create virtual environments for any Python... Create virtual environments for any Python version, without needing…
-
💭 Episode #323 Best practices for Docker in produc...
Here's my thought on 💭 Episode #323 Best practices for Docker in production - [Talk P... Great listen for anyone interested in productionizing python code…
-
💭 xxHash - Extremely fast non-cryptographic hash a...
Here's my thought on 💭 xxHash - Extremely fast non-cryptographic hash algorithm xxHash is an extremely fast non-cryptographic hash algorithm, working at RAM…
-
💭 xxhash · PyPI
Here's my thought on 💭 xxhash · PyPI I hit an issue with markata where even though a bunch of articles were cached, the site build was still slow because I…
-
Latest Page in Markata
I just implemented a latest blog post link in Markata by asking for the first post slug from the blog feed. The implementation uses the jinja_md plugin to render jinja against the markdown and a tag to redirect. Setting up the feed Feeds are setup in configuration. They provide a handy way to create an html feed, rss feed, and quickly reference a filtered set of posts like this. For more information on markata check out the full [[ markata ]] post.
-
markata
This post is a work in progress. Markata is the static site generator that I created to build my website [[ about-this-site ]]. I built it for me and I enjoy using it. I know everying it can do and I can extend it to do more easily. I have set it up for some friends to also use it and am proud that it helps them publish their content. It's a meme to create your own static site generator to make your website. Yes its funny, I don't recommend it if your not ready for the level of work that com
-
thinking about static sites in 2024
actions build k8s build fastapi sqlite
-
sick wikilink hover
Today I set up some sick wikilink hover effects using tailwind see [[ a-case-for-tailwindcss ]]. When you hover over them they show an image preview of the link that you are going to. I cant find where I have seen this but it comes from some docs sites. I'll finish this article later, just excited to see it up.
-
💭 Fields - Pydantic
Here's my thought on 💭 Fields - Pydantic and is a good pydantic combination for secret attributes such as user passwords, or hashed passwords. exclude…
-
💭 Hatch v1.10.0 - Hatch
Here's my thought on 💭 Hatch v1.10.0 - Hatch Hatch be flyin. This new release of hatch includes support for the new package installer which is just mind…
-
💭 rich.panel — Rich 13.6.0 documentation
Here's my thought on 💭 rich.panel — Rich 13.6.0 documentation This is my go to rich response container for clis written in python. It creates a nice box…
-
💭 Handling Errors - FastAPI
Here's my thought on 💭 Handling Errors - FastAPI This page shows how to customize your fastapi errors. I found this very useful to setup common templates so…
-
control minio token access
To allow access only to the , you can pass add the Resource field to the User Policy when you create a new token.
-
How to List Sqlite tables
You can inspect sqlite tables with the sqlite shell. note that you get into the shell with I also learned that , and are helper functions that query the table on the database. Here is an output from my redka database. The sqlite_master table contains all the sqlite objects type, name, tbl_name, rootpage, and sql to create them.
-
copier trust
I recently had to update my copier-gallery command to trust my own templates because some of them have shell scripts that run afterwards. Be warned that this could be a dangerous feature to run on random templates you get off the internet, but these are all mine, so if I wreck it its my own fault. All the the copier copy api can be found with help.
-
ipython f2
Today I accidentally ran f2 in ipython to discover that it opens your $EDITOR! I use this feature quite often in zsh, it is bound to for me, and since I have my environment variable set to it opens nvim when I hit . Today I discovered that Ipython has this bound to . If you know how to set it to let me know I've tried, a lot. better yet add to your .zshrc
-
python inline snapshot
is a new tool that I am trying out for python testing. It takes snapshots of your outputs and places them inline with the test. Here is the most basic starter. Now when I run my tests will fail because my assert has no value, but if I run it will fill out my snapshot values and the file will then look like this. It also works with pydantic models.
-
sqlmodel indexes
I've really been enjoying using sqlmodel for my projects that need a database. One thing that I definitely lacked on for too long was indexing my database. I hit a point with one database where it was taking 7s for pretty simple paginated queries to return 10 records. For every field that you will be querying on, you can create an index, by setting it equal to example courtesy of the docs !!! Note primary keys are indexed by default. The docs cover this pretty well, and in quite depth - Optim
-
💭 inline-snapshot
Here's my thought on 💭 inline-snapshot This is a cool snapshot testing tool that automatically creates, and updates test values for you. Starting with some…
-
Composing Typer clis
Typer makes it easy to compose your cli applications, like you might with a web router if you are more familiar with that. This allows you to build smaller applications that compose into a larger application. You will see similar patterns in the wild, namely the cli which always has the pattern. Lets setup the cli app itself first. You can put it in . Creating an app that will become a command is the same as creating a regular app in Typer. We need to create a callback that will become o
-
One Day Build - Play Outside
Inspired by Adam Savage and his One Day builds on youtube. I often build things, and want to make them generally useful for others and over configure out of the gate. This project is purely for me inspired by a need I have. play-outside !How-To This post will not directly show how to make a weather app, but document the process that I went through to make mine. It will show the tools that I used to make it, and the final result. The Situation It often goes in our house ask dad while he is bus
-
jinja macros
I am working on a page for htmx-patterns and I ran into a situation with lots of duplication. Especially when i am using tailwind I run into situations where the duplication can get tedious to maintiain. The solution I found is macros. Now I can use the same code for all of my links, and call the macro to use it.
-
jinja loop variable and htmx
jinja has a loop variable that is very handy to use with htmx. Whether you want to implement a click to load more or an infinite scroll this loop variable is very handy. Now for every chunk of contacts that we load we will trigger the infinite scroll by loading more once the last one has intersected the screen.
-
fastapi jinja url_for with query params
Out of the box Starlette does not support url_for with query params. When trying to use url_for with query params it throws the following error. In my searching for this I found starlette issue #560 quite helpful, but not complete, as it did not work for me. !!! Note "https" If you want url_for to work in production you need some way to convert http to https. Here is how I make it work, for local development I then for each environment that I am running on a server I include it in the list
-
setting up a kind cluster with argocd installed
Kind (Kubernetes in Docker) is a tool that makes it easy to create and tear down local clusters quickly. I like to use it to test out new workflows. Argocd is a continuous delivery tool that makes it easy to setup gitops workflows in kubernetes. Here is how you can setup a new kind cluster and install argocd into it using helm, the kubernetes package manager. If you want to add repos and apps to your cluster you can use the argo cli to do that, but first you will need forward the argocd port an
-
feeds
I have several feeds that you can consume my content with. Many posts will cross between them, so if you would like to follow a certain stream of content in your rss reader thats what this is for. !!! Note TLDR If you are looking for everything look here https://waylonwalker.com/archive/rss.xml All This is a feed of everything that I have published on my site to date. web rss til This is a feed of all of my TIL's (Today I Learned). These are generally short single topic posts that I write up
-
💭 fastapi decorators
Here's my thought on 💭 fastapi decorators I've been using these decorators to modify the behavior of specific routes. It will do things like 404 admin only…
-
💭 fastapi https url_for
Here's my thought on 💭 fastapi https url_for jinja's in fastapi does not account for https by default, there is probably a better way, but this is a way…
-
💭 My New Python Lsp Setup
Here's my thought on 💭 My New Python Lsp Setup I figured out the killer combination for python lsp servers, ruff and jedi! ruff does all of the diagnostics…
-
💭 flake8-to-ruff · PyPI
Here's my thought on 💭 flake8-to-ruff · PyPI Underrated python library to on board ruff, or just use it on a project where its not the norm. ruff claims…
-
💭 Creating One-To-Many Relationships in Flask-SQLA...
Here's my thought on 💭 Creating One-To-Many Relationships in Flask-SQLAlchemy - YouTube Great example from Anthony showing how easy it is to practice…
-
💭 python - Concepts of backref and back_populate i...
Here's my thought on 💭 python - Concepts of backref and back_populate in SQLalchemy? ... Today I came across some sqlalchemy models that created some…
-
💭 Read a Range of Data - LIMIT and OFFSET - SQLMod...
Here's my thought on 💭 Read a Range of Data - LIMIT and OFFSET - SQLModel Today I was running some sqlmodel queries through the sqlalchemy orm. Admittedly…
-
💭 Template Designer Documentation — Jinja Document...
Here's my thought on 💭 Template Designer Documentation — Jinja Documentation html code generated by my jinja templates generally look half garbage because of…
-
thoughts on unit tests
thoughts on unit tests cover image Theo's response puts a lot of my feelings about unit testing into words. Many of us have grown up in this world preaching unit testing. We often hear these statements "Everything must be unit tested, tests make code more maintainable." In reality when we are not writing complex low level code unit tests are probably the wrong approach. thought 192, a thought about theo's reaction to prime's unit testing Most of us are assemblers So much of software engineerin
-
💭 We Finally Agree On Unit Tests - YouTube
Here's my thought on 💭 We Finally Agree On Unit Tests - YouTube Theo's response puts a lot of my feelings about unit testing into words. It's crazy how cargo…
-
💭 Mastodon.py — Mastodon.py 1.8.1 documentation
Here's my thought on 💭 Mastodon.py — Mastodon.py 1.8.1 documentation Mastadon.py is a python api client for mastadon that makes it easy to cross post to…
-
💭 FastAPI - dependency inside Middleware? - Stack ...
Here's my thought on 💭 FastAPI - dependency inside Middleware? - Stack Overflow After struggling to get dependencies inside of middleware I learned that you…
-
💭 Handling Errors - FastAPI
Here's my thought on 💭 Handling Errors - FastAPI This page shows how to customize your fastapi errors. I found this very useful to setup common templates so…
-
💭 logs with FastAPI and Uvicorn · Issue #1508 · ti...
Here's my thought on 💭 logs with FastAPI and Uvicorn · Issue #1508 · tiangolo/fastapi Setting an additional log handler to the uvicorn logger for access logs…
-
💭 How to group FastAPI endpoints in Swagger UI?
Here's my thought on 💭 How to group FastAPI endpoints in Swagger UI? Here is a full example from the post. !!! note This post is a thought . It's a short…
-
💭 Show some equivalent list comprehensions in filt...
Here's my thought on 💭 Show some equivalent list comprehensions in filter examples · ... I often want to reach for non existing list comprehensions in jinja…
-
💭 DataDog/ddqa: Datadog's QA manager for releases ...
Here's my thought on 💭 DataDog/ddqa: Datadog's QA manager for releases of GitHub repo... DataDog ddqa is building out a textual app and deploying it with…
-
💭 Path Operation Advanced Configuration - FastAPI
Here's my thought on 💭 Path Operation Advanced Configuration - FastAPI From the Docs trailing slash I've had better luck just routing both naked and trailing…
-
💭 Protect API docs behind authentication? · Issue ...
Here's my thought on 💭 Protect API docs behind authentication? · Issue #364 · tiangol... You can protect your fastapi docs behind auth so that not only can…
-
Stripe Cancellations in FastAPI and SQLModel
Today I am working on fokais.com, trying to get to a point where I can launch by workig through stripe integrations. This is my first time using stripe, so there has been quite a bit to learn, and I am probably building in more than I need to before launching, but I am learning, and not in a rush to launch. I am building the fokais backent in python primarilyt with fastapi and sqlmodel on sqlite. My billing integration is going to be all Stripe. Stripe Subscription Cancellations Docs Here is a l
-
Looking for a Heroku replacement, What I found was...
I've long hosted my personal blog as a static site on waylonwalker.com. It's all markdown, converted to html, and shipped as is. It's been great, I've moved it from GitHub Pages, to Netlify, tried Vercel for a minute, and have landed on Cloudflare Pages. Each migration has not really been that hard, it's just pointing ci to a different host after the site has built. What about server side Now the part that I have struggled with is how to cheaply host a server rendered application that can just l
-
💭 Will McGugan on X: "Just a wee thing that came u...
Here's my thought on 💭 Will McGugan on X: "Just a wee thing that came up today. I thi... Textual is so sick, Will just made a live markdown editor in the…
-
💭 Sebastián Ramírez on X: "Now @FastAPI has 65k+ G...
Here's my thought on 💭 Sebastián Ramírez on X: "Now @FastAPI has 65k+ GitHub stars! ✨... Fastapi passes flask in GitHub stars! screenshot of…
-
Hosted Platform Brainstorm
shots shots templates gradient backgrounds premium feature upload your own background ogs scrape og info, generate og image og's build on shots, they are just hosted template endpoints that get a screenshots potentially auto approve og's from rss drops drop a file to share limited size, limited upload MB's / month archives deep permalink a website https://pypi.org/project/pywebcopy/ og-links ai-covers give me an url, and a selector, I will read your article and give you an ai generated cover ima
-
💭 Episode #433 Litestar: Effortlessly Build Perfor...
Here's my thought on 💭 Episode #433 Litestar: Effortlessly Build Performant APIs - [T... Litestar is an interesting api framework similar to fastpi, that I…
-
💭 florimondmanca/arel: Lightweight browser hot rel...
Here's my thought on 💭 florimondmanca/arel: Lightweight browser hot reload for Python... arel is a "Lightweight browser hot reload for Python ASGI web apps"…
-
💭 Automatic browser reloading in FastAPI
Here's my thought on 💭 Automatic browser reloading in FastAPI I just discovered arel for hot reloading python applications when content changes from this…
-
how to host static content with fastapi
I wanted to host some static files through fastapi. Typical use cases for this might be some static web content like html/css/js. It could also be images or some data that doesn't need dynamically rendered. From the Docs The docs cover how to host static files, and give this solution that is built into fastapi. https://fastapi.tiangolo.com/tutorial/static-files/ {.hoverlink} Authenticated Static Files Thanks to #858 . OscartGiles posted this solution to add authentication to static files. I tr
-
Setup Tailwind for Jinja
I've recently given tailwindcss a second chance and am really liking it. Here is how I set it up for my python based projects. https://waylonwalker.com/a-case-for-tailwindcss {.hoverlink} Installation is used to install the cli that you will need to configure and compile tailwindcss. Setup You will need to create a tailwind.config.js file, to get this you can use the cli. Using tailwind with jinja templates To set up tailwind to work with jinja templates you will need to point the tailwind conf
-
💭 Bigger Applications - Multiple Files - FastAPI
Here's my thought on 💭 Bigger Applications - Multiple Files - FastAPI Fastapi lets you tag your 's so that the swagger docs are grouped according to the…
-
💭 sqlite_utils Python library - sqlite-utils
Here's my thought on 💭 sqlite_utils Python library - sqlite-utils sqlite-utils is primarily a cli tool for sqlite operations such as enabling full text…
-
💭 simonw/datasette-render-markdown: Datasette plug...
Here's my thought on 💭 simonw/datasette-render-markdown: Datasette plugin for renderi... datasette really does everything doesn't it! !!! note This post is…
-
💭 simonw/shot-scraper: A command-line utility for ...
Here's my thought on 💭 simonw/shot-scraper: A command-line utility for taking automat... Daaaang, this is such an elegantly simple way to get web screenshots…
-
💭 shot-scraper: automated screenshots for document...
Here's my thought on 💭 shot-scraper: automated screenshots for documentation, built o... An interesting way to build automatically annotaatd docs with arrows…
-
💭 Create Models with a Many-to-Many Link - SQLMode...
Here's my thought on 💭 Create Models with a Many-to-Many Link - SQLModel Creating many to many relationships with sqlmodel requires a LinkTable Model. The…
-
💭 python - How to use a Pydantic model with Form d...
Here's my thought on 💭 python - How to use a Pydantic model with Form data in FastAPI... I went down the route of leveraging the extention in htmx, but…
-
💭 Debugging | pywebview
Here's my thought on 💭 Debugging | pywebview How to enable debug mode in pywebview. !!! note This post is a thought . It's a short note that I make about…
-
💭 `ValueError: Constraint must have a name` in ale...
Here's my thought on 💭 in alembic 1.10.0 · ... After a nasty time with alembic upgrades, thoughts is about to get a new users table. This may have came…
-
💭 Use Alembic Check to check for possible upgrades
Here's my thought on 💭 Use Alembic Check to check for possible upgrades Since using alembic I have been just running out a new revision checking its content…
-
💭 API — Jinja Documentation
Here's my thought on 💭 API — Jinja Documentation 🤯 jinja comes with a loader to templates! Defihnitely need to look at this for , as is till one of the…
-
💭 API — Jinja Documentation
Here's my thought on 💭 API — Jinja Documentation I've definitely been missing out on setting up a proper jinja loader on a few projects, I need to lean on…
-
💭 Read a Range of Data - LIMIT and OFFSET - SQLMod...
Here's my thought on 💭 Read a Range of Data - LIMIT and OFFSET - SQLModel Implement paging in sqlmodel with where, limit, and offset. !!! note This post is…
-
💭 Center things - Textual
Here's my thought on 💭 Center things - Textual How to center things in textual. Textual has a very unique way of styling text user interfaces for the…
-
💭 s3-tree · PyPI
Here's my thought on 💭 s3-tree · PyPI Super useful way to show a tree view of an s3 bucket's structure! !!! note This post is a thought . It's a short…
-
💭 python - SQLAlchemy ORDER BY DESCENDING? - Stack...
Here's my thought on 💭 python - SQLAlchemy ORDER BY DESCENDING? - Stack Overflow How to sort results from a sqlalchemy based orm. I needed this to enable…
-
💭 Using Rich Inspect to interrogate Python objects...
Here's my thought on 💭 Using Rich Inspect to interrogate Python objects - Textual I love rich inspect. It's one of my most often used features of rich. It…
-
💭 > htmx ~ The json-enc Extension
Here's my thought on 💭 htmx ~ The json-enc Extension json-enc extension converts url encoded form values into json encoded data, this is very useful for…
-
💭 Header Parameters - FastAPI
Here's my thought on 💭 Header Parameters - FastAPI Getting request headers in fastapi has a pretty nice stetup, it allows you to get headers values as…
-
💭 1j01/textual-paint: :art: MS Paint... in your te...
Here's my thought on 💭 1j01/textual-paint: :art: MS Paint... in your terminal. 1j01 created a complete working clone of ms paint in the terminal using the…
-
💭 sqlite-utils now supports plugins
Here's my thought on 💭 sqlite-utils now supports plugins As the title states sqlite-utils now supports plugins. I dug in just a bit and Simon implemented…
-
💭 Form Data - FastAPI
Here's my thought on 💭 Form Data - FastAPI Getting form data inside of fastapi was not intuitive to me at first. Everything I had used in fastapi leaned on…
-
💭 pywebview
Here's my thought on 💭 pywebview I am creating this post from a desktop app that I created in 3 lines. !!! note This post is a thought . It's a short…
-
💭 Column INSERT/UPDATE Defaults — SQLAlchemy 1.4 D...
Here's my thought on 💭 Column INSERT/UPDATE Defaults — SQLAlchemy 1.4 Documentation sqlalchemy server_defaults end up as defaults in the database when new…
-
💭 Template Designer Documentation — Jinja Document...
Here's my thought on 💭 Template Designer Documentation — Jinja Documentation A feature of jinja that I just discovered is including sub templates. Here is an…
-
💭 Templates - FastAPI
Here's my thought on 💭 Templates - FastAPI A guide to add Jinja2Templates to fastapi. !!! note This post is a thought . It's a short note that I make about…
-
💭 python 3.x - FastAPI redirection for trailing sl...
Here's my thought on 💭 python 3.x - FastAPI redirection for trailing slash returns no... I am trying to use htmx on a new fastapi site for my thoughts, and…
-
💭 Static Files - FastAPI
Here's my thought on 💭 Static Files - FastAPI Mounting static files in fastapi. !!! note This post is a thought . It's a short note that I make about…
-
💭 First-class session support in FastAPI · Issue #...
Here's my thought on 💭 First-class session support in FastAPI · Issue #754 · tiangolo... Here is a snippet provided by @tiangolo to store the users jwt…
-
💭 Harlequin SQL IDE - DuckDB
Here's my thought on 💭 Harlequin SQL IDE - DuckDB Harlequin is a pretty sweet example of what textual can be used to create. Its a terminal based sql ide…
-
💭 Python API - DuckDB
Here's my thought on 💭 Python API - DuckDB To persist data in duckdb you need to first make a connection to a duck db database. Then work off of the…
-
💭 SQL on Pandas - DuckDB
Here's my thought on 💭 SQL on Pandas - DuckDB duckdb can just query any pandas dataframe that is in memory. I tried running it against a list of objects and…
-
💭 pytest-subtests · PyPI
Here's my thought on 💭 pytest-subtests · PyPI pytest-subtests is a package to register multiple subtests within a similar test function. !!! note This post…
-
💭 doyensec/wsrepl: WebSocket REPL for pentesters
Here's my thought on 💭 doyensec/wsrepl: WebSocket REPL for pentesters Very inspiring textual project to check out how they set up the ui. Their intro video…
-
💭 Doyensec on Twitter: "Announcing wsrepl, the Web...
Here's my thought on 💭 Doyensec on Twitter: "Announcing wsrepl, the WebSocket testing... wsrepl is an epic websocket repl built in python on the textual…
-
💭 Filter Data - WHERE - SQLModel
Here's my thought on 💭 Filter Data - WHERE - SQLModel When fetching pydantic models from the database with sqlmodel, and you cannot select your item by id,…
-
💭 URL Decoding query strings or form parameters in...
Here's my thought on 💭 URL Decoding query strings or form parameters in Python | URLD... In order to turn url encoded links back into links that I would find…
-
💭 Python Sucks And I LOVE It | Prime Reacts - YouT...
Here's my thought on 💭 Python Sucks And I LOVE It | Prime Reacts - YouTube Prime reviews an article with some hot takes about python being slow and quirky,…
-
💭 Flask Trailing Slashes 404
Here's my thought on 💭 Flask Trailing Slashes 404 In flask apps I often get a 404 for routes with a trailing slash. This Stack Overflow post shows how to…
-
💭 python script is not found
Here's my thought on 💭 python script is not found When setting up a new machine, vm, docker image you might be installing command line tools from places like…
-
Python scandir ignores hidden directories
The next version of markata will be around a full second faster at building it's docs, that's a 30% bump in performance at the current state. This performance will come when virtual environments are stored in the same directory as the source code. "One lone jedi stands in Glowing chains of interconnected network of technological cubes, in the middle of a futuristic cyberpunk dubai city, in the art style of dan mumford and marc simonetti, atmospheric lighting, intricate, volumetric lighting, beau
-
Pycon 2023
Keynote Speaker - James Powell I don't want to be an expert python developer. https://www.youtube.com/watch?v=iKzOBWOHGFE {.hoverlink} keynote-speaker---james-powell usage of keyword only arguments to prevent pain for users of libraries In an alternate timeline the maintainer of newton could have chose to use keyword only arguments to prevent pain for users of libraries, or poor api design due to fear of changing api on users.
-
Getting Started with Pydantic
Pydantic is a Python library for serializing data into models that can be validated with a deep set of built in valitators or your own custom validators, and deserialize back to JSON or dictionary. Installation To install pydantic you will first need python and pip. Once you have pip installed you can install pydantic with pip. Always install in a virtual environment Creating a Pydantic model To get started with pydantic you will first need to create a Pydantic model. This is a python class that
-
Playing Star Wars Text Adventure with a 10 yr old
The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The following is a playthrough of StarThe following is a playthrough of Star
-
Pydantic and singledispatch
I was reading about pydantic-singledispatch from Giddeon's blog and found it very intersting. I'm getting ready to implement pydantic on my static site generator markata , and I think there are so uses for this idea, so I want to try it out. The Idea Let's set up some pydantic settings. We will need separate Models for each environment that we want to support for this to work. The whole idea is to use and type hints to provide unique execution for each environment. We might want something like
-
pipx textual devtools
I really like having global cli command installed with pipx. Since textual (the css release) is out I want to be able to pop into textual devtools easily from anywhere. "rusting tape machine robot, cinematic lighting, detailed, cell shaded, 4 k, warm colours, concept art, by wlop, ilya kuvshinov, artgerm, krenz cushart, greg rutkowski, pixiv. cinematic dramatic atmosphere, sharp focus, volumetric lighting, cinematic lighting, studio quality" -s50 -W832 -H416 -C12.0 -Ak_lms -S2404332231 Pipx In
-
textual app devtools
I am working through the textual tutorial, and I want to put it in a proper cli that I can pip install and run the command without . This is a fine pattern, but I also want this to work when I don't have a file to run. "An astronaut working in a lab, hacking on a computer terminal, htop is running, shallow depth of field beakers, test tubes, volumetric lighting, pink lighting, by victo ngai, killian eng vibrant colours, dynamic lighting, digital art" -s50 -W768 -H448 -C7.5 -Ak_lms -S3617210203
-
caching python virtual environments in GitHub Acti...
-
Markata GitHub Pages
11ty https://www.rockyourcode.com/how-to-deploy-eleventy-to-github-pages-with-github-actions/ hugo puts it in the base url https://gohugo.io/getting-started/configuration/#baseurl mkdocs uses a special cli build command https://squidfunk.github.io/mkdocs-material/publishing-your-site/#github-pages
-
Markata Supports Jinja Plugins 0.5.0.dev2
Markata now allows you to create jinja extensions that will be loaded right in with nothing more than a . From the Changelog The entry for 0.5.0.dev2 from markata's changelog Created entrypoint hook allowing for users to extend marka with jinja exensions #60 0.5.0.dev2 "cybernetic soldier working on a rusting tape machine robot, cinematic lighting, detailed, cell shaded, 4 k, warm colours, concept art, by wlop, ilya kuvshinov, artgerm, krenz cushart, greg rutkowski, pixiv. cinematic dramatic a
-
Using Django Rest Framework inside react
-
Getting Started with Django REST framework
In my adventure to learn django, I want to be able to setup REST api's to feed into dynamic front end sites. Potentially sites running react under the hood. cell shaded full body shot of a cybernetic blue soldier with glowing eyes working ina lab, llustration, post grunge, pink glowing volumetric lighting, engulfed in smoke and fog, concept art by josan gonzales and wlop, by james jean, Victo ngai, David Rubín, Mike Mignola, Laurie Greasley, highly detailed, sharp focus,alien,Trending on Artstat
-
Markata now uses hatch
Markata now uses hatch as its build backend, and version bumping tool. , and are completely gone. "An astronaut working in a lab, there is a series of eggs ready to hatch baby snakes on the table, experiments running, beakers, test tubes, cyberpunk trending on artstation, neon lighting, volumetric lighting, pink lighting" -s50 -W800 -H450 -C7.5 -Ak_lms -S4048189038 {.more-cinematic} 0.5.0 is big Markata 0.5.0 is now out, and it's huge. Even though it's the backend of this blog I don't actuall
-
django polls app
creating a new app called polls
-
django create superuser
My next step into django made me realize that I do not have access to the admin panel, turns out that I need to create a cuper user first. "cybernetic soldier working on a rusting tape machine robot, cinematic lighting, detailed, cell shaded, 4 k, warm colours, concept art, by wlop, ilya kuvshinov, artgerm, krenz cushart, greg rutkowski, pixiv. cinematic dramatic atmosphere, sharp focus, volumetric lighting, cinematic lighting, studio quality" -s50 -W832 -H416 -C12.0 -Ak_lms -S3309980874 Run Mig
-
django disallowed host
I am continuing my journey into django, but today I am not at my workstation. I am ssh'd in remotely from a chromebook. I am fully outside of my network, so I can't access it by localhost, or it's ip. I do have cloudflared tunnel installed and dns setup to a . Settings I found this in and yolo, it worked first try. I am in from my remote location, and even have auth taken care of thanks to cloudflare. I am really hoping to learn how to setup my own auth with django as this is one of the t
-
Trying out django
I have no experience in django, and in my exploration to become a better python developer I am dipping my toe into one of the most polished and widely used web frameworks Django to so that I can better understand it and become a better python developer. If you found this at all helpful make sure you check out the django tutorial "An atronaut working in a lab, there is a long snake working along side, shallow depth of field beakers, test tubes, volumetric lighting, pink lighting, by victo ngai,
-
how to escape variables in jinja templates
While updating my site to use Markata's new configurable head I ran into some escaping issues. Things like single quotes would cause jinja to fail as it was closing quotes that it shouldnt have. Nuclear core being help up by glowing neon wires, cyberpunk synthwave, intricate abstract. delicate artwork. by tooth wu, wlop, beeple, dan mumford. pink volumetric lighting, octane render, trending on artstation, greg rutkowski very coherent symmetrical artwork. cinematic, hyper realism, high detail, o
-
I've made my ipython config too complicated, let's...
When I am developing python code I often have a repl open alongside of it running snippets ofcode as I go. Ipython is my repl of choice, and I hace tricked it out the best I can and I really like it. The problem I recently discovered is that I have way overcomplicated it. What Have I done?? So in the past the way I have setup a few extensions for myself is to add something like this to my directory. It sets up some things like rich highlighting or in this example automatic imports. I even
-
markata extend in configuration
Astronauts stunting some stylish color explosion {.more-cinematic} A long needed feature of markata has been the ability to really configure out templates with configuration rather. It's been long that you needed that if you really want to change the style, meta tags, or anything in the head you needed to write a plugin or eject out of the template and use your own. Adding some Head Now you can add some extra style to your site with the existing built-in template. You can have more than one Hea
-
Create a new Python Project with the Hatch Cli
I'm really getting into using hatch as my go to build system, and I am really liking it so far. I am slowly finding new things that just work really well. is one of those things that I didn't realize I needed until I had it. Hatch new cover image creating new versions created by myself with stable diffusion hatch-new-cli Note! I am running all of these commands with pipx. I like to use pipx for all of my system level cli applications. To emphasis this point in the article I am going to use
-
Versioning Python Projects with Hatch
Hatch has an amazing versioning cli for python packages that just works. It takes very little config to get going and you can start bumping versions without worry. Hatch version cover image {.more-cinematic} creating new versions created by myself with stable diffusion project layout For trying out the cli let's make a simple project with the terrible name . pyproject.toml The main hero of this post is the . This is what defines all of our PEP 517 style project setup. statically versioni
-
From Markdown to Blog with Markata | lightning tal...
Markata is a great python framework that allows you to go from markdown to a full website very quickly. You can get up and running with nothing more than Markdown. It is also built on a full plugin architecture, so if there is extra functionality that you want to add, you can create a plugin to make it behave like you want. Full transparancy... I built markata. The talk The talk is live on YouTube. Make sure you check out the other videos from the conference. There were quite a few quality t
-
I spoke at Python Web Conf 2022
I spoke at python webconf in March 2022 about how I deploy this blog on a continuous basis. Building this blog has brought me a lot of benefits. I have a set of custom curated notes to help describe a problem and how to solve it to me. At theis point it's not uncommon to google an Issue I am having and finding my own blog with exactly the solution I need at the top. I also bump into people from time to time that recognize me from the blog, its a nice conversation starter, and street cred. The
-
Packages I Maintain
Markata I open sourced the static site framework that I use to build my-blog among other side projects. It's a plugins all the way down static site generator, that makes me happy to use. {% gh_repo_list_topic "waylonwalker", "markata" %} Repos used to build this blog my-blog is built on a number of small repos. I set it up this way so that creating content is fast and easy to do. I don't have to worry about carrying around large images with my lightweight text files just to make some posts. {%
-
I turned on 2FA for all of my pypi packages
I got all the pypi packages that I own behind 2 factor authentication. 💪 Recently this really made it's rounds in the python news since pypi was requiring critical package maintainers to have 2FA on and even offering them hardware tokens to help them turn this on. I feel like this caused a bit of confusion as turning on 2FA does not mean that you need to do anything different to deploy a package, and it DOES NOT require a hardware token. You can continue using your favorite 2FA app. You might
-
I was on Talk Python
After years of listening to talkpython.fm I had the honor to be part of episode-337 to talk about Kedro for maintainable data science. I was quite nervous to talk on a show that I helped shape my career in such a profound way. I started my journey towards software engineering near Michaels first few episodes. His discussions with such great developers over the years has made an huge impact on my skill. It has always given me great advice and topics to go deeper on. During the episode I tried m
-
use vim to remove duplicates
I just love how some features of vim are so discoverable and memorable once you really start to grasp it. Sorting and uniqing your files or ranges is one of those examples for me. I recently used this to dedupe my autogenerated links section for rich-syntax-range-style . More often I am using it to sort and uniqify objects like arrays and lists. Here is what the markdown looks like. Then typing yields a uniqly sorted list of links.
-
Highlighting text ranges with Rich | python
Today I've been playing with py-tree-sitter a bit and I wanted to highlight match ranges, but was unable to figure out how to do it with rich , so I reached out to @textualizeio for help. https://twitter.com/_WaylonWalker/status/1562469770766589952 While waiting for that reply let's show how we got this far. imports Lets import all the classes that we need from rich and setup a console to print to. some code Now we need some code to highlight. I am going to rip my from another post . prin
-
Highlighting text ranges with Rich | python
{% for year in markata.map("date.year", filter='published')|unique %} {{ year }} {% for post in markata.map('post', filter="published and date.year == "+year|string, sort='date') %} [{{ post.title }} - {{ post.date.month }}/{{ post.date.day }}](/{{ post.slug }}) {% endfor %} {% endfor %}
-
markata 0.3.0 is 15-20% faster
image from Dall-e {.more-cinematic} a sprinter edging out his opponent by Dall-e It's about time to release Markata 0.3.0. I've had 8 pre-releases since the last release, but more importantly it has about 3 months of updates. Many of which are just cleaning up bad practices that were showing up as hot spots on my reports Markata started off partly as a python developer frustrated with using nodejs for everything, and a desire to learn how to make frameworks in pluggy. Little did I know how f
-
How to Disable Kedro Viz | or any kedro hook
-
Gpu Stats in my linux statusbar
polybar gpu stats
-
get the version of any python package | python api
A common meta thing that I need in python is to find the version of a package. Most of the time I reach for , but that does not always work. but not all projects have a In searching the internet for an answer nearly every one of them pointed me to . This works for most projects, but is simply a convention, its not required. Not all projects implement a , but most do. I've never seen it lie to me, but there is nothing stopping someone from shipping mismatched versions. If you maintain a pro
-
Running My Blog on 3.11-dev
3.10.5: 109.441 3.11-dev: 108.856
-
Manage your displays with xrandr manager
a stable diffusion done with a111 web ui xrandr is a great cli to manage your windows in a linux distro using x11, which is most of them. The issue is that I can never remember all the flags to the command, and if you are using it with something like a laptop using a dock the names of all the displays tend to change every time you redock. This makes it really hard to make scripts that work right every time. Homepage Check out the deresmos/xrandr-manager for more details on it. installation xran
-
IPython q to exit
So many terminal applications bind to exit, even the python debugger, its muscle memory for me. But to exit ipython I have to type out . This is fine, but since q is muscle memory for me I get this error a few times per day. After digging way too deep into how IPython implements its I realized there was a very simple solution here. automatically imports all the scripts you put in your profile directory, all I needed was to create with the following. It was that simple. This is not a gam
-
Copier Slugify | python templating | using cookiec...
It's no secret that I love automation, and lately my templating framework of choice has been copier. One hiccup I recently ran into was having spaces in my templated directory names. This makes it harder to run commands against as you need to escape them, and if they end up in a url you end up with ugly all over. Cookiecutter has the solution Yes the solution comes from a competing templating framework. I install copier with pipx, so I need to inject cookiecutter in to my copier environment
-
Textual has devtools
Textual has devtools in the upcoming css branch, and its pretty awesome! It's still early Textual is still very early and not really ready for prime time, but it's quite amazing how easy some things such as creating keybindings is. The docs are coming, but missing right now so if you want to use textual be ready for reading source code and examples. On to the devtools As @willmcgugan shows in this tweet it's pretty easy to setup, it requires having two terminals open, or using tmux, and curren
-
Python atexit
totally guessed at this post's date I'm still trying to understand this one, but this is how you force a python object to stop atexit.
-
The one pip config you need to have
Whenever you are installing python packages, you should always use a virtual environment. pip makes this easy to follow by adding some configuration to pip. require-virtualenv Pip is the pacakage tool for python. It installs third-party packages and is configurable. One of the configuration settings that I highly reccommend everyone to add is . This will stop pip from installing any packages if you have not activated a virtualenv. why python packages often require many different dependencie
-
pyenv no module named '_sqlite3'
I've been trying to adopt pyenv for a few months, but have been completely blocked by this issue on one of the main machines I use. Whenever I start up ipython I get the following error. I talked about why and how to use pyenv along with my first impressions in this post pyenv/issues/678 According to #678 I need to install on ubuntu to resolve this issue. install libsqlite3-dev can be installed using apt But wait.... When I make a fresh env and install ipython I still get the same error an
-
Using Different versions of python with pipx | pye...
"cell shaded, long, full body, shot of a cybernetic blue soldier with glowing pink eyes, llustration, post grunge, cinebatic dramatic atmosphere, sharp focus, pink glowing volumetric lighting, concept art by josan gonzales and wlop, by james jean, Victo ngai, David Rubín, Mike Mignola, Laurie Greasley, highly detailed, sharp focus,alien,Trending on Artstation, HQ, deviantart, art by artgem" -s50 -W832 -H416 -C7.5 -Ak_lms -S70567464 I love using pipx for automatic virtual environment management o
-
LIVE-REPLAY - Python dev | Markata todoui | 4/6/20...
https://youtu.be/-42A5210HYo Super fun steam Broadcasted live on Twitch -- Watch live at https://www.twitch.tv/waylonwalker We worked on markata todoui, a command tui trello board written in python using only markdown files to store the data. I love markdown and I want to make this my workflow. During this stream we get RAIDED by TEEJ_DV! and chat about tmux a bit before calling the changes to markata-tui good and signing off. dotfiles: https://github.com/WaylonWalker/devtainer today's project:
-
convert your qutebrowser config to config.py
When you first start qutebrowser It will create some config files in your home directory for you, but they will be empty. Config As far as I know qutebrowser will create this default config out of the box for you, if it doesn't, then somehow it just appeared for me 😁. Why convert You might want to confvert if you are more comfortable with the python config, or if like me you just want config in one place and you are stealing configuration options from others who have thiers in config.py. Convert
-
qutebrowser clean up all status bars
I really like the super clean look of no status menus, no url bar, no bookmarks bar, nothing. Don't get me wrong these things are useful, but honestly they take up screen real estate and I RARELY look at them. What I really want is a toggle hotkey. I found this one from one of DT's youtube video's. I can now tap and both the status bar at the botton and the address bar at the top disappear.
-
List all git commits with GitPython
I am getting ready to do some timeseries analysis on a git repo with python, my first step is to figure out a way to list all of the git commits so that I can analyze each one however I want. The GitPython library made this almost trivial once I realized how. This returns a generator, if you are iterating over them this is likely what you want. The generator will return objects with lots of information about each commit such as , , , , and .
-
Using Git from Python
is a python api for your git repos, it can be quite handy when you need to work with git from python. Use Case I recently made myself a handy tool for making screenshots in python and it need to do a git commit and push from within the script. For this I reached for . [[ screenshot-to-blog ]] Installation is a python library hosted on pypi that we will want to install into our virtual environments using pip. Create a Repo Object Import Repo from the git library and create an instance of the
-
How I Quickly Capture Screenshots directly into My...
When I am creating blog posts it's often helpful to add screenshots to them to illustrate what I see on my screen. Sometimes I lack good screenshots in my posts because it just takes more effort than I have in the moment, and I prioritize making content over making perfect content. Making Screenshots When I have something to take a screenshot of, I need to take the shot, optimize the image, often convert it to a better format, publish it, and create a the img tag in my blog. take screenshot opti
-
Copier < 6.0.0b0 considered dangerous
Copier is a fantastic templating library written in python, but older versions have a dangerous bug if you are using it inside of existing directories. !!UPDATE!! As of May 15, 2022, the stable release of copier now includes these changes, if you have not already make sure you update. This is a PSA I Use copier several times per day and get fantastic benefit from this project, this post is not intended to crap all over copier in any way, but is rather a PSA for other users who do use copier like
-
pipx on windows
Python, click install Edit the System Environment Variables Environment Variables button Add the following path to your users Path Variable
-
PyOhio CFP's
Here are some CFP's that I used for PyOhio 2022. https://pretalx.com/pyohio-2022/cfp {.hoverlink} Idea to blog post in minutes - Shorter Markata is a plugins all the way down static site generator, that covers all the things you need to go from markdown to a blog site out of the box. Since it's plugins all the way down you can also rip out all the default plugins, and do something completely different with the lifecycle. Lets build a whole blog site in 5 minutes. Add Kedro to your Pandas Workfl
-
Python sys.excepthook
Sometimes you just want python to do something else when you hit an exception, maybe that's fire a text, slack message, email, or system notification like I wanted. I am working on a quick and dirty python script designed to take screenshots and land them on my website in a single hotkey. With it being designed to run with a hotkey, if it were to error I would not see it. I could have gone down a logging route, but honestly this is meant to be quick, dirty, and work on my system for me. I just
-
pygame events are queued | Don't make this mistake
pygame events are stored in a queue, by default the most suggested way shown in all tutorials " " the queue, which removes all the messages. start up pygame You don't necessarily need a full boilerplate to start looking at events, you just just need to and to capture any keystrokes you need a window to capture them on, so you will need a display running. get some events Let's use pygames normal method to get events. printing the events reveal this Lets get some more events. Let's say that we
-
Display Sprites in Pygame | Load and Blit
One of the most essential concepts of pygame to start making a game you will need to understand is loading images and blitting them to the screen. blit stands for block image transfer, to me it feels a lot like layering up layers/images in photoshop or Gimp. Loading an image I started by making a spotlight in Gimp, by opening a 64x64 pixel image and painting the center with a very soft brush. the spotlight I created in gimp This is what it looks like Now we can load this into pygame. Convertin
-
Dunk is my new diff pager
Dunk is a beautiful git diff tool built on top of rich . Browsing through twitter the other day I discovered it through this tweet by _darrenburns . https://twitter.com/_darrenburns/status/1510350016623394817 {.hoverlink} Dunk is beta Before I dive in deep, I do want to mention that Dunk is super new and beta at this point. I am making it my default pager, because I know what I am doing and can quickly shift back if I need to, no sweat. If you are a little less comfortable with the command
-
Pygame Boilerplate Apr 2022
I'm poking a bit into gamedev. Partly to better understand, partly because it's stretching different parts of my brain/skillset than writing data pipelines does, but mostly for the experience of designing them with my 9yo Wyatt. pygame boilerplates I've seen several pygame boilerplate templates, but they all seem to rely heavily on globl variables. That's just not how I generally develop anything. I want a package that I can pip install, run, import, test, all the good stuff. My current start
-
Did my site build just go down?
My personal Site build went down last week, and I was unable to publish a new article. This is the process I went through to get it back up and running quickly. Is it a fluke? Classic IT fix, rerun it and see if you get the same error. Everyone is busy and when you have your build go down you are probably busy doing something else. My first step is often to simply click rerun right from GitHub actions. Sometimes this will fix it, and sometimes it doesn't. It's an easy fix to run in the meant
-
Don't inherit from python BaseException, Here's wh...
I ran into a PR this week where the author was inheriting what BaseException rather than exception. I made this example to illustrate the unintended side effects that it can have. Try running these examples in a file for yourself and try to kill them with control-c. You cannot Keybard interrupt Since things such as are created as an exception that inherits from , if you except you can no longer . except from Exception or higher If you except from exception or something than inherits from
-
How I make cache-keys from python objects
When I need a consistent key for a pythohn object I often reach for It works for me and the use cases I have. diskcache Yesterday we talked about setting up a persistant cache with python diskcache. In order to make this really work we need a good way to make consistent cache keys from some sort of python object. [[ python-diskcache ]] hash does not work My first thought was to just hash the files, this will give me a unique key for each. This will work, and give you a consistant key for one
-
How I setup a sqlite cache in python
When I need to cache some data between runs or share a cache accross multiple processes my go to library in python is . It's built on sqlite with just enough cacheing niceties that make it very worth it. install diskcache Install diskcache into your virtual environement of choice using pip from your command line. setup the cache There are a couple of different types of cache, , , and , you can read more about those in the docs Adding to the cache Adding to the cache only needs a key and val
-
Cache a python function with lru_cache
The easiest way to speed up any code is to run less code. A common technique to reduce the amount of repative work is to implement a cache such that the next time you need the same work done, you don't need to recompute anything you can simply retrieve it from a cache. lru_cache The easiest and most common to setup in python is a builtin functools.lru_cache. when to use lru_cache Any time you have a function where you expect the same results each time a function is called with the same inputs,
-
Set User Agent on pandas read_csv
I keep a small cars.csv on my website for quickly trying out different pandas operations. It's very handy to keep around to help what a method you are unfamiliar with does, or give a teammate an example they can replicate. Hosts switched I recently switched hosting from netlify over to cloudflare. Well cloudflare does some work to block certain requests that it does not think is a real user. One of these checks is to ensure there is a real user agent on the request. Not my go to dataset 😭 Th
-
Get Webpage with python requests
Python's requests library is one of the gold standard apis, designed by Kenneth Reitz. It was designed with the user perspective in mind first and implementation second. I have heard this called readme driven development, where the interface the user will use is laid out first, then implemented. This makes the library much mor intuitive than if it were designed around how it was easiest to implement. Install Requests Requests is on pypi and can be installed into your virtual environtment with p
-
Style Lookatme Slides a bit more Personal
I recently gave a talk at python web conf 2022, and one of the things I did when I should have been working on my presentation was workig on how my presentation looked... classic procrastination technique. Slide One Lets use this section to show what it looks like as I change my styles. ☝ This is how my website is built write markdown build site publish default This is what the above slide looks like in lookatme. default styles Set focus to the most important element The way I write my sli
-
How I load Markdown in Python
I use a package eyeseast/python-frontmatter {.hoverlink} to load files with frontmatter in them. Its a handy package that allows you to load files with structured frontmatter (yaml, json, or toml). Install It's on pypi, so you can install it into your virtual environment with pip. 🙋 What's Frontmatter Frontmatter is a handy way to add metadata to your plain text files. It's quite common to have yaml frontmatter in markdown. All of my blog posts have yaml frontmatter to give the post metadata
-
Python's Dict Union Operator | Pep 584
Today I was watching the python web conf 2022 and saw @davidbujic use the new Dict Union Operator Live on stage during his Functional Programming talk. This operator was first introduced into python 3.9 with pep584 . Merge Dicts I've long updated dicts through the use of unpacking. Note that the last item always wins. It makes it pretty easy to make user overrides to default configurations. With pep584 landing in python 3.9 we can now leverage the operator to achieve the same result. [[
-
How I Present Markdown Slides from the Terminal
I love the freedom of writing in markdown. It allows me to write content from the comfort of my editor with very little focus on page style. It turns out that markdown is also a fantastic tool for creating slides. Present from the terminal I will most often just present right from the terminal using lookatme . Presenting from the terminal lets me see the results quick right from where I am editing. It also allows me to pop into other terminal applications quickly. reveal.js I sometimes also u
-
How I read Files in Python
When I need to read contents from a plain text file in python I find the easiest way is to just use .
-
How I glob for Files in Python
A very common task for any script is to look for files on the system. My go to method when globbing for files in python is to use pathlib. Setup I setup a directory to make some examples about globbing. Here is what the directory looks like. 1 directory, 8 files Pathlib Pathlib is a standard library module available in all LTS versions of python at this point. Creating a Path instance. Globbing Examples The path object has a glob method that allows you to glob for files with a unix style glob
-
pytest-mock Basics
Last Thursday I learned about at a local python meetup. The presenter showed how he uses for his work, and it was kinda eye opening. I knew what mocking was, but I had not seen it in this context. Discovery Watching him use I realized that mocking was not as hard as I had made it out to be. You can install , use the mocker fixture, and patch objects methods with what you want them to be. install pytest-mock is out on pypi and can be installed with pip. What I actually did Sometimes I f
-
Python Walrus Inside List Comprehension
Python 3.8 came out two and a half years ago and I have yet to really lean in on the walrus operator. Partly because it always seemed like something kinda silly (my use cases) to require a python version bump for, and partly because I really didn't understand it the best. Primarily I have wanted to use it in comprehensions, but I did not really understand how. Now that Python 3.6 is end of life, and most folks are using at least it seems time to learn and use it. What's a Walrus := The assig
-
Make Kedro Runs Beautiful
Kedro rich is a very new and unstable (it's good, just not ready) plugin for kedro to make the command line prettier. Install kedro rich There is no pypi package yet, but it's on github. You can pip install it with the git url. Kedro run You can run your pipeline just as you normally would, except you get progress bars and pretty prints. kedro rich pretty run Kedro catalog Listing out catalog entries from the command line now print out a nice pretty table. kedro rich catalog list table output G
-
Open Files with Nvim Remote
I recently found a really great plugin by mhinz to open files in neovim from a different tmux split, without touching neovim at all. Installation neovim-remote is not a neovim plugin at all, it's a python cli that you can install with pip. Unlike the repo suggests, I use pipx to install . How I use it I have this added to my that is in every one of my projects. This will tie a neovim session to that directory, and all directories under it. In my workflow I open a tmux session for each pr
-
Textual Popup Hack
As I am toying around with textual, I am wanting some popup user input to take over. Textual is still pretty new and likely to change quite significantly, so I don't want to overdo the work I put into it, So for now on my personal tuis I am going to shell out to tmux. The Problem The main issue is that when you are in a textual app, it kinda owns the input. So if you try to run another python function that calls for it just cant get there. There is a textual-inputs library that covers this,
-
GitHub Markdown now Supports Mermaid Diagrams
Mermaid diagrams provide a way to display graphs defined as plain text. Some markdown renderers support this as a plugin. GitHub now supports it. example You can define nodes like this in mermaid, and GitHub will now render them as a pretty graph diagram. Its rendered in svg, so its searchable with and everything. Here is what the example looks like on GitHub Links GitHub support announcement mermaid docs
-
Glances can watch docker processes
Glances is a system monitor with a ton of features, including docker processes. I have started using portainer to look at running docker processes, its a great heavy-weight docker process monitor. glances works as a great lightweight monitor to just give you the essentials, ( Name, Status, CPU%, MEM, /MAX, IOR/s, IOW/s, Rx/s, Tx/s, Command) install You will need to install glances to use the glances webui. We can still use to manage our virtual environment for us so that we do not need to do
-
Glances webui with pipx
Glances has a pretty incredible webui to view system processes and information like htop, or task manager for windows. The nice thing about the webui is that it can be accessed from a remote system. This would be super nice on something like a raspberry pi, or a vm running in the cloud. Its also less intimidating and easier to search if you are not a terminal junky. install You will need to install glances to use the glances webui. We can still use to manage our virtual environment for us so
-
Run glances without install with pipx
Glances is a fully featured system monitoring tool written in python. Out of the box it's quite similar to htop, but has quite a few more features, and can be ran without installing anything other than , which you should already have installed if you do anything with python. Once you run this you will be in a tui application similar to htop. You can kill processes with k, use left and right arrows to change the sorting column, and up and down to select different processes. running pipx run gl
-
Nested requirements.txt in python
python requirements text files can in fact depend on each other due to the fact that you can pass pip install arguments right into your file. The trick is to just prefix the file with a flag, just like you would if you were installing it with try it out Lets create two requirements files in a new directory to play with. Then add the following to each requirements file. Installing Installing requirements_dev.txt will install both ipython and pandas since it includes the base requirements file
-
View AWS event bridge rules with visidata
Reading eventbridge rules from the command line can be a total drag, pipe it into visidata to make it a breeze. I just love when I start thinking through how to parse a bunch of json at the command line, maybe building out my own custom cli, then the solution is as simple as piping it into visidata. Which is a fantastic tui application that had a ton of vim-like keybindings and data features.
-
Python string of letters is a string of letters, b...
In python, a string is a string until you add special characters. In browsing twitter this morning I came accross this tweet, that showed that you can use accross two strings if they do not contain special characters. https://twitter.com/bascodes/status/1492147596688871424 I popped open ipython to play with this. I could confirm on , short strings that I typed in worked as expected. Using the method on these strings does break down. If You can also see this in the id of the objects as well
-
Read stderr from python subprocess.Popen
I often run shell commands from python with Popen, but not often enough do I set up error handline for these subprocesses. It's not too hard, but it can be a bit awkward if you don't do it enough. Using Popen reading from stderr To get the stderr we must get it from the proc, read it, and decode the bystring. Note that we can only get the stderr object once, so if you want to do more than just read it you will need to store a copy of it. Better Exception Now that we can read the we can make
-
Using a Python Markdown ast to Find All Paragraphs
In looking for a way to automatically generate descriptions for pages I stumbled into a markdown ast in python. It allows me to go over the markdown page and get only paragraph text. This will ignore headings, blockquotes, and code fences. It's also super fast, previously I was rendering to html and using beautifulsoup to get only the paragraphs. Using the commonmark ast was about 5x faster on my site. Duplicate Paragraphs When I originally wrote this post, I did not realize at the time that
-
python dict get
For an embarassingly long time, til today, I have been wrapping my dict gets with key errors in python. I'm sure I've read it in code a bunch of times, but just brushed over why you would use get. That is until I read a bunch of PR's from my buddy Nic and notice that he never gets things with brackets and always with . This turns out so much cleaner to create a default case than try except. Example Lets consider this example for prices of supplies. Here we set a variable of prices as a dict
-
Find all Headings with BeautifulSoup
BeautifulSoup is a DOM like library for python. It's quite useful to manipulate html. Here is an example to find_all html headings. I stole the regex from stack overflow, but who doesn't. Make an example sample.html Lets make a sample.html file with the following contents. It mainly has some headings, and tags that I want to be able to find. Get the headings with BeautifulSoup Lets import our packages, read in our using pathlib and find all headings using BeautifulSoup. And what we get
-
Lambda Function as a Kedro Node
I keep my nodes short and sweet. They do one thing and do it well. I turn almost every DataFrame transformation into its own node. It makes it must easier to pull catalog entries, than firing up the pipeline, running it, and starting a debugger. For this reason many of my nodes can be built from inline lambdas. Examples Here are two examples, the first one is sometimes referred to as an identity function. This is super common to use in the early phases of a project. It lets you follow sta
-
Add New Dependencies to Your Kedro Project
As you work on your kedro projects you are bound to need to add more dependencies to the project eventually. Kedro uses a fantastic command under the hood to ensure that everyone is on the same version of packages at all times, and able to easily upgrade them. It might be a bit different workflow than what you have seen, let's take a look at it. git status Before you start mucking around with any changes to dependencies make sure that your git status is clean. I'd even reccomend starting a n
-
Practice making pipelines with kedro
I am a huge believer in practicing your craft. Professional athletes spend most of their time honing their skills and making themsleves better. In Engineering many spend nearly 0 time practicing. I am not saying that you need to spend all your free time practicing, but a few minutes trying new things can go a long way in how you understand what you are doing and make a hue impact on your long term productivity. [[ what-is-kedro ]] Start practicing practice building pipelines with #kedro t
-
Tmux hotkey for copier templates
I have added a hotkey to my copier template setup to quickly access all my templates at any time from tmux. At any point I can hit , thats holding control and hitting , and I will get a popup list of all of my templates directory names. Its an fzf list, which means that I can fuzzy search through it for the template I want, or arrow key to the one I want if I am feeling insane. I even setup it up so that the preview is a list of the files that come with the template in tree view. I've had t
-
Markata Filters as Telescope Pickers in Neovim
I often pop into my blog from neovim with the intent to look at just a single series of posts, , , or just see todays posts. Markata has a great way of mapping over posts and returning their path that is designe exactly for this use case. Markata listing out posts from the command line {.wide} To tie these into a Telescope picker you add the command as the find_command, and comma separate the words of the command, with no spaces. I did also in there so that the newest posts are closest to
-
Copier Tasks | Python templating post run task
Copier allows you to run post render tasks, just like cookiecutter. These are defined as a list of in your . They are simply shell commands to run. The example I have below runs an bash script after the copier template has been rendered. I have put the script in so that I know it's always on my . It will reach back into the and update the default number.
-
Gradual Typing in Python
I've referenced a video from Anthony Sotile in passing conversation several times. Walking through his gradual typing process has really helped me understand typing better, and has helped me make some projects better over time rather than getting slammed with typing errors. https://youtu.be/Rk-Y71P_9KE Step 1 Run Mypy as is, don't get fancy yet. This will not reach into any functions unless they are alreay explicitly typed. It will not enforce you to type them either. Step 2 Next we will add
-
Python Reverse Sluggify
In order to make an auto title plugin for markata I needed to come up with a way to reverse the slug of a post to create a title for one that does not explicitly have a title. !!! Note "slugs" a slug is generally all lowercase and free of spaces, and is a way to make website routes (urls) Here I have a available that gives me the articles path, ex. . An easy way to get rid of the file extension, is to pass it into pathlib.Path and ask for the stem, which returns . Then from There I chose t
-
Get Python docstring with ast
Getting docstrings from python's ast is far simpler and more reliable than any method of regex or brute force searching. It's also much less intimidating than I originally thought. Parsing First you need to load in some python code as a string, and parse it with . This gives you a tree like object, like an html dom. Getting the Docstring You can then use to get the docstring of the node you are currently looking at. In the case of freshly loading in a file, this will be the module level doc
-
Python Respect the .gitignore
Many tools such as ripgrep respect the file in the directory it's searching in. This helps make it incredibly faster and generally more intuitive for the user as it just searches files that are part of thier project and not things like their virtual environments, node modules, or compiled builds. Editors like vscode often do not include files that are .gitignored in their search either. is a pattern matching library that implements git's wildmatch pattern so that you can ignore files include
-
Rename Python Variables with nvim
I don't use refactoring tools as much as I probably should. mostly because I work with small functions with unique names, but I recently had a case where a variable name was everywhere and I wanted it named better. This was not possible with find and replace, because there were other 's in this region. I first tried the nvim lsp rename, and it failed, Then I pip installed rope, a refactoring tool for python, and it just worked! Once you have rope installed you can call rename on the variabl
-
Python Find Available Port
When running a python process that requires a port it's handy if there is an option for it to just run on the next avaialble port. To do this we can use the socket module to determine if the port is in use or not before starting our process.
-
python functools total ordering
functools.total_ordering makes adding all of six of the rich comparison operators to your custom classes much easier, and more likely that you remember all of them. From the Docs: The class must define one of __lt__(), __le__(), __gt__(), or __ge__ In addition, the class should supply an __eq__() method. one of these lt () le () gt () ge () and required to have this one eq () Total Ordering Docs Here is an example using the Enum I was working on the other day.
-
Implement --pdb in a python cli
Adding a flag to your applications can make them much easier for those using it to debug your application, especially if your applicatoin is a cli application where the user has much fewer options to start this for themselves. To add a pdb flag to your applications you will need to wrap your function call in a try/except, and start a post_mortem debugger. I give credit to this stack overflow post for helping me figure this out. Using --pdb running this example with and without --pdb flag
-
Python Enum
Python comes with an enum module for creating enums. You can make your own enum by inheriting importing and inheriting from Enum. auto incrementing Enum values can be auto incremented by importing auto, and calling as their value. using the enum Enum's are accessed directy under the class itself, and have primarily two methods underneath each thing you make, and . using the Lifecycle Enum
-
Installing Pipx on Ubuntu
I recently paired up with another dev running windows with Ubuntu running in wsl, and we had a bit of a stuggle to get our project off the ground because they were missing com system dependencies to get going. Straight in the terminal Open up a terminal and get your required system dependencies using the apt package manager and the standard ubuntu repos. Using an Ansible-Playbook I like running things like this through an ansible-playbook as it give me some extra control and repeatability next t
-
Using Copier Answers to rerun templates quickly
The copier answers file is a key component to making your templates re-runnable. Let's look at the example for my setup.py. Inside of my file is this, a message not to muck around with it, and the ansers in yaml form. The first line is just a helper for the blog post. Inside my copier.yml I have setup my _answers_file to point to a special file. This is because this is not a whole projet template, but one just for a single file. Once I change the _answers_file I was incredibly stuck Run it
-
Making a Textual Widget from a Rich Renderable
Once you have made your sick looking cli apps with rich, eventually you are going to want to add some keybindings to them. Currently Textual, also written by @willmcgugan , does this extremely well. Fair Warning it is in super beta mode and expected to change a bunch. So take it easy with hopping on the train so fast. Get the things Install them from the command line. Import make a .py file and import them in it. Make what you have a widget If you return your rich renderable out of class that
-
pipx examples
count lines of code
-
Copier Templates
%%include til/copier_endops %%include til/copier-template-variables %%include til/copier-answers
-
Changing copier template strings (_endops)
I was completely stuck for awhile. copier was not replacing my template variables. I found out that adding all these fixed it. Now It will support all of these types of variable wrappers !RTFM: Later I read the docs and realized that copier defaults to using and for its templates unlike other tools like cookiecutter.
-
copier template variables
I've been looking for a templating tool for awhile that works well with single files. My go to templating tool does not work for single files, it needs to put files into a directory underneath of it. template variables By default copier uses double square brackets for its variables. variables in files, directory_names, or file_names will be substituted for their value once you render them. note! by default copier will not inject variables into your unless you use a .tmpl suffix. Before runni
-
Running Kedro on Ubuntu 21.10 Impish Indri
I just installed a brand new Ubuntu 21.10 Impish Indri, and wanted a kedro project to play with so I did what any good kedroid would do, I went to my command line and ran But what I got back was not what I expected! This is weird, why cant I run kedro new with pipx? Lets try pip. Same issue. [[ what-is-kedro ]] Curious what kedro is? Check out this article. What's up wrong python version The issue is that kedro only runs on up to , and on Ubuntu 21.10 when you you get and the standard rep
-
A Minimal Pluggy Example
Pluggy makes it so easy to allow users to modify the behavior of a framework without thier specific feature needing to be implemented in the framework itself. I've really been loving the workflow of frameworks built with pluggy. The first one that many python devs have experience with is pytest. I've never created a pytest plugin, and honestly at the time I looked into how they were made was a long time ago and it went over my head. I use a data pipelining framework called kedro, and have bui
-
My first impressions with pyenv
pyenv provides an easy way to install almost any version of python from a large list of distributions. I have simply been using the version of python from the os package manager for awhile, but recently I bumped my home system to Ubuntu 21.10 impish, and it is only 3.9+ while the libraries I needed were only compatable with up to 3.8. I needed to install an older version of python on ubuntu I've been wanting to check out pyenv for awhile now, but without a burning need to do so. installing Based
-
Installing packages with ansible only if they do n...
Part of my neovim setup requires having the python formatter installed and callable. I install it with so that I don't have to manage a virtual environment and have it available everywhere. So far this works well for me, if there are ever breaking changes I may need to rethink this. re-installing a bunch of things that are already installed can be quite a waste and really add up to my ansible run time, so for most of my ansible tasks that install a command like this I have been following t
-
Adding __rich__ methods to python classes
Adding a method that returns a rich renderable to any python class makes it display this output if printed with rich. This also includes being nested inside a rich Layout. results of printing ShowMe with rich
-
Smoother Python with automatic imports | pyflyby
This is not a flaky works half the time kind of plugin, it's a seriously smooth editing experience. I've just started using pyflyby, and it is solid so far. I have automatic imports on every save of a python file in neovim, and automatic imports on every command in ipython. I can't tell you how pumped I am for this, and how good its felt to use over the past few weeks. It's glorious. YouTube video Listen to me rant on how great pyflyby is https://youtu.be/2QW5DJiEJH4 {.youtube-embed} Give the
-
Kedro Course
find all nodes with raw in the name use parameters make and use a logger use find-kedro in spaceflights slide in a new node vizualize your pipeline find slow nodes move the configuration directory build kedro into docker and run it pyinstrument pdb make a new cli command make a hook load catalog entries slice a pipeline by tag by name from inputs to outputs
-
kedro catalog create
I use to boost my productivity by automatically generating yaml catalog entries for me. It will create new yaml files for each pipeline, fill in missiing catalog entries, and respect already existing catalog entries. It will reformat the file, and sort it based on catalog key. https://youtu.be/_22ELT4kja4 {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Running Kedro Catalog Create The command to ensure there are catalog entries for every dataset in the pass
-
Using Nix to manage my Python Interpreter
I recently started playing with nix. goals automatically select correct python version per project activating one doesn't bleed into the other Installing nix controlling nix-env searching for packages https://search.nixos.org/ shell
-
Just Ask Ipython for help
It happens to the best of us We can't all remember every single function signature out there, it's just not possible. If you want to stay productive while coding without the temptation to hit YouTube or Twitter. Use the built in help. Here are 5 ways to get help without leaving your terminal. https://youtu.be/TZrRAP-9UMk Docstrings In any python repl you can access the docstring of a function by calling for . In Ipython we can even get some syntax highlighting with the . Source Code Sometim
-
Setting Parameters in kedro
Parameters are a place for you to store variables for your pipeline that can be accessed by any node that needs it, and can be easily changed by changing your environment. Parameters are stored in the repository in yaml files. https://youtu.be/Jj5cQ5bqcjg {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. parameters files You can have multiple parameters files and choose which ones to load by setting your environment. By default kedro will give you a and para
-
Writing your first kedro Nodes
https://youtu.be/-gEwU-MrPuA Before we jump in with anything crazy, let's make some nodes with some vanilla data structures. import node You will need to import node from kedro.pipeline to start creating nodes. func The is a callable that will take the and create the . inputs / outputs Inputs and outputs can be None, a single catalog entry as a string, mutiple catalog entries as a List of strings, or a dictionary of strings where the key is the keyword argument of the func and the value is
-
Running your Kedro Pipeline from the command line
Running your kedro pipeline from the command line could not be any easier to get started. This is a concept that you may or may not do often depending on your workflow, but its good to have under your belt. I personally do this half the time and run from ipython half the time. In production, I mostly use docker and that is all done with this cli. https://youtu.be/ZmccpLy-OEI {.youtube-embed} [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. Kedro run To run the whole darn proj
-
kedro Virtual Environment
Avoid serious version conflict issues, and use a virtual environment anytime you are running python, here are three ways you can setup a kedro virtual environment. https://youtu.be/ZSxc5VVCBhM {.youtube-embed} conda venv pipenv conda I prefer to use conda as my virtual environment manager of choice as it give me both the interpreter and the packages I install. I don't have to rely on the system version of python or another tool to maintain python versions at all, I get everything in one tool. st
-
Kedro Pipeline Create
Kedro pipeline create is a command that makes creating new pipelines much easier. There is much less boilerplate that you need to write yourself. https://youtu.be/HtyIKqlEoNw creating a new pipeline The kedro cli comes with the following command to scaffold out new pipelines. Note that it will not add it to your , to be covered later, you will need to add it yourself. results The directory structure that it creates looks like this.
-
Kedro Install
Kedro comes with an command to install and manage all of your projects dependencies. https://youtu.be/IWimEs-hHQg cd into your project directory and activate env You must start by having your kedro project either cloned down from an existing project or created from kedro new. Then activate your environment. [[ kedro-new ]] this post covers kedro new [[ kedro-environment ]] This post covers creating your virtual environment for kedro install kedro Make sure you have kedro installed in your cur
-
Kedro Git Init
Immediately after , before you start running or your first line of code the first thing you should always do after getting a new kedro template created is to . https://youtu.be/IGba3ytf_6U git init Its as simple as these three commands to get started. I don't care if this project is for learning, if it will never have a remote or not, use git.
-
Kedro New
https://youtu.be/uqiv5LAiJe0 {.youtube-embed} Kedro new is simply a wrapper around the cookiecutter templating library. The kedro team maintains a ready made template that has everything you need for a kedro project. They also maintain a few kedro starters, which are very similar to the base template. [[ what-is-kedro ]] Unsure what kedro is, Check out yesterdays post on What is Kedro. pipx I reccomend using when running kedro new. is designed for system level cli tools so that you do not
-
What is Kedro
Kedro is an unopinionated Data Engineering framework that comes with a somewhat opinionated template. It gives the user a way to build pipelines that automatically take care of io through the use of abstract that the user specifies through entries. These entries are loaded, ran through a function, and saved by . The order that these are executed are determined by the , which is a DAG . It's the 's job to manage the execution of the . https://youtu.be/Wf4rnFsaFFU {.youtube-embed}
-
Incremental Versioned Datasets in Kedro
Kedro versioned datasets can be mixed with incremental and partitioned datasets to do some timeseries analysis on how our dataset changes over time. Kedro is a very extensible and composible framework, that allows us to build solutions from the individual components that it provides. This article is a great example of how you can combine these components in unique ways to achieve some powerful results with very little work. [[ what-is-kedro ]] 👆 Unsure what kedro is? Check out this post. How
-
Manage many git repos with ease
mu-repo
-
I Started Streaming on Twitch
I recently started streaming on twitch.tv/waylonwalker and it's been a blast so far. python kedro Data Science Data Engineering webdev digital gardening Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time consuming for me. kedro-issue-606 Inspiration My introduction to twitch c
-
Upcoming Stream
!!! Caution I'm no longer streaming As much as I would really love to make streaming work, its really hard for my family situation to make large blocks of time work for me. https://stackoverflow.com/questions/16720541/python-string-replace-regular-expression {.hoverlink} I am starting to stream 3 days per week, before I start work in the morning. These streams will likely be me just talking through things I am already doing. Making DAGs do 🔮Magical Things | Open Source 🐍Python | kedro plugins |
-
Kedro Spaceflights - part 2 | Stream replay June 7...
This was my seconf time ever streaming on twitch.tv/waylonwalker , and I completely botched my mic 2x. https://youtu.be/_7MwgKu-844 Links Spaceflights Tutorial my spaceflights repo Notes to get started
-
🌱 Digital Gardening | gif to Mp4 | Stream replay J...
https://youtu.be/I4VenHqIEng Doing some Digital Gardening on stream Ahrefs Errors ahrefs large images Automatic gif to mp4 gif to mp4 After this stream all gifs on my site are converted to mp4/webm if they exist. tmux-navigation-2021
-
Kedro Spaceflights - part 1 | Stream replay June 4...
This was my first time ever streaming on twitch.tv/waylonwalker . I am excited to get going. I have been streaming early in the morning while I am still waking up, so still a bit groggy as I go. https://youtu.be/Y07UBr9Ccjs Kedro Spaceflights It all started with kedro/issues/606 , Yetu called out for users of kedro to record themselves doing a walk through of their tutorials. I wanted to do this, but was really stuck at the fact that recording or editing somewhat polished vide is quite time co
-
Comprehensive guide to creating kedro nodes
The Kedro node is an essential part of the pipeline. It defines what catalog entries get passed in, what function gets ran, and the catalog entry to save the results under. does this link work? https://waylonwalker.com/what-is-kedro/ {.hoverlink} 👆 Unsure what kedro is? Check out this post. The node function The node function is the most common and reccomended way to define kedro nodes. It is a function that constructs and returns objects for you. Creating your first kedro node function The
-
Using Kedro In Scripts
With the latest releases of kedro , it is now possible to run kedro pipelines from within scripts. While I would not start a project with this technique, it will be a good tool to keep in my back pocket when I want to sprinkle in a bit of kedro goodness in existing projects. New to Kedro [[ what-is-kedro ]] If your just learning about kedro check out this post walking through it No More Rabbit Hole of Errors as of 0.17.2 I've tried to do this in kedro and it turned into a rabbit hole of erro
-
Creating pypi-list with kedro
I had an idea come to me via twitter. Short one word name packages are becoming hard to find on pypi. Short one word readable package names that are not a play on words are easy to remember, easy to spell correctly, and quick to type out. Simple index I started with the simple index. Pypi provides a single page listing to every single package hosted on pypi via the simple-index
-
Silence Kedro Logs
Kedro can have a chatty logger. While this is super nice in production so see everything that happened during a pipeline run. This can be troublesome while trying to implement a cli extension with clean output. Silence a Python log First, how does one silence a python log? Python loggers can be retrieved by the module's function. Then their log level can be changed. Much of kedro's chattiness comes from INFO level logs. I don't want to hear about anything for my current use case unless i
-
python lsp setup
Setting up python with the native nvim>0.5 lsp was mr lsp-config https://github.com/neovim/nvim-lspconfig pyls#190 https://github.com/palantir/python-language-server/issues/190 mypy Getting mypy working with lsp was tricky for me. I had some issues trying to run mypy in ci and pyright in my editor and I really wanted them to match.
-
Python Diskcahe is locked
Running multiple processes using the same diskcache object can cause issues with locks. As I was trying to setup a rich Live display for markata I ran into issues where each part could not nun simultaneusly. As I had followed the instructions from discache it was not directly aparant to me, so I had to make a simple example to experiment and play with at a small scale. Minimum reproducible error Minimum reporducible error is one of my superpowers in development. I do this very often to sus ou
-
Vim Fugitive
Add current file and commit with diff in a split :on[ly] C-W o :on[ly] will make the current buffer the only one on the screen. This is super helpful as many of fugitive commands will open in a split by default. C-I C-O cycle through the jumplist This one has nothing to do with fugitive, but is a native vim feature that makes fugitive glorious. Before I realized how to utilize and , I would get completely lost when using fugitive. Digging deep into the log, opening a file from a specific c
-
What is if __name__ == "__main___", and how do I u...
When a python module is called it is assigned the of otherwise if it's imported it will be assigned the of the module. Concrete example Let's create a module to play with a bit. We will call this module . It is a module that we may want to run by it'self or import and use in other modules. I have set this module up to execute one of two if statements based on whether the module it'self is being ran or if the module is being imported. Note it is not common to have a block, this is just
-
Custom Kedro Logger
DRAFT -
-
How to Install micromamba on linux (from the comam...
I really like using conda ( ) as my python virtual environment manager of choice. It's simple and it includes its own python interpreter using the version that I specify at creation. Mamba from their readme Mamba is a reimplementation of the conda package manager in C++. parallel downloading of repository data and package files using multi-threading libsolv for much faster dependency solving, a state of the art library used in the RPM package manager of Red Hat, Fedora and OpenSUSE core parts
-
kedro replit
I am trying to see what an embeded replit
-
Zev Averbach Interview
Zev Averbach, Frustrated spreadsheet jockey to software developer at 36 Q: Tell me about your journey as a spreadsheet jockey into Data Engineering? A: First of all, it's hilarious that I accidentally found your questions for this interview by Googling myself. 😊 I've always been a frustrated software user, and that frustration led me to be a "power user" (keyboard shortcuts etc) of my most used applications, as well as a "visual coder" using desktop automation like Alfred and Keyboard Maestro (M
-
Pytest capsys
Testing print/log statements in pytest can be a bit tricky, capsys makes it super easy, but I often struggle to find it. capsys capsys is a builtin pytest fixture that can be passed into any test to capture stdin/stdout. For a more comprehensive description check out the docs on capsys using capsys Simply create a test function that accepts capsys as an argument and pytest will give you a capsys opject.
-
Building Rich a Dev Server
Draft Post I've really been digging @willmcgugan's rich library for creating TUI like interfaces in python. I've only recently started to take full advantage of it. Dev Server I am working on a project in which I want to have a dev server running continuously in the background. I really like dev servers theat automatically chooose an unused port and list out the running pid so that I can kill it if I need to. automatic port number auto-restart display ( port, pid, uptime ) finding the port I
-
Automatic Conda Environments
I have automated my process to create virtual environments in my python projects, here is how I did it. I've really been digging my new tmux session management setup. Now I have leveled it up by adding direnv to my workflow. It will execute a shell script whenever I cd into a directory. One thing I wanted to add to this was, automatic activation of python environments whenever I cd into a directory, or create a new environment if one does not exist. https://waylonwalker.com/tmux-nav-2021/ {.hove
-
fix crlf for entire git repo
Final Result
-
How I Review Pipeline Code
I have started doing more regular PR's on my teams Kedro pipelines. I generally take a two phase approach to the review in order to give the reviewee both quick and detailed feedback. [[ what-is-kedro ]] initial scan (Phase1) passing ci Variable Names Antipatterns No commented out code Docsttrings generally make sense Phase1 is typically a quick scan over the PR right within the PR window in my browser. Passing CI flake8 black isort interrogate pytest build The very first thing that needs to h
-
Kedro pipeline_registry.py
With the realease of came a new module in the project template . Here are some notes that I learned while playing with this new module. migrating to create a file create a function in that mirrors the register_pipelines method from your module do not bring the decorator remove register_pipelines method on your class You should now have something that looks like this in your . pipeline_registry only works in Conflict Resolution What happens If I register pipelines in both places I w
-
🐍 Pluggable Architecture with Python
pytest has open sourced their amazing plugin framework , it allows library authors to give their users a way to modify the libaries behavior without needing to submit a change that may not make sense to the entire library. Previous Experience My experience so far as a plugin user, and plugin author has been great. Building and using plugins are incredibly intuitive. I wanted to dive a bit deeper and see how they are implemented inside of a library and its a bit of a mind bend the first time yo
-
Create Og Image Covers Using Python
There are so many tutorials out there for creating OG images with or , but I have yet to see many using python. PIL Template Text Overlays
-
⚙ How Python Tools Are Configured
There are various ways to configure python tools, config files, code, or environment variables. Let's look at a few projects that allow users to configure them through the use of config files and how they do it. Motivation This will not include how they are implemented, I've looked at a few and its not simple. This will focus on where config is placed and the order in which duplicates are resolved. The motivation of this article is to serve as a bit of a reference guide for those who may want
-
Minimal Kedro Pipeline
How small can a minimum kedro pipeline ready to package be? I made one within 4 files that you can pip install. It's only a total of 35 lines of python, 8 in and 27 in . 📝 Note this is only a composable pipeline, not a full project, it does not contain a catalog or runner. Minimal Kedro Pipeline I have everything for this post hosted in this gihub repo , you can fork it, clone it, or just follow along. Installation Caveats This repo represents the minimal amount of structure to build a ked
-
Markdown Cli
This is a post that may be a work in progress for awhile, Its a collections of thoughts on managing my blog, but could be translated into anythiung that is just a collection of markdown. Listing things posts tags draft posts data frontmatter filepath content template html render content Markdown.Markdown support extentsions frontmatter cleaning. provide ways to hook in or clean up the frontmatter Markata.Markata methods load render save Markata.Post methods load render save Markata plugins befor
-
Blog Data With Python
Generating an api for a blog is much simpler than one might expect with python. Markdown Frontmatter Fill in the blanks fix missing data Fast
-
Kedro - My Data Is Not A Table
In python data science/engineering most of our data is in the form of some sort of table, typically a DataFrame from a library like pandas, spark, or dask. DataFrames are the heart of most pipelines These containers for data contain many convenient methods to manipulate table like data structures. Sometimes we leverage other data types, namely vanilla types like lists and dicts, or even numpy data types. [[ what-is-kedro ]] unfamiliar with kedro, check out this post Sometimes datasets are not t
-
Quickly Change Conda Env With Fzf
Changing conda environments is a bit verbose, I use a function with fzf that both lists environments and selects the one I want in one go. Conda I have used conda as a virtual environment tool for years now. I started using conda for its simplicity to install packages on windows, but now that has gotten so much better and it's been years since I have run a command. I'm sure that I could use a different environment manager, but it works for me and makes sense. What environment manager do you u
-
Minimal Python Package
What does it take to create an installable python package that can be hosted on pypi? What is the minimal python package setup.py my_module.py This post is somewhat inspired by the bottle framework, which is famously created as a single python module. Yes, a whole web framework is written in one file. Directory structure setup.py name The name of the package can contain any letters, numbers, "_", or "-". Even if it's for internal/personal consumption only I usually check for discrepancy with p
-
Ipython-Config
I use my ipython terminal daily. It's my go to way of running python most of the time. After you use it for a little bit you will probably want to setup a bit of your own configuration. install ipython Activate your virtual environment of choice and pip install it. Any time you are running your project in a virtual environment, you will need to install ipython inside it to access those packages from ipython. You are using a virtual environment right? Virtual environments like venv or conda ca
-
Custom Ipython Prompt
I've grown tired of the standard ipython prompt as it doesn't do much to give me any useful information. The default one gives out a line number that only seems to add anxiety as I am working on a simple problem and see that number grow to several hundred. I start to question my ability 🤦♂️. Configuration If you already have an ipython config you can move on otherwise check out this post on creating an ipython config. [[ ipython-config ]] The Dream Prompt I want something similar to the stars
-
Ipython Ninjitsu
?docstring ??sourcecode %run %debug %autoreload %history autoformat %reset !shell commands ?docstring Stop going to google everytime your stuck and stay in your workflow. The ipython is a superhero for productivity and staying on task. Note This does jump you into a pager, a j,k or up, down to navigate, q to quit. ??sourcecode Docstring not enough for you use case. I often run into cases where the docstring is not clear enough and I need to see the implementation for myself to see what a f
-
Automating my Post Starter
One thing we all dread is mundane work of getting started, and all the hoops it takes to get going. This year I want to post more often and I am taking some steps towards making it easier for myself to just get started. When I start a new post I need to cd into my blog directory, start neovim in a markdown file with a clever name, copy some frontmatter boilerplate, update the post date, add tags, a description, and a cover. Todo List for starting a post frontmatter template Title slug tags date
-
Windowing Python Lists
In python data science we often will reach for pandas a bit more than necessary. While pandas can save us so much there are times where there are alternatives that are much simpler. The itertools more-itertools` are full of cases of this. This post is a walkthrough of me solving a problem with rather than reaching for a for loop, or pandas. I am working on a one-line-link expander for my blog. I ended up doing it, just by modifying the markdown with python. I first split the post into line
-
Testing Data Pipelines
Lint/Format/Doc black flake8 interrogate mypy Pipeline Assertions pipeline constructs pipeline as expected nodes pipeline has minimum nodes test minimum tags test alternate tags Catalog Assertions test catalog follows naming structure Node Tests test function does the correct operations on test data Great Expectations
-
reasons-to-kedro
There are many reasons that you should be using kedro. If you are on a team of Data Scientists/Data Engineers processing DataFrames from many data sources should be considering a pipeline framework. Kedro is a great option that provides many benefits for teams to collaborate, develop, and deploy data pipelines [[ what-is-kedro ]] Starter Template Kedro makes it super easy to get started with their cli that utilizes cookiecutter under the hood. [[ create-new-kedro-project ]] read more about how
-
What's New in Kedro 0.16.6
Kedro 0.16.6 is out! Let's take a look through the release notes Deployment Docs This is really exciting to see more deployment options coming from the kedro team. It really shows the power of the framework. The power of some of these orchestrations options is incredible. Argo Prefect Kubeflow Batch SageMaker Most of them hinge on a sweet combination of the kedro cli, docker image, and the pipeline knowing your nodes dependencies. Argo, Prefect, and Kubeflow have an interesting technique where
-
Designing a "Router" for kedro
nodes_global I released a router-like plugin for kedro back in April 2020. This was not the first design, the idea actually came from one of the QB folks who taught me kedro nearly a year before. We were assembling our pipelines with something called . It worked fairly well but did have some issues around being set as a global variable. But... One thing in particular that it did not lend itself well to was being able to create a packagable pipeline that I could pip install and append into an
-
Reclaim memory usage in Jupyter
Today I ran into an issue where we had a one-off script that just needed to work, but it was just chewing threw memory like nothing. It started with a colleague asking me How do I clear the memory in a Jupyter notebook, these are the steps we took to debug the issue and free up some memory in their notebook. How do I clear the memory in a Jupyter notebook? Pre check the status of memory There are a number of ways that you can check the amount of memory on your system. The easiest is not necessa
-
Strip Trailing Whitespace from Git projects
A common linting error thrown by various linters is for trailing whitespace. I most often use flake8. I generally have [pre-commit]( https://waylonwalker.com/pre-commit-is-awesome hooks setup to strip this, but sometimes I run into situations where I jump into a project without it, and my editor lights up with errors. A simple fix is to run this one-liner. One-Liner to strip whitespace bash read more about how [[pre-commit-is-awesome]]
-
Three things to Automate with Python using Pandas
Here are three things that I see my non programming counterparts doing every single day. These really sum up so much of what folks do within an office. So many of us dabble in or become power users of spreadsheets without knowing there is an alternative out there that can save us time, automate boring things, and allow us to open up our minds for the part that we add value, Thinking about the data. Focus on Value Add Operations Lets face it, stitching together spreadsheets is zero value add by
-
How to Install miniconda on linux (from the comman...
miniconda is a python distribution from continuum. It's a slimmed-down version of their very popular anaconda distribution. It comes with its own environment manager and has eased the install process for many that do not have a way to compile c-extensions. It made it much easier to install the data science stack on windows a few years ago. These days windows are much better than it was back then at compiling c-extensions. I still like its environment manager, which installs to a global directory
-
What's New in Kedro 0.16.4
If we take a look at the release notes I see one major feature improvement on the list, auto-discovery of hooks. This one comes a bit surprising as it was just casually mentioned in #435 auto enabled plugins mentioned in issue 435 Think pytest As mentioned in #435 this is the model that pytest uses. Not all plugins automatically start doing things right out of the box but require a CLI argument. simplicity It feels a bit crazy that simply installing a package will change the way that your
-
Integration testing with Python, TestProject.io, a...
!!! Caution None of the testproject.io urls resolve anymore in JAN 2025, I removed all of the broken links. As I continue to build out waylonwalker.com I sometimes run into some errors that are not caught because I do not have good testing implemented. I want to explore some integration testing options using GitHub's actions. Running integration tests will not prevent bugs from happening completely, but it will allow me to quickly spot them and rollback. 🤔 What to test first? The very first t
-
🐍 Practice Python Online
When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to show up. An easy way to show up to practice with python is to use an online repl. With these you can try out something quick. Sometimes I see snippets from blogs or tweets and I need to try the out for myself to really understand. When learning a new skill it's important to practice along the way. In order for me to show up to practice I need to make it easy to sh
-
Kedro Catalog
I am exploring a kedro catalog meta data hook, these are some notes about what I am thinking. Process metadata will be attached to the dataset object under a attribute metadata will be updated metadata will be empty until a pipeline is ran with the hook on optionally a function to add metadata will be added metadata will be stored in a file next to the meta Problems This Hook Should solve what datasets have a columns with in the name what datasets were updated after last tuesday which pipe
-
How python tools configure
mypy Mypy's config parser seems to be one of the most complex. This is likely in part to it having the largest backwards compatability of all projects that I looked at. mypy/config_parser flake8 options/config.py black black portray only uses pyproject.toml portray/config.py interrogate only uses pyproject.toml
-
🐍 Parsing RSS feeds with Python
I am looking into a way to replace my google reader experience that I had back in 2013 before google took it from us. I am starting by learning how to parse feeds with python, and without much previous knowledge, it proved to be much easier than anticipated thanks to the library. This is how I used python to parse rss and setup my own custom feed. Install Install the feedparser library. Get the content The feed object The feed is a feedparser.FeedParserDict. For all intents and purposes this
-
SLIDES - understanding python \*args and \*\*kwarg...
Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. I generally post these as a carousel on LinkedIn based on a full article. Let mw know what you think of it shown inside of a blog @_waylonwalker . python args and kwargs
-
Gracefully adopt kedro, the catalog
Why use kedro catalog? While using the catalog alone will not reap all of the benefits of the framework, it does get you and your project ready for the full framework eventually. For me the full benefit of the catalog comes when you combine it with the pipeline and dont even touch read/write steps at all. Taking a step into kedro by adopting the catalog first will give you a way to organize all of your data loads in one place, and stop manually writing read/write code, which can be different fo
-
How to find things in your kedro catalog
kedro 0.16.2 just dropped last week with a long-awaited feature... catalog search ! I went as far as monkey patching this into each of my projects. I work jump between a few really big projects that have tons of datasets. Being able to quickly search for what I need is so useful. The Catalog The kedro data catalog is a key component to the kedro framework. It handles all data loading and saving for you. It is configurable and hackable. Having all your data connections listed in one place
-
How Kedro handles your inputs
Passing inputs into kedro is a key concept. Understanding how it accepts a single catalog key as input is quite trivial that easily makes sense, but passing a list or dictionary of catalog entries can be a bit confusing. *args/**args review Check out this post for a review of how work in python. [[ python-args-kwargs ]] python args and kwargs article by @_waylonwalker All Kedro inputs are catalog Entries When kedro runs your pipeline it uses the catalog to imperatively load your data, mea
-
018
Sending
-
017
Sending order matters
-
016
Recieving
-
015
Recieving order matters
-
understanding python \*args and \*\*kwargs
Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function arguments can be packed and unpacked into lists or dictionaries. Beware though, this power can lead to some really unreadable/unusable code if done wrong. Python and are super useful tools, that when used properly can make you code much simpler and easier to maintain. Large manual conversions from a dataset to function a
-
009
Combine a directory of csv's with pandas
-
010
load remote data with pandas
-
011
Load data from database into pandas
-
012
autoreload your imports in ipython for ⚡ fast development 👆 add this to your
-
005
pre-commit setup is simple Try it yourself
-
006
Setup pre-commit for isort includes automatic .isort-config
-
pre-commit is awesome
I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it for so long because it seemed like a big daunting thing to set up, but really it's easy. It will automatically run checks for you. In some cases, it will even automatically fix them for you. Out of the box, it will do things like automatically trim extra whitespace, fix file endings, and ensure file sizes are not too large for git. I recently discovered the ✨ awesomeness that is pre-commit. I steered away from it
-
002
kedro-static-viz 0.3.0 just launched with hooks support 🎉
-
Kedro Static Viz 0.3.0 is out with Hooks Support
kedro-static-viz is out with support for the newly released hooks feature. This means that you can have automatically deploy a full gatsby site keeping your visualization always up to date. Even though it is a static site there is no functionality lost. The only thing that's missing is the flask server. With kedro-static-viz you can deploy your visualization to a number of static hosting providers such as GitHub pages free of charge with wicked fast performance ⚡ It's Fast Even though
-
001
practice building pipelines with #kedro today
-
Create Custom Kedro Dataset
Kedro provides an efficient way to build out data catalogs with their yaml api. It allows you to be very declaritive about loading and saving your data. For the most part you just need to tell Kedro what connector to use and its filepath. When running Kedro takes care of all of the read/write, you just reference the catalog key. But what is happening behind the scenes Under the hood there is an that each connector inherits from. It sets up a lot of the behind the scenes structure for us so
-
Interrogate is a pretty awesome, brand new, cli fo...
As usual while listening to python bytes 181 I heard of a tool that I had to try out right away! This thing is 🔥 hot off the press folks, we're talking the first release only 3 weeks ago. Its something that the python community needed years ago, and it belongs in your CI today . I had tried several tools that tried to do docstring coverage in the past but they were a bit cumbersome and were quickly forgotten about. Not interrogate, its dead simple! Nothing I have tried has come close to being
-
creating the kedro-preflight hook
Kedro Hooks Intro - kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and (nouns). With a , or (adjective). This really reminds me of reacts lifecycle hooks, that let you hook into various state of react web components. This is going to make kedro so extendable by the community. I am super pumped to see what the community is able to do with this ability. kedro hooks are an exciting upcoming feature of kedro . They allow you to hook into , , and
-
📝 Kedro Preflight Notes
This is a very rough idea for a kedro package to prevent time lost to get partway through a pipeline run only to realize that you dont have access to data or resources. Must Haves check that inputs exist or are of a type to skip (sql) Good to haves check that all input and output databases are accessible with good credentials check for s3 bucket access check for spark install Implementation run params
-
python-deepwatch
Is it possible to deep watch a single python function for changes? Shallow Watch keeping track of a python functions hash is quite simple. There is a method attached to every python function. Calling it will return a hash of the function. If the function changes the hash will change. Using hashlib provides a consistent hash. Now we have a consistent way to hash function code. Deep hashing Find dependencies setup a function in a module with a dependency Create Generic module importer by filepath
-
TIL: Bind arguments to dynamically generated lambd...
This past week I had a really weird bug in my kedro pipeline. For some reason data running through my pipeline was coming out completely made no sense, but if I manually request raw data outside of the pipeline it matched expectations. NOTE While this story is about a kedro pipeline, it can be applied anywhere closures are put into an iterable. Debugger to the rescue After a few days of looking at it off and on, I pinpointed that it was all the way down in the raw layer. Right as data is com
-
Four Github Actions for Python
If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simple cli I have opted to use the cli for most of these, that way I know exactly what is happening and have more control over it if I need. If you are developing python packages and using GitHub here are four actions that you can use today to automate your release workflow. Since python tools generally have such a simpl
-
Variables names don't need their type
So often I see a variables inside of its name and it hurts me a little inside. Tell me I'm right or prove me wrong below. Examples Pandas are probably the worst offender that I see Sometimes vanilla structures too! Edge Cases? It's so common when you need to get inside a data structure in a special way that itsn't provided by the library.... I am not exactly sure of a good way around it. Containers are plural Always name your containers plural, so that naming while iterating is simple. Befo
-
Create New Kedro Project
This is a quickstart to getting a new kedro pipeline up and running. After this article you should be able to understand how to get started with kedro . You can learn more about this Hello World Example in the docs 🧹 Install Kedro 🛢 Create the Example Pipeline 💨 Run the example 📉 Show the pipeline visualization Create a Virtual Environment I use conda to control my virtual environments and will create a new environment called with the following command. note the latest compatible vers
-
What is Kedro
[[ what-is-kedro ]] This is my original what-is-kedro article. There is a brand new one Kedro is an open source data pipeline framework. It provides guardrails to set your project up right from the start without needing to know deeply how to setup your own python library for data pipelining. It includes really great ways to manipulate and . This article will cover the 10K view of kedro, future articles will dive deper into each one. kedro is an open-source data pipeline framework. It pr
-
Long variable names are good
🏷️ Long variable names are a good thing. Self documenting code is more important than poorly documented code. Simply adding a few characters to your variable names can go a long ways. Containers are plural Aliases are welcome Scope is important
-
simple click 2
-
simple click
cli tools are super handy and easy to add to your python libraries to supercharge them. Even if your library is not a cli tool there are a number of things that a cli can do to your library. Example Ideas Things a cli can do to enhance your library. 🆚 print version 🕶 print readme 📝 print changelog 📃 print config ✏ change config 👩🎓 run a tutorial 🏗 scaffold a project with cookiecutter 🖱 Click Click is the most popular python cli tool framework for python. There are others, some old, some new c
-
SqlAlchemy Models
Make a connection Make a session Make a Base Class Make your First Model Make your own Base Class to inherit From Use the Custom Base Class
-
Building Cli apps in Python
Packages Click Inputs Click primarily takes two forms of inputs Options and arguments. I think of options as keyword argument and arguments as regular positional arguments. Option typically aliased with a shorthand ('-v', '--verbose') **From the Docs To get the Python argument name, the chosen name is converted to lower case, up to two dashes are removed as the prefix, and other dashes are converted to underscores. Argument positional required no help text supplied by click Yaspin Yaspin Gif Cl
-
Kedro
See all of my kedro related posts in [[ kedro-feed ]]. #kedrotips I am tweeting out most of these snippets as I add them, you can find them all here #kedrotips . 🗣 Heads up Below are some quick snippets/notes for when using kedro to build data pipelines. So far I am just compiling snippets. Eventually I will create several posts on kedro. These are mostly things that I use In my everyday with kedro. Some are a bit more essoteric. Some are helpful when writing production code, some are useful mo
-
📝 Packages to Investigate Notes
jmespath Tabnine Bulwark |-|-| |github: | https://github.com/zaxr/bulwark | I definitely want to try this out with kedro. Bulwark is a package for convenient property-based testing of pandas dataframes, supported for Python 3.5+. Example
-
Debugging Python
Using pdb
-
Just Use Pathlib
Pathlib is an amazing cross-platform path tool. Import Create path object Current Directory Users Home Directory module directory Others Let's create a path relative to our current module. Check if files exist Make Directories rename files List files Glob Files recursively Write
-
Filtering Pandas
query Good for method chaining, i.e. adding more methods or filters without assigning a new variable. masking general purpose, this is probably the most common method you see in training/examples isin capable of including multiple strings to include contains Good For partial matches MASKS anything that we put inside of square brackets can be set as a variable then passed in. Operators & - and ~ - not | - or AVAILABLE and NAME AVAILABLE or NAME AVAILABLE and not NAME
-
Making good documentation in python
Tools Sphinx Portray I just started using portray and it is amazingly simple to use! Methodology
-
Pyspark
I have been using pyspark since March 2019, here are my thoughts.
-
Quick Progress Bars in python using TQDM
tqdm is one of my favorite general purpose utility libraries in python. It allows me to see progress of multipart processes as they happen. I really like this for when I am developing something that takes some amount of time and I am unsure of performance. It allows me to be patient when the process is going well and will finish in sufficient time, and allows me to 💥 kill it and find a way to make it perform better if it will not finish in sufficient time. for more gifs like these follow me
-
Clean up Your Data Science with Named Tuples
If you are a regular listener of TalkPython or PythonBytes you have hear Michael Kennedy talk about Named Tuples many times, but what are they and how do they fit into my data science workflow. Example As you graduate your scripts into modules and libraries you might start to notice that you need to pass a lot of data around to all of the functions that you have created. For example if you are running some analysis utilizing , , and data. You may need to calculate total revenue, inventory
-
Background Tasks in Python for Data Science
This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth Reitz released it. It takes away so much boilerplate from running background tasks that I use it in more places than I probably should. After taking a look at that post today, I wanted to put a better data science example in here to help folks get started. This post is intended as an extension/update from background tasks in python . I started using the week that Kenneth R
-
📝 Bash Notes
Bash is super powerful. File System Full Show Remaining Space on Drives show largest files in current directory Move files then symlink them Fuzzy One Liners edit in vim cat a file bash execute git add git reset Kill a process Finding things Files fd-find is amazing for finding files, it even respects your file 😲. Install with . ++Vanilla Bonus Content ** show matching text ** ++Vanilla Bonus ** show file names only ** ++Vanilla Bonus Recursively Replace text ++Vanilla Bon
-
Autoreload in Ipython
I have used for several years now with great success and 🔥 rapid reloads. It allows me to move super fast when developing libraries and modules. They have made some great updates this year that allows class modules to be automatically be updated. What I like about autoreload 🔥 Blazing Fast 💥 Keeps me in the comfort of my text editor 👏 Allows me to use Jupyter when I need 👟 Extremely Reliable One of the biggest benefits that I find is that it shortens the distance between my module/library co
-
Python Tips
Dictionaries Unpacking **kwargs func(**input) locals().update(d)
-
Generating Readme Tables From Pandas
Generating Readme Tables From Pandas I commonly have a need to paste the first few lines of a dataset into a markdown file. I use two handy packages to do this, and . Lets say I have a Pandas DataFrame in memory as already. All I would need to do to convert the first 5 rows to markdown and copy it to the clipboard is the following. This is a super handy snippet that I use a lot. Folks really appreciate it when they can see a sample of the data without opening the entire file.
-
Pycon 2018 Roundup
These are my notes from pycon 2018 videos. I love the python community and especially the conference talks. This year I am going to take some notes from my favorite talks and post them here. This is an Incomplete working post. Jake VanderPlas - Performance Python: Seven Strategies for Optimizing Your Numerical Code Always profile before making any optimizations. Vectorize with Numpy Looping in python can be slow Use specialized data structures. scipy.spacial pandas xarray scipy.sparse sp
-
My favorite pandas pattern
My favorite pandas pattern I work with a lot of transactional timeseries data that includes categories. I often want to create timeseries plots with each category as its own line. This is the method that I use almost data to achieve this result. Typically the data that am working with changes very slowly and trends happen over years not days or weeks. Plotting daily/weekly data tends to be noisy and hides the trend. I use this pattern because it works well with my data and is easy to explain to
-
background tasks in python
I have tried most of the different methods in the past and found that copying and pasting the threadpoolexecutor example or the processpoolexecutor example from the standard library documentation to be the most reliable. Since this is often something that I stuff in the back of a utility module of a library it is not something that I write often enough to be familiar with, which makes it both hard to write and hard to read and debug. If you are looking for a good overview of the difference
-
Pycon 2017 Roundup
Pycon 2017 Roundup Good afternoon fellow Data Geeks. Last week Pycon released 141 videos of greatness. Here are my top picks from the event. #3 Kelsey Hightower - Keynote - Pycon 2017 https://www.youtube.com/watch?v=u_iAXzy3xBA&t=1795s #2 Al Sweigart Yes, It's Time to Learn Regular Expressions PyCon 2017 https://www.youtube.com/watch?v=abrcJ9MpF60 #1 Trey Hunner Readability Counts PyCon 2017 https://www.youtube.com/watch?v=knMg6G9_XCg What's on Tap This afternoon we have a cup of from one of