The next version of markata will be around a full second faster at building
it’s docs, that’s a 30% bump in performance at the current state. This
performance will come when virtual environments are stored in the same
directory as the source code.
[1]
What happened?? # [2]
I was looking through my profiler for some unexpected performance hits, and
noticed that the docs plugin was taking nearly a full second (sometimes
more), just to run glob.
| |- 1.068 glob markata/plugins/docs.py:40
| | |- 0.838 <listcomp> markata/plugins/docs.py:82
| | | `- 0.817 PathSpec.match_file pathspec/pathspec.py:165
| | | [14 frames hidden] pathspec, <built-in>, <string>
Python scandir ignores hidden directories # [3]
I started looking for different solutions and what I found was that I was
hitting pathspec with way more files than I needed to.
len(list(Path().glob("**/*.py")))
# 6444
len([Path(f) for f in glob.glob("**/*.py", recursive=True)])
# 110
After digging into the docs I found that glob.glob uses os.scandir which
ignores ‘.’ and ‘..’ directories while Path.glob does not.
https://docs.python.org/3/library/os.html#os.scandir
results? # [4]
Now glob.py from the docs plugin does not...
Posts tagged: python
All posts with the tag "python"
312 posts
latest post 2026-05-06
Publishing rhythm
Pycon 2023
Keynote Speaker - James Powell # [1]
I don’t want to be an expert python developer.
https://www.youtube.com/watch?v=iKzOBWOHGFE
[2]
usage of keyword only arguments to prevent pain for users of libraries # [3]
# Version 1
def newton(f, x0, fprime, maxiter=100):
...
# Version 2
def newton(f, x0, fprime, tol=1e-6, maxiter=100):
...
# 🔴 Broke in Version 2
newton(f, x0, fprime, 100)
In an alternate timeline the maintainer of newton could have chose to use
keyword only arguments to prevent pain for users of libraries, or poor api
design due to fear of changing api on users.
# Version 1
def newton(f, x0, fprime, *, maxiter=100):
...
# Version 2
def newton(f, x0, fprime, *, tol=1e-6, maxiter=100):
...
# 🟢 user forced to use keyword only arguments never notices change
newton(f, x0, fprime, maxiter=100)
References:
[1]: #keynote-speaker---james-powell
[2]: https://dropper.waylonwalker.com/api/file/8275d2a5-72da-470c-a71d-86019415b303.webp
[3]: #usage-of-keyword-only-arguments-...
global Field
global BaseModel
from pydantic import BaseModel
from pydantic import Field
Pydantic is a Python library for serializing data into models that can be
validated with a deep set of built in valitators or your own custom validators,
and deserialize back to JSON or dictionary.
Installation # [1]
To install pydantic you will first need python and pip. Once you have pip
installed you can install pydantic with pip.
pip install pydantic
Always install in a virtual environment [2]
Creating a Pydantic model # [3]
To get started with pydantic you will first need to create a Pydantic model.
This is a python class that inherits from pydantic.BaseModel.
from pydantic import BaseModel
from pydantic import Field
from typing import Optional
class Person(BaseModel):
name: str = Field(...)
age: int
parsing an object # [4]
person = Person(name="John Doe", age=30)
print(person)
name='John Doe' age=30
data serialization # [5]
Pydantic has some very robust serialization methods that will automatically
coherse your data into the type specified by the type-hint in the model if it can.
person = Person(name=12, age="30")
print(f'name: {person.name}, type: {type(person.name)}')...
Playing Star Wars Text Adventure with a 10 yr old
article.blog-post {
max-width: 1200px;
}
The following is a playthrough of Star Wars Text Adventure with a 10 yr old.The
following is a playthrough of StarThe following is a playthrough of Star
❯ sw-adventure game run
[05/15/23 18:47:42] INFO marvin.marvin: Using OpenAI model "gpt-3.5-turbo" logging.py:50
18:47:42.699 | INFO | marvin.marvin - [default on default]Using OpenAI model "gpt-3.5-turbo"[/]
[18:47:42] Starting game game.py:30
generating your character
╭─ Zorin Kreez's Mission Card ─────────────────────────────────────────────────────────────────────────────────────╮
│ Zorin Kreez │ Zorin Kreez was born on Tatooine and grew up in a small farming community. He │
│ health │ 100 │ always dreamed of adventure and excitement. As soon as he was old enough, he │
│ imperial credits │ 5000 │ joined the Imperial Navy and quickly rose through the ranks. He is now a skilled │
│ fuel level │ 100 │ pilot and loyal member of the Empire. │
│ │ │
│ Imperial │ A nimble and deadly starfight...
Pydantic and singledispatch
I was reading about
pydantic-singledispatch [1]
from Giddeon’s blog and found it very intersting. I’m getting ready to
implement pydantic on my static site generator markata [2],
and I think there are so uses for this idea, so I want to try it out.
The Idea # [3]
Let’s set up some pydantic settings. We will need separate Models for each
environment that we want to support for this to work. The whole idea is to use
functools.singledispatch and type hints to provide unique execution for each
environment. We might want something like a path_prefix in prod for
environments like GithubPages that deploy to /<name-of-repo> while keeping
the root at / in dev.
Settings Model # [4]
Here is our model for our settings. We will create a CommonSettings model
that will be used by all environments. We will also create a DevSettings
model that will be used in dev and ProdSettings that will be used in prod.
We will use env as the discriminator so pydantic knows which model to use.
from typing im...
I really like having global cli command installed with pipx. Since textual
0.2.x (the css release) is out I want to be able to pop into textual devtools
easily from anywhere.
[1]
Pipx Install # [2]
You can pipx install textual.
pipx install textual
But if you try to run any textual cli commands you will run into a
ModuleNotFoundError, because you need to install the optional dev
dependencies.
Traceback (most recent call last):
File "/home/u_walkews/.local/bin/textual", line 5, in <module>
from textual.cli.cli import run
File "/home/u_walkews/.local/pipx/venvs/textual/lib/python3.10/site-packages/textual/cli/cli.py", line 4, in <module>
import click
ModuleNotFoundError: No module named 'click'
Pipx Inject # [3]
In order to install optional dependencies with pipx you need to first install
the library, then inject in the optional dependencies using the square bracket
syntax.
pipx install textual
pipx inject textual 'textual[dev]'
References:
[1]: https://stable-diffusion.waylonwalker.com/000359.2404332231.webp
[2]: #pipx-install
[3]: #pipx-inject
I am working through the textual tutorial, and I want to put it in a proper cli
that I can pip install and run the command without textual run --dev app.py.
This is a fine pattern, but I also want this to work when I don’t have a file
to run.
[1]
pyproject.toml entrypoints # [2]
I set up a new project running hatch new, and added the following entrypoint,
giving me a tutorial cli command to run.
...
[project.scripts]
tutorial = 'textual_tutorial.tui:tui'
https://waylonwalker.com/hatch-new-cli/
setup.py entrypoints # [3]
If you are using setup.py, you can set up entrypoints in the setup command.
from setuptools import setup
setup(
...
entry_points={
"console_scripts": ["tutorial = textual_tutorial.tui:tui"],
},
...
)
https://waylonwalker.com/minimal-python-package/
tui.py # [4]
adding features
Now to get devtools through a cli without running through textual run --dev.
I pulled open the textual cli source code, and this is what it does at the time
of writing.
Note: I used sys.argv as a way to implement a --dev quickly tutorial. For a
real project, I’d setup argparse, click, or typer. typer is my go to these
days, unless I am really trying to limit dependencies,...
- 11ty https://www.rockyourcode.com/how-to-deploy-eleventy-to-github-pages-with-github-actions/
- hugo puts it in the base url https://gohugo.io/getting-started/configuration/#baseurl
- mkdocs uses a special cli build command https://squidfunk.github.io/mkdocs-material/publishing-your-site/#github-pages
Markata now allows you to create jinja extensions that will be loaded right in
with nothing more than a pip install.
From the Changelog # [1]
The entry for 0.5.0.dev2 from markata’s changelog [2]
- Created entrypoint hook allowing for users to extend marka with jinja
exensions #60 0.5.0.dev2
[3]
markata-gh # [4]
The first example that you can use right now is markata-gh. It will render
repos by GitHub topic and user using the gh cli, which is available in github
actions!
Get it with a pip install
pip install markata-gh
Use it with some jinja in your markdown.
## Markata plugins
It uses the logged in uer by default.
{% gh_repo_list_topic "markata" %}
You can more explicitly grab your username, and a topic.
{% gh_repo_list_topic "waylonwalker", "personal-website" %}
How is this achieved # [5]
The jinja extension details are for another post, but this is how markata-gh
exposes itslef as a jinja extension.
class GhRepoListTopic(Extension):
tags = {"gh_repo_list_topic"}
def __init__(self, environment):
super().__init__(environment)
def parse(self, parser):
line_number = next(parser.stream).lineno
try:
args = parser.parse_tuple().items
except AttributeError:
...
npx create-react-app todoreact
import React,{useState,useEffect} from 'react';
import './App.css';
function App() {
const [data,setData]=useState([]);
const [newName,setNewName]=useState([]);
const getData=()=>{
fetch('/api'
,{
headers : {
'Content-Type': 'application/json',
'Accept': 'application/json'
}
}
)
.then(function(response){
return response.json();
})
.then(function(myJson) {
setData(myJson)
});
}
useEffect(()=>{
getData()
},[])
const addItem= async () => {
const rawResponse = await fetch('/api/add/', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({"name": newName})
});
const content = await rawResponse;
console.log(content);
getData()
}
return (
<div className="App">
{
data && data.length>0 && data.map((item)=><p>{item.id}{item.priority}{item.name}<button>raise priority</button></p>)
}
<input type='text' value={newName} onChange={(e) => (setNewName(e.target.value))} />
<button onClick={addItem} >add item</button>
</div>
);
}
export default App;
In my adventure to learn django, I want to be able to setup REST api’s to feed
into dynamic front end sites. Potentially sites running react under the hood.
[1]
Install # [2]
To get started lets open up a todo app that I created with django-admin startproject todo.
pip install djangorestframework
Install APP # [3]
Now we need to declare rest_framwork as an INSTALLED_APP.
INSTALLED_APPS = [
...
"rest_framework",
...
]
create the api app # [4]
Next I will create all the files that I need to get the api running.
mkdir api
touch api/__init__.py api/serializers.py api/urls.py api/views.py
[5]
base/models.py # [6]
I already have the following model from last time I was playing with django. It
will suffice as it is not the focus of what I am learning for now.
Note the name of the model class is singular, this is becuase django will
automatically pluralize it in places like the admin panel, and you would end
up with Itemss.
from django.db import models
# Create your models here.
class Item(models.Model):
name = models.CharField(max_length=200)
created = models.DateTimeField(auto_now_add=True)
def __str__(self):
return f"{self.priority} {self.name}"
Next I will m...
Markata now uses hatch as its build backend, and version bumping tool.
setup.py, and setup.cfg are completely gone.
[1]
0.5.0 is big # [2]
Markata 0.5.0 is now out, and it’s huge. Even though it’s the backend of this
blog I don’t actually have that many posts directly about it. I’ve used it a
bit for blog fuel in generic ways, like talking about pluggy and diskcache, but
very little have I even mentioned it.
Over the last month I made a big push to get 0.5.0 out, which adds a whole
bunch of new configurability to markata.
Here’s the changelog [3] entry.
- Moved to PEP 517 build #59 0.5.0.dev1
My Personal Simple CI/CD # [4]
Before cutting all of my personal projects over to hatch. The first thing I
did was to setup a solid github action,
hatch-action [5]that I can resue.
It automatically bumps versions, using pre-releases on all branches other than
main, with special branches for bumping major, minor, patch, dev, alha, beta,
and dev.
hatch new –init # [6]
To convert the project over to hatch, and get rid of setup.py/setup.cfg, I ran
hatch new --init. This automatically grabs all the metadata for the project
and makes a pyproject.toml that has most of what I need.
hat...
[1]
References:
[1]: https://dropper.waylonwalker.com/api/file/388f4342-8623-4ac7-9b4b-1d63cd82d2ad.png
My next step into django made me realize that I do not have access to the admin panel, turns out that I need to create a cuper user first.
[1]
Run Migrations # [2]
Right away when trying to setup the superuser I ran into this issue
django.db.utils.OperationalError: no such table: auth_user
Back to the tutorial [3]
tells me that I need to run migrations to setup some tables for the
INSTALLED_APPS, django.contrib.admin being one of them.
python manage.py migrate
[4]
yes I am still running remote on from my chromebook.
python manage.py createsuperuser
[5]
The super user has been created.
[6]
CSRF FAILURE # [7]
My next issue trying to run off of a separate domain was a cross site request
forgery error.
Since this is a valid domain that we are hosting the app from we need to tell
Django that this is safe. We can do this again in the settings.py, but this
time the variable we need is not there out of the box and we need to add it.
CSRF_TRUSTED_ORIGINS = ['https://localhost.waylonwalker.com']
I made it!! # [8]
And we are in, and welcomed for the first time with this django admin panel.
[9]
Remote Hosting # [10]
You might find these settings helpful as well if yo...
I am continuing my journey into django, but today I am not at my workstation. I
am ssh’d in remotely from a chromebook. I am fully outside of my network, so I
can’t access it by localhost, or it’s ip. I do have cloudflared tunnel
installed and dns setup to a localhost.waylonwalker.com.
Settings # [1]
I found this in settings.py and yolo, it worked first try. I am in from my
remote location, and even have auth taken care of thanks to cloudflare. I am
really hoping to learn how to setup my own auth with django as this is one of
the things that I could really use in my toolbelt.
ALLOWED_HOSTS = ['localhost.waylonwalker.com']
[2]
References:
[1]: #settings
[2]: https://stable-diffusion.waylonwalker.com/000321.3422093952.webp
I have no experience in django, and in my exploration to become a better python
developer I am dipping my toe into one of the most polished and widely used web
frameworks Django to so that I can better understand it and become a better
python developer.
If you found this at all helpful make sure you check out the django tutorial [1]
[2]
install django # [3]
The first thing I need to do is render out a template to start the project.
For this I need the django-admin cli. To get this I am going the route of
pipx it will be installed globally on my system in it’s own virtual
environment that I don’t have to manage. This will be useful only for using
startproject as far as I know.
pipx install django
django-admin startproject try_django
cd try_django
[4]
Make a venv # [5]
Once I have the project I need a venv for all of django and all of my
dependencies I might need for the project. I have really been diggin hatch
lately, and it has a one line “make a virtual environment [6] and manage it for
me” command.
hatch shell
[7]
If hatch is a bit bleeding edge for you, or it has died out by the time you
read this. The ol trusty venv will likely stand the test of time, this is w...
While updating my site to use Markata’s new configurable head I ran into some
escaping issues. Things like single quotes would cause jinja to fail as it was
closing quotes that it shouldnt have.
[1]
Jinja Escaping Strings # [2]
Jinja comes with a handy utility for escaping strings. I definitly tried to
over-complicate this before realizing. You can just pipe your variables into
e to escape them. This has worked pretty flawless at solving some jinja
issues for me.
<p>
{{ title|e }}
</p>
Creating meta tags in Markata # [3]
The issue I ran into was when trying to setup meta tags with the new
configurable head, some of my titles have single quotes in them. This is what
I put in my markata.toml to create some meta tags.
[[markata.head.meta]]
name = "og:title"
content = "{{ title }}"
Using my article titles like this ended up causing this syntax error when not
escaped.
SyntaxError: invalid syntax. Perhaps you forgot a comma?
Exception ignored in: <function Forward.__del__ at 0x7fa9807192d0>
Traceback (most recent call last):
...
TypeError: 'NoneType' object is not callable
jinja2 escape # [4]
After making a complicated system of using html.escape I realized that jinja
includ...
Hatch allows you to specify direct references for dependencies in your
pyproject.toml file. This is useful when you want to depend on a package that
is not available on PyPI or when you want to use a specific version from a Git [1]
repository. Often used for unreleased packages, or unreleased versions of
packages.
docs [2]
[project]
dependencies = ['markata', 'markata-todoui@git+https://github.com/waylonwalker/markata-todoui']
[tool.hatch.metadata]
allow-direct-references=true
References:
[1]: /glossary/git/
[2]: https://hatch.pypa.io/dev/config/dependency/#direct-references
When I am developing python code I often have a repl open alongside of it
running snippets ofcode as I go. Ipython is my repl of choice, and I hace
tricked it out the best I can and I really like it. The problem I recently
discovered is that I have way overcomplicated it.
[1]
What Have I done?? # [2]
So in the past the way I have setup a few extensions for myself is to add
something like this to my ~/.ipython/profile_default/startup directory. It
sets up some things like rich highlighting or in this example automatic
imports. I even went as far as installing some of these in the case I didn’t have them installed.
import subprocess
from IPython import get_ipython
from IPython.core.error import UsageError
ipython = get_ipython()
try:
ipython.run_line_magic("load_ext pyflyby", "inline")
except UsageError:
print("installing pyflyby")
subprocess.Popen(
["pip", "install", "pyflyby"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
).wait()
ipython.run_line_magic("load_ext pyflyby", "inline")
print("installing isort")
subprocess.Popen(
["pip", "install", "isort"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
[3]
What I missed? # [4]
I missed t...
[1]
A long needed feature of markata has been the ability to really configure out
templates with configuration rather. It’s been long that you needed that if
you really want to change the style, meta tags, or anything in the head you
needed to write a plugin or eject out of the template and use your own.
Adding some Head # [2]
Now you can add some extra style to your site with the existing built-in
template.
[[markata.head]]
text = """
<style>
img {
width: 100%;
height: auto;
}
ul {
display: flex;
flex-wrap: wrap;
}
</style>
"""
You can have more than one Head # [3]
Each text entry in markata.head just gets appended raw into the head.
[[markata.head]]
text = """
<style>
img {
width: 100%;
height: auto;
}
ul {
display: flex;
flex-wrap: wrap;
}
</style>
"""
[[markata.head]]
text = """
<script>
console.log('hey there')
</script
"""
Still need more? # [4]
If this does not take you far enough yet, you can still eject out and use your
own template pretty easy. If you are going for a full custom site it’s likely
that this will be the workflow for awhile. Markata should only get better and
make this required less often as it matures.
[markata]
post_template = "pages/tem...