Posts tagged: python

To allow access only to the , you can pass add the Resource field to the User Policy when you create a new token.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "admin:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/*"
      ]
    }
  ]
}

You can inspect sqlite tables with the sqlite shell.

note that you get into the shell with sqlite3 database.db

.tables

I also learned that .tables, .index and .schema are helper functions that query the sqlite_master table on the main database.

Here is an output from my redka database. The sqlite_master table contains all the sqlite objects type, name, tbl_name, rootpage, and sql to create them.

❯ sqlite3 database.db "SELECT * from sqlite_master;"
table|rkey|rkey|2|CREATE TABLE rkey (
    id       integer primary key,
    key      text not null,
    type     integer not null,
        version  integer not null,
    etime    integer,
        mtime    integer not null
)
index|rkey_key_idx|rkey|3|CREATE UNIQUE INDEX rkey_key_idx on rkey (key)
index|rkey_etime_idx|rkey|4|CREATE INDEX rkey_etime_idx on rkey (etime)
where etime is not null
trigger|rkey_on_type_update|rkey|0|CREATE TRIGGER rkey_on_type_update
before update of type on rkey
for each row
when old.type is not new.type
begin
    select raise(abort, 'key type mismatch');
end
table|rstring|rstring|5|CREATE TABLE rstring (
    key_id integer not null,
    value  blob not null,

    foreign key (key_id) references rkey (id)
          on delete cascade
)
index|rstring_pk_idx|rstring|6|CREATE UNIQUE INDEX rstring_pk_idx on rstring (key_id)
view|vstring|vstring|0|CREATE VIEW vstring as
  select
    rkey.id as key_id, rkey.key, rstring.value,
        datetime(etime/1000, 'unixepoch') as etime,
        datetime(mtime/1000, 'unixepoch') as mtime
  from rkey join rstring on rkey.id = rstring.key_id
  where rkey.type = 1
    and (rkey.etime is null or rkey.etime > unixepoch('subsec'))
table|rhash|rhash|7|CREATE TABLE rhash (
    key_id integer not null,
    field text not null,
    value blob not null,

    foreign key (key_id) references rkey (id)
      on delete cascade
)
index|rhash_pk_idx|rhash|8|CREATE UNIQUE INDEX rhash_pk_idx on rhash (key_id, field)
index|rhash_key_id_idx|rhash|9|CREATE INDEX rhash_key_id_idx on rhash (key_id)
view|vhash|vhash|0|CREATE VIEW vhash as
  select
    rkey.id as key_id, rkey.key, rhash.field, rhash.value,
        datetime(etime/1000, 'unixepoch') as etime,
        datetime(mtime/1000, 'unixepoch') as mtime
  from rkey join rhash on rkey.id = rhash.key_id
  where rkey.type = 4
    and (rkey.etime is null or rkey.etime > unixepoch('subsec'))

I recently had to update my copier-gallery command to trust my own templates because some of them have shell scripts that run afterwards. Be warned that this could be a dangerous feature to run on random templates you get off the internet, but these are all mine, so if I wreck it its my own fault.

copier copy --trust <template> <destination>

All the the copier copy api can be found with help.

❯ copier copy --help
copier copy 8.3.0

Copy from a template source to a destination.

Usage:
    copier copy [SWITCHES] template_src destination_path

Hidden-switches:
    -h, --help                         Prints this help message and quits
    --help-all                         Prints help messages of all sub-commands and quits
    -v, --version                      Prints the program's version and quits

Switches:
    -C, --no-cleanup                   On error, do not delete destination if it was
                                       created by Copier.
    --UNSAFE, --trust                  Allow templates with unsafe features (Jinja
                                       extensions, migrations, tasks)
    -a, --answers-file VALUE:str       Update using this path (relative to
                                       `destination_path`) to find the answers file
    -d, --data VARIABLE=VALUE:str      Make VARIABLE available as VALUE when rendering the
                                       template; may be given multiple times
    -f, --force                        Same as `--defaults --overwrite`.
    -g, --prereleases                  Use prereleases to compare template VCS tags.
    -l, --defaults                     Use default answers to questions, which might be
                                       null if not specified.
    -n, --pretend                      Run but do not make any changes
    -q, --quiet                        Suppress status output
    -r, --vcs-ref VALUE:str            Git reference to checkout in `template_src`. If you
                                       do not specify it, it will try to checkout the
                                       latest git tag, as sorted using the PEP 440
                                       algorithm. If you want to checkout always the
                                       latest version, use `--vcs-ref=HEAD`.
    -s, --skip VALUE:str               Skip specified files if they exist already; may be
                                       given multiple times
    -w, --overwrite                    Overwrite files that already exist, without asking.
    -x, --exclude VALUE:str            A name or shell-style pattern matching files or
                                       folders that must not be copied; may be given
                                       multiple times

Today I accidentally ran f2 in ipython to discover that it opens your $EDITOR! I use this feature quite often in zsh, it is bound to <c-e> for me, and since I have my environment variable EDITOR set to nvim it opens nvim when I hit <c-e>. Today I discovered that Ipython has this bound to F2. If you know how to set it to <c-e> let me know I’ve tried, a lot.

export EDITOR=nvim
ipython
<F2>

better yet add export EDITOR=nvim to your .zshrc

# ~/.zshrc
export EDITOR=nvim

I’ve really been enjoying using sqlmodel for my projects that need a database. One thing that I definitely lacked on for too long was indexing my database. I hit a point with one database where it was taking 7s for pretty simple paginated queries to return 10 records.

For every field that you will be querying on, you can create an index, by setting it equal to Field(index=True)

class Hero(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str = Field(index=True)
    secret_name: str
    age: int | None = Field(default=None, index=True)

example courtesy of the docs

 primary keys are indexed by default.

The docs cover this pretty well, and in quite depth - Optimizing Queries

inline-snapshot is a new tool that I am trying out for python testing. It takes snapshots of your outputs and places them inline with the test.

Here is the most basic starter.

import inline_snapshot

def test_one():
    assert 1 == snapshot()

Now when I run pytest my tests will fail because my assert has no value, but if I run pytest --inline-snapshot=create it will fill out my snapshot values and the file will then look like this.

import inline_snapshot

def test_one():
    assert 1 == snapshot(1)

It also works with pydantic models.

class MyModel(BaseModel):
    name: str
    age: int
    nickname: str | None = None


def test_my_model_instance():
    assert MyModel(name="Waylon", age=1) == snapshot(MyModel(name="Waylon", age=1))


def test_my_model_fields():
    me = MyModel(name="Waylon", age=1, nickname='Waylon')
    assert me.name == snapshot("Waylon")
    assert me.age == snapshot(1)
    assert me.nickname == snapshot("Waylon")

Today I learned how to VACUUM a sqlite database and cut its size in about half. It’s a database that I have had running for quite awhile and has some decent traffic on it.

Why is it important to do a VACUUM? In short its becuase the file system gets fragmented with as data is updated. On delete the files are removed from the database and marked as available for reuse in the filesystem, but the space is not reclaimed.

To VACUUM a database, run the following sql command. You can do it right form the sqlite shell by running sqlite3.

You will need about double the current size of the database as free space to do the VACUUM, you need space for a full copy, journaling or write ahead logs, and the existing database.

VACUUM;

The docs are fantastic for vacuum.

Typer makes it easy to compose your cli applications, like you might with a web router if you are more familiar with that. This allows you to build smaller applications that compose into a larger application.

You will see similar patterns in the wild, namely the aws cli which always has the aws <command> <subcommand> pattern.

Lets setup the cli app itself first. You can put it in project/cli/cli.py.

import typer

from project.cli.api import api_app
from project.cli.config import config_app
from project.cli.user import user_app
from project.cli.run import run_app

app = typer.Typer()

app.add_typer(api_app, name="api")
app.add_typer(config_app, name="config")
app.add_typer(user_app, name="user")
app.add_typer(run_app, name="run")

Creating an app that will become a command is the same as creating a regular app in Typer. We need to create a callback that will become our command, and a command that will become our subcommand in the parent app.

import typer
from rich.console import Console

from project.config import get_config

config_app = typer.Typer()

@config_app.callback()
def config():
    "model cli"


@config_app.command()
def show(
):
    project_config = get_config(env)
    Console().print(fokais_config)

Setting up the entrypoint in pyproject.toml.

[project.scripts] # <- this project is part of the config DO NOT change it
project = "project.cli.cli:app" # <- This project is the project name, DO change it

Now you can see each cli application as a sub command.

❯ project --help

 Usage: project [OPTIONS] COMMAND [ARGS]...

╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion  [bash|zsh|fish|powershell|pwsh]  Install completion for the specified shell.│
│                                                        [default: None]                            │
│ --show-completion     [bash|zsh|fish|powershell|pwsh]  Show completion for the specified shell,   │
│                                                        to copy it or customize the installation.  │
│                                                        [default: None]                            │
│ --help                                                 Show this message and exit.                │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ────────────────────────────────────────────────────────────────────────────────────────╮
│ api                        model cli                                                              │
│ config                     config cli                                                             │
│ user                       user cli                                                               │
│ run                        run cli                                                                │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯

In the example above we can run the command project config show to see the current configuration of our project.

I am working on a page for htmx-patterns and I ran into a situation with lots of duplication. Especially when i am using tailwind I run into situations where the duplication can get tedious to maintiain. The solution I found is macros.

Now I can use the same code for all of my links, and call the macro to use it.

{% macro link(id, text, boosted=false) -%}
<a
    class="
    {% if id is none %}
      pointer-events-none bg-terminal-950 text-terminal-900 ring-terminal-900
    {% else %}
      bg-terminal-950 hover:bg-terminal-900 hover:text-terminal-400 text-terminal-500 shadow-lg shadow-terminal-300/20 hover:shadow-terminal-300/30 ring-terminal-300
    {% endif %}
      cursor-pointer block text-center font-bold py-2 px-4 rounded w-full ring-1
    "
    {% if id is not none %}
    href="{{ url_for('boosted', id=id) }}"
    {% endif %}
    {% if boosted %}
    hx-boost="true"
    {% endif %}>
    {{ text }}
</a>
{%- endmacro %}

<h2 class='text-3xl font-light mt-0 max-w-xl text-center prose-xl mt-8 text-terminal-500'>
    Boosted Links
</h2>

<div class='flex flex-row gap-4'>
    {{ link(prev_id, 'Previous', boosted=True) }}
    {{ link(next_id, 'Next', boosted=True) }}
</div>

<h2 class='text-3xl font-light mt-0 max-w-xl text-center prose-xl mt-8 text-terminal-500'>
    Normal Links
</h2>

<div class='flex flex-row gap-4'>
    {{ link(prev_id, 'Previous', boosted=False) }}
    {{ link(next_id, 'Next', boosted=False) }}
</div>

jinja has a loop variable that is very handy to use with htmx. Whether you want to implement a click to load more or an infinite scroll this loop variable is very handy.

{% for person in persons %}
<li
{% if loop.last %}
    hx-get="{{ url_for('infinite', page=next_page) }}"
    hx-trigger="intersect once"
    hx-target="#persons"
    hx-swap='beforeend'
    hx-indicator="#persons-loading"
{% endif %}
    {{ person.name.upper() }} -
    {{ person.phone_number }}
</li>
{% endfor %}

Now for every chunk of contacts that we load we will trigger the infinite scroll by loading more once the last one has intersected the screen.

Out of the box FastAPI.">Starlette does not support url_for with query params. When trying to use url_for with query params it throws the following error.

starlette.routing.NoMatchFound: No route exists for name "infinite" and params "page"

In my searching for this I found starlette issue #560 quite helpful, but not complete, as it did not work for me.

import jinja2

if hasattr(jinja2, "pass_context"):
    pass_context = jinja2.pass_context
else:
    pass_context = jinja2.contextfunction

@pass_context
def url_for_query(context: dict, name: str, **params: dict) -> str:
    request = context["request"]
    url = str(request.url_for(name))
    if params == {}:
        return url
    from urllib.parse import parse_qs, urlencode, urlparse, urlunparse

    # Parse the URL
    parsed_url = urlparse(url)

    # Parse the query parameters
    query_params = parse_qs(parsed_url.query)

    # Update the query parameters with the new ones
    query_params.update(params)

    # Rebuild the query string
    updated_query_string = urlencode(query_params, doseq=True)

    # Rebuild the URL with the updated query string
    updated_url = urlunparse(
        (
            parsed_url.scheme,
            parsed_url.netloc,
            parsed_url.path,
            parsed_url.params,
            updated_query_string,
            parsed_url.fragment,
        )
    )

    if os.environ.get("ENV") in ["dev", "qa", "prod"]:
        updated_url = updated_url.replace("http", "https", 1)

    return updated_url

def get_templates():
    templates = Jinja2Templates(directory="templates")
    templates.env.globals["url_for"] = url_for_query
    return templates

 If you want url_for to work in production you need some way to convert http

    if os.environ.get("ENV") in ["dev", "qa", "prod"]:
        updated_url = updated_url.replace("http", "https", 1)

The route might look something like this.

@infinite_router.get("/")
async def home(request: Request, page: int = 1, n: int = 10):
  ...

To access the home route using url_for in a jinja template you can use the following, once you have applied the url_for_query function as your default url_for

<a href="{{ url_for('home', page=1) }}">Home</a>

Kind (Kubernetes in Docker) is a tool that makes it easy to create and tear down local clusters quickly. I like to use it to test out new workflows.

Argocd is a continuous delivery tool that makes it easy to setup gitops workflows in kubernetes.

Here is how you can setup a new kind cluster and install argocd into it using helm, the kubernetes package manager.

kind create cluster --name argocd

# your first time through you need to add the argocd repo
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update

# install argocd into the cluster
helm install argo argo/argo-cd --namespace argocd --create-namespace

# deploy the app of apps
kubectl apply -f apps/apps.yaml

If you want to add repos and apps to your cluster you can use the argo cli to do that, but first you will need forward the argocd port and login.

# Wait until Argo CD API server is available
echo "Waiting for Argo CD API server to be available..."
while ! kubectl wait --for=condition=available --timeout=60s deployment/argo-argocd-server -n argocd; do
  echo "Waiting for Argo CD API server to be ready..."
  sleep 10
done


kubectl port-forward svc/argo-argocd-server -n argocd 8080:443 &
argocd_admin_password=$(kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}" | base64 -d)
argocd login localhost:8080 --insecure --username admin --password $argocd_admin_password
argocd repo add https://github.com/fokais-com/app.fokais.git --username waylonwalker --password ${GH_ARGO_PAT}
argocd app create app-fokais-local --repo https://github.com/fokais-com/app.fokais.git --path k8s/overlays/local --dest-server https://kubernetes.default.svc --sync-policy automated --sync-option Prune=true