Posts tagged: dev

from kedro.pipeline import node

node(
    input="raw",
    output="int",
    func=my_func,
    tags=["one"],
)

npx create-react-app todoreact

import React,{useState,useEffect} from 'react';
import './App.css';

function App() {
  const [data,setData]=useState([]);
  const [newName,setNewName]=useState([]);
  const getData=()=>{
    fetch('/api'
    ,{
      headers : {
        'Content-Type': 'application/json',
        'Accept': 'application/json'
       }
    }
    )
      .then(function(response){
        return response.json();
      })
      .then(function(myJson) {
        setData(myJson)
      });
  }
  useEffect(()=>{
    getData()
  },[])

  const addItem= async () => {
    const rawResponse = await fetch('/api/add/', {
      method: 'POST',
      headers: {
        'Accept': 'application/json',
        'Content-Type': 'application/json'
      },

    body: JSON.stringify({"name": newName})
    });
    const content = await rawResponse;

    console.log(content);
    getData()
  }




  return (
    <div className="App">
     {
       data && data.length>0 && data.map((item)=><p>{item.id}{item.priority}{item.name}<button>raise priority</button></p>)
     }
    <input type='text' value={newName} onChange={(e) => (setNewName(e.target.value))} />
    <button onClick={addItem} >add item</button>
    </div>
  );
}

export default App;

In my adventure to learn django, I want to be able to setup REST api’s to feed into dynamic front end sites. Potentially sites running react under the hood.

Install #

To get started lets open up a todo app that I created with django-admin startproject todo.

pip install djangorestframework

Install APP #

Now we need to declare rest_framwork as an INSTALLED_APP.

INSTALLED_APPS = [
    ...
    "rest_framework",
    ...
]

create the api app #

Next I will create all the files that I need to get the api running.

mkdir api
touch api/__init__.py api/serializers.py api/urls.py api/views.py

base/models.py #

I already have the following model from last time I was playing with django. It will suffice as it is not the focus of what I am learning for now.

Note the name of the model class is singular, this is becuase django will automatically pluralize it in places like the admin panel, and you would end up with Itemss.

from django.db import models

# Create your models here.

class Item(models.Model):
    name = models.CharField(max_length=200)
    created = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.priority} {self.name}"

Next I will make some dummy data to be able to return. I popped open ipython and made a few records.

from base.models import Item

Item.objects.create(name='first')
Item.objects.create(name='second')
Item.objects.create(name='third')

api/serializers.py #

Next we need to set up a serializer to seriaze and de-serialize data between our model and json. You can specify each field individually or all of them by passing in __all__.

from rest_framework import serializers

from base.models import Item


class ItemSerializer(serializers.ModelSerializer):
    class Meta:
        model = Item
        fields = '__all__'

api/views.py #

Now we need a view leveraging the djangorestframework. The serializer we just created will be used to serialize all of the rows into a list of objects that Response can handle.

Note: to return a collection of model objects we need to set many to True

from rest_framework.decorators import api_view
from rest_framework.response import Response

from base.models import Item

from .serializers import ItemSerializer


@api_view(["GET"])
def get_data(request):
    items = Item.objects.all()
    serializer = ItemSerializer(items, many=True)
    return Response(serializer.data)

@api_view(['POST'])
def add_item(request):
    serializer = ItemSerializer(data = request.data)
    if serializer.is_valid():
        serializer.save()
    return Response()

api/urls.py #

Now we need to setup routing to access the views through an url.

from django.urls import path

from . import views

urlpatterns = [
        path('', views.get_data),
        path('add/', views.add_item),
        ]

todo/urls.py #

Then we need to include these urls from our api in the urls specified by settings.ROOT_URLCONf

from django.urls import path

urlpatterns = [
    ...
    path("api/", include("api.urls")),
]

Run it #

python manage.py runserver

Running the developement server and going to localhost:8000/api we can see the full list of items in th api.

My next step into django made me realize that I do not have access to the admin panel, turns out that I need to create a cuper user first.

Run Migrations #

Right away when trying to setup the superuser I ran into this issue

django.db.utils.OperationalError: no such table: auth_user

Back to the tutorial tells me that I need to run migrations to setup some tables for the INSTALLED_APPS, django.contrib.admin being one of them.

python manage.py migrate

python manage.py createsuperuser

The super user has been created.

CSRF FAILURE #

My next issue trying to run off of a separate domain was a cross site request forgery error.

Since this is a valid domain that we are hosting the app from we need to tell Django that this is safe. We can do this again in the settings.py, but this time the variable we need is not there out of the box and we need to add it.

CSRF_TRUSTED_ORIGINS = ['https://localhost.waylonwalker.com']

I made it!! #

And we are in, and welcomed for the first time with this django admin panel.

Remote Hosting #

You might find these settings helpful as well if you are trying to run your site on a remote host like aws, digital ocean, linode, or any sort of cloud providor. I had it running in my home lab while I was out of the house and ssh’d in over with a chromebook.

I am continuing my journey into django, but today I am not at my workstation. I am ssh’d in remotely from a chromebook. I am fully outside of my network, so I can’t access it by localhost, or it’s ip. I do have cloudflared tunnel installed and dns setup to a localhost.waylonwalker.com.

Settings #

I found this in settings.py and yolo, it worked first try. I am in from my remote location, and even have auth taken care of thanks to cloudflare. I am really hoping to learn how to setup my own auth with django as this is one of the things that I could really use in my toolbelt.

ALLOWED_HOSTS = ['localhost.waylonwalker.com']

I have no experience in django, and in my exploration to become a better python developer I am dipping my toe into one of the most polished and widely used web frameworks Django to so that I can better understand it and become a better python developer.

If you found this at all helpful make sure you check out the django tutorial

install django #

The first thing I need to do is render out a template to start the project. For this I need the django-admin cli. To get this I am going the route of pipx it will be installed globally on my system in it’s own virtual environment that I don’t have to manage. This will be useful only for using startproject as far as I know.

pipx install django
django-admin startproject try_django
cd try_django

Make a venv #

Once I have the project I need a venv for all of django and all of my dependencies I might need for the project. I have really been diggin hatch lately, and it has a one line “make a virtual environment and manage it for me” command.

hatch shell

If hatch is a bit bleeding edge for you, or it has died out by the time you read this. The ol trusty venv will likely stand the test of time, this is what I would use for that.

python -m .venv --prmpt `basename $PWD`
. ./.venv/bin/activate

Start the webserver #

Next up we need to start the webserver to start seeing that development content. The first thing I did was run it as stated in the tutorial and find it clashed with a currently running web server port.

python manage.py runserver

I jumped over to that tmux session, killed the process and I was up and running.

What’s running #

The default django hello world looks well designed. You are first presented with this page.

Next #

I opened up the urls.py to discover that the only configured url was at /admin. I tried to log in as admin, but was unable to as I have not yet created a superuser. Next time I play with django that is what I will explore.

I recently attended python web conf 2022 and after seeing some incredible presentations on it I am excited to give htmx a try.

The base page #

Start with some html boilerplate, pop in a script tag to add the htmx.org script, and a button that says click me. I added just a tish of style so that it does not sear your delicate developer your eyes.

<!DOCTYPE html>
<html lang="en">
  <head>
    <title></title>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <style>
      html  { background: #1f2022; color: #eefbfe; font-size: 64px; }
      button {font-size: 64px;}
      body { height: 100vh; width: 100vw; display: flex; justify-content: center; align-items:center; }
    </style>
    <!-- Load from unpkg -->
    <script src="https://unpkg.com/[email protected]"></script>
  </head>
  <body>
  <!-- have a button POST a click via AJAX -->
  <button hx-get="/partial" hx-swap="outerHTML">
    Click Me
  </button>

  </body>
</html>

Save this as index.html and fire up a webserver and you will be presented with this big beefcake of a button.

If you don’t have a development server preference I reccomend opening the terminal and running python -m http.server 8000 then opening your browser to localhost:8000

The Partial #

Now the page has a button that is ready to replace itself, notice the hx-swap="outerHTML">, with the contents of /partial. To create a static api of sorts we can simply host a partial page in a file at /partial/index.html with the following contents.

<p>
hello
</p>

Tree #

To make it a bit clearer here is what the file tree looks like after setting this up.

~/git/htmx  v3.9.7 (git)
❯ tree
.
├── clicked
│   └── index.html
└── index.html

Demo #

I added htmx to this page and setup a partial below, check out this easter egg.

Links #

• python web conf 2022
• htmx

• 

• 

Let’s make a vim command to automatically collect all the links in these posts at the end of each article. Regex confuses the heck out of me… I don’t have my regex liscense, but regex can be so darn powerful especially in an editor.

Step one #

Before you run someone’s regex from the internet that you don’t fully understand, check your git status and make sure you are all clear with git before you wreck something

Inspiration #

Something that I have always appreciated form Nick Janetakis is his links section. I often try to gather up the links at the end of my posts, but often end up not doing it or forgetting.

Making a Links section #

Searchng through the internet I was able to find an article from Vitaly Parnas called vim ref links that did almost exactly what I needed, except it was more complicated and made them into ref liks.

Here is my interpretation of the code I took from Vitaly’s post. It makes a Links section like the one at the bottom of this post.

function! MdLinks()
    $norm o## Links
    $norm o
    g/\[[^\]]\+\]([^)]\+)/t$
    silent! '^,$s/\v[^\[]*(\[[^\]]+\])\(([^)]+)\)[^\[]*/* \1(\2)/g
    nohl
endfunction
command! MdLinks call MdLinks()

So far it is working for me and saving me a few seconds off each post I make.

Links #

• Nick Janetakis
• vim ref links

Mermaid gives us a way to style nodes through the use of css, but rather than using normal css selectors we need to use style <nodeid>. This also applies to subgraphs, and we can use the name of the subgraph in place of the nodeid.

graph TD;
    a --> A
    A --> B
    B --> C

    style A fill:#f9f,stroke:#333,stroke-width:4px
    style B fill:#f9f,stroke:#333,stroke-width:4px

    subgraph one
        a
    end

    style one fill:#BADA55

produces the following graph

Mermaid provides some really great ways to group or fence in parts of your graphs through the use of subgraphs.

Here we can model some sort of data ingest with some raw iot device and our warehouse in different groups.

graph TD;

    subgraph raw_iot
        a
    end

    subgraph warehouse
        A --> B
        B --> C
    end

subgraph raw_iot
    a
end

subgraph warehouse
    A --> B
    B --> C
end

connecting subgroups #

If we want to connect them, we can make a connection between a and A outside of the subgraphs.

graph TD;

    subgraph raw_iot
        a
    end

    a --> A

    subgraph warehouse
        A --> B
        B --> C
    end

subgraph raw_iot
    a
end

a --> A

subgraph warehouse
    A --> B
    B --> C
end

separation of concerns #

It’s also possible to specify subgraphs separate from where you define your nodes. which allows for some different levels of grouping that would not be possible if you were to define all your nodes inside of a subgraph.

graph TD;
    a --> A
    A --> B
    B --> C

    subgraph one
        A
        C
    end

subgraph warehouse
    A
    C
end

Since GitHub started supporting mermaid in their markdown I wanted to take another look at how to implement it on my site, I think it has some very nice opportunities in teaching, documenting, and explaining things.

The docs kinda just jumped right into their mermaid language and really went through that in a lot of depth, and skipped over how to implement it yourself, turns out its pretty simple. You just write mermaid syntax in a div with a class of mermaid on it!

<script src='https://unpkg.com/[email protected]/dist/mermaid.min.js'></script>
<div class='mermaid'>
graph TD;
a --> A
A --> B
B --> C
</div>

You just write mermaid syntax in a div with a class of mermaid on it!

The above gets me this diagram.

This feels so quick and easy to start getting some graphs up and running, but does lead to layout shift and extra bytes down the pipe. The best solution in my opionion would be to forgo the js and ship svg. That said, this is do dang convenient I will be using it for some things.

In looking for a way to automatically generate descriptions for pages I stumbled into a markdown ast in python. It allows me to go over the markdown page and get only paragraph text. This will ignore headings, blockquotes, and code fences.

import commonmark
import frontmatter

post = frontmatter.load("post.md")
parser = commonmark.Parser()
ast = parser.parse(post.content)

paragraphs = ''
for node in ast.walker():
    if node[0].t == "paragraph":
        paragraphs += " "
        paragraphs += node[0].first_child.literal

It’s also super fast, previously I was rendering to html and using beautifulsoup to get only the paragraphs. Using the commonmark ast was about 5x faster on my site.

Duplicate Paragraphs #

When I originally wrote this post, I did not realize at the time that commonmark duplicates nodes. I still do not understand why, but I have had success duplicating them based on the source position of the node with the snippet below.

from itertools import compress

import commonmark
import frontmatter

post = frontmatter.load("post.md")
parser = commonmark.Parser()
ast = parser.parse(post.content)

# find all paragraph nodes
paragraph_nodes = [
    n[0]
    for n in ast.walker()
    if n[0].t == "paragraph" and n[0].first_child.literal is not None
]
# for reasons unknown to me commonmark duplicates nodes, dedupe based on sourcepos
sourcepos = [p.sourcepos for p in paragraph_nodes]
# find first occurence of node based on source position
unique_mask = [sourcepos.index(s) == i for i, s in enumerate(sourcepos)]
# deduplicate paragraph_nodes based on unique source position
unique_paragraph_nodes = list(compress(paragraph_nodes, unique_mask))
paragraphs = " ".join([p.first_child.literal for p in unique_paragraph_nodes])

BeautifulSoup is a DOM like library for python. It’s quite useful to manipulate html. Here is an example to find_all html headings. I stole the regex from stack overflow, but who doesn’t.

Make an example #

sample.html

Lets make a sample.html file with the following contents. It mainly has some headings, <h1> and <h2> tags that I want to be able to find.

<!DOCTYPE html>
<html lang="en">
  <body>
    <h1>hello</h1>
    <p>this is a paragraph</p>
    <h2>second heading</h2>
    <p>this is also a paragraph</p>
    <h2>third heading</h2>
    <p>this is the last paragraph</p>

  </body>
</html>

Get the headings with BeautifulSoup #

Lets import our packages, read in our sample.html using pathlib and find all headings using BeautifulSoup.

from bs4 import BeautifulSoup
from pathlib import Path

soup = BeautifulSoup(Path('sample.html').read_text(), features="lxml")
headings = soup.find_all(re.compile("^h[1-6]$"))

And what we get is a list of bs4.element.Tag’s.

>> print(headings)
[<h1>hello</h1>, <h2>second heading</h2>, <h2>third heading</h2>]

I recently added a heading_link plugin to markata, you might notice the 🔗’s next to each heading on this page, that is powered by this exact technique.

Today I discovered a sweet new cli for compressing images. squoosh cli is a wasm powered cli that supports a bunch of formats that I would want to convert my website images to.

  > Unfortunately, due to a few people leaving the team, and staffing issues

Web App #

First the main feature of squoosh is a web app that makes your images smaller right in the browser, using the same wasm. It’s sweet! There is a really cool swiper to compare the output image with the original, and graphical dials to change your settings.

CLI #

What is even cooler is that once you have settings you are happy with and are really cutting down those kb’s on your images, there is a copy cli command button! If you have npx (which you should if you have nodejs and npm) already installed it just works without installing anything more.

Converting all of my png’s to webp #

I copied the command that it gave me for converting to webp, and set it up to run on all of my pngs.

npx @squoosh/cli --webp \
  '{"quality":75 \
    "target_size":0 \
    "target_PSNR":0 \
    "method":4 \
    "sns_strength":50 \
    "filter_strength":60 \
    "filter_sharpness":0 \
    "filter_type":1 \
    "partitions":0 \
    "segments":4 \
    "pass":1 \
    "show_compressed":0 \
    "preprocessing":0 \
    "autofilter":0 \
    "partition_limit":0 \
    "alpha_compression":1 \
    "alpha_filtering":1 \
    "alpha_quality":100 \
    "lossless":0 \
    "exact":0 \
    "image_hint":0 \
    "emulate_jpeg_size":0 \
    "thread_level":0 \
    "low_memory":0 \
    "near_lossless":100 \
    "use_delta_palette":0 \
    "use_sharp_yuv":0 \
    }' \
    static/*.png -d squoosh-webp

I opened my images repo and converted all pngs to webp using the command above. I got 94% compression on my existing pngs without resizing anything. This is dang impressive, and not too hard to do. I do want to refactor my images site at some point and include this as part of the ci system.

I also converted to avif, but it sent all my cpus to 100 for quite awhile, for only another 2MB total. Not sure if its worth it or not.