Posts tagged: webdev

Mermaid gives us a way to style nodes through the use of css, but rather than using normal css selectors we need to use style <nodeid>. This also applies to subgraphs, and we can use the name of the subgraph in place of the nodeid.

graph TD;
    a --> A
    A --> B
    B --> C

    style A fill:#f9f,stroke:#333,stroke-width:4px
    style B fill:#f9f,stroke:#333,stroke-width:4px

    subgraph one
        a
    end

    style one fill:#BADA55

produces the following graph

Mermaid provides some really great ways to group or fence in parts of your graphs through the use of subgraphs.

Here we can model some sort of data ingest with some raw iot device and our warehouse in different groups.

graph TD;

    subgraph raw_iot
        a
    end

    subgraph warehouse
        A --> B
        B --> C
    end

subgraph raw_iot
    a
end

subgraph warehouse
    A --> B
    B --> C
end

connecting subgroups #

If we want to connect them, we can make a connection between a and A outside of the subgraphs.

graph TD;

    subgraph raw_iot
        a
    end

    a --> A

    subgraph warehouse
        A --> B
        B --> C
    end

subgraph raw_iot
    a
end

a --> A

subgraph warehouse
    A --> B
    B --> C
end

separation of concerns #

It’s also possible to specify subgraphs separate from where you define your nodes. which allows for some different levels of grouping that would not be possible if you were to define all your nodes inside of a subgraph.

graph TD;
    a --> A
    A --> B
    B --> C

    subgraph one
        A
        C
    end

subgraph warehouse
    A
    C
end

Since GitHub started supporting mermaid in their markdown I wanted to take another look at how to implement it on my site, I think it has some very nice opportunities in teaching, documenting, and explaining things.

The docs kinda just jumped right into their mermaid language and really went through that in a lot of depth, and skipped over how to implement it yourself, turns out its pretty simple. You just write mermaid syntax in a div with a class of mermaid on it!

<script src='https://unpkg.com/[email protected]/dist/mermaid.min.js'></script>
<div class='mermaid'>
graph TD;
a --> A
A --> B
B --> C
</div>

You just write mermaid syntax in a div with a class of mermaid on it!

The above gets me this diagram.

This feels so quick and easy to start getting some graphs up and running, but does lead to layout shift and extra bytes down the pipe. The best solution in my opionion would be to forgo the js and ship svg. That said, this is do dang convenient I will be using it for some things.

In looking for a way to automatically generate descriptions for pages I stumbled into a markdown ast in python. It allows me to go over the markdown page and get only paragraph text. This will ignore headings, blockquotes, and code fences.

import commonmark
import frontmatter

post = frontmatter.load("post.md")
parser = commonmark.Parser()
ast = parser.parse(post.content)

paragraphs = ''
for node in ast.walker():
    if node[0].t == "paragraph":
        paragraphs += " "
        paragraphs += node[0].first_child.literal

It’s also super fast, previously I was rendering to html and using beautifulsoup to get only the paragraphs. Using the commonmark ast was about 5x faster on my site.

Duplicate Paragraphs #

When I originally wrote this post, I did not realize at the time that commonmark duplicates nodes. I still do not understand why, but I have had success duplicating them based on the source position of the node with the snippet below.

from itertools import compress

import commonmark
import frontmatter

post = frontmatter.load("post.md")
parser = commonmark.Parser()
ast = parser.parse(post.content)

# find all paragraph nodes
paragraph_nodes = [
    n[0]
    for n in ast.walker()
    if n[0].t == "paragraph" and n[0].first_child.literal is not None
]
# for reasons unknown to me commonmark duplicates nodes, dedupe based on sourcepos
sourcepos = [p.sourcepos for p in paragraph_nodes]
# find first occurence of node based on source position
unique_mask = [sourcepos.index(s) == i for i, s in enumerate(sourcepos)]
# deduplicate paragraph_nodes based on unique source position
unique_paragraph_nodes = list(compress(paragraph_nodes, unique_mask))
paragraphs = " ".join([p.first_child.literal for p in unique_paragraph_nodes])

BeautifulSoup is a DOM like library for python. It’s quite useful to manipulate html. Here is an example to find_all html headings. I stole the regex from stack overflow, but who doesn’t.

Make an example #

sample.html

Lets make a sample.html file with the following contents. It mainly has some headings, <h1> and <h2> tags that I want to be able to find.

<!DOCTYPE html>
<html lang="en">
  <body>
    <h1>hello</h1>
    <p>this is a paragraph</p>
    <h2>second heading</h2>
    <p>this is also a paragraph</p>
    <h2>third heading</h2>
    <p>this is the last paragraph</p>

  </body>
</html>

Get the headings with BeautifulSoup #

Lets import our packages, read in our sample.html using pathlib and find all headings using BeautifulSoup.

from bs4 import BeautifulSoup
from pathlib import Path

soup = BeautifulSoup(Path('sample.html').read_text(), features="lxml")
headings = soup.find_all(re.compile("^h[1-6]$"))

And what we get is a list of bs4.element.Tag’s.

>> print(headings)
[<h1>hello</h1>, <h2>second heading</h2>, <h2>third heading</h2>]

I recently added a heading_link plugin to markata, you might notice the 🔗’s next to each heading on this page, that is powered by this exact technique.

Today I discovered a sweet new cli for compressing images. squoosh cli is a wasm powered cli that supports a bunch of formats that I would want to convert my website images to.

  > Unfortunately, due to a few people leaving the team, and staffing issues

Web App #

First the main feature of squoosh is a web app that makes your images smaller right in the browser, using the same wasm. It’s sweet! There is a really cool swiper to compare the output image with the original, and graphical dials to change your settings.

CLI #

What is even cooler is that once you have settings you are happy with and are really cutting down those kb’s on your images, there is a copy cli command button! If you have npx (which you should if you have nodejs and npm) already installed it just works without installing anything more.

Converting all of my png’s to webp #

I copied the command that it gave me for converting to webp, and set it up to run on all of my pngs.

npx @squoosh/cli --webp \
  '{"quality":75 \
    "target_size":0 \
    "target_PSNR":0 \
    "method":4 \
    "sns_strength":50 \
    "filter_strength":60 \
    "filter_sharpness":0 \
    "filter_type":1 \
    "partitions":0 \
    "segments":4 \
    "pass":1 \
    "show_compressed":0 \
    "preprocessing":0 \
    "autofilter":0 \
    "partition_limit":0 \
    "alpha_compression":1 \
    "alpha_filtering":1 \
    "alpha_quality":100 \
    "lossless":0 \
    "exact":0 \
    "image_hint":0 \
    "emulate_jpeg_size":0 \
    "thread_level":0 \
    "low_memory":0 \
    "near_lossless":100 \
    "use_delta_palette":0 \
    "use_sharp_yuv":0 \
    }' \
    static/*.png -d squoosh-webp

I opened my images repo and converted all pngs to webp using the command above. I got 94% compression on my existing pngs without resizing anything. This is dang impressive, and not too hard to do. I do want to refactor my images site at some point and include this as part of the ci system.

I also converted to avif, but it sent all my cpus to 100 for quite awhile, for only another 2MB total. Not sure if its worth it or not.