Posts tagged: k8s

k3s system-upgrade minor by minor

2025-12-05T09:25:39Z

The k3s system-upgrade controller is a fantastic tool for upgrading k3s automatically. It has done a fantastic job for me every time I’ve used it. Today I ran it on a cluster that needed to upgrade several minors and I learned that the controller does not pick up on changes to the channel url if you change from minor to minor.

The solution I came up with was to name the plan with the version it supports. Then on each patch upgrade, change both the plan name and the channel. I use gitops with argocd, it automcatically cleaned up old plans, created new plans, and the system-upgrade-controller picked up the plan and started applying immediately.

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/k3s-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33

I’d love to see a better way if you have a way to upgrade through minors, or manually control the minor of your cluster let me know.

💭 K8s Diagram Builder - Free Visual Kubernetes Architecture Desi...

2025-12-01T14:10:55Z

K8s Diagram Builder - Visual Kubernetes YAML Generator

Free Kubernetes diagram builder with drag-and-drop design. Auto-generate production-ready YAML for Ingress, Services, Deployments, ConfigMaps, Secrets & more. No signup required.

K8s Diagram Builder · k8sdiagram.fun

This looks like great prototyping tool for k8s. I too often ask ai to get me going with the things I need. I’ve used k8s long enough that I can generally remember all the things I need, roughly where they go, would probably forget a few things and need to iterate, but I cannot remember exactly what goes where and need examples at a minimum. I need to give this a go from desktop and see if it will work for me. Right now looking through mobile looks promising.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 k8s-monitoring-helm/charts/k8s-monitoring/docs/examples/privat...

2025-05-23T19:58:59Z

k8s-monitoring-helm/charts/k8s-monitoring/docs/examples/private-image-registries/globally/values.yaml at main · grafana/k8s-monitoring-helm

Contribute to grafana/k8s-monitoring-helm development by creating an account on GitHub.

GitHub · github.com

k8s-monitoring requires setting imageregistry and pullsecrets twice

global:
  image:
    registry: my.registry.com
    pullSecrets:
      - name: my-registry-creds
  imageRegistry: my.registry.com
  imagePullSecrets:
    - name: my-registry-creds

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Chris Biscardi's Digital Garden

2025-05-12T13:05:33Z

External Link

christopherbiscardi.com

Interesting take on kubernetes from a front end perspective. All valid arguments to me, and really the answer to any do you need to any specific implementation of tech is probably no. We got along just fine before k8s ever existed and you still can, but its really nice in a lot of cases. If your skills lean toward backend or infrastructure I encourage you to give it a try.

k8s distros #

There are a lot of beginner friendly k8s distros that you can setup with relative ease, kind and k0s are great for single node, If you want multi-node k3s is what I generally use. If you want a very lightweight OS that you only interact with through an api, and has a very small attack surface talos is an amazing product.

When else might you want k8s #

Internal, on-prem, self hosted. If you are trying to avoid the cloud for cost, rules, regulations, red tape, kubernetes is a great option to manage your container workflows yourself without needing to have a cloud budget, get approvals and sign offs on running workflows in a public cloud.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

grafana dashboards as k8s configmap

2025-05-06T20:24:35Z

I’m trying to learn proper logs, monitoring, otel, and grafana. Today I imported a bunch of pre-made k8s dashboards and made a few of my own for specific apps, and it made me want to know how I can turn my own custom dashboards into infrastructure as code. Turns out grafana makes it pretty easy to do this, if you have the grafana dashboard sidecar running. It will pick up any ConfigMap with the grafana_dashboard label and import it.

Go to Dashboards -> Pick a Dashboard -> Export -> JSON.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-dashboard
  namespace: meta
  labels:
    grafana_dashboard: "1"
data:
  my-dashboard.json: |
    {
      "annotations": {
        "list": [
      ...
      "uid": "fel2uhjhepg5ce",
      "version": 3
    }

💭 Diun

2025-04-07T13:33:20Z

Diun

Receive notifications when a Docker image is updated on a Docker registry

crazymax.dev

Diun, looks like a very interesting tool to monitor for image updates, it does not make any change, it only makes notifications. This feels like an easy start to getting image updates started with low effort, keep git ops, but requires manual updates. I see this as a tool that would be a great start and pair well with automated image updaters to ensure they are working as expected.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Keel

2025-04-07T13:30:49Z

Keel

Kubernetes Operator to automate Helm, DaemonSet, StatefulSet & Deployment updates

keel.sh

Keel looks interesting, I might give it a try as a simple image updater. I’m unsure if it fits my gitops patterns though. I like to keep everything defined in git, I don’t like drift outside of that so Keel might not be the thing I want.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Changing k8s Storage Class - Migration Job

2025-04-04T09:57:50Z

I’m setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I’m all in on gitops so option 1 was not an option. So changing storageclass it is.

Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute.

Migration Job #

This migration job will create a new pvc with the new storageclass and move the data from the old pvc to the new pvc.

Existing Pods

 This migration job will not work if you have a pod using the old pvc.  You

will need to shutdown the pod and delete it.

# old pvc with longhorn storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
# new pvc with longhorn-backup storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn-backup
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
# migration job to move the data to the new pvc
apiVersion: batch/v1
kind: Job
metadata:
  name: pvc-migration
  namespace: waylonwalker-com
spec:
  template:
    spec:
      containers:
      - name: pvc-migration
        image: ubuntu:22.04
        command: ["/bin/bash", "-c"]
        args:
          - |
            echo "Starting migration..."
            cd /data
            cp -av source/. destination/
            echo "Migration complete!"
        volumeMounts:
        - name: source-vol
          mountPath: /data/source
        - name: dest-vol
          mountPath: /data/destination
      restartPolicy: Never
      volumes:
      - name: source-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn
      - name: dest-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn-backup

Apply the manifests and wait for the job to complete.

kubectl apply -f pvc-migration.yaml

Cleanup #

I had chatgpt create me a script to help me find what is using the pvc so that it can be deleted.

#!/bin/bash

NAMESPACE="waylonwalker-com"
PVC_NAME="site-pvc-longhorn-new"

echo "⏳ Checking if PVC exists..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" || {
  echo "✅ PVC already deleted."
  exit 0
}

echo "🔍 Describe PVC..."
kubectl describe pvc "$PVC_NAME" -n "$NAMESPACE"

echo -e "\n🔗 Checking if any pod is using this PVC..."
kubectl get pods -n "$NAMESPACE" -o json | jq -r \
  --arg PVC "$PVC_NAME" \
  '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName == $PVC) | .metadata.name'

echo -e "\n🧹 Checking finalizers..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq '.metadata.finalizers'

echo -e "\n🔎 Checking associated VolumeAttachment..."
PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}')
echo "🔗 PVC is bound to PV: $PV_NAME"

kubectl get volumeattachment -A -o json | jq \
  --arg PV "$PV_NAME" \
  '.items[] | select(.spec.source.persistentVolumeName == $PV) | {name: .metadata.name, node: .spec.nodeName, attached: .status.attached}'

echo -e "\n🚀 Done."

I had still had cronjob pods completed, so I had to delete them first.

🔗 Checking if any pod is using this PVC...
pvc-migration-ndv92
waylonwalker-com-cronjob-29057840-8s92p
waylonwalker-com-cronjob-29057850-4rvm9
waylonwalker-com-cronjob-29057860-6g89j

kubectl delete pod pvc-migration-ndv92 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057840-8s92p -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057850-4rvm9 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057860-6g89j -n waylonwalker-com

💭 AI workloads on Talos Linux - Sidero Labs

2025-02-14T19:27:54Z

AI workloads on Talos Linux

Running GPU accelerated workloads on Kubernetes? System extensions let you load kernel modules without compromising security on Talos Linux.

Sidero Labs · siderolabs.com

cool article for setting up talos linux with an nvidia gpu. What a wild world it we are living in where these devices that started out being only for hardcore gamers are becoming commonplace in servers and slowly entering the homelab space.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 poolers.postgresql.cnpg.io CRD metadata.annotations Too long ·...

2025-01-21T17:06:21Z

poolers.postgresql.cnpg.io CRD metadata.annotations Too long · Issue #325 · cloudnative-pg/charts

Unable to deploy helm chart using ArgoCD. Getting following error Failed sync attempt to : one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io "poolers.postgr...

GitHub · github.com

I’ve never seen or needed to use a serversideapply in kubernetes before, but I ran into this same issue in my k3s homelab while installing cloudnative-pg.

You can do it with argo

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
syncPolicy:
syncOptions:
- ServerSideApply=true

and you can do it with kubectl

kubectl apply --server-side --force-conflicts -f cnpg-1.25.0.yaml

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Keycloak

2025-01-19T05:34:32Z

Keycloak

Keycloak - the open source identity and access management solution. Add single-sign-on and authentication to applications and secure services with minimum effort.

Keycloak · keycloak.org

Keycloak looks like an interesting way to setup sso. It’s part of the cncf so it’s got a good backing. I want something better for argo workflows and this might be it. I’m curious what else I can tie into it.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 GitHub - ngalaiko/tree-sitter-go-template: Golang template gra...

2025-01-06T18:49:13Z

GitHub - ngalaiko/tree-sitter-go-template: Golang template grammar for tree-sitter

Golang template grammar for tree-sitter. Contribute to ngalaiko/tree-sitter-go-template development by creating an account on GitHub.

GitHub · github.com

This setup fixed my nvim syntax highlighting in helm templates.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

slow nfs performance

2025-01-02T20:23:10Z

I’m running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I’m already learning why this is not reccommended.

Slow #

I’ve been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don’t care if they take a bit longer to keep my master node freed up for more critical work.

I just started trying to build this site in a cronjob, It was taking 20 minutes to build, and something I noticed was that markata was taking minutes to run glob ( search for files ), normally this happens in a few ms and I never notice this step.

I just moved into the master node and the results were wild at ~30x faster

Permissions #

I have seen where you can get diffent permissions on the nfs share based on user id. Since I’m homelabbing here I only have one user per machine. As you step into enterprise level VMs with tighter controls and dozens of users for all the different services that might run on it.

I’ve ran into maybe one issue where I was root in one place and not another, other than that it’s been fine.

And it only got better #

As the cache was warm subsequent runs only got better.

I just checked again and we are now 80x faster

Conclusion #

I don’t have the answers yet, it might be my network, it might be nfs settings, it might be ext4 filesystem. I have some things to try, but what I do know is that it is not as easy as I thought it would be just to have the same file system mounted on both ends and share data between nodes.

I might even go with a complete alternative and use minio as a storage backend, and sync the files in on each run, this will add some latency to do the sync each time though.

💭 Changelog on X: "🗣️@dhh on Kubernetes' migration pitch: “Oh, w...

2024-11-25T02:30:44Z

External Link

X (formerly Twitter) · x.com

switching cloud providers, theres no easy way. K8s was supposed to get us there, haha, the deep integrations with each vendor just keep locking us in

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Inside Argo: Automating the Future - YouTube

2024-11-22T02:29:00Z

This is a really amazing documentary of argocd. I got into k8s pretty late in the game. Which is pretty typical for me. As I went to use k8s for the first time i was using workflows, then cd. both of these tools had a level of polish that made them seem like they had been there forever and not quite as young as they actually are.

I thought it was interesting how they focused on how the name must be two syllables or less, start with a or b, logo needs to be cutesy funny and recognizable seemed interesting, but puts them at the top of lists and makes them look like they’ve been there forever.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Cluster API book

2024-11-05T04:17:37Z

Provider List - The Cluster API Book

cluster-api.sigs.k8s.io

Cluster API book

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

k3s config after first install

2024-09-15T16:57:41Z

After first setting up a new k3s instance your kubeconfig file will be located in /etc/rancher/k3s/k3s.yaml.

You cans use it from here by setting $KUBECONFIG to that file.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Or you can copy it to ~/.kube/config

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

If you have installed k3s on a remote server and need the config on your local machine then you will need to modify the server address to reflect the remote server.

scp user@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Warning

only do this if you don’t already have a ~/.kube/config file, otherwise copy it to a new file and set your $KUBECONFIG env variable to use it.

Now you will need to open that file and change the server address, making sure to keep the port number.

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: ****
      server: https://<server-ip>:6443
    name: default

k8s kustomize diff

2024-07-06T09:42:42Z

I’ve started leaning in on kubernetes kustomize to customize my manifests per deployment per environment. Today I learned that it comes with a diff command.

kubectl diff -k k8s/overlays/local

You can enable color diffs by using an external diff provider like colordiff.

export KUBECTL_EXTERNAL_DIFF="colordiff -N -u"

You might need to install colordiff if you don’t already have it.

sudo pacman -S colordiff

sudo apt install colordiff

Now I can try out kustomize changes and see the change with kustomize diff.

kubectl dash k

2024-07-05T20:15:11Z

Kubernetes ships with a feature called kustomize that allows you to customize your manifests in a declarative way. It’s a bit like helm, but easier to use. I found this useful.

kustomization.yaml #

In order to use kustomize you need to have a kustomization.yaml, kustomization.yml, or Kustomization file in the directory you are applying.

kubectl apply -k .
error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/tank/git/kustomize-playground'

The kustomization.yaml file cannot be empty.

❯ kubectl apply -k pod
error: kustomization.yaml is empty

lets make an mvp #

mkdir pod
touch pod/pod.yaml
touch pod/kustomization.yaml

# pod.yaml

# kustomization.yaml

Overlays #

Overlays must point to yaml or a directory with a kustomization.yaml file, and cannot reside inside the same directory causing a circular reference.

Layout #

k8s
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
├── overlays
│   ├── local
│   │   └── kustomization.yaml
│   ├── dev
│   │   └── kustomization.yaml
│   └── prod
│       ├── special-prod.yaml
│       └── kustomization.yaml

base #

Place all of the common k8s manifests here in the base directory along with kustomization.yaml. You can split them up into separate files or just one file, I’m keeping it to one for simiplicity.

base - kustomization.yaml #

In the base kustomization.yaml file add all of the manifests that you want to use as a resource.

resources:
  - deployment.yaml

overlays #

Now for each separate environment that you want to have create a directory in overlays with a kustomization.yaml file, and any special manifests that only apply to a particular environment.

overlays - kustomization.yaml #

resoures #

resources:
  - ../../base
# any other specific resources, maybe special ones for prod here
  - sealed-dotenv.yaml

images #

images:
  - name: my-image
    newName: docker.io/myrepo/myimage
    newTag: 1.0.0

💭 Kustomize: The Best Way to Manage Your Kubernetes Configs - Yo...

2024-07-03T13:31:47Z

Great intro into kustomize. This helped me get started with kustomize.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts