Posts tagged: k3s

k3s system-upgrade minor by minor

2025-12-05T09:25:39Z

The k3s system-upgrade controller is a fantastic tool for upgrading k3s automatically. It has done a fantastic job for me every time I’ve used it. Today I ran it on a cluster that needed to upgrade several minors and I learned that the controller does not pick up on changes to the channel url if you change from minor to minor.

The solution I came up with was to name the plan with the version it supports. Then on each patch upgrade, change both the plan name and the channel. I use gitops with argocd, it automcatically cleaned up old plans, created new plans, and the system-upgrade-controller picked up the plan and started applying immediately.

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/k3s-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33

I’d love to see a better way if you have a way to upgrade through minors, or manually control the minor of your cluster let me know.

slow nfs performance

2025-01-02T20:23:10Z

I’m running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I’m already learning why this is not reccommended.

Slow #

I’ve been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don’t care if they take a bit longer to keep my master node freed up for more critical work.

I just started trying to build this site in a cronjob, It was taking 20 minutes to build, and something I noticed was that markata was taking minutes to run glob ( search for files ), normally this happens in a few ms and I never notice this step.

I just moved into the master node and the results were wild at ~30x faster

Permissions #

I have seen where you can get diffent permissions on the nfs share based on user id. Since I’m homelabbing here I only have one user per machine. As you step into enterprise level VMs with tighter controls and dozens of users for all the different services that might run on it.

I’ve ran into maybe one issue where I was root in one place and not another, other than that it’s been fine.

And it only got better #

As the cache was warm subsequent runs only got better.

I just checked again and we are now 80x faster

Conclusion #

I don’t have the answers yet, it might be my network, it might be nfs settings, it might be ext4 filesystem. I have some things to try, but what I do know is that it is not as easy as I thought it would be just to have the same file system mounted on both ends and share data between nodes.

I might even go with a complete alternative and use minio as a storage backend, and sync the files in on each run, this will add some latency to do the sync each time though.

k3s config after first install

2024-09-15T16:57:41Z

After first setting up a new k3s instance your kubeconfig file will be located in /etc/rancher/k3s/k3s.yaml.

You cans use it from here by setting $KUBECONFIG to that file.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Or you can copy it to ~/.kube/config

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

If you have installed k3s on a remote server and need the config on your local machine then you will need to modify the server address to reflect the remote server.

scp user@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Warning

only do this if you don’t already have a ~/.kube/config file, otherwise copy it to a new file and set your $KUBECONFIG env variable to use it.

Now you will need to open that file and change the server address, making sure to keep the port number.

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: ****
      server: https://<server-ip>:6443
    name: default

💭 Manual Upgrades | K3s

2024-04-19T12:51:03Z

Manual Upgrades | K3s

You can upgrade K3s by using the installation script, or by manually installing the binary of the desired version.

docs.k3s.io

You can give k3s an install channel to install stable, latest, or specific versions like 1.26. This is handy to make sure that you install the same version on all of your workers.

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=latest <EXISTING_K3S_ENV> sh -s - <EXISTING_K3S_ARGS>

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Uptime Kuma

2023-11-11T02:46:12Z

Uptime Kuma

A self-hosted monitoring tool

uptime.kuma.pet

Uptime kuma is a fantastic self hosted monitoring tool. One docker run command and you are up and running. Once you are in you have full control over checking status of urls, frequency, allowed timeouts, and a HUGE list of notification providers

docker run -d --restart=always -p 3001:3001 -v uptime-kuma:/app/data --name uptime-kuma louislam/uptime-kuma:1

I deployed it in my homelab today.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Creating k8s jobs with python

2023-10-28T00:52:24Z

External Link

thoughts.waylonwalker.com

I was looking to add running kubernetes jobs to a python cli I am creating, and I found this solution, mostly thanks to ollama run mistral:7b-instruct-q4_K_M and my loose understanding of what the yaml syntax is supposed to look like for a kubernetes job. This will let me create a job in the cluster, choose the image that runs, the command that is called, and how long until the job expires and is cleaned up. While the job still exists I can go in and look at the logs, but once its ttl has expired they are gone.

from kubernetes import client, config

# Load the default kubeconfig
config.load_kube_config()

# Define the API client for batch jobs
api_instance = client.BatchV1Api()

# Create a new job object
job = client.V1Job(
    api_version="batch/v1",
    kind="Job",
    metadata=client.V1ObjectMeta(name="myjob"),
    spec=client.V1JobSpec(
        ttl_seconds_after_finished=100,
        template=client.V1PodTemplateSpec(
            metadata=client.V1ObjectMeta(labels={"app": "myjob"}),
            spec=client.V1PodSpec(
                containers=[
                    client.V1Container(
                        name="myjobcontainer",
                        image="busybox",
                        command=["ls", "/"],
                    ),
                ],
                restart_policy="Never",
            ),
        ),
        backoff_limit=1,
    ),
)

# Call the Kubernetes API to create the job
api_instance.create_namespaced_job(namespace="default", body=job)

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Translate a Docker Compose File to Kubernetes Resources | Kube...

2023-10-22T02:04:18Z

Translate a Docker Compose File to Kubernetes Resources

What

Kubernetes · kubernetes.io

kompose is a sick cli to convert docker-compose.yml to kubernetes manifest.

# install

curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-linux-amd64 -o kompose

kompose convert
kompose convert -o deployment.yaml

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Kubernetes Persistent Volumes with Deployment and StatefulSet

2023-10-21T01:47:58Z

Kubernetes Persistent Volumes with Deployment and StatefulSet

How to use Kubernetes persistent volumes with deployment and stateful set and also when you should use one or another.

Alen Komljen · akomljen.com

Example of how to add a pvc to a deployment.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Inspect a Kubernetes PersistentVolumeClaim | Frank Sauerburger

2023-10-21T01:34:08Z

I was curious to see what was going on inside of my minio object storage. Great technique here by Frank to create an inspector pod, then you can do as you wish with the data.

I created the manifest as pvc-inspector.yml

apiVersion: v1
kind: Pod
metadata:
  name: pvc-inspector
spec:
  containers:
  - image: busybox
    name: pvc-inspector
    command: ["tail"]
    args: ["-f", "/dev/null"]
    volumeMounts:
    - mountPath: /pvc
      name: pvc-mount
  volumes:
  - name: pvc-mount
    persistentVolumeClaim:
      claimName: pvc-name

Then used it like this.

# create pvc-inspector pod
kubectl apply -f pvc-inspector.yml
# exec into inspector
kubectl exec -it pvc-inspector -- sh
# explore data
ls /pvc
# cleanup
kubectl delete -f pvc-inspector.yml

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Can't create Secret in Kubernetes: illegal base64 data at inpu...

2023-10-21T00:49:39Z

External Link

stackoverflow.com

In order to use k8s secrets manifest you first need to encode the data values.

echo -n 'mega_secret_key' | openssl base64

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Can I access k3s using just kubectl (no sudo and no k3s comman...

2023-10-20T22:52:00Z

Can I access k3s using just kubectl (no sudo and no k3s command)

Reddit · reddit.com

Right after installing k3s you are going to need to use sudo to use any kubectl command. The reason for this is that the default config is owned by root. To get around this you will need to make your own config and set the KUBECONFIG environment variable

To do this I used sudo one last time to copy the k3s.yaml file into my own directory and take ownership of it.

sudo cp /etc/rancher/k3s/k3s.yaml /home/waylon/.config/kube

sudo chown -R waylon:waylon ~/.config/kube

export KUBECONFIG=~/.config/kube/k3s.yaml

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Quick-Start Guide | K3s

2023-10-20T21:32:03Z

Quick-Start Guide | K3s

This guide will help you quickly launch a cluster with default options. Make sure your nodes meet the requirements before proceeding.

docs.k3s.io

I recently spun up k3s in my homelab. I’m trying to offload some work off of my free tier fly.io app in order to keep it free tier without crashing.

# install and start k3s
curl -sfL https://get.k3s.io | sh -
# check to see if your nodes are started
sudo kubectl get nodes

My main hiccup so far was the machine I am running on runs zfs on root, and it would not start the master node. Rather than figuring out how to make zfs play nice I just pointed k3s to a drive that is not zfs.

# manuallly
sudo k3s server -d /mnt/vault/.rancher/k3s
# without editing systemd service
sudo ln -s /mnt/vault/.rancher/k3s /var/lib/rancher/k3s

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts