Posts tagged: k8s Archive

k3s system-upgrade minor by minor

2025-12-05T09:25:39Z

The k3s system-upgrade controller is a fantastic tool for upgrading k3s automatically. It has done a fantastic job for me every time I’ve used it. Today I ran it on a cluster that needed to upgrade several minors and I learned that the controller does not pick up on changes to the channel url if you change from minor to minor.

The solution I came up with was to name the plan with the version it supports. Then on each patch upgrade, change both the plan name and the channel. I use gitops with argocd, it automcatically cleaned up old plans, created new plans, and the system-upgrade-controller picked up the plan and started applying immediately.

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan-v1.33 # <- This is important if you want to change the channel name
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/k3s-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/k3s-upgrade
  channel: https://update.k3s.io/v1-release/channels/v1.33

I’d love to see a better way if you have a way to upgrade through minors, or manually control the minor of your cluster let me know.

💭 K8s Diagram Builder - Free Visual Kubernetes Architecture Desi...

2025-12-01T14:10:55Z

K8s Diagram Builder - Visual Kubernetes YAML Generator

Free Kubernetes diagram builder with drag-and-drop design. Auto-generate production-ready YAML for Ingress, Services, Deployments, ConfigMaps, Secrets & more. No signup required.

K8s Diagram Builder · k8sdiagram.fun

This looks like great prototyping tool for k8s. I too often ask ai to get me going with the things I need. I’ve used k8s long enough that I can generally remember all the things I need, roughly where they go, would probably forget a few things and need to iterate, but I cannot remember exactly what goes where and need examples at a minimum. I need to give this a go from desktop and see if it will work for me. Right now looking through mobile looks promising.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 k8s-monitoring-helm/charts/k8s-monitoring/docs/examples/privat...

2025-05-23T19:58:59Z

k8s-monitoring-helm/charts/k8s-monitoring/docs/examples/private-image-registries/globally/values.yaml at main · grafana/k8s-monitoring-helm

Contribute to grafana/k8s-monitoring-helm development by creating an account on GitHub.

GitHub · github.com

k8s-monitoring requires setting imageregistry and pullsecrets twice

global:
  image:
    registry: my.registry.com
    pullSecrets:
      - name: my-registry-creds
  imageRegistry: my.registry.com
  imagePullSecrets:
    - name: my-registry-creds

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Chris Biscardi's Digital Garden

2025-05-12T13:05:33Z

External Link

christopherbiscardi.com

Interesting take on kubernetes from a front end perspective. All valid arguments to me, and really the answer to any do you need to any specific implementation of tech is probably no. We got along just fine before k8s ever existed and you still can, but its really nice in a lot of cases. If your skills lean toward backend or infrastructure I encourage you to give it a try.

k8s distros #

There are a lot of beginner friendly k8s distros that you can setup with relative ease, kind and k0s are great for single node, If you want multi-node k3s is what I generally use. If you want a very lightweight OS that you only interact with through an api, and has a very small attack surface talos is an amazing product.

When else might you want k8s #

Internal, on-prem, self hosted. If you are trying to avoid the cloud for cost, rules, regulations, red tape, kubernetes is a great option to manage your container workflows yourself without needing to have a cloud budget, get approvals and sign offs on running workflows in a public cloud.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

grafana dashboards as k8s configmap

2025-05-06T20:24:35Z

I’m trying to learn proper logs, monitoring, otel, and grafana. Today I imported a bunch of pre-made k8s dashboards and made a few of my own for specific apps, and it made me want to know how I can turn my own custom dashboards into infrastructure as code. Turns out grafana makes it pretty easy to do this, if you have the grafana dashboard sidecar running. It will pick up any ConfigMap with the grafana_dashboard label and import it.

Go to Dashboards -> Pick a Dashboard -> Export -> JSON.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-dashboard
  namespace: meta
  labels:
    grafana_dashboard: "1"
data:
  my-dashboard.json: |
    {
      "annotations": {
        "list": [
      ...
      "uid": "fel2uhjhepg5ce",
      "version": 3
    }

💭 Diun

2025-04-07T13:33:20Z

Diun

Receive notifications when a Docker image is updated on a Docker registry

crazymax.dev

Diun, looks like a very interesting tool to monitor for image updates, it does not make any change, it only makes notifications. This feels like an easy start to getting image updates started with low effort, keep git ops, but requires manual updates. I see this as a tool that would be a great start and pair well with automated image updaters to ensure they are working as expected.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Keel

2025-04-07T13:30:49Z

Keel

Kubernetes Operator to automate Helm, DaemonSet, StatefulSet & Deployment updates

keel.sh

Keel looks interesting, I might give it a try as a simple image updater. I’m unsure if it fits my gitops patterns though. I like to keep everything defined in git, I don’t like drift outside of that so Keel might not be the thing I want.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Changing k8s Storage Class - Migration Job

2025-04-04T09:57:50Z

I’m setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I’m all in on gitops so option 1 was not an option. So changing storageclass it is.

Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute.

Migration Job #

This migration job will create a new pvc with the new storageclass and move the data from the old pvc to the new pvc.

Existing Pods

 This migration job will not work if you have a pod using the old pvc.  You

will need to shutdown the pod and delete it.

# old pvc with longhorn storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
# new pvc with longhorn-backup storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn-backup
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
# migration job to move the data to the new pvc
apiVersion: batch/v1
kind: Job
metadata:
  name: pvc-migration
  namespace: waylonwalker-com
spec:
  template:
    spec:
      containers:
      - name: pvc-migration
        image: ubuntu:22.04
        command: ["/bin/bash", "-c"]
        args:
          - |
            echo "Starting migration..."
            cd /data
            cp -av source/. destination/
            echo "Migration complete!"
        volumeMounts:
        - name: source-vol
          mountPath: /data/source
        - name: dest-vol
          mountPath: /data/destination
      restartPolicy: Never
      volumes:
      - name: source-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn
      - name: dest-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn-backup

Apply the manifests and wait for the job to complete.

kubectl apply -f pvc-migration.yaml

Cleanup #

I had chatgpt create me a script to help me find what is using the pvc so that it can be deleted.

#!/bin/bash

NAMESPACE="waylonwalker-com"
PVC_NAME="site-pvc-longhorn-new"

echo "⏳ Checking if PVC exists..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" || {
  echo "✅ PVC already deleted."
  exit 0
}

echo "🔍 Describe PVC..."
kubectl describe pvc "$PVC_NAME" -n "$NAMESPACE"

echo -e "\n🔗 Checking if any pod is using this PVC..."
kubectl get pods -n "$NAMESPACE" -o json | jq -r \
  --arg PVC "$PVC_NAME" \
  '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName == $PVC) | .metadata.name'

echo -e "\n🧹 Checking finalizers..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq '.metadata.finalizers'

echo -e "\n🔎 Checking associated VolumeAttachment..."
PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}')
echo "🔗 PVC is bound to PV: $PV_NAME"

kubectl get volumeattachment -A -o json | jq \
  --arg PV "$PV_NAME" \
  '.items[] | select(.spec.source.persistentVolumeName == $PV) | {name: .metadata.name, node: .spec.nodeName, attached: .status.attached}'

echo -e "\n🚀 Done."

I had still had cronjob pods completed, so I had to delete them first.

🔗 Checking if any pod is using this PVC...
pvc-migration-ndv92
waylonwalker-com-cronjob-29057840-8s92p
waylonwalker-com-cronjob-29057850-4rvm9
waylonwalker-com-cronjob-29057860-6g89j

kubectl delete pod pvc-migration-ndv92 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057840-8s92p -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057850-4rvm9 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057860-6g89j -n waylonwalker-com

💭 AI workloads on Talos Linux - Sidero Labs

2025-02-14T19:27:54Z

AI workloads on Talos Linux - Sidero Labs

Companies are exploring how to run GPU accelerated workloads on Kubernetes. It doesn’t matter if you have a business use case for AI or not, knowing how it works is important.

siderolabs.com

cool article for setting up talos linux with an nvidia gpu. What a wild world it we are living in where these devices that started out being only for hardcore gamers are becoming commonplace in servers and slowly entering the homelab space.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 poolers.postgresql.cnpg.io CRD metadata.annotations Too long ·...

2025-01-21T17:06:21Z

poolers.postgresql.cnpg.io CRD metadata.annotations Too long · Issue #325 · cloudnative-pg/charts

Unable to deploy helm chart using ArgoCD. Getting following error Failed sync attempt to : one or more objects failed to apply, reason: CustomResourceDefinition.apiextensions.k8s.io "poolers.postgr...

GitHub · github.com

I’ve never seen or needed to use a serversideapply in kubernetes before, but I ran into this same issue in my k3s homelab while installing cloudnative-pg.

You can do it with argo

apiVersion: argoproj.io/v1alpha1
kind: Application
spec:
syncPolicy:
syncOptions:
- ServerSideApply=true

and you can do it with kubectl

kubectl apply --server-side --force-conflicts -f cnpg-1.25.0.yaml

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Keycloak

2025-01-19T05:34:32Z

Keycloak

Keycloak - the open source identity and access management solution. Add single-sign-on and authentication to applications and secure services with minimum effort.

Keycloak · keycloak.org

Keycloak looks like an interesting way to setup sso. It’s part of the cncf so it’s got a good backing. I want something better for argo workflows and this might be it. I’m curious what else I can tie into it.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 GitHub - ngalaiko/tree-sitter-go-template: Golang template gra...

2025-01-06T18:49:13Z

GitHub - ngalaiko/tree-sitter-go-template: Golang template grammar for tree-sitter

Golang template grammar for tree-sitter. Contribute to ngalaiko/tree-sitter-go-template development by creating an account on GitHub.

GitHub · github.com

This setup fixed my nvim syntax highlighting in helm templates.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

slow nfs performance

2025-01-02T20:23:10Z

I’m running a two node k3s cluster at home, I thought I could simply mount an nfs share on each worker node, and essentially have the same storage accross all nodes. I’m already learning why this is not reccommended.

Slow #

I’ve been running some cronjobs and argo workflows on the second node for awhile, these are things that run in the background and I don’t care if they take a bit longer to keep my master node freed up for more critical work.

I just started trying to build this site in a cronjob, It was taking 20 minutes to build, and something I noticed was that markata was taking minutes to run glob ( search for files ), normally this happens in a few ms and I never notice this step.

I just moved into the master node and the results were wild at ~30x faster

Permissions #

I have seen where you can get diffent permissions on the nfs share based on user id. Since I’m homelabbing here I only have one user per machine. As you step into enterprise level VMs with tighter controls and dozens of users for all the different services that might run on it.

I’ve ran into maybe one issue where I was root in one place and not another, other than that it’s been fine.

And it only got better #

As the cache was warm subsequent runs only got better.

I just checked again and we are now 80x faster

Conclusion #

I don’t have the answers yet, it might be my network, it might be nfs settings, it might be ext4 filesystem. I have some things to try, but what I do know is that it is not as easy as I thought it would be just to have the same file system mounted on both ends and share data between nodes.

I might even go with a complete alternative and use minio as a storage backend, and sync the files in on each run, this will add some latency to do the sync each time though.

💭 Changelog on X: "🗣️@dhh on Kubernetes' migration pitch: “Oh, w...

2024-11-25T02:30:44Z

External Link

X (formerly Twitter) · x.com

switching cloud providers, theres no easy way. K8s was supposed to get us there, haha, the deep integrations with each vendor just keep locking us in

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Inside Argo: Automating the Future - YouTube

2024-11-22T02:29:00Z

This is a really amazing documentary of argocd. I got into k8s pretty late in the game. Which is pretty typical for me. As I went to use k8s for the first time i was using workflows, then cd. both of these tools had a level of polish that made them seem like they had been there forever and not quite as young as they actually are.

I thought it was interesting how they focused on how the name must be two syllables or less, start with a or b, logo needs to be cutesy funny and recognizable seemed interesting, but puts them at the top of lists and makes them look like they’ve been there forever.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Cluster API book

2024-11-05T04:17:37Z

Provider List - The Cluster API Book

cluster-api.sigs.k8s.io

Cluster API book

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

k3s config after first install

2024-09-15T16:57:41Z

After first setting up a new k3s instance your kubeconfig file will be located in /etc/rancher/k3s/k3s.yaml.

You cans use it from here by setting $KUBECONFIG to that file.

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Or you can copy it to ~/.kube/config

cp /etc/rancher/k3s/k3s.yaml ~/.kube/config

If you have installed k3s on a remote server and need the config on your local machine then you will need to modify the server address to reflect the remote server.

scp user@<server-ip>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

Warning

only do this if you don’t already have a ~/.kube/config file, otherwise copy it to a new file and set your $KUBECONFIG env variable to use it.

Now you will need to open that file and change the server address, making sure to keep the port number.

apiVersion: v1
clusters:
  - cluster:
      certificate-authority-data: ****
      server: https://<server-ip>:6443
    name: default

k8s kustomize diff

2024-07-06T09:42:42Z

I’ve started leaning in on kubernetes kustomize to customize my manifests per deployment per environment. Today I learned that it comes with a diff command.

kubectl diff -k k8s/overlays/local

You can enable color diffs by using an external diff provider like colordiff.

export KUBECTL_EXTERNAL_DIFF="colordiff -N -u"

You might need to install colordiff if you don’t already have it.

sudo pacman -S colordiff

sudo apt install colordiff

Now I can try out kustomize changes and see the change with kustomize diff.

kubectl dash k

2024-07-05T20:15:11Z

Kubernetes ships with a feature called kustomize that allows you to customize your manifests in a declarative way. It’s a bit like helm, but easier to use. I found this useful.

kustomization.yaml #

In order to use kustomize you need to have a kustomization.yaml, kustomization.yml, or Kustomization file in the directory you are applying.

kubectl apply -k .
error: unable to find one of 'kustomization.yaml', 'kustomization.yml' or 'Kustomization' in directory '/tank/git/kustomize-playground'

The kustomization.yaml file cannot be empty.

❯ kubectl apply -k pod
error: kustomization.yaml is empty

lets make an mvp #

mkdir pod
touch pod/pod.yaml
touch pod/kustomization.yaml

# pod.yaml

# kustomization.yaml

Overlays #

Overlays must point to yaml or a directory with a kustomization.yaml file, and cannot reside inside the same directory causing a circular reference.

Layout #

k8s
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
├── overlays
│   ├── local
│   │   └── kustomization.yaml
│   ├── dev
│   │   └── kustomization.yaml
│   └── prod
│       ├── special-prod.yaml
│       └── kustomization.yaml

base #

Place all of the common k8s manifests here in the base directory along with kustomization.yaml. You can split them up into separate files or just one file, I’m keeping it to one for simiplicity.

base - kustomization.yaml #

In the base kustomization.yaml file add all of the manifests that you want to use as a resource.

resources:
  - deployment.yaml

overlays #

Now for each separate environment that you want to have create a directory in overlays with a kustomization.yaml file, and any special manifests that only apply to a particular environment.

overlays - kustomization.yaml #

resoures #

resources:
  - ../../base
# any other specific resources, maybe special ones for prod here
  - sealed-dotenv.yaml

images #

images:
  - name: my-image
    newName: docker.io/myrepo/myimage
    newTag: 1.0.0

💭 Kustomize: The Best Way to Manage Your Kubernetes Configs - Yo...

2024-07-03T13:31:47Z

Great intro into kustomize. This helped me get started with kustomize.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

kind cluster

2024-07-02T08:01:20Z

kind{.hoverlink} is a very useful tool to quickly standup and teardown kubernetes clusters. I use it to run clusters locally. Generally they are short lived clusters for trying, testing, and learning about kubernetes.

Kind is Kubernetes in Docker, its very fast to get a new cluster up and running. Other than checking a box in docker desktop it is the easiest way currently to get a cluster up and running. I’ve used docker desktop for k8s before I really developed on k8s and it was buggy at the time and sometimes started and sometimes didn’t, when it didnt I had no idea how to fix it. I’d suggest kind as the best option to get a cluster up and running locally.

Not Production #

If you are looking for a production ready cluster this is not it. I really like k3s{.hoverlink}. At the time that I chose k3s it was the most lightweight option that easily supported multi-node clusters.

Starting a kind cluster #

The first step, and maybe only one that you need is to create a cluster and give it a name. This command will edit your $KUBECONFIG file, and set the kind cluster as your default cluster to interact with.

kind create cluster --name <CLUSTER_NAME>

Using podman as a backend #

I use podman as my docker engine, kind works with docker and podman, but docker by default, in order to switch to podman you need to set an environment variable.

export KIND_EXPERIMENTAL_PROVIDER=podman

This will tell kind to use podman as the backend provider instead of docker.

Loading images #

If your images are not publically available from a registry, you can load them in kind using the kind load docker-image command.

kind load docker-image $REPOSITORY:$TAG --name <CLUSTER_NAME>

Note

 the CLUSTER_NAME is the name that you gave kind when you started the kind cluster.

Argocd #

Argocd is a great way to setup gitops workflows in kubernetes. compared to just hand-rolling kubectl apply, argo holds the state and is able to not only apply new and change, but cleanup removed things.

setting up a kind cluster with argocd installed

You can stand up argocd in kind for learning argo or getting a nice visual. But often when I use kind its overkill. The cluster is not long lived, I don’t care if things are not cleaned up, and I want to quickly apply changes without a commit and push to a git repo.

Install sealed-secreats via manifest

2024-07-02T07:54:01Z

Yesterday I realized that I have overlooked the default installation method of the sealed secrets controller for kubernetes kubeseal this whole time an jumped straight to the helm section. I spun up a quick kind cluster and had it up quickly. I can’t say this is any better or worse than helm as I have never needed to customize the install. According to the docs you can customize it with [[ kustomize ]] or helm.

# option if you don't have a cluster try with kind

kind create cluster

curl -L https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.27.0/controller.yaml > controller.yaml

kubectl apply -f controller.yaml

💭 Releases · stackrox/kube-linter

2024-06-24T17:09:37Z

GitHub - stackrox/kube-linter: KubeLinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices.

KubeLinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices. - stackrox/kube-linter

GitHub · github.com

A linter for linting kubernetes manifests and help charts.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 argoproj/argo-events: Event-driven Automation Framework for Ku...

2024-06-09T14:30:26Z

GitHub - argoproj/argo-events: Event-driven Automation Framework for Kubernetes

Event-driven Automation Framework for Kubernetes. Contribute to argoproj/argo-events development by creating an account on GitHub.

GitHub · github.com

Argo events is an event driven automation framework for kubernetes that can create kubernetes objects among other things based on events. I’ve been using native kubernetes cronjobs to kick off jobs based on a cron trigger.

For instance I am running reader.waylonwalker.com every hour, to rebuild the site and re-deploy it. It takes about two minutes to fetch every rss feed, so this is a nice application of a job compared to a web server fetching the feeds live. Now my posts may be up to an hour stale but they load fast.

Argo events takes event drien architecture to the next level allowing to be triggered by many more things, and do many more things than creating a cron job. I’m definitely thinking about dropping this in my homelab.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 How to Restart All Pods in a Kubernetes Namespace | Boot.dev

2024-04-25T21:59:56Z

https://boot.dev/blog/devops/how-to-restart-all-pods-in-a-kubernetes-namespace/

blog.boot.dev

As of kubernetes 1.15 there is an easy way to restart all pods in a deployment.

kubectl -n {NAMESPACE} rollout restart deploy

Thanks Lane give him a follow @wagslane

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 argocd automated sync

2024-04-19T19:36:47Z

![[none]]

---

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kanboard
  namespace: argocd
spec:
  project: default
  destination:
    namespace: kanboard
    server: 'https://kubernetes.default.svc'
  source:
    path: kanboard
    repoURL: 'https://github.com/waylonwalker/homelab-argo'
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Manual Upgrades | K3s

2024-04-19T12:51:03Z

Manual Upgrades | K3s

You can upgrade K3s by using the installation script, or by manually installing the binary of the desired version.

docs.k3s.io

You can give k3s an install channel to install stable, latest, or specific versions like 1.26. This is handy to make sure that you install the same version on all of your workers.

curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=latest <EXISTING_K3S_ENV> sh -s - <EXISTING_K3S_ARGS>

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Sealed Secrets

2024-03-28T01:07:06Z

External Link

sealed-secrets.netlify.app

kubeseal is a pretty simple to get started with way to manage secrets such that they can be stored in a git repo and be picked up by your continuous delivery service.

Sealed Secrets provides declarative Kubernetes Secret Management in a secure way. Since the Sealed Secrets are encrypted, they can be safely stored in a code repository. This enables an easy to implement GitOps flow that is very popular among the OSS community.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

kubernetes kubeseal

2024-03-27T20:02:57Z

In my homelab kubernetes cluster I am using kubeseal to encrypt secrets. I have been using it successfully for a few months now wtih great success. It allows me to commit all of my secrets manifests to git with out risk of leaking secrets.

You see kubeseal encrypts your secrets with a private key only stored in your cluster, so only the cluster itself can decrypt them using the kubeseal controller.

KubeSeal #

https://sealed-secrets.netlify.app/

installation #

Installation happens in two steps. You need the kubernetes controller and the client side cli to create a sealed secret.

For a more complete instruction see the [docs#installation](https://github.com/bitnami-labs/sealed-secrets?tab=readme-ov-file#installation]

installation - controller #

Warning

 **context**

Make sure that you are in the right context before running any kubectl commands.

kubectl config current-context

sealed-secrets is installed using the helm package manager. To install sealed-secrets run the following command.

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm install sealed-secrets -n kube-system --set-string fullnameOverride=sealed-secrets-controller sealed-secrets/sealed-secrets

installation - client #

For the client you can check your OS package manager, brew, or the github-releases. For me I found it in the main arch repos.

paru -S kubeseal
# or
sudo pacman -S kubeseal
# or
brew install kubeseal

Note

 You will need to install kubeseal on every device that you will want to

create sealed secrets on.

Example #

Most of these commands come straight from the docs. From my experience I have always specified the namespace, my projects go per namespace and I don’t have any reason that other namepsaces should see the secret, and if they do I deploy another secret in that namespace.

# Create a json/yaml-encoded Secret somehow:
# (note use of `--dry-run` - this is just a local file!)
echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml -n thenamespace > mysecret.yaml

note that the key of the secret is foo and the value is bar

results

apiVersion: v1
data:
  foo: YmFy
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: thenamespace

Note

 The data is base64 encoded.

echo -n bar | base64
# YmFy

# This is the important bit:
kubeseal -f mysecret.yaml -w mysealedsecret.yaml

# At this point mysealedsecret.json is safe to upload to Github,
# post on Twitter, etc.

# Eventually:
kubectl create -f mysealedsecret.yaml -n thenamespace
# sealedsecret.bitnami.com/mysecret created

# Profit!
kubectl get secret mysecret
kubectl get secret mysecret -n thenamespace
# NAME       TYPE     DATA   AGE
# mysecret   Opaque   1      27s

cat mysealedsecret.yaml | kubeseal --validate

echo -n bar | kubectl create secret generic mysecret --dry-run=client --from-file=foo=/dev/stdin -o yaml \
  | kubeseal -o yaml -n thenamespace > mysealedsecret.yaml
echo -n baz | kubectl create secret generic mysecret --dry-run=client --from-file=bar=/dev/stdin -o yaml \
  | kubeseal -o yaml -n thenamespace --merge-into mysealedsecret.yaml

Results

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: thenamespace
spec:
  encryptedData:
    bar: AgBLkamltcLfH1dC1JxQ3qd8lJ8aBZF2ARoq3uo055hnzXOy8g2T5liTx5UPvyPV8yyWqABU8eOnwjNhDtzSATvYeBB3fGkucdOZziWEoiNsWTR9ZtFEkod7Ya6uGkzZOJwi3IkrHFVIT9oWZQUxxJZ6vFhPiFcx9Dorr8TNSzG4KOug25+PhWPPiHDgSup5N3CkWCZaYOF7dbZRVSA4nGP1fZxjFByHP4AsdjLCHptyVbkpLRKeiXTkLxfLX4K+JLZGM41S1On5bSP56mCfv1daTJx619kDXkRLw9l21Ot283/L0NMNAiw781AefYMVoO3aHmYgcT6wAtsQAKje9fyL7DQRHt8a5NZOWukp/P6XjdXRz/nfQasQlbSTrRkDpplKIM5/WdPcBoKi+yyoOL0rZ8x1X7YzUI3BggZmzWyEPD01BK1YAHGZnYIZbbCy1JSm8JCBvP+xWMg+i0Z/DCD8nclAhH1GX2Q7/NrNHF//589AJfuriymd2+mk7uaLA4RRsY0l5QeZD6HVAqSv5jWsVQQtSftWmI9vn9oL/Pno7sEUjSDpXPfF4nnsULhxsPEe2DFAMm1kZAjdF06ueF4/x2Fdy80ZQNyycaDx2CWm4z3b14A75WGyOXl2wJZQqxrFCz8el4hD2nH3zQFEzd6AIh49myqVAGuu2qGlYP4p94LJghVa+mQjztLD/2ZUZjY+anQ=
    foo: AgAducXW6iUCY900cPDdmRfuj7tKnh2hY4C1+2hFoAtjyvjepsKNWsiPJ81t8anaMfFPat4ta060l5VtTrceFE8oS/rViz1tvNWGPBjL+GwL/QjkGl6H8ju87vKERKQn5qw7B/V5j24VM8AxOd+/vJNt/IeRIHLubvFft4hyMq4b0xmIxaemBSTxchQX/5364T3VJH2kHaqpqd+JJgQnTbiTQe/XnyZokDX8GSxw4rAbJUJSRUtY9DB9ZDu2zC5VngX+GJjbwHGbv9EKs8LShJIPrD8xHqrDmlSXGkkP01D4A6268Qoi3x5S0H5aqDgtrgBiWsNkzdKwjfrTNx7pKecOi41lyFdffHOGUew4aPPMqjzWR2TEms9WNNQXwnBdDHKMkFsisocF52BolEkcjF8g/u5h2Af92abMu6k16VybJrB7TV1set5A9W7rqG1iXI4+1W6XQfFnpja8xL/zJBvZcyHgeYMNaxa3C3s6PANhPzAUVaXV9eedAkptGJLN13IZj4LujpoAxRKo6bEdydv/5P23R3fx5PgTOpVI7riECAOIg2PThFsEoVCUwStmKCvIx1I2+YixIlv/OiaUWo4lrI/3ve5WGp4ZnuiJPk34JoYAlRbR6+sX14d8Ek6viq/pJUUIfVpNIkNMboUL4u+KpT47eyQ/mWih/KFduQyX9II/vQ+/IJGzEEHIipxAhdmV+K4=
  template:
    metadata:
      creationTimestamp: null
      name: mysecret
      namespace: thenamespace

backing up your sealing key #

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml >main.key

converting .env files to a secret #

Working with web applications .env is a common way to store credentials. Let’s look at how we can convert these to secrets.

kubectl create secret generic mysecret --from-env-file=.env -n thenamespace --dry-run=client -o yaml > mysecret.yaml

Now you have a secret that looks like this.

apiVersion: v1
data:
  foo: YmFy
kind: Secret
metadata:
  creationTimestamp: null
  name: mysecret
  namespace: thenamespace

Seal it up just like before.

kubeseal -f mysecret.yaml -w mysealedsecret.yaml

Using the secrets #

I typically use the secrets in the container spec.

containers:
  - name: myapp
    envFrom:
      - secretRef:
          name: mysecret
    # You can still have other env vars
    env:
      - name: foo
        value: bar

Sometimes I want to mount the secret as a volume.

containers:
  - name: myapp
    volumeMounts:
      - name: mysecret
        mountPath: /mysecret
volumes:
  - name: mysecret
    secret:
      secretName: mysecret

Image Pull Secrets #

I also need to use imagePullSecrets. Let’s walk through the whole process. Starting with the secret.

kubectl create secret docker-registry regcred --docker-server=myprivateregistry.example.com --docker-username=foo --docker-password=bar --dry-run=client -o yaml

Generates the following secret.

apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJteXByaXZhdGVyZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6ImZvbyIsInBhc3N3b3JkIjoiYmFyIiwiYXV0aCI6IlptOXZPbUpoY2c9PSJ9fX0=
kind: Secret
metadata:
  creationTimestamp: null
  name: regcred
type: kubernetes.io/dockerconfigjson

---

the secret

Now we we can seal that secret.

kubeseal -f regcred.yaml -w regcred-sealed.yaml

And that gives us the following sealed secret that we can deploy into our cluster.

---
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: regcred
  namespace: default
spec:
  encryptedData:
    .dockerconfigjson: AgATYVEywkyYEaoErQbJo6xEZfOnRn1ydNTLkO3Jt/NF/UH+0o9lHpDecRpN0XnVu8xJUcdjkgD9q2XkwP8e6qQDS2mMPTiTNIN+8gbJx97WrD1YQDT0lXBuoyi9I/iwlXxx6MgH/6GY6CeGTz5SRlvoU0Xhlt4d11s7/xapdE8QMLsAReqPEv8oZHEyAxDRrjXX0V+tO8dV+G+GXjUDMBBceLael9rvGzSKIwDVXACVqQhLkB6FoP98M+yyBE46RBNnSnS0ShQM5PprL24HKpRZ43x4RM53KBrQ7R/MxeshafY+B6vUvrolmVox4sud8xngMOcjTO28LLOrck5V8ZiDabhN7ajHEf03IESr1o/ADGf5k9988Vv1txJtsZW0K2mpRu0D7/BLVL9KzbZ5ywULqIoD/Ur2GIGnZqMAKOq4laGp/GJtMKLrhmEvekT397wC/Gf/xdDKVhHf2p4ocsPu7LKFuS5H/Auel/Q5grdn8L5wwrO4VWRv3eJroKh/Hux7Qd7f64O7qdi0XthDocf+gmtjys+Gy72M7tyf8f/O+3oKbS4CWQVTj4ZThMc9znrFnHqt2q/7pAyytTQCpk51wlzOsNvOhCueJM/jmeahaL0LuBrqngqISpnd65sgVzBcZpwK9i2Fckyt0DrZLH+NoIuvaqNhzlF+OMbAft/ylWWKCH4WUP+FKG+1LXM7ud7AA3MMbGSBxHL0/WK/INa7MB56xZKMqqyvvLLQHFTQUROJjkgkzsumdOgwZTRgIFnAZ4+vOX3/1Rtt3mAs3vdoJhL4GuKUYCnEHt908eKkWEVEs7eMk5SdSRtIsbaXO2s0dtADwg==
  template:
    metadata:
      creationTimestamp: null
      name: regcred
      namespace: default
    type: kubernetes.io/dockerconfigjson

Now that we have our sealed registry secret, we can deploy it into our cluster.

kubectl apply -f regcred-sealed.yaml

Now we can use it to pull images from our private registry.

containers:
  imagePullSecrets:
    - name: regcred

Full example #

Here is a full deployment example using all the secrets we have created.

regcred
mounting a secret
envFrom secret

---
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    service: myservice
  name: myservice
  namespace: mynamespace
spec:
  replicas: 1
  selector:
    matchLabels:
      service: myservice
  strategy:
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        service: myservice
    spec:
      containers:
        - envFrom:
            - secretRef:
                name: mysecret
          env:
            - name: foo
              value: bar
          image: private-registry.io/myimage:1.0.0
          name: myimage
          ports:
            - containerPort: 5000
              protocol: TCP
          resources: {}
          volumeMounts:
            - mountPath: /mysecret
              name: mysecret
      restartPolicy: Always
      volumes:
        - name: mysecret
          secret:
            secretName: mysecret
      imagePullSecrets:
        - name: regcred

Downside #

Now the main downside I see with kubeseal is that it does not provide a way to store your secrets in a way that you can access outside of your cluster. So you need to make sure that you have another solution in place to store your secrets so that you still have them if you ever were to take the cluster down or move from k8s to something else.

Overall the likelyhood of you loosing a production cluster is pretty low, so maybe it’s ok to just trust it depending on what the secrets are. Especially for things you control and can rotate anyways its fine.

💭 Configure Liveness, Readiness and Startup Probes | Kubernetes

2024-03-15T14:38:02Z

Configure Liveness, Readiness and Startup Probes

This page shows how to configure liveness, readiness and startup probes for containers. For more information about probes, see Liveness, Readiness and Startup Probes. Before you begin You need to h...

Kubernetes · kubernetes.io

What is the difference between health, liveness, readiness, and startup? This article does a great job at a full writeup description of how it works in kubernetes, here is my TLDR.

health 200 OK - I’m still responding to requests
health ERR - something happened and I cant respond to requests
liveness 200 OK - I’m ready for more work
liveness ERR - I’m still responding to requests, and i’m already working send requests to another pod, or scale up

Z-pages #

These probes are commonly deployed at /healthz and /livez endpoints.

Why the z?

z is a convention that comes from google for meta endpoints to reduce conflict with actual endpoints, and can be deployed to any application.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Kubernetes Secrets in 5 Minutes! - YouTube

2023-10-30T16:47:19Z

I am converting my docker compose env secrets over to k8s secrets. This guide was clear and to the point how I can replicate this exact workflow.

First set the secret, the easiest way is to use kubectl wtih –from-literal because it automatically base64 encodes for you.

kubectl create secret generic minio-access-key --from-literal=ACCESS_KEY=7FkTV**** -n shot

If you don’t use the --from-literal you will have to base64 encode it.

echo "7FkTV****" | openssl base64

Once you have your secret deployed, you have to update the container spec in your deployment manifest to get the valueFrom secretKeyRef.

    spec:
      containers:
        - env:
            - name: ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: ACCESS_KEY
                  name: minio-access-key
            - name: SECRET_KEY
              valueFrom:
                secretKeyRef:
                  key: SECRET_KEY
                  name: minio-secret-key
          image: registry.wayl.one/shot-scraper-api
          name: shot-wayl-one
          ports:
            - containerPort: 5000
              protocol: TCP
          resources: {}
      restartPolicy: Always

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 Waylon Walker 🐍 on X: "Which is more complicated" / X

2023-10-30T13:25:53Z

External Link

X (formerly Twitter) · twitter.com

Wow, shocked at these results. All this time I’ve been told and believed that k8s is incredibly hard, and you need a $1M problem before you think about it because it will take a $1M team to maintain it. So far my experience has been good, and I definitely do not have a $1M problem in my homelab.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

💭 mkimuram/k8sviz: Generate Kubernetes architecture diagrams fro...

2023-10-22T21:07:26Z

GitHub - mkimuram/k8sviz: Generate Kubernetes architecture diagrams from the actual state in a namespace

Generate Kubernetes architecture diagrams from the actual state in a namespace - mkimuram/k8sviz

GitHub · github.com

This is a sick kubernetes architecture diagran generation tool.

Here is an example

installation #

$ curl -LO https://raw.githubusercontent.com/mkimuram/k8sviz/master/k8sviz.sh
$ chmod u+x k8sviz.sh

Usage #

./k8sviz.sh --kubeconfig ~/.config/kube/falcon-k3s.yaml -t png -o k8sviz.png

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts