💭 Performance Difference between RWX and RWO volumes · longhorn/...

2025-08-15T19:13:56Z

Performance Difference between RWX and RWO volumes · longhorn longhorn · Discussion #6964

Hey all, because of some internal testing I made a couple of experiments on our Cluster related to performance of RWX and RWO volumes. Because this might be of interest to some people I thought I s...

GitHub · github.com

Interesting longhorn storage performance test, author does highlight right away that this is a simulation and not a REAL test. I did not fully understand the storage semantics before reading through this.

RWO - Always presents a filesystem ext4 or xfs
RWX/ROX - Always presents a network share nfs to the pod.

This is an important distinction for applications that use sqlite or a tool on top of sqlite such as diskcache. With sqlite it is not recomended to run over nfs due to missing required file locking mechanisms.

Longhorn storage still provides a lot of benefits to these applications as the storage is automatically replicated, if the node that your application is running on goes offline a new pod will start on an existing node. If you have planned downtime, you can cordon and drain a node. Since the data is available in another location you will be able to start a new pod on anther node. barring your PodDisruptionBudget settings, taints, and affinity, this may happen automatically.

Note

This post is a thought. It’s a short note that I make about someone else’s content online #thoughts

Changing k8s Storage Class - Migration Job

2025-04-04T09:57:50Z

I’m setting up longhorn in my homelab, and I ran into an issue where I initially setup some pvcs under longhorn, and later realized that to get longhorn to snapshot and backup I needed to hand edit volumes after the fact or change storage class. I’m all in on gitops so option 1 was not an option. So changing storageclass it is.

Now the issue is that you CANNOT mutate storageclass on a provisioned pvc, it is an immutable attribute.

Migration Job #

This migration job will create a new pvc with the new storageclass and move the data from the old pvc to the new pvc.

Existing Pods

 This migration job will not work if you have a pod using the old pvc.  You

will need to shutdown the pod and delete it.

# old pvc with longhorn storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
# new pvc with longhorn-backup storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: site-pvc-longhorn-backup
  namespace: waylonwalker-com
spec:
  storageClassName: longhorn-backup
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
---
# migration job to move the data to the new pvc
apiVersion: batch/v1
kind: Job
metadata:
  name: pvc-migration
  namespace: waylonwalker-com
spec:
  template:
    spec:
      containers:
      - name: pvc-migration
        image: ubuntu:22.04
        command: ["/bin/bash", "-c"]
        args:
          - |
            echo "Starting migration..."
            cd /data
            cp -av source/. destination/
            echo "Migration complete!"
        volumeMounts:
        - name: source-vol
          mountPath: /data/source
        - name: dest-vol
          mountPath: /data/destination
      restartPolicy: Never
      volumes:
      - name: source-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn
      - name: dest-vol
        persistentVolumeClaim:
          claimName: site-pvc-longhorn-backup

Apply the manifests and wait for the job to complete.

kubectl apply -f pvc-migration.yaml

Cleanup #

I had chatgpt create me a script to help me find what is using the pvc so that it can be deleted.

#!/bin/bash

NAMESPACE="waylonwalker-com"
PVC_NAME="site-pvc-longhorn-new"

echo "⏳ Checking if PVC exists..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" || {
  echo "✅ PVC already deleted."
  exit 0
}

echo "🔍 Describe PVC..."
kubectl describe pvc "$PVC_NAME" -n "$NAMESPACE"

echo -e "\n🔗 Checking if any pod is using this PVC..."
kubectl get pods -n "$NAMESPACE" -o json | jq -r \
  --arg PVC "$PVC_NAME" \
  '.items[] | select(.spec.volumes[].persistentVolumeClaim.claimName == $PVC) | .metadata.name'

echo -e "\n🧹 Checking finalizers..."
kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o json | jq '.metadata.finalizers'

echo -e "\n🔎 Checking associated VolumeAttachment..."
PV_NAME=$(kubectl get pvc "$PVC_NAME" -n "$NAMESPACE" -o jsonpath='{.spec.volumeName}')
echo "🔗 PVC is bound to PV: $PV_NAME"

kubectl get volumeattachment -A -o json | jq \
  --arg PV "$PV_NAME" \
  '.items[] | select(.spec.source.persistentVolumeName == $PV) | {name: .metadata.name, node: .spec.nodeName, attached: .status.attached}'

echo -e "\n🚀 Done."

I had still had cronjob pods completed, so I had to delete them first.

🔗 Checking if any pod is using this PVC...
pvc-migration-ndv92
waylonwalker-com-cronjob-29057840-8s92p
waylonwalker-com-cronjob-29057850-4rvm9
waylonwalker-com-cronjob-29057860-6g89j

kubectl delete pod pvc-migration-ndv92 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057840-8s92p -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057850-4rvm9 -n waylonwalker-com
kubectl delete pod waylonwalker-com-cronjob-29057860-6g89j -n waylonwalker-com

Posts tagged: longhorn

💭 Performance Difference between RWX and RWO volumes · longhorn/...

Changing k8s Storage Class - Migration Job

Migration Job #

Cleanup #