Volume ownership changes: keep fsGroup from turning recovery into a long mount

Last updated: 1 Oct 20266 min read

tutorial

A Pod fsGroup can make mounted files accessible to a process group, but changing ownership and permissions across a large volume may delay startup. The fsGroupChangePolicy field can skip recursive work when the volume root already matches the expected ownership for supported volume types. Some CSI drivers perform group handling themselves; in that case the Kubernetes-side policy may not control the mount time. Permission tuning must preserve data access, not merely shorten a readiness graph.

Operational decision

A receipt archive Pod spends several minutes before its application container starts after moving to a new node. Compare scheduling, attach, mount, ownership-change, and application-start timestamps. Inspect the security context, volume root ownership, CSI driver capabilities, and file count before changing policy. In a disposable copy of the volume, test OnRootMismatch as shown below, first with a matching root and then with an intentionally wrong root. Confirm the application can read and write a sample file owned by the actual workload identity after both mounts. If the driver owns the group change, inspect its mount behavior rather than assuming the field altered it. Avoid a broad root-level chmod or chown across a production volume during an incident; it can be slow and can change access beyond the target service. Record cold and warm mount time separately.

apiVersion: v1
kind: Pod
metadata:
  name: receipt-archive-check
  namespace: checkout
spec:
  securityContext:
    runAsUser: 2047
    runAsGroup: 2047
    fsGroup: 2047
    fsGroupChangePolicy: OnRootMismatch
  containers:
    - name: archive-check
      image: registry.internal/receipt-archive@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
      command: ["/app/check-volume"]
      volumeMounts:
        - name: archive
          mountPath: /data
  volumes:
    - name: archive
      persistentVolumeClaim:
        claimName: receipt-archive

Cost and verification

Skipping a recursive walk can cut mount latency for a large, correctly owned tree. It also means nested files with unexpected ownership are not repaired just because the root matches. Measure mount duration, file count, permission-denied errors, and recovery time after node loss. The image digest is a placeholder for an approved internal image and must be replaced before use. A root mismatch can still trigger the full walk, so plan for that slow path during failover.

Common Mistakes

Do not equate a faster mount with correct permissions on every nested file.
Do not assume fsGroupChangePolicy controls a CSI driver that handles group mounts itself.
Do not run an unbounded recursive ownership change during a production incident.

Connected lessons

Practice and check

Advanced follow-up

Supplemental groups: remove unexpected access inherited from an image

devops

operations