Kubernetes 1.24: Prevent unauthorised volume mode conversion

Author: Raunak Pradip Shah (Mirantis)

Kubernetes v1.24 introduces a new alpha-level feature that prevents unauthorised users from modifying the volume mode of a PersistentVolumeClaim created from an existing VolumeSnapshot in the Kubernetes cluster.

The problem

The Volume Mode determines whether a volume is formatted into a filesystem or presented as a raw block device.

Users can leverage the VolumeSnapshot feature, which has been stable since Kubernetes v1.20, to create a PersistentVolumeClaim (shortened as PVC) from an existing VolumeSnapshot in the Kubernetes cluster. The PVC spec includes a dataSource field, which can point to an existing VolumeSnapshot instance. Visit Create a PersistentVolumeClaim from a Volume Snapshot for more details.

When leveraging the above capability, there is no logic that validates whether the mode of the original volume, whose snapshot was taken, matches the mode of the newly created volume.

This presents a security gap that allows malicious users to potentially exploit an as-yet-unknown vulnerability in the host operating system.

Many popular storage backup vendors convert the volume mode during the course of a backup operation, for efficiency purposes, which prevents Kubernetes from blocking the operation completely and presents a challenge in distinguishing trusted users from malicious ones.

Preventing unauthorised users from converting the volume mode

In this context, an authorised user is one who has access rights to perform Update or Patch operations on VolumeSnapshotContents, which is a cluster-level resource.
It is upto the cluster administrator to provide these rights only to trusted users or applications, like backup vendors.

If the alpha feature is enabled in snapshot-controller, snapshot-validation-webhook and external-provisioner, then unauthorised users will not be allowed to modify the volume mode of a PVC when it is being created from a VolumeSnapshot.

To convert the volume mode, an authorised user must do the following:

1. Identify the VolumeSnapshot that is to be used as the data source for a newly created PVC in the given namespace.
2. Identify the VolumeSnapshotContent bound to the above VolumeSnapshot.

kubectl get volumesnapshot -n <namespace>

3. Add the annotation snapshot.storage.kubernetes.io/allowVolumeModeChange to the VolumeSnapshotContent.

4. This annotation can be added either via software or manually by the authorised user. The VolumeSnapshotContent annotation must look like following manifest fragment:

kind: VolumeSnapshotContent
metadata:
  annotations:
    - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true"
...

Note: For pre-provisioned VolumeSnapshotContents, you must take an extra step of setting spec.sourceVolumeMode field to either Filesystem or Block, depending on the mode of the volume from which this snapshot was taken.

An example is shown below:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  annotations:
  - snapshot.storage.kubernetes.io/allowVolumeModeChange: "true"
  name: new-snapshot-content-test
spec:
  deletionPolicy: Delete
  driver: hostpath.csi.k8s.io
  source:
    snapshotHandle: 7bdd0de3-aaeb-11e8-9aae-0242ac110002
  sourceVolumeMode: Filesystem
  volumeSnapshotRef:
    name: new-snapshot-test
    namespace: default

Repeat steps 1 to 3 for all VolumeSnapshotContents whose volume mode needs to be converted during a backup or restore operation.

If the annotation shown in step 4 above is present on a VolumeSnapshotContent object, Kubernetes will not prevent the volume mode from being converted. Users should keep this in mind before they attempt to add the annotation to any VolumeSnapshotContent.

What's next

Enable this feature and let us know what you think!

We hope this feature causes no disruption to existing workflows while preventing malicious users from exploiting security vulnerabilities in their clusters.

For any queries or issues, join Kubernetes on Slack and create a thread in the #sig-storage channel. Alternately, create an issue in the CSI external-snapshotter repository.

• ←Previous
Next→