I recently installed rook as the storage provider for my kubernetes cluster, managed by ArgoCD. On first use of trying to provision a PVC, I came across a small issue which proved difficult to diagnose, but easy to fix. Hopefully this post is favoured by the search index gods, and finds its way to some poor sap facing the same issue to help.

The initial symptom

A new PVC is created, and stays perpetually in “Pending state”

Initial investigation

Following the rook troubleshooting guides, we check the provisioner logs, and see a whole bunch of logs of the provisioner trying and failing to provision the PV:

kubectl -n rook-ceph logs deploy/csi-rbdplugin-provisioner -c csi-provisioner
W0501 10:24:23.798564       1 controller.go:934] Retrying syncing claim "0f1a6d42-0f9b-47a0-9e62-aa36310f3ea5", failure 385
I0501 10:24:23.798780       1 event.go:364] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"pvc-demo",\
Name:"my-new-pvc", UID:"7eaef04d-6af8-4104-8cae-8b0edf5f2ed8", APIVersion:"v1", ResourceVersion:"89671786", FieldPath:""}):\
type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "cluster-ceph": rpc error:\
code = InvalidArgument desc = provided secret is empty
code = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is empty

Plugging some of these errors into your search engine of revealed a grand total of… 6 search results, and nothing relevant.

The issue - background

Reading through the rook storage configuration docs, the block storage StorageClass definition needs to define a number of standard csi.storage.k8s.io parameters defining secret names with the credentials required for connectivity.

Indeed, having a look at the values.yaml of the rook cluster helm chart used to install the release via argo, the default block pool does indeed define these parameters.

The issue - my mistake

However, I had overridden the cephBlockPools list in my own values, creating a pool with a custom name and replication specification. What I didn’t do was provide the credentials parameters for its StorageClass, as I’d assumed that these were fairly standard, and would be included by default if I didn’t provide overrides.

I assumed wrong.1

The fix

Ctrl-c, Ctrl-v the secrets parameters from the default values to my block pool’s StorageClass definition, let Argo redeploy the cluster, and wait.

A few minutes later, success! Our PVs are now provisioned!

  1. As a high school maths teach of mine was fond of saying - When you assume, you make an ass out of u and me.2 ↩︎

  2. The man was himself, indeed, an ass. ↩︎