A Desipient Blog - pvcZola2024-05-01T00:00:00+00:00https://desipient.com/tags/pvc/atom.xmlRook CSI Provisioner Empty Secret2024-05-01T00:00:00+00:002024-05-01T00:00:00+00:00
Unknown
https://desipient.com/blog/rook-csi-provisioner-empty-secret/<p>I recently installed <a rel="external" href="https://rook.io">rook</a> as the storage provider for my kubernetes
cluster, managed by ArgoCD. On first use of trying to provision a PVC, I came across
a small issue which proved difficult to diagnose, but easy to fix. Hopefully this
post is favoured by the search index gods, and finds its way to some poor sap facing
the same issue to help.</p>
<h2 id="the-initial-symptom"><a class="zola-anchor" href="#the-initial-symptom" aria-label="Anchor link for: the-initial-symptom"><i class="icon"></i></a>
The initial symptom</h2>
<p>A new PVC is created, and stays perpetually in “Pending state”</p>
<h2 id="initial-investigation"><a class="zola-anchor" href="#initial-investigation" aria-label="Anchor link for: initial-investigation"><i class="icon"></i></a>
Initial investigation</h2>
<p>Following the rook <a rel="external" href="https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-csi-common-issues/#provisioning-volumes">troubleshooting guides</a>,
we check the provisioner logs, and see a whole bunch of logs of the provisioner
trying and failing to provision the PV:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">kubectl</span><span style="color: #A6E3A1;"> -n rook-ceph logs deploy/csi-rbdplugin-provisioner -c csi-provisioner</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">W0501</span><span style="color: #A6E3A1;"> 10:24:23.798564</span><span style="color: #FAB387;"> 1</span><span style="color: #A6E3A1;"> controller.go:934] Retrying syncing claim "0f1a6d42-0f9b-47a0-9e62-aa36310f3ea5", failure</span><span style="color: #FAB387;"> 385</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">I0501</span><span style="color: #A6E3A1;"> 10:24:23.798780</span><span style="color: #FAB387;"> 1</span><span style="color: #A6E3A1;"> event.go:364] Event</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">v1.ObjectReference</span><span style="color: #A6E3A1;">{Kind:</span><span style="color: #89B4FA;font-style: italic;">"PersistentVolumeClaim",</span><span style="color: #A6E3A1;"> Namespace:"pvc-demo",</span><span style="color: #F5C2E7;">\</span></span>
<span class="giallo-l"><span>Name:</span><span style="color: #A6E3A1;">"my-new-pvc"</span><span>,</span><span style="color: #A6E3A1;"> UID:"7eaef04d-6af8-4104-8cae-8b0edf5f2ed8", APIVersion:"v1", ResourceVersion:"89671786", FieldPath:""}</span><span style="color: #9399B2;">)</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\</span></span>
<span class="giallo-l"><span>type:</span><span style="color: #A6E3A1;"> 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "cluster-ceph": rpc error:</span><span style="color: #F5C2E7;">\</span></span>
<span class="giallo-l"><span>code</span><span style="color: #A6E3A1;"> = InvalidArgument desc = provided secret is empty</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">code</span><span style="color: #A6E3A1;"> = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is empty</span></span></code></pre>
<p>Plugging some of these errors into your search engine of revealed a grand total
of… 6 search results, and nothing relevant.</p>
<h2 id="the-issue-background"><a class="zola-anchor" href="#the-issue-background" aria-label="Anchor link for: the-issue-background"><i class="icon"></i></a>
The issue - background</h2>
<p>Reading through the rook storage configuration <a rel="external" href="https://rook.io/docs/rook/latest-release/Storage-Configuration/Block-Storage-RBD/block-storage/#provision-storage">docs</a>,
the block storage StorageClass definition needs to define a number of standard
<code>csi.storage.k8s.io</code> parameters defining secret names with the credentials required
for connectivity.</p>
<p>Indeed, having a look at the values.yaml of the rook cluster helm chart used to
install the release via argo, the default block pool does indeed <a rel="external" href="https://github.com/rook/rook/blob/release-1.14/deploy/charts/rook-ceph-cluster/values.yaml#L491">define</a>
these parameters.</p>
<h2 id="the-issue-my-mistake"><a class="zola-anchor" href="#the-issue-my-mistake" aria-label="Anchor link for: the-issue-my-mistake"><i class="icon"></i></a>
The issue - my mistake</h2>
<p>However, I had overridden the <code>cephBlockPools</code> list in my own values, creating a
pool with a custom name and replication specification. What I <em>didn’t</em> do was provide
the credentials parameters for its StorageClass, as I’d assumed that these were
fairly standard, and would be included by default if I didn’t provide overrides.</p>
<p>I assumed wrong.<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup></p>
<h2 id="the-fix"><a class="zola-anchor" href="#the-fix" aria-label="Anchor link for: the-fix"><i class="icon"></i></a>
The fix</h2>
<p><code>Ctrl-c</code>, <code>Ctrl-v</code> the secrets parameters from the default values to my block pool’s
StorageClass definition, let Argo redeploy the cluster, and wait.</p>
<p>A few minutes later, success! Our PVs are now provisioned!</p>
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn-1">
<p>As a high school maths teach of mine was fond of saying - When you <em>assume</em>,
you make an <em>ass</em> out of <em>u</em> and <em>me</em>.<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">2</a></sup> <a href="#fr-1-1">↩</a></p>
</li>
<li id="fn-2">
<p>The man was himself, indeed, an ass. <a href="#fr-2-1">↩</a></p>
</li>
</ol>
</section>