A Desipient Blog - kubernetesZola2024-05-01T00:00:00+00:00https://desipient.com/tags/kubernetes/atom.xmlRook CSI Provisioner Empty Secret2024-05-01T00:00:00+00:002024-05-01T00:00:00+00:00
Unknown
https://desipient.com/blog/rook-csi-provisioner-empty-secret/<p>I recently installed <a rel="external" href="https://rook.io">rook</a> as the storage provider for my kubernetes
cluster, managed by ArgoCD. On first use of trying to provision a PVC, I came across
a small issue which proved difficult to diagnose, but easy to fix. Hopefully this
post is favoured by the search index gods, and finds its way to some poor sap facing
the same issue to help.</p>
<h2 id="the-initial-symptom"><a class="zola-anchor" href="#the-initial-symptom" aria-label="Anchor link for: the-initial-symptom"><i class="icon"></i></a>
The initial symptom</h2>
<p>A new PVC is created, and stays perpetually in “Pending state”</p>
<h2 id="initial-investigation"><a class="zola-anchor" href="#initial-investigation" aria-label="Anchor link for: initial-investigation"><i class="icon"></i></a>
Initial investigation</h2>
<p>Following the rook <a rel="external" href="https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-csi-common-issues/#provisioning-volumes">troubleshooting guides</a>,
we check the provisioner logs, and see a whole bunch of logs of the provisioner
trying and failing to provision the PV:</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="shellscript"><span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">kubectl</span><span style="color: #A6E3A1;"> -n rook-ceph logs deploy/csi-rbdplugin-provisioner -c csi-provisioner</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">W0501</span><span style="color: #A6E3A1;"> 10:24:23.798564</span><span style="color: #FAB387;"> 1</span><span style="color: #A6E3A1;"> controller.go:934] Retrying syncing claim "0f1a6d42-0f9b-47a0-9e62-aa36310f3ea5", failure</span><span style="color: #FAB387;"> 385</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">I0501</span><span style="color: #A6E3A1;"> 10:24:23.798780</span><span style="color: #FAB387;"> 1</span><span style="color: #A6E3A1;"> event.go:364] Event</span><span style="color: #9399B2;">(</span><span style="color: #89B4FA;font-style: italic;">v1.ObjectReference</span><span style="color: #A6E3A1;">{Kind:</span><span style="color: #89B4FA;font-style: italic;">"PersistentVolumeClaim",</span><span style="color: #A6E3A1;"> Namespace:"pvc-demo",</span><span style="color: #F5C2E7;">\</span></span>
<span class="giallo-l"><span>Name:</span><span style="color: #A6E3A1;">"my-new-pvc"</span><span>,</span><span style="color: #A6E3A1;"> UID:"7eaef04d-6af8-4104-8cae-8b0edf5f2ed8", APIVersion:"v1", ResourceVersion:"89671786", FieldPath:""}</span><span style="color: #9399B2;">)</span><span style="color: #A6E3A1;">:</span><span style="color: #F5C2E7;">\</span></span>
<span class="giallo-l"><span>type:</span><span style="color: #A6E3A1;"> 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "cluster-ceph": rpc error:</span><span style="color: #F5C2E7;">\</span></span>
<span class="giallo-l"><span>code</span><span style="color: #A6E3A1;"> = InvalidArgument desc = provided secret is empty</span></span>
<span class="giallo-l"><span style="color: #89B4FA;font-style: italic;">code</span><span style="color: #A6E3A1;"> = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is emptycode = InvalidArgument desc = provided secret is empty</span></span></code></pre>
<p>Plugging some of these errors into your search engine of revealed a grand total
of… 6 search results, and nothing relevant.</p>
<h2 id="the-issue-background"><a class="zola-anchor" href="#the-issue-background" aria-label="Anchor link for: the-issue-background"><i class="icon"></i></a>
The issue - background</h2>
<p>Reading through the rook storage configuration <a rel="external" href="https://rook.io/docs/rook/latest-release/Storage-Configuration/Block-Storage-RBD/block-storage/#provision-storage">docs</a>,
the block storage StorageClass definition needs to define a number of standard
<code>csi.storage.k8s.io</code> parameters defining secret names with the credentials required
for connectivity.</p>
<p>Indeed, having a look at the values.yaml of the rook cluster helm chart used to
install the release via argo, the default block pool does indeed <a rel="external" href="https://github.com/rook/rook/blob/release-1.14/deploy/charts/rook-ceph-cluster/values.yaml#L491">define</a>
these parameters.</p>
<h2 id="the-issue-my-mistake"><a class="zola-anchor" href="#the-issue-my-mistake" aria-label="Anchor link for: the-issue-my-mistake"><i class="icon"></i></a>
The issue - my mistake</h2>
<p>However, I had overridden the <code>cephBlockPools</code> list in my own values, creating a
pool with a custom name and replication specification. What I <em>didn’t</em> do was provide
the credentials parameters for its StorageClass, as I’d assumed that these were
fairly standard, and would be included by default if I didn’t provide overrides.</p>
<p>I assumed wrong.<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup></p>
<h2 id="the-fix"><a class="zola-anchor" href="#the-fix" aria-label="Anchor link for: the-fix"><i class="icon"></i></a>
The fix</h2>
<p><code>Ctrl-c</code>, <code>Ctrl-v</code> the secrets parameters from the default values to my block pool’s
StorageClass definition, let Argo redeploy the cluster, and wait.</p>
<p>A few minutes later, success! Our PVs are now provisioned!</p>
<section class="footnotes">
<ol class="footnotes-list">
<li id="fn-1">
<p>As a high school maths teach of mine was fond of saying - When you <em>assume</em>,
you make an <em>ass</em> out of <em>u</em> and <em>me</em>.<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">2</a></sup> <a href="#fr-1-1">↩</a></p>
</li>
<li id="fn-2">
<p>The man was himself, indeed, an ass. <a href="#fr-2-1">↩</a></p>
</li>
</ol>
</section>
Multiple ingress-nginx Instances With MetalLB and ArgoCD2024-04-28T00:00:00+00:002024-04-28T00:00:00+00:00
Unknown
https://desipient.com/blog/multiple-ingress-nginx-instances-with-metallb-and-argocd/<p>In the process of updating my <a rel="external" href="https://kubernetes.github.io/ingress-nginx/">ingress-nginx</a> controllers to
a <a href="../my-clusters-journey/">ArgoCD application</a> on my k8s cluster, I found myself needing to have
multiple instances running within a single namespace and Argo application. OK, maybe I didn’t <em>need</em> to,
but my sense of organisation needed to…</p>
<p>Of course, the first go at setting this up was to add an Argo application sourcing the helm chart twice, and
basically copy-pasting the helm values from the FAQ
<a rel="external" href="https://kubernetes.github.io/ingress-nginx/user-guide/k8s-122-migration/#i-cant-use-multiple-namespaces-what-should-i-do">specifically</a>
about installing multiple instances within the same namespace for the second instance. But this approach was
thwarted by the documentation being wrong. Using these values results in at least one resource with a
duplicate name (the controller <code>service</code> at least, but I didn’t check for more). Had I attempted the install
with helm, this should have failed as the combination of name and resource type is unique within a namespace.
Argo, however, will happily accept the duplicates and ignore all but the last generated.<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup></p>
<p>So the fix? Easy - the chart provides a <code>nameOverride</code> value which can be used to override the generated names
of the resources. This includes, by the way, the <code>electionID</code>, so we can drop that specification in our values.</p>
<p>The resulting application definition looks a little something like the below. Note as well the annotations for
the IPs requested from MetalLB. This ensure that the assigned IPs for each don’t change when moving between
namespaces.</p>
<pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="yaml"><span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 1</span><span style="color: #89B4FA;">apiVersion</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> argoproj.io/v1alpha1</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 2</span><span style="color: #89B4FA;">kind</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> Application</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 3</span><span style="color: #89B4FA;">metadata</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 4</span><span style="color: #89B4FA;"> name</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> ingress-nginx</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 5</span><span style="color: #89B4FA;"> namespace</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> argo</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 6</span><span style="color: #89B4FA;"> finalizers</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 7</span><span style="color: #9399B2;"> -</span><span style="color: #A6E3A1;"> resources-finalizer.argocd.argoproj.io</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 8</span><span style="color: #89B4FA;">spec</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;"> 9</span><span style="color: #89B4FA;"> project</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> default</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">10</span><span style="color: #89B4FA;"> sources</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">11</span><span style="color: #9399B2;"> -</span><span style="color: #89B4FA;"> repoURL</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> https://kubernetes.github.io/ingress-nginx</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">12</span><span style="color: #89B4FA;"> chart</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> ingress-nginx</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">13</span><span style="color: #89B4FA;"> targetRevision</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> 4.10.1</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">14</span><span style="color: #89B4FA;"> helm</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">15</span><span style="color: #89B4FA;"> valuesObject</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">16</span><span style="color: #89B4FA;"> controller</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">17</span><span style="color: #89B4FA;"> service</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">18</span><span style="color: #89B4FA;"> annotations</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">19</span><span style="color: #89B4FA;"> metallb.universe.tf/loadBalancerIPs</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> 192.168.0.1</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">20</span><span style="color: #9399B2;"> -</span><span style="color: #89B4FA;"> repoURL</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> https://kubernetes.github.io/ingress-nginx</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">21</span><span style="color: #89B4FA;"> chart</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> ingress-nginx</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">22</span><span style="color: #89B4FA;"> targetRevision</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> 4.10.1</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">23</span><span style="color: #89B4FA;"> helm</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">24</span><span style="color: #89B4FA;"> valuesObject</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">25</span><span style="color: #89B4FA;"> nameOverride</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> ingress-nginx-secondary</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">26</span><span style="color: #89B4FA;"> controller</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">27</span><span style="color: #89B4FA;"> ingressClassResource</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">28</span><span style="color: #89B4FA;"> name</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> nginx-secondary</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">29</span><span style="color: #89B4FA;"> controllerValue</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> k8s.io/ingress-nginx-secondary</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">30</span><span style="color: #89B4FA;"> enabled</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> true</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">31</span><span style="color: #89B4FA;"> ingressClassByName</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> true</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">32</span><span style="color: #89B4FA;"> ingressClass</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> nginx-secondary</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">33</span><span style="color: #89B4FA;"> service</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">34</span><span style="color: #89B4FA;"> annotations</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">35</span><span style="color: #89B4FA;"> metallb.universe.tf/loadBalancerIPs</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> 192.168.0.2</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">36</span><span style="color: #89B4FA;"> destination</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">37</span><span style="color: #89B4FA;"> server</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> https://kubernetes.default.svc</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">38</span><span style="color: #89B4FA;"> namespace</span><span style="color: #94E2D5;">:</span><span style="color: #A6E3A1;"> ingress-nginx</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">39</span><span style="color: #89B4FA;"> syncPolicy</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">40</span><span style="color: #89B4FA;"> syncOptions</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">41</span><span style="color: #9399B2;"> -</span><span style="color: #A6E3A1;"> CreateNamespace=true</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">42</span><span style="color: #9399B2;"> -</span><span style="color: #A6E3A1;"> PruneLast=true</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">43</span><span style="color: #89B4FA;"> automated</span><span style="color: #94E2D5;">:</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">44</span><span style="color: #89B4FA;"> selfHeal</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> true</span></span>
<span class="giallo-l"><span aria-hidden="true" class="giallo-ln" style="color: #7F849C;">45</span><span style="color: #89B4FA;"> prune</span><span style="color: #94E2D5;">:</span><span style="color: #FAB387;"> true</span></span></code></pre><section class="footnotes">
<ol class="footnotes-list">
<li id="fn-1">
<p>Yes, this is expected behaviour. Yes, I am aware that this says more about the fact that I am completely abusing the multiple source feature of ArgoCD. <a href="#fr-1-1">↩</a></p>
</li>
</ol>
</section>