A Desipient Blog - rust

A Desipient Blog - rust Zola 2026-05-14T00:00:00+00:00 https://desipient.com/tags/rust/atom.xml Ridiculously Tiny, Auditable Images with Rust and Distroless 2026-05-14T00:00:00+00:00 2026-05-14T00:00:00+00:00 Unknown https://desipient.com/blog/ridiculously-tiny-auditable-images/ <h2 id="the-white-rabbit-trivy"><a class="zola-anchor" href="#the-white-rabbit-trivy" aria-label="Anchor link for: the-white-rabbit-trivy"><i class="icon"></i></a> The White Rabbit - Trivy</h2> <p>I recently had fun building a little <a rel="external" href="https://github.com/sumebrius/ai-operator">toy app</a> in rust, and when the time came to containerising and deploying it, I found myself going down a rabbit hole of miniaturisation. This is the story of that rabbit hole.</p> <p><strong>TLDR:</strong> We’re going to wind up building a tiny image with a minimal surface area for vulnerabilities and full auditability - jump to the end for the final Containerfile<sup class="footnote-reference" id="fr-1-1"><a href="#fn-1">1</a></sup></p> <p>This leporid journey starts with insecurity. That is, on initially building the application into an image based (arbitrarily) on a debian image and scanning with <a rel="external" href="https://github.com/aquasecurity/trivy">Trivy</a>, I find myself irrationally annoyed by a distressing number of vulnerabilities<sup class="footnote-reference" id="fr-2-1"><a href="#fn-2">2</a></sup>.</p> <p>On further investigation, I’m further irrationally annoyed by the fact that not only are all of the vulnerabilities from my arbitrary base, but Trivy evidently has no knowledge of my own app’s dependencies.</p> <h2 id="phase-1-application-auditability"><a class="zola-anchor" href="#phase-1-application-auditability" aria-label="Anchor link for: phase-1-application-auditability"><i class="icon"></i></a> Phase 1 - Application Auditability</h2> <p>Fortunately, letting Trivy know about what dependencies we use in an application is trivial<sup class="footnote-reference" id="fr-3-1"><a href="#fn-3">3</a></sup>, courtesy of some existing tooling - namely the <a rel="external" href="https://crates.io/crates/cargo-auditable">cargo auditable plugin</a>. This works by generating an SBOM (Software Bill Of Materials) and stuffing it right into the compiled binary.</p> <p>To make use of this, we simply install the plugin and use that for building our binary in the build stage of our Containerfile:</p> <pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="docker" data-name="Containerfile"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># ...</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">RUN</span><span> cargo install cargo-auditable</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">COPY</span><span> Cargo.lock Cargo.toml ./</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">COPY</span><span> src/ src/</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">RUN</span><span> cargo auditable build --release</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># ...</span></span></code></pre><h2 id="phase-2-getting-rid-of-base-image-vulnerabilities"><a class="zola-anchor" href="#phase-2-getting-rid-of-base-image-vulnerabilities" aria-label="Anchor link for: phase-2-getting-rid-of-base-image-vulnerabilities"><i class="icon"></i></a> Phase 2 - Getting rid of base image vulnerabilities</h2> <p>Again existing tooling comes to the rescue, in the form of <a rel="external" href="https://github.com/GoogleContainerTools/distroless">“Distroless” Container Images</a>, designed specifically for the problem of runaway dependencies. Considering our application is a single binary, we don’t really need much at all from our base and we’re right in the target audience.</p> <p>Swapping out the base for our final application image from <code>debian:trixy-slim</code> to <code>distroless/cc-debian13:nonroot</code><sup class="footnote-reference" id="fr-4-1"><a href="#fn-4">4</a></sup> is fairly straight-forward, and immediately gets our image’s vulnerability count down to zero.</p> <p>As a nice little side-effect, it also drops our image size from an already respectable 34MiB to an impressive 14.3MiB.</p> <p>As a side-side-effect, it also gives me an irrational urge to see just how small we can make this thing.</p> <h2 id="phase-3-static-compilation"><a class="zola-anchor" href="#phase-3-static-compilation" aria-label="Anchor link for: phase-3-static-compilation"><i class="icon"></i></a> Phase 3 - Static Compilation</h2> <p>What we really, really want is to<sup class="footnote-reference" id="fr-5-1"><a href="#fn-5">5</a></sup> be able to use the distroless <code>static</code> base - but to do this we need to make sure our binary is truly statically compiled.</p> <p>Static compilation with rust is easy-ish enough, we just need to set a <del>magic incantation</del> <code>RUSTFLAG</code> to target the feature <a rel="external" href="https://doc.rust-lang.org/reference/linkage.html#r-link.crt.crt-static">`-C target-feautre=+crt-static</a>.</p> <p>And immediately we’re struck with a wall of linking errors to symbols in <code>libcrypto-lib-c_zlib.o</code>…</p> <h3 id="the-side-quest-rustls"><a class="zola-anchor" href="#the-side-quest-rustls" aria-label="Anchor link for: the-side-quest-rustls"><i class="icon"></i></a> The Side Quest - rustls</h3> <p>The reqwest library, used extensively by our toy app considering it mostly just wraps around calling an API, has two available backends for dealing with TLS:</p> <ul> <li><code>native-tls</code> - which wraps the OS TLS frameworks (or the ubiquitous OpenSSL library)</li> <li><code>rustls</code> - a rust native library</li> </ul> <p><code>rustls</code>, for hopefully evident reasons, is what we want for our use case. It’s also the default choice with the default reqwest feature flags. However, watching compilation and digging through our cargo.lock manifest, for some undocumented<sup class="footnote-reference" id="fr-6-1"><a href="#fn-6">6</a></sup> reason it appears that when combined with the <code>tokio</code> asynchronous runtime (also used extensively by our toy app) then the <code>native-tls</code> dependency is preferred instead.</p> <p>Fortunately, it’s easy enough, if fiddly, to simply be very specific about our feature flags in our cargo dependencies:</p> <pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="toml" data-name="cargo.toml"><span class="giallo-l"><span>reqwest</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> {</span><span> version</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "0.12.24"</span><span style="color: #9399B2;">,</span><span> features</span><span style="color: #94E2D5;"> =</span><span style="color: #9399B2;"> [</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> "json"</span><span style="color: #9399B2;">,</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> # Force rustls</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> "rustls-tls"</span><span style="color: #9399B2;">,</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"> # Defaults except native-tls</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> "charset"</span><span style="color: #9399B2;">,</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> "http2"</span><span style="color: #9399B2;">,</span></span> <span class="giallo-l"><span style="color: #A6E3A1;"> "system-proxy"</span><span style="color: #9399B2;">,</span></span> <span class="giallo-l"><span style="color: #9399B2;">],</span><span> default-features</span><span style="color: #94E2D5;"> =</span><span style="color: #FAB387;"> false</span><span style="color: #9399B2;"> }</span></span></code></pre><h3 id="where-were-we"><a class="zola-anchor" href="#where-were-we" aria-label="Anchor link for: where-were-we"><i class="icon"></i></a> Where were we?</h3> <p>And back from our sidequest, we’re now compiling again. Swapping to the static distroless base for the final image, we’re now down to ~4.9MiB. Pow! now we’re in truly tiny image territory!</p> <h2 id="phase-4-smaller-aka-using-musl"><a class="zola-anchor" href="#phase-4-smaller-aka-using-musl" aria-label="Anchor link for: phase-4-smaller-aka-using-musl"><i class="icon"></i></a> Phase 4 - Smaller! (aka using musl)</h2> <p>We may be tiny already, but by this stage my irrational urge for miniaturisation has become a full-blown obsession. Next step, let’s try replacing those old, stodgy glibc std lib with musl - the standard library used by the famously tiny (but not tiny enough) alpine distro based images.</p> <blockquote class="markdown-alert-caution"> <p>If you care about such inane things as “performance” or “reliability” - you probably want to either ignore this section<sup class="footnote-reference" id="fr-7-1"><a href="#fn-7">7</a></sup> or do some thorough benchmarking. musl is not a drop in replacement for glibc with regards to compatibility or performance.</p> </blockquote> <p>There’s a few steps to this, but nothing too onerous.</p> <ol> <li>Use a musl friendly alpine base for our build image</li> <li>Install the <code>musl-dev</code> - this isn’t installed by default in the base as it’s not exactly a common requirement.</li> <li>Ensure we target the correct platform triplet</li> </ol> <p>With these changes, we’re now down to a smidge under 4.5MiB. glibc added 400 whole KiB of bloat to our image.</p> <h2 id="phase-5-even-smaller-aka-distrolessless"><a class="zola-anchor" href="#phase-5-even-smaller-aka-distrolessless" aria-label="Anchor link for: phase-5-even-smaller-aka-distrolessless"><i class="icon"></i></a> Phase 5 - Even Smaller! (aka distrolessless)</h2> <p>While doing this write-up, a now obvious-in-hindsight thought came to me… Our binary is now fully self-contained, is there anything we even need from our static distroless base? Long answer, all it really provides are tzinfo (not needed, as long as we’re happy to keep everything in UTC) and ca-certificates (not used, as we’ve baked our trust store directly into the binary with rustls).</p> <p>Short answer, no.</p> <p>Replacing our base for the final image with <code>FROM scratch</code> to build it from a literal bare base, we now get a (yes, working) image with a total size of: <strong>3.61MiB</strong></p> <pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="docker" data-name="Containerfile"><span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># Image build base</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">FROM</span><span> rust:1-alpine </span><span style="color: #CBA6F7;">as</span><span> builder</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">WORKDIR</span><span> /build</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># Our musl build dependency</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">RUN</span><span> apk add --no-cache musl-dev</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># For our SBOM generation</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">RUN</span><span> cargo install cargo-auditable</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># Tell rustc to statically link everything</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">ENV</span><span> RUSTFLAGS=</span><span style="color: #A6E3A1;">"-C target-feature=+crt-static"</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">COPY</span><span> Cargo.lock Cargo.toml ./</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">COPY</span><span> src/ src/</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">RUN</span><span> cargo auditable build --release --target x86_64-unknown-linux-musl</span></span> <span class="giallo-l"></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># Runtime image "base"</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">FROM</span><span> scratch</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># The binary is literally the only thing we need</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">COPY</span><span> --from=builder /build/target/x86_64-unknown-linux-musl/release/ai-operator /bin/</span></span> <span class="giallo-l"><span style="color: #9399B2;font-style: italic;"># Oh and a bit of metadata about how to actually run...</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">WORKDIR</span><span> /app</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">EXPOSE</span><span> 3000/tcp</span></span> <span class="giallo-l"><span style="color: #CBA6F7;">CMD</span><span> [</span><span style="color: #A6E3A1;">"/bin/ai-operator"</span><span>]</span></span></code></pre><h2 id="phase-6-still-smaller-aka-compiler-optimisation"><a class="zola-anchor" href="#phase-6-still-smaller-aka-compiler-optimisation" aria-label="Anchor link for: phase-6-still-smaller-aka-compiler-optimisation"><i class="icon"></i></a> Phase 6 - Still Smaller! (aka compiler optimisation)</h2> <p>At this stage, there’s nothing more we can do to reduce this from an image build perspective. The image layers are literally just the binary itself and metadata - there’s simply nothing left to strip out.</p> <p>Any further reductions will need to target shrinking the binary itself, but fortunately(?) the rust compiler defaults don’t optimise for binary size - and we have some levers to pull here.</p> <p>Updating our release <a rel="external" href="https://doc.rust-lang.org/cargo/reference/profiles.html#profile-settings">profile settings</a> with a fairly naive set of size-oriented optimisations, we’re now down to 2.1MiB:</p> <ul> <li>Stripping all symbols out of the binary - yes these are injected by default in the release profile!</li> <li>Setting our optimisation level for size, not speed</li> <li>Enabling link time optimisation, and forcing a single codegen unit to eliminate optimisation boundaries.</li> </ul> <pre class="giallo" style="color: #CDD6F4; background-color: #1E1E2E;"><code data-lang="toml" data-name="cargo.toml"><span class="giallo-l"><span style="color: #9399B2;">[</span><span>profile.release</span><span style="color: #9399B2;">]</span></span> <span class="giallo-l"><span>strip</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "symbols"</span></span> <span class="giallo-l"><span>opt-level</span><span style="color: #94E2D5;"> =</span><span style="color: #A6E3A1;"> "z"</span></span> <span class="giallo-l"><span>lto</span><span style="color: #94E2D5;"> =</span><span style="color: #FAB387;"> true</span></span> <span class="giallo-l"><span>codegen-units</span><span style="color: #94E2D5;"> =</span><span style="color: #FAB387;"> 1</span></span></code></pre> <p>There’s something else we can do here to shave off a surprising number of bytes - eliminating runtime panic unwinds. By setting <code>panic = "abort"</code> we remove any stdlib stack traces for any possible panics.<sup class="footnote-reference" id="fr-8-1"><a href="#fn-8">8</a></sup></p> <p>This gives us a final score of… <strong>1.89MiB</strong> for the entire image.</p> <h2 id="where-next"><a class="zola-anchor" href="#where-next" aria-label="Anchor link for: where-next"><i class="icon"></i></a> Where next?</h2> <p>Aside from nerfing the aforementioned panic handling, we’re starting to run of things to eliminate without taking a hatchet to the code itself.</p> <p>If I were to really let the obsession take me well and truly beyond reason, there are some static assets bundled in the binary that would make obvious targets to explore eliminating:</p> <ul> <li><strong>Ignoring Phase 1</strong> and just don’t add our SBOM. Who cares about security, anyway?</li> <li><strong>Stripping superfluous trust certs</strong> - We’re currently bundling a full trust store with reqwest. Considering we’re only really hitting a single endpoint, we could get very fiddly and add ourselves some maintenance burden by manually building our own trust store consisting of the single ca at the root of our target endpoint (and pray the provider doesn’t change it)</li> </ul> <hr /> <section class="footnotes"> <ol class="footnotes-list"> <li id="fn-1"> <p>Or Dockerfile, if you’re one for brand names. <a href="#fr-1-1">↩</a></p> </li> <li id="fn-2"> <p>Even if none of them are <em>particularly</em> high. <a href="#fr-2-1">↩</a></p> </li> <li id="fn-3"> <p>One might even say… <em>Trivyal</em> 🥁 <a href="#fr-3-1">↩</a></p> </li> <li id="fn-4"> <p>Turns out while we don’t need much from our base, we still need <em>something</em>. The distroless base gives us glibc + libssl, and the <a rel="external" href="https://github.com/GoogleContainerTools/distroless/blob/main/cc/README.md">cc</a> variants also gives us glibcc1 - all of which turn out to be required by our binary courtesy of dependencies. <a href="#fr-4-1">↩</a></p> </li> <li id="fn-5"> <p>zig-a-zig-ah <a href="#fr-5-1">↩</a></p> </li> <li id="fn-6"> <p>If anybody has any clue as to why, I would be glad to acquire this knowledge from you… <a href="#fr-6-1">↩</a></p> </li> <li id="fn-7"> <p>Actually you should probably just ignore this entire blog <a href="#fr-7-1">↩</a></p> </li> <li id="fn-8"> <p>This setting hasn’t made it into the final image though, as this is likely an optimisation beyond reason. Stack traces are too useful when things go pear shaped. <a href="#fr-8-1">↩</a></p> </li> </ol> </section>