Intro

For my new venture as a freelance DevOps Engineer, I decided to build a website using Flask as the backend and a template (named “Simone - Personal Portfolio Template”), purchased from ThemeForest, as the frontend.
Nothing too complicated; HTML, CSS, and Javascript do 90% of the work, and the remaining 10% consists of the Flask backend, which handles internal functionalities like the contact form with Captcha, page routing, the endpoint for Kubernetes livenessProbe, and all the search engine optimization bits (robots.txt and sitemap).

For a couple of months now, I’ve been hosting this website lazzarotto.dev on my homelab using Kubernetes on my 3-node K3s cluster, with two replicas for HA (definitely overkill) and Longhorn for storage (actually only used for Flask’s log files). Until last week, I was using the most common image for a Flask website, which is python:3.14-slim; it’s a Docker image based on the well-known Linux distro Debian Trixie.

Non-Root and Distroless Image

The website was working fine with the previously mentioned Docker image, but for a while now, I had been hearing about these distroless and non-root images on Reddit (and elsewhere), which offered a higher level of security compared to traditional Docker images.
For this reason, and because I wanted to “get my hands dirty” and learn, I decided to convert my Dockerfile to have a root-less and distro-less Docker image for my website.

So, I still used the python:3.14-slim image as the builder stage in my Dockerfile, but I used gcr.io/distroless/python3-debian13:nonroot as the release stage.
So, I built the image, deployed it to Kubernetes with the new image, and the pod started, but it would crash after a few seconds. Looking at the pod’s logs, I discovered that loguru (the logging library) couldn’t write the log file to the folder that was supposed to be mounted by the Longhorn PVC.

Troubleshooting a Distroless Container

If you’ve ever worked with a distroless container, you’ll surely know how complicated it is to troubleshoot an app inside one. For those who don’t know, a distroless image is a Docker image that is missing any basic Linux programs, including: apt/yum, ls, tar, sh, or bash. And so, it’s impossible to use docker exec or kubectl exec to get into the container and check the status of the mountpoint in question.

At this point, my thought was, “I’ve changed two things: the distro and the user running the app.” And I had no idea which of the two changes to the image had introduced the log writing issue.
What I had left to do was to rebuild the Docker image, still without a “root” user, but with a base distro in order to have access to at least ls, touch, etc.

I should point out that all of this was happening while my website was offline because I don’t have a testing environment in my homelab (not yet, at least).

Once I deployed the pod with the new “debug” image, I was able to log into the pod (or rather, into the pod’s container) and realize that the /app/logs folder wasn’t owned by the current user (if I remember correctly, it should be a user with UID 1000), but was instead owned by root.

How to Waste Time Asking Gemini for the Solution

I initially turned to Gemini 2.5 Flash (I know, I should have used the Pro model, but I thought the solution would be simple) to find a solution to the permissions problem with the logs folder.
Its answers seemed really good, and it was always very self-confident, as is always the case with its hallucinations, ça va sans dire. I spent about half an hour on it, but in the end, I gave up and went back to searching on Google, the “old-fashioned” way.

The Solution

The solution came from an issue on the Longhorn project’s GitHub repo that saved me. In short, the problem arises because Longhorn, when it “mounts” the PVC in the container, sets the owner user and group to root, which prevents the non-root user in my image from accessing that PVC.

It still took me some time to put the pieces together, I must admit, but in the end, I chose solution number 7 (Case 7), which was the best for my use case because it didn’t require me to create custom StorageClasses or overhaul my manifests, but to act directly on the securityContext of my Deployment.
Basically, it’s about telling Kubernetes: ‘Hey, when you mount this volume, please make sure it belongs to a group that my application user can write to.’ Simple and effective.

Conclusions and Manifest

So there you have it, an entire afternoon of debugging to add three lines of YAML to my securityContext. But beyond the technical fix, this experience reminded me of two important things.
First: minimal and non-root images are fantastic for security, but they can make debugging a real nightmare if you’re not prepared. Second: never underestimate the power of an “old-fashioned” search on GitHub issues when AI leads you astray.

Now my site is back online, more secure than before, and I’ve learned a valuable lesson about storage permissions in Kubernetes.

I’m adding the relevant part of my manifest that I used for the app’s deployment below.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lazzarotto-dev
  namespace: dmz
  labels:
    app: lazzarotto-dev
spec:
  replicas: 2
  selector:
    matchLabels:
      app: lazzarotto-dev
  revisionHistoryLimit: 2
  template:
    metadata:
      labels:
        app: lazzarotto-dev
    spec:
      securityContext:
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 2000
      containers:
        - name: lazzarotto-dev
          image: git.mlazzarotto.it/marco/lazzarotto_dev:20260205-113032
          imagePullPolicy: Always
    [...]