Most of what you see on my website dates from 2021 when I kicked out the old blog and started using Wordpress. The reason for this was simple: I just wanted to write some blog posts without being held back by making it perfect. However, that backfired somewhat because I realized after several blog posts that the template I was using was holding me back, once again. So here we are, a pandemic and 3 years later doing a blog post on a new platform (Hugo).

About the blog

In 2021 I had an itch to write on several subjects, and I felt the same itch a couple of times over the past few months and decided that I should take matters in my own hands to get writing again. Having 2 weeks of PTO from work over the summer (as I’m writing this), I upgraded my local Hugo instance, found a lovely template for the website and started the work to convert all my blog posts (and all drafts that were never released as blog posts) to Markdown.

Whilst converting my blog posts from Wordpress to Markdown files, I encountered several shortcomings with the template. Because Hugo templates are straightforward, I was able to make the changes I needed to the template I chose. I’ve put up all the changes on my own fork of the typo repo here.

The main visible shortcomings of the template were the way it handles syntax highlighting and code blocks, and how it handles images. I added a lightbox (the only gallery software in javascript that I know by name) dependency in the template so that you can click on the images and get the full resolution picture.

Is this perfect? No, not by far. There’s still a lot I want to improve personalize like a cool way to add side notes to a paragraph, much like Crafting Interpreters does. And more options when it comes to images (ideally breaking them out of the width of the text on larger screens).

“Isn’t the title of this blogpost about containers?”. Yes, that’s true. Let’s get to that.

Container setup

In the last year or so, I’ve been using k3s for a side project. It gives you all the main features of Kubernetes but with opinionated choices and it’s all just one binary. I’ve had a lot of fun playing around with this, but in the end it’s a complex layer to add to any side project if you start thinking about troubleshooting, monitoring (metrics, traces, logs, …), …

But then with the drama and buy-out of Hashicorp, I found myself reading more about the Nomad product than I expected. Like k3s and k8s, it offers an orchestration platform for workloads but it doesn’t constrain itself to containers. You can happily run raw executables or JAR files without using containers. Like k3s it also comes in the format of one binary and unlike k3s/k8s, it has a simple UI builtin.

At some point whilst playing around with it, I found myself thinking about moving my public services (this website and other very sikrit things) to containers and run them with Nomad. It seems to have the right balance between the flexibility I want and the simplicity I need for managing it all.

So there you have it, if you’re reading this, you’re reading an HTML being served by low-privileged container that is orchestrated by a Nomad instance and served behind an nginx ingress gateway (outside of Nomad).

Nomad and Podman

As you’ve read above, I’m running my containers rootless using the Nomad podman driver. This is an implementation of a driver that can run workloads under Nomad. Agents that have the driver installed, can run these type of workloads.

Changes to the podman driver that I needed

Now that’s all good and well but I had some issues with running rootless containers. As a first, the current implementation of the driver (0.6.1 at the time of writing) only allows you to specify one podman socket. So either you run that socket with the root user or you pick one low-priv user that runs the pods. This didn’t really seem like a good idea to me because it breaks the security of having containers run as different users on the host. It also means that if two containers run with their own UID 1000 that they’re effectively the same user on the host system, allowing them to (potentially) read files they shouldn’t be able to access from an application perspective.

So I did what I usually do, try to understand how the driver works and then making tweaks here and there to implement multi-socket support. At the time of writing, the pull request has not yet been accepted. The PR was accepted! But I’m using my modified podman driver for my setup (including other fixes and changes that I stumbled upon whilst testing out this setup).

The current setup of a VPS server looks like:

The user flow involves a User connecting to the nginx ingress, which proxy passes (and provides authentication/mTLS where needed) connections to localhost. The nomad job spec defines the podman socket to call (I have one podman socket per application), the ports that will be exposed and mapped and the volumes that are mounted (read-only where possible) and Nomad makes sure the podman containers stay running. But Nomad and the driver are not involved in actually handling user traffic, it’s only an orchestrator.

Downsides of running Nomad

Unlike when I’m running k3s, the internet is less helpful when you’re googling duckduckgoing errors and problems with Nomad. Furthermore, there’s just a lot less examples floating around that do what you want, so there’s more effort required to figuring things out yourself. I often find myself browsing the Nomad source code to see what it’s doing. But in the end, it’s always good to have a deep understanding of each piece that makes up the pie.

Using podman and systemd quadlet

I tried using podman with systemd quadlet for managing the pods but after a few hours of poking around, I realized that Debian ships with a version that does not include quadlet support. It ships with podman 4.3 and the systemd support is in 4.4. And using Nomad seemed like a more “fun” solution than using systemd files.

Ansible automation

Whilst playing around with making containers for each service, I was also thinking that I’m not gonna make the entire thing a bespoke, manual install on each VPS. So I resorted to the automation tool that I’m most familiar with (but not a huge fan tbh) to get things moving in the right direction without spending too much thought and time on this process.

I created Ansible roles for installing my baseline tools (sysadmin habits are hard to break), nginx as ingress, nomad agents and servers, … This installs a functioning IAAN node in minutes. Although, at this point in time, there’s no reason to believe that I will need to scale up and down a lot, it’s nice to know that when I need to move to a new Debian version or when a datacenter burns down from my VPS provider, I can quickly move my VPS to another datacenter or provider.

Where we go from here

I’ll keep a watch on how Nomad performs and how stable all workloads are, as well as the direction Hashicorp is taking with Nomad to decide if Nomad is a long term fit as my orchestrator. If I’m doubling down on the Hashicorp stack, I can start using their Vault and Consul products in my Nomad setup. If I want to get out of the Hashicorp stack, I’ll probably switch to k3s which should be straightforward given that Podman can produce Kubernetes manifests from (Podman) pods.

I also still have the intention to release several of the draft posts that I already made. There’s some good SIEM stuff in there that I’ve been wanting to share.

Questions & remarks

Since I lost the Wordpress comment functionality, you can reply on Mastodon to the thread that announces this blog post.