About containers and VMs

Yesterday, I spotted a discussion that I think lacked the terms “application containers” vs. “system containers”. (Unfortunately, the article does not have either on the title.)

This article provides these useful definitions, that I find interesting to understand the typical way to use Docker from the “alternatives to virtual machines” that systems such as Proxmox enable.

Personally, I think both useful. I think application containerization, despite being hugely popular, is a bit messy. When you use application container images provided by third parties, you end up with a random variety of underlying distribution base images. Although application containers should be an opaque box, frequently the underlying distribution matters, and you end up having to know how multiple distributions work.

Of course, it’s also rare for all services you want to run to be packaged for a single distribution you choose to use, and most people find packaging hard, so the apparent ease of packaging stuff as containers is a significant factor.

I use Proxmox (and maybe soon Incus) as the hypervisor in my infrastructure. Most of my services run on system containers based on a single distribution. I feel this helps make my infrastructure easier to manage. However, I have caved in and run some services as application containers (in k8s), because that’s how their authors mean to distribute their software. It’s OK, but my infrastructure has evolved a lot to handle “system containers”, but not “application containers”, so it’s a bit janky still. (A significant part of the self-hosted crowd has the opposite- infrastructures completely geared towards application containers.)

As a parallel point, I prefer Linux workstations, but I find the popularity of just using application containers for anything in development quite hostile to non-Linux users. Running Docker containers on Linux is lightweight and networking can be quite straightforward (esp. because host networking is available).

But although application container tools for macOS and Windows are improving all the time, due to using Linux containers, you require a VM to run them. At that makes everything difficult.

My usual criticism of Docker is in the vein of “easy, not simple”. Windows and macOS have it easier for just using “binaries”, and every day we learn how to make binaries that run on more Linux distributions (even though NixOS is an outlier), leading to projects such as pg-embed that you can use to provide a PostgreSQL database for development environments without the use of containerization, and thus not requiring a heavyweight VM for Windows and macOS developers.

I wrote something about this topic in my cinematic universe.

There’s nothing preventing you from running multiple processes in a docker container. Seems like a false distinction.

Also maybe I should not spam the top level comments so much, but although it seems the application vs. system thing only applies to containers, there are (application) “container” runtimes, such as Kata Containers that use virtual machines.

So while traditionally VMs have been “systems”, you can have application VMs too. So the difference between virtual machines and containers in my opinion keeps reducing- perhaps just to “do you want a kernel with that or not?”