I've gone full Nix: Proxmox to NixOS + Incus
13 points by bew
13 points by bew
This seems like it's reaching the complexity bar of k8s. Anybody have reasons why going nix is better than something made to manage castle and not pets?
mulling over a nixos -> incus stack for a cloud hypervisor, self service API for local compute (vms) once that's sorted, run something like k0s overtop with https://github.com/lxc/cluster-api-provider-incus
incus has incusos, but it might not be quite ready... original poster liked having more flexibility w/ his baremetal, but it's because it's homelab grade stuff
the ideas here scale, though
this has been my stack for a while now. Not only is a rolling distro great for a hypervisor, incus really solved the VM + container problems.
I still use docker compose for a lot of stuff as well but that's mostly because i consume services in this format and don't really bother to do anything more complicated.
I started doing the same migration too some time back. However, I ran into some odd network issues (after a while containers got 2nd, 3rd,... IPs, and became unreachable to eachother until restarted).
After some time attempting to fix, I dropped incus. I used podman+systemd to run the containers as defined in my configuration.nix file, and it works well.
I might add a Nomad server later as a way to run short-term or dev workloads, but I haven't needed that yet.
I even went the step further and use NixOS modules directly instead of containers. That works pretty well. Are you trying to achieve better isolation by using Podman?
A bunch of software only has docker containers but no NixOS modules (eg Wolf). A small amount of software has both but the NixOS module is a disaster and the docker container works perfectly (eg Netbird).
Containers are unavoidable in a homelab that has a non-trivial collection of services. I think I’m using ~5 now despite my strong preference to be native NixOS when possible.
And I’m definitely not using Podman as an isolation primitive: every service I run gets its own microvm, whether it’s a nix package or a podman container, and that’s how I isolate things.
Its mostly just been using containers that various apps ship as. I run postgres as a container for filesystem isolation, so I can test upgrades without breaking the main instance etc.
I do also run a bunch of things directly with systemd units in my configuration.nix file too.
Congratulations, and indeed fully declarative saves a lot of headaches.
Note that to some extent you can actually have your cake and eat it too (truly declarative configuration of virtual machines with NixOs inside Proxmox), courtesy of the wonderful Nix community:
I've been running a small proxmox setup at home for a while but recently started going with a similar approach to this post of getting everything declarative in a single git repo using NixOS + terraform. The idea of getting rid of the proxmox side and going full NixOS all the way down seems pretty appealing, though.