Running microVMs in Proxmox VE, The Easy Way
18 points by lproven
18 points by lproven
It was just nice reading a human author, at least as far as I can detect without any of the usual llm words in them. I might have to look at micro VMS with incus. PVE is just too complicated!
PVE is just too complicated!
I do not use it myself but as I understand it, its main selling point is that it's really simple. It's the FOSS equivalent of VMware ESXi.
If so, it's inferior in many aspects - networking, for example. I miss ESXi's visual switches
This is an interesting approach and I like the concept. It eliminates most virtual hardware devices except virtual disks.
I feel there is room to combine it with the method described here:
https://github.com/Technohacker/central-diskless-boot
Any opinions?
The VirtIO-FS stuff was designed to do this efficiently. It's basically FUSE-over-VirtIO, but it's also has some support for read-only pages that can be directly added to the buffer cache on guests. For the base filesystem, every VM not only shares the space on disk, they also share the space in the RAM cache.
This is great for things like containers because each of the common layers can be shared and each VM has only its own unique layers. For ephemeral VMs, such as CI runners, it can also be very good because the hot pages stay in RAM across VM invocations.
That sounds like a very satisfying setup! I can only really imagine it being useful for something like a shell server, but I'm curious if you have other things in mind?
For a shell server use-case, I think it would be especially cool to also expose a common nix or guix store to the guests, so the read-only root filesystem isn't as restrictive.
I can only really imagine it being useful for something like a shell server
Why so? The author lists a whole set of things he uses it for.
the read-only root filesystem isn't as restrictive.
[looks very puzzled] Where did you get that from?
The phrase "read only" does not occur in the article. There's nothing to indicate anything is read-only. The VMs' boot filesystem will be R/O, I think, but that's all.
Why so? The author lists a whole set of things he uses it for.
Sorry, I was meaning to reply very narrowly to the idea of combining the microvm support from the article article with the specifics from https://github.com/Technohacker/central-diskless-boot
I didn't mean to imply the ro-root was a technical limitation, or that microvms were only useful for a shell server. The article does also mention CI for a git host, which also seems like a great use-case, and the nix angle seems like it would work there too (Tangled seems to leverage this for their CI).
You can use nfsroot since forever: https://www.kernel.org/doc/html/latest/admin-guide/nfs/nfsroot.html
and you don't need a bios to handle PXE
but if you use firecracker you can also assign read-only rootfs, which can be shared by all machines.
I think on a practical level I will not actually use this unless/until it is upstream. At that point, though, I think I would replace almost all of my KVM VMs in my proxmox node with microvms (save for the ~2 windows-based ones I am stuck with).
Since I'm just using proxmox as a nice UI for non-production stuff with limited interaction outside of my LAN, I would be interested in a docker-compose-file-per-microvm setup, where all of the microvms share a root FS that is just alpine+docker+docker-compose+docker-compose-openrc-service, and the corresponding compose.yaml file for the vm just gets mounted at e.g. /srv/compose.yaml. Maybe I'm over-thinking it and there is a nicer way to do this?