Maybe we should revisit microkernels
48 points by bobpoekert
48 points by bobpoekert
If linux were a microkernel system then the linux kernel team wouldn't be responsible for merging every driver for every hardware device, and wouldn't be responsible for vetting code that they would have to become experts on the inner workings of every chip supported by the linux kernel to properly vet.
The reason Linux contains all the drivers is that they don’t have a stable API for out of tree kernel modules. You don’t need to be a micro kernel to have such an API.
And nor does a microkernel prevent such having an unstable internal API, it would just be cross-process.
I remember seeing complaints on a LKML post about (hypothetically making all) Linux drivers in userpace about not being able to do internal API changes .
The Linux kernel actually has a dedicated document by Greg Kroah-Hartman that many people do not agree with explaining why Linux doesn't have a stable driver API: https://github.com/torvalds/linux/blob/master/Documentation/process/stable-api-nonsense.rst
A thread about it: https://lwn.net/Articles/204197/
The gist of it is that kernel developers worry that having stable driver API will make major kernel refactors difficult, since there might be pressure or an expectation to continue supporting drivers that use an older driver API version. And if they drop support for an older stable driver API, many drivers might never be ported over to new API version, forcing people to use an old kernel.
Basically, keeping all the drivers inside the kernel tree means that we all get free driver upgrades / porting, whenever internal kernel structures change. Anyone refactoring the kernel is forced to update all drivers that depend on whatever they're refactoring.
It's so interesting how the kernel community makes this out to be such a huge problem. In the rest of the world, we deal with changing public APIs all the time: we define a version 1.0 of a public API, we change it gradually and backwards compatibly, going through 1.1, 1.2, 1.3 etc, until such a time that we have built up enough technical debt and annoyances from maintaining backwards compatibility that we release version 2.0. Consumers of the API are expected to upgrade from version 1 to version 2 in a timely manner, and if they don't, they're left behind.
Instead of that, we have hardware vendors independently maintaining their own kernel forks with extremely limited hardware support in upstream, oftentimes lagging vendor forks by half a decade or more. Apparently, this is a non-issue, while a versioned public API is the end of the world.
Playing devil's advocate a bit here, but to some extent it makes sense. Having a monorepo means that any such API changes can be done all at once for all the drivers when merging such a change. Not having to maintain a versioning system does make your life as a maintainer much easier. Carrying around code and drivers that (could, say) have concurrency bugs or crashes in your kernel is also not a good thing.
But whether that tradeoff is actually worth it... not sure.
The difference is that with a microkernel, the API crosses process boundaries, which means the collateral damage that buggy drivers can cause is contained and minimized. Imagine your graphics driver is buggy and does garbage memory accesses everywhere in process memory, or even that your hard disk driver crashes. You just restart the driver, the microkernel keeps running, remaining rock solid. You have to have worked with microkernels to develop a deep appreciation for the incredible robustness of these systems.
Meanwhile, on Linux, when my Xbox controller has low batteries and turns off and on quickly, the controller driver gets stuck in such a way that a Linux thread is blocked forever and the kernel module cannot be reloaded. I need to reboot the machine. Insanity.
Anyway, an API that crosses process boundaries forces the programmer to think about object lifetimes, ownership, concurrency, initialization, etc. These things are neglected in various areas of the Linux kernel, which is one of the reasons why the Rust people stepped on so many toes as they tried to integrate the C interfaces into Rust code.
Linux could keep all drivers inside the kernel tree / linux repo, and while doing so, transform into a microkernel. This would address the concerns raised in the “stable api nonsense” document, while reaping the benefits of moving to a microkernel design.
I agree about the USB controller, but I don't think the solution to completely broken graphics and storage(!) drivers is just restarting them. They are not dependable either way.
macOS is kinda going hybrid, by making third-party kexts increasingly impossible and providing userspace APIs for typical use cases for them.
L4 kernels can be fast, but they're only really usable on very static embedded systems.
Genode technically supports various kernels, but it's comically bad on e.g. seL4. (My coworkers have tried booting a Linux VM… only 32-bit was supported and it took many many minutes to halfway boot it…) The Genode fork of the NOVA microhypervisor is the default platform for a good reason (it actually works and performs well enough).
Kry10 OS is a dynamic system built on the seL4 microkernel. It's also performant enough to run Windows in a VM, even without APICv support (something QEMU can't even do without).
A lot of the usability aspects of L4-based systems being restricted to static embedded systems is because of who wants to fund projects: civilian (or unfortunately military) systems integrators who want high assurance guarantees about the systems, which is much easier for static userland.
There is a Swedish company that has reported they run seL4 as a hypervisor for BSD* systems.
Hey! both, our HQ and datacenter are based in Iceland, and we definitely are not a Swedish company. 😄
Regarding the seL4 as a hypervisor, it happened to be a short romance due to its threading implications (and MT not being verified yet). I've eventually employed two people who work full-time on our fork of FreeBSD, and the system is now based on stateless, hardened, and immutable NanoBSD appliances that are easy to swap on-demand.
Oh, cool! Sorry, must have confused Scandinavian with Swedish in my mind.
I'd be interested in hearing more of your thoughts about why you stopped using it.
I want to play devil's advocate and argue that statically-defined systems might be worth considering. A lot of the difficulty with modern OS security is that we're dynamically configuring and administering them. For example by first logging in as a normal user and then punching a hole in the system by using su to elevate privileges so that we can install some piece of software or driver. In a statically-defined system you'd first first define the system while in a restricted barebones administrator mode, then use that definition to boot into the system as a user with normal privileges without even the capacity to escalate. A two-phase system that fundamentally separates administration from everyday use.
One can remove both the su and sudo commands to remove the ability to elevate privileges.
We can imagine a much richer system than just a Unix without sudo. I could have one trusted machine that I use to edit and generate the system definition, which then flashes that definition into the firmware of the untrusted machine. Nothing I do in the untrusted machine would have the ability to flow backwards into the trusted machine. And the system definition could be much richer than just what programs are installed, it could also govern things like what capabilities each program has, what parts of the system they can access (like subsets of the filesystem, hardware devices, dedicated communication channels between other programs), whether or not they can launch subprocesses, and more. The microkernel would set all of this up at boot time from the system definition, then the root process with universal capability would terminate itself before yielding to normal operation, locking down the system in a way that cannot be undone without a reboot.
Are you imagining that "trusted machine" to be physically in front of you? That doesn't help you if the "trusted machine" is elsewhere. I have a public server (somewhere far from where I'm at) that used to just handle my email, DNS and website, but have since added gopher and gemini. I'd hate to think I'd have to take the "trusted machine" a few hundred miles to reflash it. Or if I'm a thousand miles from my "trusted machine" and have to reflash it.
Such a system could still permit some amount of dynamic reconfiguration if you chose to define it that way. Even today remotely administering a system introduces additional complications. But one nice thing about a locked-down system such as the one described above is that you could feasibly leave it unchanged and unupdated for years, even exposed to the network. This is because things like denying processes from launching subprocesses while strictly limiting what hardware and filesystem resources programs can access greatly reduces the ability for potential malware to do anything useful on the machine.
I looked at redox docs after reading this and it looks like it still does a context switch to assemble the request message - presumably a validation step - but it does seem to use a circular buffer and maybe there isn’t a need to do a second context switch if the receiver is already running on another core.
I imagine that context-switch-free message parsing is problematic because the receivers need to validate the messages, which maybe creates an unacceptably large attack surface? But this could be mitigated by kernel-supplied dynamic library, I guess.
I’m as far from an expert on this as I could be, but I feel like the article skipped a few important considerations.
I'm no expert, but qnx is supposedly a fast microkernel. What does it do right?
My experience was, it didn't.
Years ago, a buddy and me got way excited about the elegance of microkernels and QNX in particular. We had a problem to solve in image processing that involved firewire. The code was simple and cool. And it was dismally slow. We picked up the 1394 driver for Linux at the time, added isochronous support using DMA to it (took about 12 hours) and were off to the races. We were relatively new to Linux as well at the time, and it was early enough that there were skeptics in the company about using something like Linux for our optical sorting machines. This event changed that.
I remain of the belief that the ideals behind QNX/microkernels, and message passing, are awesome, but to take hold at that level, something has to be done to make the message passing of data between processes way cheaper.
Ironically, I do web applications with Elixir these days, where I enjoy all the same process isolation as QNX advertised. I just don't care a lot about the performance (e.g. I'm not doing optical processing), because if I did Elixir/BEAM has the same data copying issues.
something has to be done to make the message passing of data between processes way cheaper.
Something like io_uring?
I'll admit I haven't looked at the costs of message-passing for micro-kernels. But I suspect that there a huge amount of context switches and copying of data. My (minor) experience in embedded systems is that lowering that overhead can substantially increase the performance of the system.
i.e. You'd hope that the computer is spending most of its time running your application, but in reality the OS overhead and memory management overhead is enormous.
Yes. Context switching multiple times per (e.g. every network packet) in a synchronous fashion is never going to be particularly fast. There's a reason why you don't do that in Linux either. Batching comes into play, as well as asynchronous shared-memory-queue designs.
That's also to say nothing about the numerous security benefits of address space isolation and code not literally being able to read & write to any memory if compromised.
More precisely: let's say you're running 12Gb/s ethernet (basically 10 but it makes some head-math easier). This is about 1.5GB/s, and for 1500 byte packets that's about 1 Mpps (million packets per second). On a ~3GHz CPU, that gives you around 3000 cycles to process each packet without batching. Just memcpy without batching comes with issues, as (assuming non-cache-coherent DMA) main memory accesses on the order of 100+ cycle latencies can easily exceed the actual time to memcpy one packet and that's 10% of your CPU utilisation gone already. Doing all the packet processing inside the kernel means you pay less of the address-switch costs, but you still pay the mode-switch costs (easily another few hundred cycles on modern x86_64 processors) to go to the application.
Yes, ping-ponging a singular packet through several user-level processes and then a response back is going to never work. But that's the same reason you don't design a web server that processes a single request at a time. You can probably get away with processing a single packet in 3000 cycles if all you do is a UDP echo server, and driver, and multiplexer, but at that point you're spending most of your time context switching. Bump network speeds up to 25Gbe or even 100Gbe and now you have 300 cycles, and (still assuming one CPU) now you have no headroom, even if you only ever need to stay in kernel space.
Of course, anyone who needs the absolute lowest latency where the delay of batching is an issue tends to use full kernel-bypass stacks where you utilise a full core polling, and a full core running an application and everything in shared memory, then you're not paying any context switching costs. But that's easier in a microkernel, because your network (& drivers) already (usually) live in userspace, and your interrupt handling path is faster because the micro-ness of the microkernel causes less cache pollution - although likely you're not even using interrupts and never need to drop into the kernel.
A good example is what happened to the Mach kernel, which is the ancestor of XNU, the kernel of macos and ios. They gradually moved all the userspace processes back into the kernel for performance reasons until it was just a regular monolithic kernel.
I don't know a lot about Mach, but I have heard the argument that it is not "just a regular monolithic kernel" -- even if its components may not be as truly independent as originally envisioned, the architecture still provides some benefits.
P.S. Also, obviously a microkernel is going to do badly on benchmarks like a classic POSIX file listing, where you need to readdir() every entry and stat() it. Batch-oriented APIs like io_uring might change the picture here. If you only do one "system call" once every while, higher latency might not be such a drawback.
Liedkte's "on μ-Kernel Construction", 1995 was a pretty solid deconstruction that Mach's slowness was down to it having a large cache footprint - i.e. it wasn't minimal enough.
P.S. Also, obviously a microkernel is going to do badly on benchmarks like a classic POSIX file listing, where you need to readdir() every entry and stat()
So will a Linux if you do a single syscall per network packet, you won't be able to keep up with line rate. Batching is important on Linux and for Microkernels.
The biggest issue with Mach was that it appended a security context trailer to every message. Calculating this consumed a disproportionate amount of time but it was necessary only because Mach ports were multi-producer, single-consumer and, even then, not actually necessary unless you wanted to make policy decisions about what to do based on the source of a message rather than making policy decisions about who gets to hold a write handle to the port.
MkLinux is an example of the pitfalls. Running Linux as a task under Mach had clear performance impacts, and the fact it was a big pain to maintain all those hacks in the Linux kernel was the final nail in the coffin.
Now, there's a touch of no-true-Scotsman here because that sort of architecture (one big monolithic server as opposed to multiple domain-specific ones) isn't the ideal plan for a Mach-based OS, but it's still a precautionary tale.
There are three key things that the kernel has to do in a microkernel system: scheduling, inter-process communication, and managing access to I/O devices.
What about managing memory protection and address spaces, i.e. controlling the MMU? That seems essential … unless your processes are all written in a memory-safe language with no ability to access ‘unsafe’ mode, which is impractical. (And even then you’d have to worry about safety of the language runtime, like the bugs people exploit against V8 to break out of a browser sandbox.)
Yes, also something a microkernel does need to do. There's a few other things that end up existing too. (and arguably scheduling does not need to completely be in the kernel).
Unless the cost of moving data reduces by an order or magnitude, I don't think microkernel will become competent enough.
Theoretical they are elegent and cleaner but world is a messy place. Perhaps we need messy kernel to deal with it
I love proper architecture, but Linux specifically is the machine for secondary to raw tech reasons. I'd love to read on a multidimensional take at this.
I've heard those microkernel talking points back when I was in university in the early 2000s, and, back then and today, the key talking points appear like extremely poor arguments to me. It all sounds like a disguised excuse to have poorly written software.
I don't find it at all obvious that limiting the impact of a crash is, in the long run, a good thing. It takes away pressure to fix things and have software with fewer bugs. (Just think about this: we went from "integer overflow is undefined behavior" in C to "integer overflow is a clean crash" in Rust.)
If I read something like this: "If linux were a microkernel system then the linux kernel team wouldn't be responsible for merging every driver for every hardware device, and wouldn't be responsible for vetting code that they would have to become experts on the inner workings of every chip supported by the linux kernel to properly vet." I can't help to think that this would be way worse. The fact that every driver for every hardware device is part of the Linux kernel may be annoying, but it acts as a basic quality check that every driver has to go through. This vetting means you have a baseline quality of drivers that you wouldn't have otherwise.
I'm not an operating systems kernel expert, and there may be good reasons to have microkernels. But the common arguments brought forward don't convince me.