How I think about Kubernetes
20 points by jado
20 points by jado
Yep, Kubernetes is a data center operating system built atop a container scheduler built atop a controller/operator framework. I often think about how handy it would be if we could divorce the container stuff from the controller/operator framework so we could more easily use the controller framework for application development without choosing between (1) polluting the underlying Kubernetes infrastructure with a bunch of application resources and (2) running a full Kubernetes--with the incumbent container stuff--when we really just want etcd + apiserver + controller-runtime. I'm sure this is possible, but it would be great if someone would package it up nicely.
I saw a talk about such a project. I think it was Kelsey Hightower. I cannot find it again though.
That said, if you start an apiserver and no scheduler, kubelets or controller-manager, you can still use all you're CRDs as you like.
I've been thinking about this as well. Do you think it would be helpful to be compatible with existing k8s operators in some way (trick them into thinking they run on k8s) or start again from first principles?
The container stuff is largely contained in the kubelet, if you don't want it then you can provide your own node agent instead. My old job used to do this: https://github.com/stackabletech/agent, there's also https://github.com/krustlet/krustlet (also abandoned) which does a similar thing.
That’s very interesting and I didn’t realize you could roll your own node agent; however, to be clear, I don’t want any node agent nor a scheduler nor a Pod resource nor a Deployment resource or any of the other things pertaining to running code—I just want the reconciliation framework (etcd + controller-runtime).
You can run it without a node agent at all! (I've done this as part of some testing infrastructure.) Pods will still exist, but any you create will just get stuck as Pending since there's nowhere to schedule them to (nor would you probably run the scheduler that would attempt it :p).
Interesting read for me - based on my experience I more or less expect orchestration tooling to be declarative and do some kind of continuous reconciliation. But the author is absolutely right that this isn't necessarily the case, and being more explicit about what kuberbetes does is useful.
That said, I think this definition begins to break down a bit once you get beyond basics - kuberbetes does quite a lot of other things under the hood apart from reconciling resources. I'm thinking of networking, scheduled jobs, policy enforcement... the list could go on.
The thing that makes Kubernetes so stupid powerful is all those other things are "just" reconciling resources.
It also makes failure modes quite difficult to fix.
But I think you could make that argument about pods and deployments too. There’s nothing particularly simple about scheduling a pod nor is there anything special about configuring a network. In the end containers, networking, scheduling and policies are simply reconciling different kinds of resources.
I’ve personally started to think of Kubernetes as the operating system for a cluster. Linux is the API that it sits on. I’ve long been tired of the POSIX way of thinking about computers and the more I use K8s the happier I am to leave the POSIX world behind.
This is how I describe it too, but with a small addition: "a declarative cluster OS". If you have time for a lengthier definition, you can also add that it's extensible: you can add new "types" (CRDs) and add new operators that reconcile them.
I think the extensibility is important. Without it, Kubernetes is way too low level for my tastes. I don't think you're really leaving the POSIX world behind unless you can run all your workloads without building your own container images.
When you build container images, you need to learn Kubernetes AND you need to learn POSIX. And given how we "abuse" container images, you likely need to learn quite a few Linux distributions too!
I'm sad buildpacks are not more popular, because the idea of just pointing a system to source code and tell it to run it is much nicer. And ultimately, most uses of Kubernetes involve building your own ad-hoc janky system to do that.
But then, we should be able to create buildpacks that do not require "containers as chroots" as a base. You could write buildpacks for a single Linux distribution or for some more suitable runtime. I guess WASM might be it. Maybe then you could have a system where you could point at a bunch of source code and have the system run that... without having to ever think about Linux nor Linux distributions- only about this system.
(Yes, I'm aware that Nix exists and how it might fit into what I'm saying.)
I’m not familiar with buildpacks (looks cool) but I agree that you still need to know POSIX at the container level. I look forward to a world where we don’t need POSIX any more, and maybe WASM/WASI is that world? I love it in theory - I guess we’ll see what it’s like in practice.
I am not entirely convinced by Nix. I certainly think declarative systems are the way forward, and I’ve even used Nix in production (for Nomad worker nodes), but there is something impenetrable and overly idiosyncratic about it, that has never sat well with me.
I also agree that you need to learn k8s AND posix - I would add Dockerfiles too! But I think k8s is a good starting point. I’ve started using FROM SCRATCH in Go, and I barely ever think about POSIX any more.
i went from Ansible -> Guix -> Qubes + Salt, think they'd all benefit from k8s-inspired scope-creep and that one's got it right yet... no time to elaborate, but i appreciate this thread! as i bridge the good, bad, of ugly of k8s with these common ideals and ponder what that system's form, workflows, and api could look like c: there will prob always need to be posix hooks, alpine... not really interested in buildpack beyond how to better expose eg. guix deploy or manage qubes, from a user-flow perspective, sounds like there might be smth there
I don't know. I think that the K8S complexity and the control loop pattern are bad outside clusters, and certainly not everything needs to be a cluster. I also dislike stateful workloads on K8S.
I don't know if WASM is efficient and powerful enough. In the meantime, POSIX is the best we have, and Nix is one of the most promising approaches. My particular desire is to separate defining the state from applying the state. Nix uses a single process to do both things, and I wish it was possible to define system state with whatever language/templating you wanted, and then apply that using a generic tool. (I'm working on this to explore the first part of this.)
I think some of the immutable distros will soon get to something similar- a declarative way to build images, so that you can get the best of Nix, without having to learn Nix. Although I'm not sure if this will be better than Nix or not.