Minecraft Server in FreeBSD Jails Container
12 points by vermaden
12 points by vermaden
Please stop calling things containers when they have almost none of the properties of an OCI container. The term has actual meaning in the industry. There are at least three container implementations on FreeBSD. This is using the same isolation mechanism (jails) that they use but none of the other benefits of containerisation. It just gives ammunition to people who say ‘FreeBSD has no container solution, look they think jails are containers they don’t even understand what containers are’. It is one of the most harmful things you can do to FreeBSD adoption at the moment.
Either call them jails, or use containers (which use jails for isolation on FreeBSD).
I’m not sure I agree. When talking about OCI containers, I think it can be useful to explicitly call them “OCI containers”.
Even on Linux there are other kinds of containers. E.g. LXC and LXD.
You can have single process Jails just like you can with Docker/Podman:
You can use Bastillefile exactly the same as you use Dockerfile and so on:
Again, you miss the point.
Containers have an image abstraction that lets you build filesystems out of deltas applied to CoW snapshots. These serve as a building block in the distribution and deployment model. Containers separate the image from the per-deployment state (either ephemeral, built as an additional layer, or as a volume that persists across multiple container invocations and can be shared).
Jails are one of the FreeBSD features that implement the container model. ZFS is another. Racct and pf are others.
Just using jails is not building a container deployment, it is just a jail deployment. That is a useful thing for some use cases, but every time you post something like this claiming that it’s a container you make Linux users less likely to switch because they see this and think FreeBSD has no container solution, which is untrue (containerd, Podman, xc, and potluck are all viable container systems on FreeBSD, though I think only the first three are OCI compliant).
Please stop making it easy for people to dismiss FreeBSD.
I have read this definition now the second time and I still don’t see the argument why this is the better definition of containers. Yes it’s wildly used, but I don’t see why it is sound.
To explain it a bit better: When I use for example podman, I get/create some application image (or image description) and give it to podman. Podman now creates an container and start the application in the container. On linux this is done by a mix out of cgroups and namespaces. On freebsd only jails are necessary. So I would call jails a container.
The difference is on Linux is not so easy to manual create your own container. You mostly need some container deployment solution like docker, podman, … On FreeBSD you can very easy play around with manual crafted container.
but every time you post something like this claiming that it’s a container you make Linux users less likely to switch because they see this and think FreeBSD has no container solution
This is why I hate docker and all modern container stuff. It implies you need docker like container deployment to use software in a bigger system. Which causes docker to be your primary OS. I know devops love this, but I think devops is a big mistake.
I have read this definition now the second time and I still don’t see the argument why this is the better definition of containers. Yes it’s wildly used, but I don’t see why it is sound.
Because, to people who manage large container deployments, containers are first and foremost a model for packaging, distribution, deployment, and orchestration. They depend on an isolation technology (of which there are many options, including Kata Containers on Linux that create very lightweight firecracker VMs that share a filesystem with the host OS), but that is an implementation detail.
On linux this is done by a mix out of cgroups and namespaces. On freebsd only jails are necessary. So I would call jails a container.
Turn that around. If someone wrote a tutorial that described how to set up cgroups and namespaces and then called it a container tutorial, would you think it made sense? Or would you think that the person didn’t understand containers.
Now imagine if FreeBSD were the dominant open source platform, and everyone used Docker or Podman for container deployments on FreeBSD. You come across an article explaining how to set up namespaces on Linux and restrict them with cgroups, then install a program into them and run it. Would your reaction be ‘oh, great, Linux has a good container solution’ or ‘Linux people have no idea what containers are’?
The difference is on Linux is not so easy to manual create your own container. You mostly need some container deployment solution like docker, podman, … On FreeBSD you can very easy play around with manual crafted container.
I have no problem with people writing tutorials on how to do things with jails on FreeBSD. The fact that they are a self-contained abstraction that can be used with or without containers is valuable. Writing an article that says ‘FreeBSD has several great container solutions but they all use this core primitive for isolation and you can use that without all of the container infrastructure if you want to, here’s how’ is great FreeBSD marketing.
But when then industry understands ‘containers’ to mean something and you write an article with ‘Container’ in the title that does not do any of these things, it is good anti-FreeBSD marketing. And I wish @vermaden would stop it. This isn’t the first such article and every single one hurts FreeBSD adoption, yet he insists on keeping doing it.
The term ‘Containers’ is not reserved to Linux Podman/Docker solutions. The term Containers was in the BSD/UNIX land long before Docker or Podman saw light of day on Linux - and yet - now the new kids on the block are to decide what to call a Container or not?
Solaris Containers [1] (also known as Zones) were introduced 4 years after FreeBSD Jails happened in 2000 and they both do exactly the same OS Level Virtualization.
HP with its HP-UX UNIX - which LAST official release was in 2007 (there were some patches after that) has
HP-UX Virtual Partitions which are also officially called vPars Containers [2] and HP-UX Containers (SRP) and HP 9000 Containers [3] [4] solutions.
Only IBM within its AIX system named OS Level Virtualization solution a WPAR - Workload Partition - not a Container.
Lets compare that to cars. There are so called Muscle Cars with large 5L-6L-7L V8 engines from 70s and 80s. Now in 1990 Honda starts to call its 1.4L R4 engine based Civic the Muscle Car and says that from now on the cars from the 70s and 80s will be called Trucks … I do not like this rewriting of history.
[1] https://en.m.wikipedia.org/wiki/Solaris_Containers
[2] https://docs.microfocus.com/SA/10.51/Content/V12N/hpux_v12n/ManagingHP-UXVirtualServers.htm
[3] https://support.hpe.com/connect/s/product?language=en_US&kmpmoid=4164838
[4] https://serviceitdirect.com/modernizing-legacy-hp-ux-servers-and-application/
EDIT:
It seems that even Microsoft uses term Containers [5] when it comes to Windows systems - did not knew that before:
TL;DR: If you hate FreeBSD and want to put people off the platform, keep doing what you’re doing. If you don’t, understand that it doesn’t matter what you think a word means or how it was used in the past, it matters how your readers interpret the word.
The term Containers was in the BSD/UNIX land long before Docker or Podman saw light of day on Linux
We called them jails. Solaris called them Zones. It wasn’t until they grew the additional distribution and orchestration features that people started calling them Containers (even in Solaris, Containers were built on Zones, ZFS snapshots, and some other things).
But even if that were true, it doesn’t matter. Terms change. The term ‘container’ now has a clear meaning to people across the industry and every single time you decide to use it to describe something that has a subset of those properties you make FreeBSD look bad. As someone who cares about FreeBSD adoption I am asking you to please stop making FreeBSD look bad.
If you hated FreeBSD and wanted people not to use it, I would understand what you are doing, but most of the time you seem to be advocating for FreeBSD, so why do you insist on using terminology in a way that puts people off the platform?
It seems that even Microsoft uses term Containers [5] when it comes to Windows systems - did not knew that before:
You should because I mentioned that the last two times I asked you not to say ‘container’ when you mean something different I used this as an example. Windows supports both Linux Containers on Windows (LCOW) and Windows Containers on Windows (WCOW). Both of these use Hyper-V for isolation (WCOW can also use isolated kernel namespaces with a shared kernel). Both are container platforms. They can run OCI containers. The OCI container spec describes how to create Windows container images, how to distribute them, and so on.
I just do not accept that Linux oriented people can ‘steal’ terminology just like that.
Duck is a duck and car is a car.
I do not intend to harm FreeBSD - I will make separate article on the blog to make that clear - about Containers term - to make that clear.
Thanks, ver
It’s not Linux people. Containers (with the same definition) work on Linux, Windows, Solaris derivatives, and FreeBSD. And FreeBSD is a great platform for containers because everything works with jails. VNET makes it easy to assign pf rules to jails, racct understands jails as an object for resource accounting, ZFS delegated administration can apply to a jail, and so on. All of these are important building blocks for creating a container solution and, when combined, they add up to a great implementation of containers. I hope the VirtIO-FS patches will also be merged soon so we can have bhyve as an alternative for jails as the isolation mechanism, at which point we will have a way of running FreeBSD and Linux containers on FreeBSD that does not use jails.
Before Docker/Podman existed - the problem you describe did not existed - so yes - its Linux people.
I have to disagree with you here. “Container” has been used to mean many things long before your idea of “container” existed. Neither of us dispute that.
Where we seem to be in disagreement is that popular but incorrect use of a term does not somehow make it correct. People can refer to a Trojan as a virus, and many people would not care about the distinction, but when technical people, particularly those who are writers, do it, they’re doing technical and non-technical people alike a disservice.
A tarball is a container, and that term has been used to describe self-contained (see how “contained” is related to “container”?) applications and environments that have everything you need to run inside of the tarball since at least the ’90s. “Give me a container of that” means give me a tarball of the program, its dependencies, and all of the shared libraries it needs to run.
If we’re talking about terminology creep, we could also talk about how every Unix-like thing these days is referred to as “Linux”, so by your logic, common use of the term would mean that NetBSD can be referred to as a Linux distro. I do that sometimes to poke fun at this, but obviously know the difference.
So does the popular usage of “container” outweigh the proper definition? Does the popular usage of “Linux” outweigh the proper definition? Would you argue yes for one and no for the other? If so, why?
Where we seem to be in disagreement is that popular but incorrect use of a term does not somehow make it correct.
I’ll chime in to say: Yes, I 100% disagree. There is no objectively correct or incorrect use, words do not have inherent meaning. Their meaning is formed by their use for communication. And it evolves, all the time. It’s a bit scary, but it’s also just how natural language works.
We say a program “dumps core” even though core memory has not been around for many, many decades. We call terminal emulators simply “terminals” now, because nobody has a real terminal anymore. We say we’re “printing” a string even though there is no printer or teletype in sight. A linguistic prescriptivist may call all of these “wrong”. And yet all of this is fine as long as everyone involved knows what’s up.
I don’t fully disagree with you, but I mostly disagree with you. A terminal is a terminal because it acts terminal-like; conflating a virus and a Trojan is still precisely wrong when technical people are talking.
What distinction am I making here? A terminal, like “Terminal” in macOS acts like a terminal, so in 99% of the cases where the distinction between a VT420 and a macOS terminal window doesn’t matter, we don’t care about the distinction. Where the distinction matters, we care, and we should care, just as if someone refers to a user-installed program that can’t self-replicate as a virus, they’re incorrect, no matter how many other (non-tecnical) people would make the same mistake.
Using the word “container” to refer to things that predate the contemporary common use of the word can’t be incorrect, else we’d never stop rewriting things. People don’t cease to try this, though - the term “PC” predates the IBM PC, but people want to take exception to personal computers being called PCs, even pre-IBM PC.
So unless you (or someone else) can point out how and why a definition ceases to be valid and retroactively becomes a new usage, I disagree.
Now imagine if FreeBSD were the dominant open source platform, and everyone used Docker or Podman for container deployments on FreeBSD. You come across an article explaining how to set up namespaces on Linux and restrict them with cgroups, then install a program into them and run it. Would your reaction be ‘oh, great, Linux has a good container solution’ or ‘Linux people have no idea what containers are’?
I don’t think you have understand my argument. I say Docker/Podman/… is not a container(solution), It’s container management (or something like this). Yes when someone explain how you build something like jails with cgroups and namespaces on linux I would also accept this has “container” in the title. I also accept when some other article go there and use “the industry definition” for there article.
What bugs me about your comment: You came here bring your definition and claim this is the only one which can be used. I would argue the term “container” has become a buzzword like cloud. So because the term has lost it’s technical meaning I would avoid it.
I think we can stop this discussion here. I feel like you just want to be right. I haven’t read on argument why your definition is the right one or why it’s sound. Only pointing to “the industry” or claiming something will happen when using other definitions.
I did not know FreeBSD had snapshot-based container support until @david_chisnall pointed it out as possible with some combination of ZFS and jails, because I’ve only ever seen @vermaden (and others) talk about jails as FreeBSD’s solution to containers. It never occurred to me Podman ran on FreeBSD either, for the same reason. To most people a jail is like a lightweight VM, not a container solution.
I would argue the term “container” has become a buzzword like cloud.
Most people would probably disagree. Containers are fairly well-defined, even if most people can’t explicitly define them - you know a container solution when you see it.
To most people a jail is like a lightweight VM, not a container solution.
Being precise - FreeBSD Jails are OS Level Virtualization - not a VM (does not matter if lightweight or not).
VMs on FreeBSD is being done by technologies such as Bhyve/VirtualBox/XEN.
I did not know FreeBSD had snapshot-based container support until @david_chisnall pointed it out as possible with some combination of ZFS and jails, because I’ve only ever seen @vermaden (and others) talk about jails as FreeBSD’s solution to containers. It never occurred to me Podman ran on FreeBSD either, for the same reason.
But why? I have never seen claims or implication that filesystem snapshots aren’t possible. Also there are a lot jails management tools (i.e. ezjail) with snapshot support. Podman support is quite new. There are other tools which have different approaches for jail management to choose from, depending on what your requirements and constraints are. Some work more like in the OCI Container world, some looks more like some sort of libvirt hypervisor abstraction and others are a mixture out of both (and also some complete different concepts).
I can understand that it’s might be surprising to hear someone claim that jails are containers, when only know the OCI container idea. But on FreeBSD you had all the tools to build something like docker way before docker (or OCI) was a thing. But till the podman port[0], nobody had writen or ported a docker like solution. But there are solutions to have some big infrastructure managed with FreeBSD, you just need to look a bit around and not only look through the lens of OCI.
Also: If you like the clean OCI world, I don’t think it’s good for you to switch to FreeBSD. Because it doesn’t make a noticeable different[1] to run podman on a Linux distribution or on FreeBSD. When you want to see different ways of managing your Services, FreeBSD (and jails) might be a good OS to play around with.
To most people a jail is like a lightweight VM, not a container solution
Are we talking about containers or container solutions? What is a lightweight VM and why is it not a container? This kind of sloppy use of the term container brings me to my conclusion:
I would argue the term “container” has become a buzzword like cloud.
Most people would probably disagree. Containers are fairly well-defined, even if most people can’t explicitly define them - you know a container solution when you see it.
I think here we can agree to disagree. This loose definition, overloading and the previous mentioned sloopy use of the term are the reasons why I think “container” is a buzzword.
[0] I’m not sure, but I think podman was the first OCI like jail manager. But correct me, if I’m wrong.
[1] For the container part, there are some other nice features you might want to try.
But why? I have never seen claims or implication that filesystem snapshots aren’t possible. Also there are a lot jails management tools (i.e. ezjail) with snapshot support.
Because you have to build it yourself. Containers, first and foremost, are a distribution and orchestration model. If you have a container system, you can:
And you can do all of this with off-the-shelf tools. Being able to build all of this yourself doesn’t help. It’s like claiming that X.org is a desktop environment. Sure, it’s an important part of DEs. If you have X11, you can build a DE, but you don’t automatically have one. They’re different levels of abstraction.
And, as with X11 and DEs, the lowest level bits are replaceable. Most X11 DEs have been ported to Wayland. In the container model, the isolation mechanism is implemented by a shim. Docker originally did this with runc using Linux namespaces and cgroups (crun does it the same way but in C instead of Go, so with far less per-container memory overhead). But gVisor does it with ptrace and Kata Containers provide a shim that uses a lightweight KVM VM. On FreeBSD, runj and ocirun both use jails but there are plans for a bhyve-based version.
Containers on FreeBSD currently use:
You can assemble all of these yourself to deploy containers, but most people don’t consider a pile of ingredients to be equivalent to a meal. If you want to use containers on FreeBSD today, you can use Podman, containerd, or xc for OCI containers. Or you can use Bastille or Potluck for non-OCI containers.
There are lots of container systems for FreeBSD that build on top of jails, just as there are lots of desktop environments that build on top of X.org.
I’m not sure, but I think podman was the first OCI like jail manager. But correct me, if I’m wrong
Samuel Karp wrote runj and ported Moby (Docker) and containerd first but the Podman port (started about a year later by Doug Rabson) works better.
This is kinda like Docker/Podman thing on Linux – but secure instead.
What’s insecure about a rootless Podman container?
Sure - lets have a discussion about differences between security of FreeBSD Jails and Linux Podman containers.
Isolation: With rootless Podman it seems to be on the same level as Jails - but only if You run Podman with SELinux or AppArmor enabled. Without SELinux/AppArmor the Jails offer better isolation. When you run Podman with SELinux/AppArmor and then you add MAC Framework (like mac_sebsd/mac_jail/mac_bsdextended/mac_portacl) the Jails are more isolated again.
Kernel Syscalls Surface: Even rootless Podman has ‘full’ syscall access unless blocked by seccomp (SELinux). Jails have restricted use of syscalls without any additional tools - and that can be also narrowed with MAC Framework on FreeBSD.
Firewall: You can not run firewall inside rootless Podman container. You can run entire network stack and any firewall like PF or IPFW independently from the host inside VNET Jail - which means more security.
TL;DR: FreeBSD Jails are generally more secure out-of-the-box compared to Podman containers and even more secure if you take the time to add additional layers of security.
Also …
Jails are in production since 1999/2000 when they were introduced - so 25 years strong - very well battle tested.
Docker is with us since 2014 so that means about 10 years less - but we must compare to Podman …
Rootless support for Podman first appeared late 2019 (1.6) so only less then 6 years to test.
That means Jails are the most battle tested of all of them.
Hope that helps.
Regards, vermaden
Instead of tuning the JVM memory ceiling to an absolute value , can you set the jail itself to have a memory ceiling?
Setting a limit on the jail might make it crash instead of making it allocate less.
Modern JVMs detect Linux container limits and factor them in. I don’t think openjdk upstream has support to do the same for freebsd jails, nor whether freebsd has a downstream patch to add support. The existing implementation was written with easily extending it in mind.
Both are interesting to set. You can control many memory parameters within the JVM, but some are outside the JVM. I don’t know much about minecraft, but for example everything that Non-Blocking IO (NIO), would allocate on the heap, but outside of the JVM’s managed heap.
Setting the max heap (Xmx) on the JVM is smart to give it enough room to avoid too frequent GC etc. You can indeed limit the heap also, but that will just trigger the GC more often and decrease the performance of the app. Setting the max memory on the jail would be an upper limit to avoid messing with the rest of the system.
So it’s 2 different goals.
Where/when is the user mcserver defined? That seems to be in the hosts’s passwd db. Is that right?
It’s created by make install on the port, see https://cgit.freebsd.org/ports/tree/games/minecraft-server/Makefile and https://cgit.freebsd.org/ports/tree/Mk/bsd.port.mk#n1439 and https://cgit.freebsd.org/ports/tree/Mk/Scripts/do-users-groups.sh which runs pw useradd
Thanks. Re-reading the article I see the mcserver user is only shown in the context of the jail; I had thought it was shown in the host context as well.