We're (now) moving from OpenBSD to FreeBSD for firewalls
31 points by wezm
31 points by wezm
This is helpful, as I'm considering switching from OPNsense to something simpler. My two top candidates are OpenBSD and FreeBSD.
I've read that OpenBSD is all about simplicity, security, and clear documentation, but that hasn't been my experience with it so far. I tried installing it on my Pi 4, and the first gotcha is that you have to enter a special command during boot unless you're plugging into the Pi via a USB to TTL cable. And they don't document how to interrupt the boot to enter the command or how to resume boot once you've entered it. And then Ethernet didn't work, but I didn't realize until after I completed the Ethernet step in the installer, and the installer won't let you go backwards. And lots of other little issues where it makes me feel like "good documentation" means "bad UX, so you're forced to read the entire manual to understand anything."
I tried installing FreeBSD on the Pi, and it just worked with no fuss or gotchas.
you can type ! at any of the prompts in the installer and it will take you to a shell, and typing exit will take you back to where you were or install to go back to the start or upgrade will start the upgrade script, which is covered in the INSTALL.arch file for every architecture supported by OpenBSD. Unfortunately, Raspberry Pi's are not a good arm choice for OpenBSD due to the closed nature of broadcom chipsets used, I'm amazed they support as much hardware as they do…
Out of curiosity, what are you using that Pi with FreeBSD for? As a headless home server? Which Pi model was it?
I’m asking because I’m considering myself to get a Pi as a super cheap headless home server in which I can do local backups and maybe host some web apps for my LAN and it’s nice to hear that the installation was uncomplicated :)
It was a Pi 4B.
I'm not actually planning to use it long-term, but I'm using it as a trial run for my new firewall. My actual firewall is running OPNsense on a server with 10 Gbps ports, so I want to see if I can recreate the config on my Pi 4 with FreeBSD + pf + Unbound even though Pi only has 1 Gbps ports and might be CPU bound on routing packets.
If everything works on my Pi, I'll install FreeBSD on my real firewall server and copy over the config that worked on the Pi.
I'll back up my OPNsense config before wiping it, but I'd like to see everything running on a real system before I switch over.
Thanks for the explanation!
Are you hosting stuff on your home networking that must be accessible from outside your LAN, or why do you need a firewall server at home?
I use pf in all of my FreeBSD VPSes but haven’t thought of running a firewall at home (yet), maybe I am missing out on something?
Paranoia mainly. I have a separate WiFi for guests, but guest laptops/phones could very well be running malware, and I don't want them to port scan my network and start attacking my internal services, so the guest WiFi has its own VLAN that can only access the Internet and nothing else on my internal network.
And then I have stuff like a random IP camera that I use as a security camera, but who knows what hidden features are in its firmware, and I don't want it reaching out to some overseas server feeding stills or video from my house, so I have the IP camera on its own VLAN with firewall configured so nothing can get in or out except for me fetching the RTSP feed.
With OPNsense you already have a fully working FreeBSD system under the hood using pf for the firewall.
Yeah, but lately I've been feeling like OPNsense is adds too much complexity for my needs. I think I'll be better off managing config files directly and writing convenience scripts for what I want.
I see, in your use case to setup a new VLAN often, then it may make sense to look into the API if that could be done with some script. I guess you are aware of the Forum, maybe someone already has a solution for that.
Yeah, I considered scripting something on top of the OPNsense API, but I feel like at that point, it's just so many layers of abstraction on top of pf when using pf directly or building my own simple abstraction seems not that difficult.
OpenBSD performance is the single painful point of that OS for me. That is to be expected I guess - all the security mitigations come with a cost.
I never ran (nor was interested in) a proper benchmark, but at some point the difference between OpenBSD and NetBSD on some old hardware was "Literally Unusable vs Usable".
That is to be expected I guess - all the security mitigations come with a cost.
That's not entirely the case here. For the networking stack, a lot of it historically has struggled in the multi-core world due to global locks and how interrupts are delivered in OpenBSD. Many of the recent OpenBSD releases have improved this area, but it's still a game of catch-up.
A good deep dive into the networking stack has been presented a few times by bluhm@: https://www.youtube.com/watch?v=Kn2XEW4Qre0
This year Bluhm presented this slides about networking performance: https://www.openbsd.org/papers/eurobsdcon2025-bluhm-protomp.pdf
I have to imagine a lot of OpenBSD firewalls calcify on an old version because no one wants to rewrite pf.conf syntax on production firewalls. FreeBSD is a lot better at the "don't change unless there's a good reason to" thing that a lot of BSD fans tout.
Should really consider LionsOS for router/firewall.
I know that you love microkernels, and LionsOS looks very promising and interesting, but isn't it extremely early for that? There's only been a couple alpha releases so far.
That being said, it would be a great submission in its own right!
It is only "extremely early" if ignoring the incredibly unreliable foundation other firewall/router systems stand on; seL4 exists on a class of its own.
I agree re: submission.
A "firewall" that considers "TCP connection tracking" and "IPv6 support" as things to be implemented in the future is "extremely early" for any real-world use, regardless of how old or reliable some of its other code is.
seL4 is impressive. It’s also tiny. 99% of what you think of as an OS (including device drivers and the network stack: all of the bits that matter for this use case) are not part of seL4, they are layered on top.
A large part of the reason for switching from OpenBSD to FreeBSD here is better multicore scalability (last benchmarks I saw, OpenBSD was faster for firewall workloads on a single core but on multicore systems FreeBSD was a lot faster).
LionsOS is built around an SPSCQ abstraction, which means that you need to have a lot of static partitioning, which makes it easy to have unbalanced performance. And the verification of seL4 on multicore systems is still quite early.
It might be worth watching seL4 Summit 2025's talk by Gernot Heiser.
A very real TCP/IP stack scenario was shown, and excellent performance was demonstrated, outscaling that of Linux.
He and I gave keynotes at the same SOSP workshop, so I got to see the extended version, including that example. That particular example was not at all well received, it could have been an entry into the benchmarking crimes paper. So many confounding variables in the comparison and no attempt to do any root-cause analysis on the performance, just an assertion that it was due to their design, in spite of the fact that Linux normally gets about an order of magnitude better performance than his baseline was showing on comparable hardware. Oh, and it was run on a single-core machine, which is not where anyone is focusing optimisation for networking anymore.
Sure Gernot wouldn't commit benchmarking crimes?
https://microkerneldude.org/2025/04/27/benchmarking-crimes-meet-formal-verification/
He made a point in the Summit talk to note that they tried their absolute hardest to make Linux perform better than it did, but couldn't.
I think it is not unreasonable to go with the simplest explanation as "root cause", which is that seL4 does perform better in this scenario.
cks doesn’t want to rewrite their pf rules, which limits the choice of OS
Firewalls have been around for decades now, I'm surprised that the industry hasn't coalesced around a standard rules format.
Firewall rule languages are often very leaky abstractions over the underlying implementations. Creating a generic abstraction layer limits the shape of implementations. And there are commercial incentives not to do so, because your firewall rule design is how you expose the differentiating features of your firewall to users.
For simple things such as blocking ports, forwarding ports, or blocking specific remote IPs, there are basically only two ways of doing it (inline rules or rules that match table entries), but more advanced features like traffic shaping, QoS, and so on are much more complex, and the interactions between different features add a lot more complexity.
Not only leaky but also often terrible. There was a reason I was using ferm on personal machines for like 10 years because I didn't want to spend another minute with iptables.
And the software development industry hasn't coalesced around a single standard programming language, either. :)
I think the reasons are similar. The main cause is that the "firewall" rule language in a network OS is rarely, if ever, just for the firewall in the strict sense — it's a policy language that is also used for traffic classification and marking, policy-based routing definitions, and other features. Which kinda makes sense because it would be wasteful to invent a whole different language for marking packets for QoS and PBR purposes and make users learn it, when you can add a new action to the "firewall" language.
Since those additional features can vary between OSes a lot and the way they work can vary dramatically as well, a "least common denominator" language could only be a small subset of the functionality that pf, nft, Cisco's firewalls, etc. actually provide.
Well, and proprietary vendors also don't want to make it easy to migrate away from them, I suppose that's also a big reason...
There were some projects like https://fwbuilder.sourceforge.net/ for example. Never seemed to see much adoption.