Why We’re Moving on From Nix
9 points by uncenter
9 points by uncenter
I want to be clear, we don’t have any problem with Nix itself. But there is a problem with how we were using it. Trying to abstract all the parts of Nix that make Nix… Nix, just fundamentally doesn’t work.
I don’t want to be critical here but I do feel like they were trying to subvert the paradigm.
A key idea of using Nixpkgs is that the entirety of it should work together. So yes, users cannot mix-and-match versions of dependencies between different versions of Nixpkgs. This is something I also struggle to come to terms with. I think the right way to use Nix for something like Python is to tell Nix your packages, choose a version of Nixpkgs and let it handle the specific versions. You pin to that Nixpkgs. The dependencies are also pinned as a result.
As for the image issues, if you let Nix build a layered image, you’ll basically get a layer for each dependency. This is great because as your application changes, only your application layer gets rebuilt. How big is your source code? It’s probably measured in KB or tens of MB. That’s how big your final image layer will be. The other layers will be for each of your dependencies that Nix calculated, and there will be a lot of them, but they will be cached for as long as you leave things pinned with Nix. This is a great improvement IMO. But you have to let Nix build your images rather than using Docker.
It looks like the product from this company wants to do a mixture of Nix choices and developer-facing choices. I think devenv is probably a better approach to this problem, because in the end Nix is in charge of what happens. Nix is kind of like Kubernetes; you don’t want to put it into a box and build up abstractions around it, you kind of have to make it the outermost layer and if you want abstraction, build it within the box.
That said, it looks like an interesting product, and I clicked the link because even though I am pretty enthusiastic about Nix, it isn’t without its shortcomings and getting things fully moved into it is not trivial, so I remain curious about why and how people abandon it.
Yeah, their setup honestly looks pretty bizarre. This is like running the equivalent of
FROM maven
COPY . /app
WORKDIR /app
RUN mvn package
and wondering why your builds are so slow and your images are so massive.
Nix can do amazing things, but it really wants to be in charge of the whole build to do that. If you don’t keep your /nix/store between builds, you’re going to have a bad time. If you copy your whole /nix/store rather than just the output closure, you’re going to have a bad time. If you treat /nix/store as one monolithic layer, rather than split it into a layer per derivation… you’re going to have a bad time.
At my previous work we did exactly that: our whole toolchain (dev and production runtime) was provisionned by Nix and used the approach you describe (pin Nixpkgs to a specific commit), and then in edge cases (build issues with Nix, CVE fix not available in Nix) we would let our build system do the install locally. We were building OCI layered images, and then adding our overlay on top of the Nix runtime. Additionally, we could deploy the required subset of the Nix store on a remote Linux with the same architecture to run out application packages as-is. It worked great, albeit was a bit slow at startup due to provisioning Nix and then restarting in a Nix sandboxed shell. These days a prefer mise as it’s faster, but I miss the ability to create an OCI.
We’re also doing the toolchain split today but we’re slowly moving more and more stuff into native Nix. The OCI stuff is, arguably, the killer app. We also have a lot of folks using macOS and deploying on Linux who do not want to - understandably - spend their entire day running tools in Docker containers. (Maybe the new container stuff from Apple in macOS 26 will change that; I find WSL pretty pleasant.)
Getting people to nix run company#internal-tool has been rewarding - on the horizon all software at the company might be installed via Nix. For compliance reasons having an internal overlay that eliminates packages with difficult licenses also helps. Stuff like overriding all jdk references to default to the company-preferred version? I’m not sure how you’d do that with other tools.
We tried devenv a year or two ago and it had a lot of sharp edges in the enterprise - dealing with stuff like netrc and certificate issues over and over again - we didn’t see a huge benefit to it over flakes + nix-direnv. Well-written and commented flakes go a long way even if LLMs struggle with Nix syntax. :)
Even with a bunch of optimizations in place I’ll agree Nix feels slow especially when you’re doing home-manager stuff that would normally be a one-line edit; it’s also really, really terrible in the enterprise when you’ve got various security tools scanning every new file in /nix/store and every symlink…
nix run company#internal-tool
Given most of our stuff is Typescript and people need to have access to our company package directory setup anyway, this for us would be as simple as: pnpm dlx @chocoapp/internal-tool.
For our Python stuff I imagine it would be the same with uv.
Ok, now do postgres and a jdk with internal certificates in the trust store. (Npm and variants are also mildly awful with internal certs, or were two years ago when I last had to mess with it.)
The point/advantage is having one tool that covers everything instead of the user needing to know which language a tool was written in to figure out which install command to use.
A key idea of using Nixpkgs is that the entirety of it should work together.
That’s fine for NixOS, an operating system, but it’s not fine for a package management system, which is how Nix is being used here, and how it’s often used.
It’s definitely the case that (1) a ton of software just doesn’t depend on each other and (2) people might need very precise versions of software on their system.
If I need a very specific version of Node.js and Playwright for my system, and they aren’t both in the same commit of nixpkgs, I’m subverting the nix paradigm? That isn’t right, and if it was right, that’s a horrible paradigm and doesn’t make any sense.
If I need a very specific version of Node.js and Playwright for my system, and they aren’t both in the same commit of nixpkgs, I’m subverting the nix paradigm?
You are not subverting the nix paradigm. nixpkgs is meant for NixOS, because it is also the largest collection of nix ‘packages’ for software it is used a baseline in most scenarios. If you need a specific version of node you would override attributes (overrideAttrs) of the node definition in nixpkgs to use the version you want. Because most package definitions use the version to to construct the URL to fetch one would only need to modify the version and the sha256 attributes.
If one needs to provide multiple versions a package, like railway does, one needs to provide their own derivations. They can use a nixpkg derivation as a base but modify attrs for each version. One way to do so is to write a json file that has a declarative description of what are the inputs/variations for each versions, cf. versions.json. One can even write glue code scrape the information and write the json file automatically so that it can be kept up to date by a cron job. This is the approach that nixpkgs-ruby takes. You can see its not a lot of code. They even provide a helper that so that the user points to the .ruby-version file instead of specifying the version manually.
Now, in the case of the article, they mention Swift. Ruby, unlike Swift, has few dependencies and probably can be build with a range of versions for their dependencies. ej. One version of OpenSSL probably works for a multitude of Ruby releases. Swift depends on LLVM and likely you’ll need multiple LLVM versions to cover the range they mention inthe article so keeping a flake/overlay to provide multiple versions of Swift is likely a bigger lift than to do so for Ruby.
At no point is switching nixpkgs version because you need a different version of a package a reasonable idea. It is not even a reasonable stop gap solution.
I don’t understand this part:
But even this was problematic because versions are tied to a single commit SHA. Updating the commit hash to support the latest version of a package meant all other package versions would also update.
I don’t understand this. In a Nix context, only all of the dependencies of the package commit hash that has been updated will in turn get updated, but those are also proven to be the ones that won’t fail the build of that package.
If a default version changed, there was a high likelihood that a user’s build would suddenly fail with unexpected errors.
This is literally the risk that Nix is designed to fix- sudden unexpected failures based on updated but incompatible dependencies!! So you are either using Nix wrong or are using it as a hack as part of some greater system that reintroduces all the issues that Nix was designed to be immune from, gaining the worst of both worlds!
We feel bad when users can’t access the latest packages, but feel worse when previously functional builds suddenly fail.
I guess this is the same problem that functional languages have when you try to blend them with procedural/OO languages in the same stack- You lose all the guarantees that functional languages provide, which means… you sort of lose the whole point of making functional languages part of the stack in the first place. It’s like marrying someone who is only guaranteed to be faithful for 6 out of 7 days… Why bother with the marriage if it doesn’t guarantee faithfulness anymore? (Obviously taking liberties with the analogy, here. Marriages are complicated… technology marriages arguably even moreso!)
Which is probably why you have a minority of functional-only shops (:: waves at my people ::)… It’s all-in, or nothing.
I mean I’m waiting for more people to come to this point or for Nix to make some serious steps forward.
Similar to this write-up, all the things that are easy to do in nix are also trivially available elsewhere, and the things that are hard to do in nix are not really worth it.