nixpkgs-multiverse: every version that ever existed
18 points by domenkozar
18 points by domenkozar
I got really excited when Farid showed me this!
He has fixed the oldest and most upvoted issue on devenv: https://github.com/cachix/devenv/issues/16
Basic example in devenv:
{ multiverse, ... }: {
packages = [
multiverse.cmake."3.16.5"
multiverse.bun."0.7.0"
];
}
Docs: https://devenv.sh/pinning/#pinning-an-individual-package-version
Wow, this feels huge.
nixpkgs has a large collection of binaries available on many OSs that was previously only available as either rolling release, or a stable set that's cut every 6 months.
But now, this pinning supports a more familiar and granular way to depend on packages.
I expect this will enable all sorts of unforeseen usecases
(Won't someone think of the NixOS global cache hit ratio?!)
I was just thinking recently about the possibility of doing something like this, but the problem that I got hung up on was how to minimize the number of duplicate dependencies between different packages. When I say "duplicate" I mean semantically the same dependency but there being some trivial difference causing it to be a different instantiation of it.
Every package version exists in a whole range of revisions. Across that range of revisions the package may end up having multiple instantiations due to changes in its dependencies (or in stdenv or the per-language build environment). Which one do you pick? This blog post says to pick the newest revision, which is a simple answer but can be inefficient. If I want 3 different packages, maybe all 3 could be fetched from a single nixpkgs revision, but a naive "use the latest revision" approach likely means ending up with 3 different nixpkgs revisions. And not only does that mean 3 copies of nixpkgs, but if these 3 packages have a bunch of common dependencies, using different nixpkgs revisions for each package may mean getting different instantiations of the dependencies too.
So it would be great if there was a way to try to minimize the number of nixpkgs revisions fetched for a set of packages by looking for overlap in the revision ranges for those packages. But for each nixpkgs input you need there's still a range of revisions to pick from, which means there's opportunity here for picking revisions that minimize the number of instantiations of shared dependencies. I don't know how much data you'd need to collect and download to be able to actually solve that problem though.
You are right. It gives you the latest revision that had "python3.10" let's say, but that might have used a different version of a dependency than "python3.10" at the commit before.
We could encode more information but accessing it through JSON is a bit of a bottleneck. I thought about a SQLite builtin for Nix could solve this but started with this approach so far.
Does this not include nested package sets? I just fired up the repl and I don't see python3Packages in here. I assume that python3.pkgs would exist but that would be tied to changes to the python3 derivation, meaning no way to fetch individual python packages by version.
Right now it's top level attr. It's a work in progress -- happy to accept contribution to make it more robust.
Perhaps a very stupid question. But would it make sense to try and upstream this? Or is there little chance this would survive nixpkgs' review?
It would require a whole process around regenerating the json files periodically and making lots more commits that need to be synchronised against incoming PRs. And that's for a small minority of users. Logistically, I believe it's way better for it to live in an external project. You're not losing anything that way and having it in nixpkgs wouldn't provide any special benefit.
We could do it inline with the unstable bump. It's not that many commits and I'm running the GitHub action now....
I think though that incubating as a separate project to work out any kinks is good though.