Nix CI Benchmarks

16 points by rudolfs

jameshh

Using garnix at work, and it's been great. In my experience Nix comes with a lot of upfront costs, getting the Nix flake setup properly for our monorepo (and then maintaining it) can be a pain. But at least this work pays off by giving us access to fast and stable CI, for which we don't need to worry about caching at all. I've sunk more time than I'd like to admit trying to get caching working well on GitHub actions.

jkarni

If anyone has any idea what we might be missing or doing wrong with magic-nix-cache, let me know! You can see an example workflow file here.

dpc_pw

Iirc github cache api is heavily rate limited so bigger building artifacts fare better than a lot of small ones and it was a problem for magic nix cache. This might play a factor here.
- jkarni
  
  This could be it. There were a lot of files. But it also seems like uploading is just often really slow even when it works. E.g. here.
adamcstephens

Did you consider GitHub actions with a self hosted runner that can access the nix store?
- jkarni
  
  Yeah, this and Hydra would make sense to have. They have very different characteristics in terms of cost and scalability, so I think at that point it probably makes sense to include cost estimates and concurrent benchmarks. Which would be good anyhow, but a chink more work!
imcquee

I always get the same magic cache unauthorized error in my workflows. Is this expected when not using flakehub?

https://github.com/garnix-io/benchmark-github/actions/runs/18619774000/job/53089372464
- jkarni
  
  That's what I assumed. It does successfully upload and download things from the cache even with that message, and the docs seem to describe the FlakeHub Cache as optional.