Composition: the feature I've wanted in Flox since I joined the company

7 points by zmitchell

altano

This creates a “merged” manifest from which we lock and build a “composed” environment.

Is this something you can commit to your project’s repository and then re-use by pulling the repo, without having to re-merge anything from floxhub until you’re ready to do an upgrade? As in, does the final, composed environment have a commitable manifest that lives entirely with the project?

If so, this sounds neat.

If not, this still sounds neat, but just not for me. One of the appeals of nix and devbox for me is that every project is a sealed environment that prevents bit rot, and when I open my toy project 15 years from now it’ll build on the first try. I wouldn’t want changes in my root layer, my dev environment, messing that up. But I can see how some (most?) people aren’t treating that as a goal.

zmitchell
Short answer: yep!

Here’s how that works:
- Each environment has a manifest.toml that on its own would get locked to a manifest.lock.
- All you really need to build the environment is the lockfile, and the lockfile itself contains the manifest that it was locked from.
- For composed environments we store the “composing” manifest, the “included” manifests, and the “composed” manifest in the lockfile, which is created from the composed manifest itself.
- So, at the end of the day, the .flox/env/manifest.{toml, lock} that would normally be committed to your repo already has everything it needs to be rebuilt.
We intentionally designed this so that editing the composing manifest (e.g. add a new package, edit an environment variable, etc) is disconnected from needing to fetch any new manifests. The manifests are re-merged every time the environment is built, but because we store them in the lockfile, we don’t need to fetch them every time.
- taras
  
  Nice blog, I wanna try this now. As a non-nix user, what’s the story for incorporating own binaries,etc into flox environments. Am I forced to declare nix packages? Eg on your website you suggest going via flox instead of rustup..but what if I wanna fetch some binary off github, etc?
  - altano
    
    Yep, you can use nix to package most software. Flox (and most of the tools like it) let you install those packages. Here’s a walkthrough of an example with Flox: https://flox.dev/blog/extending-flox-with-nix-flakes/
    
    But if what you’re really asking is “I don’t want to package anything with Nix I just want to fetch an arbitrary binary off GitHub” then it sounds like you want curl and not Nix/Flox.
  - zmitchell
    
    For the moment we don’t have a great way to incorporate your own binaries into a Flox environment, but we’re working on a feature that will address this. Your options right now are (1) write a flake and install that way via the link altano posted, (2) in the “hook.on-activate” script of your manifest you can use curl to download the binary off of GitHub, stick it in a directory, and add that directory to PATH. (2) is probably easier if you don’t know Nix, but it’s a bit of a hack and makes the environment not totally reproducible.
- altano
  
  Seems really well thought out, will check it out!
altano

Another use case for this sort of composition: sometimes we want a subset of a developer environment in CI. Right now with devbox I have a ci devbox and a local dev devbox (purely to speed up ci), and I’ve been toying with the idea of having two ci configs to speed up non-e2e jobs in ci, and these are all defined separately. It’s not hard to keep things in sync for my small setup but devbox wasn’t made for this, so adding packages at my repo root doesn’t work, for example.

I guess I should checkout flox and see if this solves those problems.
- zmitchell
  Oh that’s a cool use case I hadn’t even considered! Yeah, you could totally do that, and since each environment would be a subset of the necessary CI tools, spinning up the CI job wouldn’t take as long (I’m assuming your bottleneck is populating a Nix store from a cache somewhere, I think that’s what we see in our own CI).
  
  We have some documentation for using Flox in CI: https://flox.dev/docs/tutorials/ci-cd/
  
  I just realized the docs don’t point this out, but at least for the GitHub Action you can specify which environment to activate, so in your case you could have a repo with multiple environments like this:
  
  repo/ envs/ ci-minimal/.flox ci-e2e/.flox ci-all/.flox <-- includes ci-minimal and ci-e2e dev-extra/.flox .flox
  
  where .flox/env/manifest, which I’m assuming is the main environment you would use for development, would contain
  
  [include] environments = [ { dir = "envs/ci-minimal" }, { dir = "envs/ci-e2e" }, { dir = "envs/dev-extra" } ]
  
  and in your CI jobs you would flox activate -d envs/ci-all or flox activate -d envs/ci-minimal.
  - zmitchell
    
    I created an issue to explain more of those options in the docs: https://github.com/flox/floxdocs/issues/223