Super Mario Derivations
10 points by npiazza
10 points by npiazza
This is border level insane (in a good way). I loved the super Mario Bros 3 example, unconventional but just what I needed to keep reading and I genuinely learned something about Nix that I had failed to grasp in many of my previous attempts at learning it...
Sometimes I'm at a shell, interactively transforming a dataset like curl $URL | transform1 | transform2 | filter, and debugging takes a long time since curl or transform1 takes a long time. So I cache it, and I'm left with:
cat intermediate.csv | transform2 | filter
After a few steps I have many such intermediate files, and a confusing shell history.
I wish my shell did smarter caching here, so I could just do:
curl $URL | transform1 | transform3 | filter
and have my shell know that it can reuse outputs.
Has anyone tried Nix, or similar, to get this transparent caching? Similar to this supermario example.
I know this isn't trivial, since $URL might change, and transform1 might be non-deterministic ,or have side-effects. Nix operates with the same constraints.
After all, my issue is ultimately a build problem.
If I were to think about your idea: It would be a shell that transforms every command into a Nix derivation. If you used pipes, those are inputs to the derivation so they change.
The command itself and the argv (maybe the environment variables) are the inputs to calculate the hash? A neat idea I think worth exploring.