Forwardly-evaluated build systems
22 points by jkarni
22 points by jkarni
How much would a fast evaluator help? I asked around during the Lix fork and it seemed like there wasn't any appetite for fast evaluation. I broke ground on a JIT'able evaluator around that time but put it away in favor of more personally-interesting work. It seems like if there were a fast-enough evaluator than a frontend in TypeScript, Dhall, etc. would be more attractive.
I'm not sure why you heard there wasn't an appetite for fast evaluation. As we move Lix to be more RPC-shaped, we're going to replace the evaluator with a bytecode interpreter, which should significantly enable more work towards performance.
I wasn't asking loudly at the time due to the optics of any critique of Lix during the fork; my opinion is that neither C++ nor Rust are a good choice for writing fast interpreters. Now that I'm running Lix daily throughout my homelab, I feel a little bit better about voicing my position directly. Also, now that Lix has had a moment to differentiate itself, there's great documentation on the wiki; I read a few pages (1, 2, 3, 4) before writing the next paragraph.
I'm considering replacing the evaluator entirely, including all of libexpr and the connected commands, with something that is amenable to JIT from the start. I just demonstrated (notes, commit) that RPython can be statically linked, even with JIT generation enabled, which I recall is a structural requirement for any evaluator that wants community adoption. Last time, I stopped because I used an LR parser generator foolishly thinking that Nix is LR(k); this time, I'd pick a PEG generator. The hardest part is probably replicating the porcelain, although maybe there will be low-level socket fun in talking to the daemon.
If you like, I can idle in Matrix and answer questions about going fast. But I suspect that it will be more productive if I hack out a prototype and demonstrate that it can form a hybrid with Lix first.
The first benchmark (~460ms vs ~880ms) is, I think, largely because of TypeScript's faster evaluation. (Though there are a lot of confounding factors: in Nix parsing TOML is a builtin, whereas in garn it's not; garn does a few things in inefficient ways when dealing with derivation trees; in TypeScript many operations can be async so user code can e.g. kick off expensive tasks like path() (which needs to filter, hash and copy a directory to the nix store) without blocking.)
Whether this matters: I've already heard 3 or 4 companies say eval times are the most negative part of using Nix at scale. It isn't only at that scale that it matters; even for us, rebuilding our internal devshell when something changes is annoying enough.