Parallelizing Compilation: In-Process vs Multi-Process
5 points by Riolku
5 points by Riolku
I'm not sure if there are already good names for this, but I can't find anything useful when I look online.
Obviously, when compiling programs, parallelizing the work is crucial. There seem to be two main approaches:
I'm looking into this because I'm looking at designing a programming language, and I have fast compilation as an explicit goal. Zig and Rust obviously have some of the same ideas but wildly different compilation speeds. C is known to be quite fast to compile, while C++ templates are slow(?).
In terms of parallelization, why is it that a parallel compiler seems to be the choice for newer languages? Is this a coincidence or explicitly required because of e.g. circular imports?
P sure it’s because modern languages don’t compile single files without being affected by what’s in other files. C has that property which makes spawning compilers feasible.
The concept you are looking for is compilation unit. Compilation unit is code that needs to be compiled together. Separate compilation is compilation of different compilation units. Multi-process compilation parallelization can parallelize between compilation units but can't parallelize within a compilation unit.
So the size of a compilation unit is the variable affecting decision of multi-process compilation parallelization vs in-process compilation parallelization. Yes, circular imports can increase the size of a compilation unit. Rust, Zig, and Haskell have large compilation unit. C and OCaml have small compilation unit.
At some level the two approaches are equivalent in that you could implement the same program with either, with enough machinery. But in practice with threads it is much easier to share data between tasks and with processes it is relatively harder.
Sharing data is nice when your language benefits from it (e.g. if piece A and B both need C, it's nice to be able to have them work in parallel off a shared C). Not sharing data is nice because it keeps pieces separable (e.g. it's relatively easier to shove separate processes onto separate cloud workers) and it makes it easier to integrate with other systems (e.g. two unrelated compilers can each spawn processes, with the process limiting controlled by a shared build system).
A historical note: threads only came to Linux in 1996, before that you only had separate processes. Similarly I guess (just skimming now) older unixes didn't have threads. So there's a bit of path dependence in why older tools relied on processes. (PS A fun bit of trivia is that the initial LinuxThreads was from Xavier Leroy, better known as the author of OCaml!)
You might want to take a closer look at what Rust and Zig do. Rust is still fairly close to C++ in its compilation model, cargo creates multiple instances of rustc. Zig, on the other hand, forgoes separate compilation altogether.
Maybe there are great resources on similar topics in compiler design that I've failed to find
Contrary what others are suggesting in this thread, I don't think this is necessarily about compilation units. Even if my compilation units are small and independent I wouldn't choose multi-processing.
Rather, in-process multithreading is just the obvious default to me for writing a parallel program. Multiprocessing makes coordination and data sharing between workers either awkward or slow.
why is it that a parallel compiler seems to be the choice for newer languages?
Because the overhead is lower. You get to share resources between threads instead of re-processing them.
Is this a coincidence or explicitly required because of e.g. circular imports?
It's not "required" in the sense that you can of course write the same compiler as a single-threaded process. However, if your compiler has compilation performance as an explicit goal, there aren't many convincing reasons to do so. The "simplicity" you gain from making your compiler single-threaded is the same abstraction you can do in-process by making your core compile function single-threaded.
Even if you spawn a compiler per-TU like C/C++ and get the OS to schedule them, you have some hairy coordination problems with i.e. jobservers. Not to mention being able to parallelize things within a single TU or across different TUs without having the linker step be a bottleneck.