Where are all the low-level JITs?

The early .NET marketing played up a lot the idea of doing install-time compilation. This is quite a nice idea because you can have binaries that are tailored to your specific CPU model (not just a single distribution format for Alpha and x86 Windows, but also one where a Pentium II and a K6 could have different tuning in the binary). Unfortunately, it turned out to be much too slow. It's not uncommon for a complex desktop app to need at least tens of minute of CPU time to compile, and hours are not unusual. Spending this time on every client is not great, and starting with an interpreter then moving hot paths to the JIT gives a more immediate experience.

This is not at all true. In fact, it's closer to the opposite of what you said.

First, the product you're referring to here is the .NET desktop framework, i.e. the version of .NET that ships in Windows. Since the beginning of .NET it was widely recognized that startup was a big problem for JITed languages -- the bytecode is not directly executable so you have to spend time to JIT it. There are two main ways to address this problem:

1. Start with an interpreter, then switch to JIT when a certain condition is met.
2. Precompilation

.NET has always relied heavily on (2). In .NET Framework this process is called NGEN and it has been in Windows since .NET 1.0. It runs on install of the .NET Framework and it precompiles the entire framework. As to why it runs on install -- it's mostly not about producing highly optimized code for different architectures. .NET interoperates with a large number of different native APIs and platform features in Windows and this interaction can have subtle code changes for the target platform. Combined with the huge number of Windows machines and huge number of Windows configurations, that means that the best strategy to compile code for every possible client configuration was to compile the code on the machine at install time.

This process can take a while -- hours or maybe a day -- but this is mostly because the NGEN process runs in the background at low priority, it has to compile the entire .NET Framework, and the whole thing isn't well optimized for finishing as quickly as possible. However, this wasn't about apps. You certainly could queue your app for NGEN but this is a relatively rare thing and would produce significant results only for very large apps, since the startup path of most applications is heavily dominated by code in the framework, not the application. For various reasons it also requires admin permissions, so there are also plenty of reasons not to do it.

Now, to switch over to modern cross-platform .NET (.NET 5+), we have a replacement technology that we call "ReadyToRun" or "crossgen". In contrast to NGEN, it's specifically designed to remove those platform dependencies and intermodule dependencies that required code to be compiled on the target machine. Nowadays we ship with the framework precompiled for all the targets, so the benefit is provided for free to every .NET application. It's still an essential part of the startup story -- .NET is much slower to start without precompilation.

Finally, you mention an interpretation. .NET has never had an interpreter. Interpretation and precompilation serve many of the same purposes and we went the precompilation route, so there wasn't a lot of reason to build it. This is in contrast to Java, which has a smaller precompilation story and a much larger interpreter story. The other reason for this, aside from path dependence, is that it's much harder to build a good interpreter for .NET IL vs Java bytecode. Reified generics, value types, and specialization mean that the .NET IL is much more complicated, so building a good interpreter requires a lot more work. Moreover, offering generic specialization as a feature causes .NET users to write code that depend on it for performance and interpreting that code can sometimes mean slowdowns so large that the entire performance class of the application changes.

.NET Framework was in fact very simple. It had no tiering (only one JIT optimization mode) and precompilation. There were no optimizations for "on-stack replacement" where a particular method could be recompiled on the fly into a more optimized form.

This is no longer true for modern .NET -- we have multiple JIT tiers and regularly recompile and replace code -- but precompilation is still a critical part of our startup performance story.

Re install-time compilation, isn't this exactly what Android currently does to Dalvik bytecode? IIUC when you install a program it interprets it or maybe does a quick-and-dirty compilation so you can use it immediately, while in the background it compiles it to target-specific native code. Do you know anything about why this seems to work for Android and not for 2000's-era Windows?

(After doing some wikipedia-crawling it appears that the Android Runtime is the component responsible for this now, replacing the Dalvik VM, and the pipeline is more complicated than I thought. It doesn't necessarily AoT the entire program to native code, just portions of it, and it will also try to JIT some of the code as the program is running. I suspect I could happily spend years digging into the details of how it works.)

my understanding is that it's more or less just a binary representation of Java.

This isn't true. The JVM has a stack-based instruction set, just like WASM. Unlike WASM, though, JVM bytecode doesn't require managing memory (as you have a GC) and it understands the concept of objects and inheritance, which simplifies that aspect of compilation.

Here's an example of Java Bytecode:

Compiled from "SimpleAlgorithm.java"
public final class SimpleAlgorithm {
  public SimpleAlgorithm();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String[]);
    Code:
       0: aload_0
       1: arraylength
       2: ifle          12
       5: aload_0
       6: iconst_0
       7: aaload
       8: astore_1
       9: goto          17
      12: iconst_1
      13: invokestatic  #2                  // Method java/lang/System.exit:(I)V
      16: return
      17: aload_1
      18: invokestatic  #3                  // Method java/lang/Integer.parseInt:(Ljava/lang/String;)I
      21: istore_2
      22: new           #4                  // class DivisorPrinter
      25: dup
      26: iload_2
      27: invokespecial #5                  // Method DivisorPrinter."<init>":(I)V
      30: astore_3
      31: aload_3
      32: invokevirtual #6                  // Method DivisorPrinter.print:()V
      35: return

I haven't explored WASM much, but my understanding was that the JIT compilation is pretty rudimentary, right? That is, the WASM runtime assumes that the bytecode was already optimized by a full AOT compiler and it "just" needs to translate it to native code with local optimizations, without inlining, vectorization, devirtualization and similar larger-scale transformations.

Perhaps GraalVM is what you are looking for

Where are all the low-level JITs?

Baked into the very circuitry of the microchip you're running your code on (register file allocation, speculative execution, branch prediction, etc.)

This is only somewhat a tongue-in-cheek response: many of the supposed advantages that JIT-ed code has over AOT code vanishes when you've got a sufficiently smart CPU, as most of us do nowadays.

Julia uses LLVM at runtime to generate specialized code as a sort of JIT. It does this for flexibility and dynamism rather than exceeding AOT performance though (it typically matches AOT performance instead). Sometimes this flexibility makes it quicker to implement more ambitious algorithms which can have better performance but that’s sort of a separate consideration.

My suspicion is that (although not in theory) JIT is heavily constrained by compile time in practice and with that constraint can't compete with AOT. HotSpot already gets "it's slow to warm up" complaints, and hypothetical AOT-competitive or AOT-beating JIT would be even more slow to warm up.