Nobody knows how the whole system works

I think the key difference is determinism. I probably understand more about how compilers work than most programmers (I spent a lot of my career working on them), but for 90% of the code I write that doesn’t matter. The thing that the compiler provides is an abstract machine. You can write code in a source language targeting the abstract machine and understand the semantics of it in terms of that abstract machine. Most languages abstract machines don’t have any notion of timing (which is why writing crypto code in them is so hard) and the compiler is free therefore to make changes that affect timing but not things that are visible in the abstract machine. When you care about performance, you may need to look below.

In a large part, these abstractions exist for portability. I can compile my code with a selection of compilers and target a large variety of different hardware and be confident that the semantics, with respect to the abstract machine, are preserved. And, contrary to popular belief, compilers are not actively hostile so there is a naïve lowering that gives me a worst-case performance and the compiler will typically do better than that.

This is true for the networking things as well. The way the analogue layer works in modern wireless protocols is entirely magic but the abstraction remains the same as wired networks: you send a packet and it will probably arrive, you send two packets and they will probably arrive in the right order, but there’s a chance that they will arrive in the other order or one or both might not arrive. TCP lets you pretend that you have a circuit-switched network and, aside from performance, that abstraction is preserved. The same applies for higher levels.

All of this means that the abstraction layers let you reason about correctness by building things in terms of the abstract machine that they provide.

Probabilistic tools like LLMs lack this property. They do not present a coherent abstract machine. They do not give you a foundation for reasoning about the top of a system by targeting a simplified model of the real system with well-defined semantics. They just give you a way of creating an approximation of what you wanted with no bounds on the failure modes.

Reminds me of:

"At last, there breezed into my office the most senior manager of all, a general manager of our parent company, Andrew St. Johnston. I was surprised that he had even heard of me. "You know what went wrong?" he shouted -- he always shouted -- "You let your programmers do things which you yourself do not understand." I stared in astonishment. He was obviously out of touch with present day realities. How could one person ever understand the whole of a modem software product like the Elliott 503 Mark II software system? I realized later that he was absolutely right; he had diagnosed the true cause of the problem and he had planted the seed of its later solution." -- The emperor's old clothes, Tony Hoare (1980)

and:

"The belief that complex systems require armies of designers and programmers is wrong. A system that is not understood in its entirety, or at least to a significant degree of detail by a single individual, should probably not be built." -- A Plea for Lean Software by Niklaus Wirth (1995)

Maybe no one person knows it all, but you can know your part and rest assured that somebody, somewhere knows the other parts. So you know how to dial the phone and know there's people at the phone company who know the rest of how it works, so if it breaks, you can ... well, not call if the phone is broken, but reach out to them via alternative means of communication and they can fix it.

Is that also true of the new stuff?