The NX bit is not just about security
21 points by fanf
21 points by fanf
We ran into this in the seL4 boot loaders as well(unverified code). More specifically, someone from STM32 contributing platform support for the STM32MP2 did, on a Cortex-A35.
Slightly different, it was ret vs br x30, but same idea - ret got predicted whilst br x30 did not. This time it was memory protected by STM's RIF (resource isolation framework).
We solved it a slightly different way, by only mapping the necessary memory, but then we're a bootloader not a hypervisor.
Funnily enough, the ZynqMP (ZCU102) board had been having periodic failures booting in 32-bit mode every so often (not consistent even with the same binary), and this solved it
CPUs don’t signal page faults for speculative accesses
This is not strictly true - the CPU core won't (which is what produces page faults, so technically correct), but on ARM, such as in the case with the RIF, speculative accesses that make it out to the system bus and to various peripherals can produce SErrors.
Oh wow. The first issue found on the way is a lot more "nightmare fuel" than the main event:
modifications to the data do not automatically propagate to the instruction fetches
I guess I never really even realized that that was the case because… well, modifying instructions is generally not something I would do just for fun, but that's quite the footgun for those rare cases when that happens.
x86 is somewhat weird here not requiring it: they have special code in the frontend that handles even self-modifying-code and will flush the pipeline (that doesn't work when modified through a different address - that requires a pipeline flush); and then i-cache and d-cache are coherent as you mentioned.
I'm more of an ARM person, I've barely touched x86, but it seems weird to me that they'd be coherent: after all, that's extra work the processor needs to do to maintain coherency for a relatively rare operation. (This is true also for loading programs into memory - not just modifying existing operations; which is arguably the same thing).
Self-modifying code used to be relatively common back in the DOS days, when x86 chips didn't have cache (that was introduced in the 486). The coherent behavior in later chips was presumably to keep backwards compatibility with existing self-modifying code.