Endian wars and anti-portability
16 points by runxiyu
16 points by runxiyu
I can't really agree, tbh. Improved cross-architecture portability is great, indeed. But, depending on the language, practices, etc., community-contributed ports might still be a lot more code to maintain; accepting a few patches to convert endianness when necessary is fine, but adding an entire architecture's backend to your compiler project and expecting you to maintain it, and particularly saying doing otherwise is a disservice to the community, meh.
And if the majority of programmers in the 90s had their way, everything would have been on Windows. Unix? Mac? Screw those systems! They don't have any market share anyway. Supporting non-Windows systems means more code to maintain, accepting patches to do thing in a OS-agnostic way, adding entire subsystems to support non-Windows. Madness!
I should have made this clearer. It's certainly nice to support more architectures and operating systems, I had fun porting QBE and Hare to ppc64le (with the ELFv2 ABI), pass all tests, and I do want to upstream it when I sort out the code quality issues. Yet if they did not accept it, I would definitely not claim that they are doing a disservice to the community. Perhaps this is just because I view most sentences constructed to imply that FOSS maintainers have an automatically implied responsibility to the community with skepticism.
The 1990s was a portability nightmare, especially on the server side. You often needed to support Sparc, MIPS, x86, PA-RISC, 68k, Alpha, Power, et al. There were even more operating systems, including dozens of UNIX variants that were only superficially interoperable. This had an insanely high overhead cost when writing software. You also had to at least support Solaris on Sparc to be relevant; many Silicon Valley devs programmed on pizza boxes in the 1990s.
The only reason anyone bothered with portability is because the market was so badly fragmented that it was difficult to find enough customers to justify the development effort unless you ported it to enough environments. Testing, debugging, and maintaining all of that portability was outrageously expensive.
I find any nostalgia for that world deeply weird. It objectively sucked for productivity and the portability requirements were a big reason why. That we've arrived at a place where 64-bit Linux on x86 or ARM is virtually the entire server market is an unmitigated success. I don't miss it at all.
And what system in the 90s had the most malware and exploits? Windows, precisely because it had 90% of the market. Monocultures are bad for agriculture and computers.
It's completely fair for an individual project to decide to only support Windows. I say this as someone who practically never touches Windows.
I think rather than "everyone settling on the most common architecture/port/etc. is best" runxiyu is intending to say "it's not as black-and-white as accepting a port = good and rejecting a port = bad". As with many (most?) other things there are tradeoffs involved and there's no single correct answer for all scenarios.
Indeed, I would love to see improved support for ppc64, ppc64le, riscv64, and even aarch64be. My disagreement is its claim about FOSS governance
Is this about compiler backends? My impression is that they're talking about compiling a program and running it on another architecture, not emitting code for that architecture.
Fair, though that's not entirely clear
I thought it was fairly clearly about software above the compiler - no one would seriously claim that the various compilers don’t support big endian or non-64bit CPUs (they go to great lengths to perform arithmetic etc without relying on host architecture, and the “correctness” w r t target endianness is a product of the source being compiled).
That said the cost of supporting multiple endian hardware is in many ways higher in than other compatibility issues because it is a feature of runtime rather than something where errors show up at compile time (MTE, pointer auth, and CHERI features also cause this kind of issue but they’re generally designed to work with existing software so by design don’t have these issues as often)
What type of code causes issues with endianess? The only aspect I can think of is networking (since the IP protocols are big-endian) and some file formats (IFF [1] is big endian, PNG [2] little endian), and in both cases, the endian issues should be handled right at the I/O boundary, not spread through out the code base.
[1] Used extensively on the Amiga platform.
[2] Based on the IFF format, but not IFF.
There's a bunch of types of code (mostly IO, yes) where you wanna read data straight into a buffer and then use it from that buffer. But this only works if the endianness of the source matches the endianness of the CPU. (Yes you must be careful not to violate aliasing and alignment rules but once you know how, that's not difficult.)
What I usually do is to specify that the source is little endian, put a static_assert in my code to leave useful diagnostic when compiling when compiling for big endian, and then write the rest of the IO code assuming little endian. If someone who's for some god forsaken reason running big endian POWER or something wants to use my code, they can fork my project and add the requisite byte swap code themselves. I have no intention of having such code in my repository because I have no intention to find a way to test it. I don't have big endian hardware and have no desire to keep around a big endian VM for testing purposes.
Another case where endianness can cause problems is packed data structures that use type punning to reduce their size. C unions make the problem worse, not better. It’s often possible to make the code reasonably portable by using a consistent word size and shifts/masks to access fields, but that can make it hard to save space by using unaligned words (such as a 12 byte struct with an 8 byte field).
For software like database engines, most in-memory data structures must be pageable to storage. These commonly use highly optimized bit-packed structures that don't translate well across endian-ness. If storage can be read by architectures with different endian-ness then much of your logic must be endian-aware and it isn't always a simple byte swap. It can be pretty fussy.
In practice, most people don't bother making this type of software endian-aware anymore unless it is legacy. Instead, they optimize for little-endian and then add a static assert or compiler directive that fails to build on big-endian architectures so that someone doesn't accidentally target a big-endian architecture.
I agree with a lot, but the reason too drop Alpha support is that the memory model is awful. It is impossible to implement the C++11 memory model and writing correct code for an Alpha SMP system is very, very hard. Unless it’s being regularly tested, the odds are that any multithreaded code is broken on Alpha.
Could you elaborate on this a bit? I know Alpha has a very relaxed memory model (to put it in modern terms) but is it completely lacking fences/locks or other mechanisms to guarantee ordering?
It's worth noting that some of the advocates, at least in the Linux world, for forcibly ending support for big endian, 32 bit and non-mainstream architectures are corporate shills who're paid to work on open source.
That is, they not only want to do only what matters to their corporate daddies, but they also want to stop others from working on things that don't matter to their corporate daddies.
I agree with the post. I'm still a bit sad that big-endian systems aren't popular, thanks to the overwhelming marketing of Microsoft in propping up the Wintel dupopoly. Of the systems I learned to program in assembly, the most fun were the big-endian systems (6809, 68000, MIPS), as opposed to the misery of the little endian systems I knew (started on the 8088, and no one I knew really liked it).
And it's funny---at my previous job, we had endian issues with our code, but it wasn't because it was little-endian specific, but big-endian specific. I was the one that went in and fixed the code so we could generate the data on Intel platforms.
I find big-endian easier to wrap my head around, though in fairness the processor I've probably written the most lines of assembly language code for is the 6502, which is pervasively little.
I specifically added endianness information to my (possibly best-in-class) arbitrary-length binary integer encoding https://github.com/pmarreck/BLIP in order to avoid this entire class of issue.
From its README: "BLIP is 4x faster than LEB128/Protobuf for large value encoding (1.3 vs 5.1 ns/op) because it writes raw LE bytes with a single memcpy instead of shifting and masking 7 bits at a time."
There's a whole section on endianness in there, with data, as well.