What Sun got wrong
88 points by j11g
88 points by j11g
Sun did a lot of other things wrong. I only saw it from the outside but when your corporate dysfunction is obvious from thousands of miles away it’s clear you need to do something different.
With SPARC, they had a fairly solid architecture that, with a few tweaks (asynchronous spilling of register windows!) could have been an amazing architecture. And they structured the company to guarantee that this would never happen.
They had a hardware team and a software team that could have done co-design but they rarely involved the software team early. I can’t remember when SPARC got a broadcast invalidate for instruction caches but it was long after Solaris identified that this was a bottleneck because each mmap with an executable mapping needed an IPI and rendezvous with all cores. This really hurt process creation time on multicore systems and UNIX really encourages short-lived processes. When you added JIT compilers (hmm, Java, I think Sun had some interest in that…) then it was much worse.
Near the end they had the Niagara and Rock teams working on SPARC. One was building simple in-order cores that would scale to massive multicore and multithreaded systems. The other was focused on single-threaded performance. These should have been targeting different market segments and aiming to eventually produce heterogeneous SoCs with a small number of Rock-project cores and a large number of Niagara-project cores. Instead, they were set up as competing teams. And they were joint custodians of the SPARC ISA. Nothing went into SPARC unless both teams agreed and, because they hated each other, nothing went in because if one team liked it then the other would hate it on principle.
The biggest example of this was Project Maxwell. This added a backwards-compatible object-indexed memory model on SPARC, ran young-generation GC in hardware in the cache and provided a mechanism for promoting objects to the old generation on cache eviction. With this, the Java+Solaris+SPARC stack would have performed amazingly well. Internal infighting killed it and Java ran better on x86 Linux than SPARC Solaris.
When your own software runs better on your competitor’s hardware (and better on their OS) than your own, you’ve done a lot of things wrong.
When your own software runs better on your competitor’s hardware (and better on their OS) than your own, you’ve done a lot of things wrong.
I realize this is quite off topic, but this sentence reminds me a lot of how many Windows games now run better and more performant on Linux using the Wine compatibility layer. I suppose it shows the difference in priorities Microsoft has compared to OSS devs.
With SPARC, they had a fairly solid architecture that, with a few tweaks (asynchronous spilling of register windows!) could have been an amazing architecture.
I think it's fair to say that if "a few tweaks" could have made it an amazing architecture then it would at least slowly have improved over the years in that direction. I'm also entirely unconvinced that asynchronous spilling would be the great unlock here. I found this to be an excellent read on the register window issue: https://danielmangum.com/posts/retrospective-sparc-register-windows/
One could argue that these are all elements of "Running a business"; which would fit in Bryan's extremely concise summation. The specifics are useful to learn about though. Your examples read like internal culture problems manifesting.
I feel this (and your other comments here) is much more on-topic than the original submission. (I expected more technical and specific thoughts about what Sun got wrong, maybe about Solaris, or Java, rather than how Sun was bad at sales.)
I'm into computing history, and what you shared about SPARC reminds me of other promising technologies that "could have been". I feel most of them are ultimately intertwined with this kind of organisational dysfunction.
To me, this is more of an example of how tech excellence is just a small factor in success.
Just Java or ZFS are enough to get into computing history, and they did so much more.
(And I feel Oracle was responsible for a big chunk of accessibility of GNOME. My theory is that they were one of the last companies trying to sell UNIX-like workstations to governments and they had to provide accessibility.)
I'm quite nostalgic for them.
I was in university 1997-2002. They had a lab with Sun workstations and we all wanted to have the subjects that required going there. Some departments had Linux labs, but the main labs did not offer Linux as a boot option for a while. So the Solaris workstations were the first chance for many of us to use "serious" UNIX workstations.
At my first job, we had Sun servers for production. I have fond memories of learning the hard way that not all UNIX are the same, by running killall in a production server. I think that was very helpful and nowadays I still try to stick to POSIX in scripts (and switch to Python quickly).
I also think Sun used to throw one of these mousemats in every order. I got one of these that I kept after I left that job. I loved that mousemat, so nice quality and... well, I'm also nostalgic of the Sun logo. (Many of my university notes had scribbles of the logo all over the corners.) Unfortunately, I left the mousemat too much time in the Sun (ironic), and it is nearly destroyed. I wish I could get one again, or even the graphic assets for the mousemat to use somewhere else.
edit: NOOO. Apparently, Cloudflare bought the slogan "the network is the computer".
Just Java or ZFS are enough to get into computing history, and they did so much more.
Most importantly: NFS. Which they then open sourced and which became the de-facto standard file-sharing protocol for UNIX-like systems.
The OpenStep specification was a collaboration between NeXT and Sun (this is why the class names start with NS for NeXT-Sun: NeXT used NX for their pre-collaboration class names). Cocoa is an implementation of OpenStep and most Mac or iOS apps depend on it. As part of this, Sun shipped the XDPS (Display Postscript within an X11 window) extension on their X server. Unfortunately, this included code licensed from Adobe and was never open sourced. The open-source UNIX desktop would have been a much happier place if they'd had resolution-independent print-quality text and vector graphics a couple of years before Windows 95 shipped.
Other notable things that Sun did:
They defined what unix shared libraries should look like from the programmer and build system point of view; ELF followed Sun’s a.out design and made it more elaborate.
Pluggable authentication modules
Name service switch
Both of which have a regrettable reliance on loading random shared libraries into random programs. Solaris had a name service cache daemon which was single threaded and therefore acted as a bottleneck for the whole system especially due to slow DNS lookups. I wish they had built NSS on an asynchronous RPC layer instead…
All of these are pretty bad, in various ways. The modern UNIX shared object model tries very hard to provide an easy migration path from static linking. Any kind of symbol can be exported. This ends up with horrific things like copy relocations.
Windows has a more MULTICS-inspired model where shared libraries are both richer (you can define shared memory as a special kind of symbol in a Windows DLL and all processes in the same security context that load the same DLL will see the same copy of the global, whereas UNIX requires explicit mmap, shmat, or similar) and more constrained. Windows requires explicit annotation of exported symbols and does not allow one DLL's globals to be initialised to point to others. This gives a cleaner set of guarantees at library boundaries.
Completely agreed on both PAM and NSS. They weren't the only things Solaris did like this. On Solaris, setlocale is another thing wrapper around dlopen. Each locale is a separate .so file and it gets opened and replaces the various locale-aware functions in the symbol table. Fortunately, the code for doing this was based on some IBM code that Sun didn't have a license to open source and so OpenSolaris derivatives don't do this.
It's quite surprising that the company that built an RPC layer as a building block for a networked filesystem didn't use that layer for DNS and so on. NSS would make much more sense to be an RPC API, PAM probably would as well, since it would be much easier for things that want to run with more (or different) privileges than the program in question to work. I think systemd fixes (though, amusingly, not with the same RPC layer that they use for everything else, because that depends on working name resolution). A load of PAM things end up building their own RPC code to talk to a daemon, whereas we could have had a single generic one.
Windows requires explicit annotation of exported symbols and does not allow one DLL's globals to be initialised to point to others. This gives a cleaner set of guarantees at library boundaries.
AIUI, AIX and NeXT shared libraries were more like that. (Apple’s tooling has gradually converged towards a more Sun-style DX.)
I have a copy of John Levine’s book on “Linkers and Loaders” which is recent enough to still be occasionally useful 25 years after it was written, but too recent to get into the weeds of unix-wars diversity. One of the things that annoyed me about libtool ever since it first turned up (around the same time) was that it was solving its problem a decade too late, just as the problem was disappearing; and not only was it already obsolete, its implementation is shockingly wasteful and inefficient, even worse than autoconf.
Whoops accidental 1990s build tooling rant.
You can specify exports manually with AIX but in practice everyone uses some form of automatic symbol exporting (be it from the linker or some tooling like libtool). libtool also provides some extra AIX specific knobs that just end up exploding the different build type permutations and obfuscate stuff like rtl vs. nortl by conflating it with other things.
Oh, I still use NFS in my personal infra! I guess it's a sign of its brilliance that nowadays I'm frustrated there aren't really so many better alternatives.
(Also, as an RPC person, it's an interesting success story.)
Huh, today I learned about the ns_ prefix, I had never thought about that. I only toyed a bit with WindowMaker because NeXT boxes were yet another unattainable workstation for me in my youth. I recently played a bit with this WASM easy emulator of NeXT OSes. (They also have other Apple-related operating systems to play with.)
(Offtopic, but concerning vector graphics, I'm still a bit miffed that much of the way to get vector graphics in browsers is to use fonts with non-text symbols. For a good while I had a dream that instead of doing bitmap screenshots in documents, I'd be able to use "vector screenshots". I played a bit with printing web pages as PDF for some stuff, but that was not viable.)
Note that WindowMaker is not an OpenStep implementation. It has a thing called WINGS (WINGS Is Not GNUstep) to provide the look of OpenStep and none of the APIs. It's a shame that it's become the thing most associated with NeXT on non-NeXT / Apple systems because it copies the superficial bits without copying most of the bits that made the underlying system good.
Yeah, for completeness I did some browsing and came across:
https://www.gnustep.org/information/aboutGNUstep.html
GNUstep is not a window manager. And the libraries are not used by WindowMaker at present (it uses WINGs), although WindowMaker shares a similar.
Remember: as WINGs means WINGs is not GNUstep, it is also true that GNUstep is not WindowMaker. If you like the classic look, the vertical menus, the square tiles, you can use GNUstep with WindowMaker as window manager: they integrate quite well.
The link to Etoile in the alternatives they list is timing out for me, the other two look quite bare. I'd love to play with NeXT-inspired/-lineaged stuff, but unfortunately it seems to be a bit in the retrocomputing neighborhood, that is, not something you'd try to get into daily driving for most purposes.
The link to Etoile in the alternatives they list is timing out for me
Yup, I got sidetracked from Étoilé designing hardware because the things I wanted to do with Étoilé are very hard (if not impossible) with existing hardware. I hope to get back to it any decade now.
I feel like Sun lost the plot with Java. Instead of making it a great language that ran on Solaris it felt like they were trying to reinvent the operating system inside the JVM… poorly. Imagine a world where instead of all that time wasted on JavaEE and app containers, it was spent building isolation at the OS level, and making Java the best language to run on top of it. Instead we got one technology and “reference implementation” after another, with horrible OS integration and things like OSGi which just didn’t work (or at least we could never get it to work).
The thing that upsets me is not just the waste of energy but also the missed opportunity to really push the technology world forward.
Wrapping the nasty mutually-incompatible worlds of all the different OSes of the ‘90s was the main sales pitch though. Why make a language only for a single kind of computer you sell when you could run everywhere, including on clients embedded in a web page? JS basically wasn’t a thing yet, and over a quarter century later still doesn’t have some of Java’s features like “integers.”
It's really important to remember the context for a lot of these things.
When Java was released, web browsers were very simple. JavaScript wasn't a thing yet and, when JavaScript was first released, most of the things needed for dynamic UIs weren't there yet. Browsers did support plugins. Flash existed but I never heard about FutureWave Flash, the first I heard of it was Macromedia Flash, which shipped after Java.
Java Applets let you run things that looked like real applications in a web browser. And that was really a completely new thing. And it can't be overstated how much this panicked Microsoft. Internet Explorer had been planning to use ActiveX as its plugin mechanism (ActiveX was an evolution COM and the Visual Basic extension model) but that rapidly pivoted to being a way of embedding arbitrary Win32 code into a web page so that people would write Windows-only web apps, rather than migrating to web apps.
Writing cross-platform GUIs was even harder than it is today because most GUIs actually had a well-defined set of usability rules that most software complied with and so anything portable looked out of place. Sun tried very hard to address this with the Abstract Windowing Toolkit, which let you use native controls, let you build abstractions over them so that common behaviour was factored out, and also let you plug in per-platform behaviour rules that made things behave more like native apps (it didn't work very well, but they at least tried).
And GUIs were a lot more heterogeneous. Not just the proliferation of X11-based things, but also OPENSTEP, which used Display PostScript, and consumer platforms like Amiga, Atari, and Acorn were still non-trivial market share: they'd mostly lost out in offices, but home users had a variety of different platforms.
Yes, Linux was a thing but it was new and not widespread. The big proprietary UNIX systems from IBM, HP, DEC, SGI, and so on were still around and more important to be able to target. And VMS was still actively sold, as were IBM's big iron platforms (in some circles, UNIX was regarded as the toy OS, real computers ran a mainframe-derived OS).
The promise of something that let you write software and run it on dozens of different operating systems, a variety of architectures, and have the ability to tailor the UI for the few where you really cared while still supporting the others, was a powerful message. The ability to embed these programs in web pages so that you could deploy them in your org, across all of the kinds of computer that your teams used, by just uploading them to the Intranet, was an amazing promise.
They didn't really deliver, but that's a different story.
WORA was interesting marketing but I don’t know how important it was in the scheme of things. I mean yes it was the pitch, but it just never struck me as all that interesting, I mean I had been doing cross platform work in C for years when Java hit the scene.
I think Sun should have pushed most of the JavaEE ideas up into the OS and made Solaris the best place to run enterprise apps, they had everything they needed to invent k8s years before anyone else. Instead they tried to do everything in Java and it was just bad, for the most part.
I mean I have no insider insight or anything. This is just how it looked to me. A company with great technology and huge potential just going off on a tangent.
I had also been doing cross-platform C and C++ before Java, and it was a freaking nightmare. We spent 90% of our time on figuring out the details of "cross platform support" and 10% of the time on what we were actually trying to build. Even supporting Windows 16 and Windows 32 from the same code base was an absolute nightmare. (Not impossible, but painful enough to leave deep scars.)
Within 30 minutes of downloading the JDK beta, I told my co-workers "I'm never writing C++ again."
I was wrong, of course. But since then, I was able to do 95% of everything we needed (across several different companies) without dropping down to C or C++. The productivity increase leaving C++ was probably on an order of magnitude. And most of my work was systems level; I haven't done any real app dev since before Java appeared.
I was in the same situation re C/C++ but we only had to support multiple Unix platforms so it was I guess much easier for us. And yeah I had the same reaction to Java when I downloaded it. I haven’t used C for 20 years.
But my point is not that Java was bad but that Sun sunk a huge amount of energy into making it so complex, yet it never really worked properly.
I think they made a massive strategic error by spending so much of their innovation budget on JavaEE instead of making Java + Solaris the “enterprise” system.
This would have required fixing Java’s startup times, and footprint much earlier, and I mean look at Valhalla. Imagine how much easier it would have been to add value types to Java 1.0. But now it’s never going to happen.
That narrative is already one step off course, as originally chartered. The write once run anywhere promise of java was originally referring to processor architecture. The operating system was going to shrink to architecture-specific shims that would look like boot loaders.
I don't mean here that you misinterpreted anything. Their marketing and dev tools very much promised write once, run on any operating system. I'm just pointing out that Sun itself lost the thread with Java before the first release!
Hmm, certainly OSGi predates Solaris Zones/Containers (2000 vs. 2004),but how would you implement cross-OS containerization? To do so on top of the JVM doesn't seem like such a bad idea.
(I checked and indeed Sun participated in OSGi, but I always thought it was more of an IBM thing, esp. because Eclipse apparently made it work?)
I actually think that in most cases, containers of any kind are a waste. Java-based webapps were so much nicer when everyone realized you could just run embedded Jetty in your process and forget about wars and stuff. You can add Linux containers to wrap it, but in most cases, what for?
Agree 100% that Jetty and JavaSE was reasonably sane. By the time I quit using Java I’d found a way to enjoy using it, but I would struggle to go back.
I might have misremembered OSGi but there was also JavaEE which promised so much and didn’t deliver on anything, and ultimately I came to see it as wrong headed long before it was cancelled.
Java containers were a massive waste of energy IMO but In terms of OS containers, I mean k8s solves so many of my problems that I can’t imagine life without it for the work I do. But I can certainly imagine a system that’s better than k8s and I think Sun could have built it. They had the skill, they had the ideas but they seemed fixated on doing everything in Java, huge apps were supposed to run entirely inside one VM, in one not particularly efficient language, and useful primitives like processes were cast aside.
OSGi and Jetty weren't from Sun. FWIW, I also was unable to make heads or tails of OSGi.
The JVM also significantly predates VM and container work. Things would likely have looked much different if Gosling had started building Java after Amazon EC2 took off.
For a while, though, a lot of the large investments in the Java "standards" were wasted effort. Most of the complexity in J2EE was just a complete waste, unfortunately. There were plenty of good ideas in there, but complexity is a killer.
I started using Java in '96, and was able to almost instantly build cross platform UIs (sure, they looked terrible on every platform, but they worked!) and even UIs that would deploy in the browser via that awful Applet thingy. It was the breakthrough that we needed to wrest control of the industry away from the then-evil monopoly (Microsoft and its Windows dominance). Had it not incorporated so many OS aspects into the runtime libs, the language would have never taken off. So the thing you hate about it is the reason why millions of developers jumped on it as fast as they could. (That's not a right vs wrong thing, just an explanation of what you saw.)
I still do a fair bit of work in Java, and while it has its (design) flaws, there's really nothing in that space that can hold a candle to it.
The JVM also significantly predates VM and container work.
It’s a pity that IBM’s VM technology took so long to be redone for use on mass market hardware.
I think it’s another of those things (like decoupling service names from host names in the DNS, as I have written about here before) which in retrospect would have made many higher-level things neater and easier if we had thought of the right infrastructure sooner, but which didn’t seem like a pressing need until after mass www hosting took off. (I suppose that before then, boxes were small enough that running multiple kernels would have been far too wasteful.) And after VMs became clearly necessary, even lack of hardware support didn’t stop VMware and Xen from coming up with creative workarounds.
VMWare was just getting going around y2k, and the first real working server support that I saw was around '03. What made it take off was the SMT and multi-core support from Intel. Up until that point, it didn't make much sense to virtualize workloads of any substance on a single hardware threaded machine with a pittance of RAM. The e220/e250/e420/e450 servers from Sun (coming out around the time VMWare was getting started) were the first "affordable" multi-CPU servers, and the $80,000 e450 from Sun couldn't accept more than 4GB RAM, so virtualization could have been useful, but obviously limited.
Intel's Pentium Pro had brought decent 2-CPU configs down to earth in the late 90s, but no real server OSs could run on it. Windows NT was "kind of" a server at the time, but mostly used for printer servers, department email servers, and file sharing. Linux didn't work as a server for more than the simplest workloads until the early aughts. Solaris for x86 was a nice idea but Sun kept flip-flopping on whether they were going to actually support it or not; it was a bit of a "red-headed step child", and hardware support was pretty poor. I used all of these OSs and more for server work, and (early on) the most stable was probably Solaris and the most usable was probably Linux.
But as Linux matured (and quite nicely, and quite quickly), getting VMs into use was a no-brainer, and VMWare was well positioned. Then Xen matured pretty rapidly behind that. Then EC2 appeared and the rest is history.
The 2-CPU infrastructure that Intel delivered in the early aughts was amazingly cost-effective, and combined with the growing multi-core was well timed to support virtualization. Intel servers also were able to quickly grow their max memory support up into the TBs. Hardware support for virtualization followed pretty quickly as well.
Intel's Pentium Pro had brought decent 2-CPU configs down to earth in the late 90s, but no real server OSs could run on it.
Aww, I was running FreeBSD on that kind of machine very nicely!
I feel you could implement a system that has many of the innovations of k8s without containers at all! That is, a distributed declarative operating system. Instead of containers you could run binaries and provide runtimes (with stuff such as the standalone Python builds, you can easily run many different Python applications using different versions of Python) and that would might actually save you a ton of complexity (building and pushing images is an unnecessary pain in many cases).
IIRC, JavaEE implementations did implement clustering, and they enabled dropping a war and having it run in a cluster just like k8s does. I'm pretty sure people who designed Borg and k8s learned quite a bit from Java environments and built the new systems with lessons learned.
Personally, I think the JVM approach of building an OS on top of an OS "had to be done". It was hugely useful, even if it was a bit too early for the hardware available, and so much was learned from it. Heck, I don't see so many better alternatives to the JVM today, 30 years later. I'm very likely to go to that if I want to write something in a static language that can afford trading some resource efficiency for developer productivity.
Yeah look we can disagree and be friends :) I use Go today and whenever I have to work in legacy Java I remember why I hate it.
I take your point about containers and I agree, Nomad could just run binaries, and the containers I deploy to k8s are basically just wrapped Go binaries. Orchestration and isolation is what I need and it’s easy to imagine this without containers. Containers are just how we (kinda) do it in Linux.
Not sure why the OS had to be done in a VM. Once our throw away the basic UNIX execution primitive of a process there seems to be a ceiling to what you can do, and certainly that was my experience with JavaEE, WARs, EARs and all that stuff. It was always about resource leaks, thread leaks, eventually you couldn’t redeploy any more and it would all have been so much better if it was just Unix.
I don't think we disagree so much, at least in this thread :)
Maybe you need isolation in many cases. I recently had to design and deploy something that included an online "public" IDE, so process isolation would not really cut it. Linux containers maybe were not enough either- many recommend KVM VMs.
About the later point, you might be right. I'm pretty sure JavaEE stuff was just too complex to get right at a reasonable cost, and we've learned better approaches later. What I'm saying is just... JavaEE was necessary to learn that the JavaEE approach was wrong. Otherwise, in a world were containers were not popular, it was the natural approach, and maybe helped bring us to containers faster.
:)
My understanding is that Solaris already had most of what it needed to do the kind of isolation we now associate with containers.
I mean I’m not here to argue over counterfactuals, it’s just my opinion, but I think if Sun had spent the time making Java and Solaris the “enterprise” Java (instead of JavaEE) then I think the industry would be a bit different today. Linux would have copied it and we would have nicer things.
(Maybe I am just jaded, I am pretty bored with POSIX generally, it feels to me like userspace hasn’t changed in decades)
