How a false smoke alarm almost drowned Jane Street's only NYC datacenter
84 points by scanner_brightly
84 points by scanner_brightly
Bash wouldn’t cut it for this. Nor would Perl. No, a job of this magnitude called for the most reliable and type-safe programming language imaginable. So I fired up emacs and wrote it in good old OCaml:
good joke :)
i imagine working in that language all the time must put into perspective how hard systems design/operation is and how little type-safety does to help that. hardware failures are a good reminder of this, fun story!
It was only a half joke.
Working in OCaml all of the time makes the status quo feel so much worse by comparison. Trading strategies written in OCaml could be understood by non-programmers but configuration languages for so much industry standard stuff, including open source stuff, are practically indecipherable even by sys admins.
Really makes you want to rewrite the entire world.
re ocaml - is there any open source/blogs/books that demonstrate this sort of readability?
Would also love to see some examples around this! Especially how it relates to system administration. Using OCaml for it has never even crossed my mind.
Anecdotally, whenever I'm reading some algorithms paper and they start presenting actual code, I have a much easier time reading it if it's ocaml than if it's haskell, Java, python, etc. I have never learned ocaml or written a single line of it. I've written Java and Python before.
This neatly explains a problem with contemporary amd64 "server" grade hardware that people really should consider: the OS should have awareness of environmental factors.
I don't use "server" grade, because IMPI is too insecure, I can get ECC on consumer motherboards, and companies like Supermicro don't seem to take security seriously until the media lights a fire under their asses.
I build my own servers, and each one can run at ridiculously high temperatures. Not only can they run at high temperatures, they're tested extensively at intentionally high temperatures long before they're colocated.
When a server starts thermal throttling, instead of just letting the throttling happen and ignoring it everywhere else, the systems will automatically clock down to lower speeds, then, if the temps stay above a certain threshold, all but one CPU core will cpuctl offline. If the temperatures still stay above a certain threshold, a more attention grabbing alert is sent and a machine will shut itself down.
Why this is almost never set up elsewhere is beyond me.
Well I'll tell you why I would veto this idea. It sounds like it would make reasoning about failures more difficult overall, not less.
A known, predictable failure mode that doesn't depend on people taking action within certain timeframes is, in my opinion, desirable.
I don't understand how this would make reasoning about failures more difficult. I don't even understand why we need to reason about failures. Could you explain?
I would not couple CPU performance to temperature and would actively work to decouple it on server grade stuff because I would not want to rely on tracking this variable.
To the greatest extent possible I would rather have a loud temperature warning than have the CPU performance change based on an environmental factor.
Oh! I understand now, thank you. I wouldn't agree myself, but I suppose we have different priorities. I'd rather something possibly fail at its workload because of not enough CPU than to possibly fail catastrophically.
Having the CPU change speed on you is a form of failing catastrophically depending on your industry. I also don't know what you mean by failing catastrophically? Not that I would rely on this as a first line, but CPUs reset and refuse to boot if they get too hot.
A room getting too hot, causing the release of sprinker water is failing catastrophically in my book.
My thinking is that a system that proactively takes action to avoid runaway temperatures that could lead to sprinklers and hours or days of downtime is not preferable to a system that slows itself, yet can be recovered remotely.
I understand that certain industries would be significantly affected by CPU throttling, but I don't see how throttling can be worse than a machine getting killed completely, unless throtlling prevents proper failover (although that can be programmed in).
Many of my servers are thousands of miles away, so my priorities are heavily influenced by that :)
Is it actually common to have water sprinklers in server rooms? That seems dumb. Isn't something like halon or argon used for this very reason?
Wait but isn’t this already what the hardware does? If you hold a blow dryer to a modern CPU, it will throttle frequency, if that doesn’t work it’ll move cores to deep sleep and ultimately it’ll turn off?
Or are there modern CPUs that don’t do this in hardware or is there some other benefit to doing it in software?
Yes, pretty much any modern CPU (at least say, Intel in the past 25 years) will throttle first, then shut off if reducing temperature doesn't mitigate the issue.
No, they don't do that. They'll throttle, but that's about it.
The system controller can be configured to shut down at a certain temperature, but often programming the system controller requires entering the BIOS or installing and running special software that has limited OS support, so it's quite common for the defaults to be used.
I'm surprised that it wasn't possible to remotely power them off (or maybe each server is maintained by a different person/team, so getting access to everything on short notice wasn't feasible).