20 Years on AWS and Never Not My Job
53 points by fuzzy
53 points by fuzzy
Personal responsibility seems to be an increasingly difficult trait to find in people.
There's some real graybeard/open source contribution for the greater good here. It's good to see a megacorp giving back a little bit. I can't help thinking it's probably not as much as he deserves.
Connecting this to the "organizations like legibility" meme/train of thought, a lot of how large institutions work seems to be about avoiding personal responsibility.
I work in a small company with a very strong engineering culture of "it's okay to admit you fucked up", and it's been an incredible experience. It's also a company where fucking up can cost 4-8 figures of broken hardware in a second or two. It's wayyyyy easier to get people to take personal responsibility, and relatedly admit "I screwed this up" without wasting energy on evasion or bickering or finger-pointing, when the response is a serious nod and "okay, how do we change things how we do things so this doesn't happen next time?"
I know we’ve worked in similar industries. One of the moments early on in my current role that really made me feel like I’d instilled the right ethos in my junior reports was when one of them came to find me outside while I was having a smoke break. “Tony, I fucked up. I mixed up some cables and put 48V into [expensive equipment that runs on 12V]. It let out all the smoke.” Ballparking from memory, somewhere around $15k.
“…ok, thanks for telling me. Your task for the afternoon is to decide on which connectors we’re going to use for 5V, 12V, 24V, and 48V. I should’ve pushed to standardize earlier.”
I also mentioned — in fact in one of Jeff Barr's AWS user meetups in Second Life — that I wanted a way for an EC2 instance to be launched with a read-only root disk and a guaranteed state wipe of all memory on reboot, in order to allow an instance to be "reset" into a known-good state; my intended use case for this was building FreeBSD packages, which inherently involves running untrusted (or at least not-very-trusted) code. The initial response from Amazonians was a bit confused (why not just mount the filesystem read-only) but when I explained that my concern was about defending against attackers who had local kernel exploits, they understood the use case. I was very excited when EC2 Instance Attestation launched 18 years later.
18 years later!
I wasn't sure whether to place the security tag on this post. In retrospect, it should have been there.