Flipping the bozo bit on flips the learning off
20 points by eatonphil
20 points by eatonphil
The value of the bozo bit is that some people are reliable sources of anti-information -- they habitually and repeatedly come to incorrect conclusions, misunderstand even the most directly-worded documentation, and create designs that introduce new problems without solving their stated objectives. You flip a bozo bit on someone when you've decided that any time spent trying to glean knowledge from their communications is time wasted.
The article is conflating the bozo bit with something related but different, which is a firmly-held belief that a particular problem can't happen and then working backwards from there to come up with reasoning. The term I've heard for this is "post hoc justification", and there's been a lot of ink spilled over the centuries (millennia?) about why it's to be avoided.
Going back to the article's example, the company that gave an LLM admin credentials to their production database, the response of "that couldn't happen to me because X" is justified if X is along the lines of "giving an LLM admin credentials to my production database seems so bizarre and insanely dangerous that I just wouldn't do it".
It's like people who eat dangerous things and then something catastrophic happens. You'll read a news story about someone ate a slug and got a brain parasite and died, and as you're reading it you might be confident that you are not going to die from that particular cause. The article's author imagines this is because you believe yourself better/smarter than the guy who died, but the actual thought process is more along the lines of "I am 100% confident that I will not voluntarily eat a slug even if I was starving on a desert island full of juicy slugs, therefore slug-borne parasites are not part of my risk model".
(As an aside, “they should have known…” is an incoherent sentence. It’s one thing if somebody deliberately took on excessive risk. But it’s another thing if they unknowingly took on excessive risk. How can you blame a person for not knowing something?)
People are blamed for not knowing things all the time. It's part of being an adult. I wouldn't blame a 3 year old for eating a slug, but if my 30 year old coworker ate a slug I would absolutely blame them. There are some actions that are so clearly and obviously unwise that you're allowed to blame people for performing them.
How can you blame a person for not knowing something?)
People are blamed for not knowing things all the time. […] There are some actions that are so clearly and obviously unwise that you're allowed to blame people for performing them.
There's literally a term for this, ‘wilful ignorance’. I think the article was advancing a good point I hadn't considered before, but has a couple obvious (to me) flaws. That's the first one.
The second one is that I found myself questioning the story about the Americans, and ironically — or perhaps quite aptly — thinking, “yeah, but was their distancing through differencing the same as what I'm doing when I dismiss the AI production deletion incident?”.
In other words, if they dismissed the chance of the danger reaching them without having any clue how it'd happened at the other factory, that's unjustified,
but if they heard that the manager had brought in some new, experimental, non-deterministic humanoid robots onto the factory floor and given them free reign to help the senior factory workers do their job faster, then their dismissal can't be cast in the same light.
I do accept the conclusion, though, because recklessness isn't a binary. The next time I hear an outlandish story, I guess I'll still pause and think to myself, “what am I doing that might be worth taking more care to put safeguards around, even if it's not quite as careless as what they did?”.
I wouldn't blame a 3 year old for eating a slug, but if my 30 year old coworker ate a slug I would absolutely blame them.
I wouldn’t ask why a 3 year old ate a slug. I would investigate thoroughly the circumstances that led to my 30 year old coworker eating a slug. If I felt myself tempted to “blame” them, I might first ask why we hired a slug eater and if we still are?
Some reasonable points.
I want to talk about why this reaction is counter-productive. I also want to call out the technical term for this phenomenon, which is a cousin of flipping the bozo bit. It’s called distancing through differencing.
It's both a productive and a counter-productive reaction. It's productive, because not having to evaluate every input is a huge optimization in a world that is already firing too many events at us to handle. And of course it's counter-productive for the reasons cited in the blog.
Knowing which things to take seriously and pay attention to is the real trick. I'll get back to you after I master this trick.
What learning should I be taking away from someone who gave AI direct access to prod?
A somewhat generous interpretation of the LLM story is: prod access should be access-controlled, and catastrophic actions should require two-person signoff.
Story time. I was working at a company that was experiencing rapid growth and many of their engineering practices were those of a garage startup rather than 1000-employee multinational enterprise.
The following three things were true at the time:
cd-ing into a Git checkout on their laptop and running <tool> deploy <service-name>. Whichever Git commit was currently checked out was what got deployed.So one day the database team has a new intern, who is following the Database Team 101 wiki tutorial, and they run the wiki's recommended command to verify their deploy permissions work: <tool> deploy --prod <database>.
Turns out the laptop provisioning image was out of date, and that particular intern hadn't yet attended the spinup class where they did git pull origin master.
The "LLM breaking prod" story and the "intern breaking prod" story are similar, but the latter is easier to learn from because the individual mistakes are smaller.
This article notably skims past explaining why flipping the bozo bit on the main example bozo it presents was incorrect. It goes off into an example from a paper, and doesn't address its own main example. This comes across as if the author is trying to sneak through some other bozo.
From a fault analysis point of view, 'flipping the bozo bit' is a form of stopping at a single contributing factor (‘human error!’) when you should look at the rest of the fault tree, too.
In the given example:
Oh I have learned something: Don't use "AI" tools.