In defense of lock poisoning in Rust

This is restating some of the OP but just to be on the same page: I come from a background where we separate runtime failures into programmer errors and operational errors -- the same as what the OP calls unexpected and expected errors, respectively. Operational errors are handleable and in Rust they're generally handled with Result and error propagation. Panicking is for programmer errors. The key point about programmer errors is that as the programmer, you cannot reliably reason about the behavior of the program after it's hit a programmer error. You may think you get better availability by swallowing them, but you're rolling the dice because you don't really know what the blast radius was. By construction: this is a thing that you thought couldn't happen and didn't write code to handle. Here's the most concrete example from my career: in a [2013-era] Node.js program I worked on, folks installed a global uncaughtException handler for exactly the reason you said: to keep the program running in the face of whatever strange thing came up. One day, the program threw an exception while processing a database result while hanging onto a database connection with a session open that had row locks held in the database. If the program had simply crashed on the uncaught exception, this would have been a non-issue. The process exits, the file descriptor is closed, the TCP socket is shut down, the database cleans up the session, and the lock gets dropped. Instead, everything just barreled along with this lock held and tons of queries started backing up. The whole application ground to a halt and required operator intervention to correct. The problem here is that while, as an industry, we have done a lot of work to make the Unix process a meaningful fault domain, it's much harder to construct meaningful fault domains within a process such that you can reason about the rest of the program.

Now, you could make a credible case that with Rust, it's possible to create smaller fault domains. I'm not convinced, but it's at least a lot harder for one thread to randomly corrupt data in another one or for state (like the database session above) to not get cleaned up on unwind or whatever.

But then we get to the second point: with Rust, you just don't really need to do any of that. At least for me, Rust has almost completely eliminated these implicit fatal failures (invalid memory access, bounds check violations, division by zero, etc.). (Some of this depends on how much you lean into it. If you write code using as and array indexes, you can still have these; if you switch to try_from() and iterators, etc., then all these implicit failures become explicit ones.) What's left are the explicit failures: calls to panic!, unwrap(), expect(), etc. We do use those for things that really shouldn't be possible (e.g., I just checked the length of this collection, it was non-zero, and then I called iter().next()). But I almost never see those fire. And if a case is reasonably possible, even if it "shouldn't" happen (e.g., got garbage data from the database), we just ... handle those cases. It's usually not hard -- define some useful error types and use ? instead of .unwrap(). In the end (and I do feel kind of crazy saying this -- I know how it sounds), it just hasn't been that hard to have the program not panic. (I'm not saying Rust eliminates all bugs. Programs still have bugs. But they mostly only panic because someone wrote code that explicitly panics.) Of course, we have a restarter and we have redundancy so that when it happens, the system survives. But it's not like stuff's panicking all the time in production.

I can't say how many companies operate like this, but I can say we're definitely not unique among companies seeking to build highly reliable software in a variety of languages (I've been working this way in C, JavaScript, and Rust).

I am not sure what code you have to write where the poisoning actually helps you in practice.

Imagine you have some state like this:

struct Server {
    state: Arc<RwLock<State>>,
}

struct State {
    foo: Vec<Foo>,
    bar: Vec<Bar>,
}

And you have some methods like this (this is contrived, bear with me)

impl Server {
    fn add_entry(&self) {
        let mut state = self.state.write().unwrap();
        state.foo.push(Foo::new());
        state.bar.push(Bar::new());
    }

    fn use_state(&self, f: impl Fn(&Foo, &Bar)) {
        let state = self.state.read().unwrap();
        for index in 0..self.state.foo.len() {
             f(&self.foo[index], &self.bar[index]);
        }
    }
}

use_state has a precondition that foo and bar have the same number of elements. If Bar::new panics then that precondition is violated. Without poisoning, you would rewrite use_state to assert!(state.foo.len() == state.bar.len()), but with poisoning you don't need to explicitly assert on all your preconditions for State whenever acquiring a lock for it.

Of course you can rewrite this code such that lock poisoning is useless, but I'd also say that you can rewrite your locking code such that critical sections are small enough that if a panic really occurs while that lock is held it is meaningful.

In ACID terms, poisoning the lock asserts that the guarded state is consistent. Whether or not it's required depends on what your updates are. Personally I think it's a bit heavy handed but since much of Rust defaults to "do the safe thing" it's better than the alternative (for that, there's parking_lot)