Borrow-checking surprises
73 points by jamii
73 points by jamii
The examples in the post are all used with primitive types (usize, i32), which are the special case where x += y evaluates differently from x.add_assign(y). Quoting the Rust Reference:
If the types of both operands are known, prior to monomorphization, to be primitive, the right hand side is evaluated first, the left hand side is evaluated next, and the place given by the evaluation of the left hand side is mutated by applying the operator to the values of both sides.
Otherwise, this expression is syntactic sugar for using the corresponding trait for the operator (...)
In other words, += evaluates left-to-right or right-to-left depending on the type of its operands: https://gist.github.com/Lysxia/972d06eb7d73a1bbea71a03f0c1b00d5
Anyone know what the reason is for this? Why not evaluate left to right always?
In the first couple years after Rust 1.0, code like vec.push(vec.len()) (or x.add_assign(x)) was actually rejected by the compiler because "two-phase borrows" didn't exist yet. I didn't know about this special case right-to-left behaviour for += applied to primitive types, but I would bet that it predates the more general left-to-right solution, and cannot be changed due to backwards compatibility.
EDIT: Here is a demonstration of x.add_assign(x) failing on Rust 1.8.0 but succeeding on Rust 1.94.0, whereas if you switch it to x += x, there was already a special case on 1.8.0 to make it work: https://godbolt.org/z/qv3E44zo6
Whoa!! Consider me surprised; I too thought I had a good mental model but a lot of these got me. Good presentation, thanks for taking the time to present it clearly
I was actually looking up the evaluation order of += and = with indexing for a bunch of languages this week to see what we should do for Typst (since our current evaluation order is ad-hoc).
Given this Rust code:
fn incr(x: &mut usize) -> usize { *x += 1; *x }
let mut array = [0, 0, 0];
let mut x = 0;
array[incr(&mut x)] += incr(&mut x);
At the end, array will either be [0, 2, 0] if the language evaluates the index of the assignment first (left-to-right) or [0, 0, 1] if the language evaluates the value being assigned first (right-to-left).
Here's a table of language behaviors using the array[incr(..)] = incr(..) syntax or an equivalent with both = and +=:
= +=
Rust R-L R-L
Zig L-R L-R
Swift L-R L-R
Go L-R L-R
Java L-R L-R
JS L-R L-R
Ruby L-R L-R
C L-R R-L (yikes)
Python R-L L-R (big yikes)
As mentioned by @lyxia, Rust has more edge cases here, but (with some exceptions) languages seem to prefer the left-to-right evaluation order, which I was not expecting!
C L-R R-L (yikes)
It’s undefined behaviour, so the “yikes” is the sound of demons flying out of your nose.
languages seem to prefer the left-to-right evaluation order, which I was not expecting!
I'm curious why you weren't expecting that. Is there a downside to L-R outside of borrow-checking?
In most languages everything else evaluates textually left-to-right, so having a few exceptions to that rule seems confusing. In rust the exception was worth it for the ergonomic improvements (although now that two-phase borrows exist, using them in assignment would be equally ergonomic and would avoid the need for exceptions to evaluation order).
I believe these surprises is due to the fact that Rust's borrow checker was developed using a syntax-first approach (write the program we want to write and let's figure out the semantics later), as opposed to semantics-first (and let the syntax fall out from the semantics).
It's hard to find a nice semantics once you've commited to a syntax, as the many attempts at formalising Rust has shown. If you're doing this from scratch and without having to bother with Rust's syntactic choices, then you might want to consider what the denotational semantics needs to be.
(Denotational semantics means giving your programs meaning in terms of mathematical objects which have already proved themselves useful in other settings. This reuse of established mathematical structure is what avoids surprising behaviour. It's easier said than done though.)
What are some examples of languages which have been developed with a semantics-first approach?
I think it's more common for features to be developed "semantic-first" rather than whole languages (too much work to do it in one go). Some examples I know of:
?-chaining is a special case of this;I should also say that "semantics-first" doesn't necessarily mean that the feature wasn't introduced until the semantics was developed, but often the feature was cleaned up / made more intuitive by the semantics (which is I think is the stage we are at with respect to Rust's borrow checker).
I think ML family of langs fall into this category, given their drive to have very straightforward semantics.
I'm not a Rust guy yet. Maybe some day. I just lurk and scan for now. Posts like this do not encourage me.
Years ago, I would try to convince C and C++ programmers to use some of the higher level "easier" languages that were available. Their teammates were and management wanted them to.
I observed an effect that I came to call the "hillclimbers effect". It was poignant when a peer told me they had moved to another less high powered language. Said they "I was getting kinda bored actually. It was just too easy". They found the "challenge" of solving the same problems in less helpful languages more challenging and engaging. From afar, borrow checker memory management seems like it could be a similar effect. This isn't meant to devalue the merits of better memory management. It's just that I wonder sometimes what the real motivations are for the "craftsmen" programmer.
I also begin to suspect there's a sort of Hari Seldon's Foundation "Tech Priesthood" feedback that occurs with these. It creates a skills barrier. You have to pay the price to learn how to do this. Since time isn't exhaustable, it creates a class of people that have paid the price to manipulate memory this way, and thereby the class of people that can collect revenue for practicing it.
That all said. I may try it. I've got to prove I have an open mind despite my skepticism.
I've shipped a lot of rust projects in the past and I only found all these edge cases when trying to implement my own borrow checker. When actually writing rust, things that you expect to work just work and you maybe don't think much about exactly how the analysis is implemented.
So my default language is mostly TypeScript, and 90% of the time having fully automatic memory management via garbage collection is very convenient, and being able to, for example, easily create complex graph structures without needing to convince the compiler that they are valid is very convenient to me, and it's not something I'd go back on just for the challenge.
That said, I do find myself regularly missing Rust's ownership model in TypeScript because it's so convenient to be able to describe not just the type of some data, but also who owns that data - whether a given function has taken control of an object and therefore can do what it likes, or whether it's just referencing data, and if it is just a reference to data living elsewhere, whether that reference is exclusive or shared. This is a useful property for reasoning about the behaviour of a function, and more importantly about the flow of data in a program. If I can visualise this property of data in my system, then it's easier to both communicate about it (e.g. by clearly indicating to another reader of my code that a given function requires mutating data), but also to reason about for myself (e.g. understanding where a final .close() call needs to live, and whether I'll still be using a variable after I've called that close method).
So I don't think it's just about writing more complex code for the sake of the challenge. It's more that using Rust has given me the language and tools to describe a phenomenon that I've always understood implicitly but had limited capability to express until now. If Rust was merely about manually doing what a garbage collector could do for me, I'd stick to TypeScript (and for most projects that's what I do). But specifically by requiring this additional programmer bookkeeping, I get a feature that makes my life easier, so in the right circumstances it's worth it.
Exactly. I came from Python and I like Rust because:
panic!, all return paths are visible in the type signature.ignore, etc.QWidget bindings or an SQL interface with SQLAlchemy+Alembic-level or Django ORM-level abstraction over SQLite's limitations, the differences between SQLite and PostgreSQL, and draft migration autogeneration by diffing the schema against the database.Granted, what I want to write tends to just work first time because, aside from non-obvious things like needing to restrict the impl for constructors on object-safe types, I'd already developed a bunch of patterns Rust likes while working to make my Python projects more testable and maintainable. (eg. Basic frameworkless dependency injection, leaning toward data-oriented design because I got burned by "Enterprise OOP", etc.)
The OP is more of a close up on technical details of the language. You don't need to be knowledgeable about those to be proficient in day to day programming in Rust. You can do plenty by sticking to simple data structures and tolerating a bit of copying to avoid dealing with borrows.
As a Python guy who came to Rust, I can say it's not as scary as it looks in this.
This is about the kinds of knowledge you need to write a Rust compiler, or if you just like technical trivia... it'll generally just work in normal use.
As for your example, C/C++ and Rust "show the complexity" in different ways. C and C++ expect you to understand it all or things will blow up. Rust's "make costs explicit" philosophy just makes it tempting to prematurely optimize and it's the premature optimization that gets you fighting the compiler over the correct use of features too advanced to even be an option in languages like TypeScript.
For example, in a garbage-collected language, every variable just lives until the last thing referencing it lets go. In Rust, if you don't explicitly ask for that with something like Rc<T> or Arc<T>, then the compiler will take you at your word and error out when you try to violate the semantics you asked for.
Most commonly, something like doing let parsed = foo.download()?.parse()?; with a zero-copy parsing library, which would give you an error because you didn't assign the output of foo.download() to a variable, so the memory that parsed is holding references into is going to be freed at the end of the line.
The fix there being either this...
let raw = foo.download()?;
let parsed = raw.parse()?;
...or, if parsed is a Cow<T> (common for things like JSON where character escapes may need to be resolved but the parser doesn't want to copy unconditionally), then let parsed = foo.download()?.parse()?.into_owned(); and let the compiler's optimizer skip making a copy since the original copy wouldn't be used for anything after .into_owned().
(Note the emphasis on "zero-copy parsing library" there. All this complexity is in service of combining a performance optimization that may not be expressible in simpler languages with the ability to have the compiler stop you before you silently corrupt data or open up a vulnerability.)
The ? operators are explicit analogues to raising/throwing an exception, like a more composable counterpart to Java's checked exceptions. Fallible Rust functions return either Result<T, E> (T being what you expected and E being an error) or Option<T> (T being what you expected. The alternative being None) and you have to say what you want to do in both cases to get access to one of them. ? means "If it's T, give it to me. Otherwise, early-return the Err(E) or None variant.)