Debunking "Faster than Rust": Malicious Benchmarks in Tov
18 points by ettolrach
18 points by ettolrach
The second i saw the tov benchmarks i thought: no way these benchmarks are well done if thats the result, just a single look at the used datatypes in the benchmarks make this obvious. Nice to see someone actually took the time to put some numbers for the compare
I'm kind of confused about the dispatch benchmark. I took a look at the tov code and it seems to be testing dynamic dispatch. Changing the Rust code seems to be changing what the benchmark is actually measuring.
Obviously, "faster than rust" is hyperbole, you can basically always produce faster code in a lower level language for any given task, we've gone through this sort of thing a million times, but it is entirely possible that tov produces faster code for dynamic dispatch in this particular case
Tov's optimizer might be monomorphizing the dynamic dispatch into something that works the way the fast Rust version does. Which is a point in its favor, except that dynamic dispatch as in this example isn’t idiomatic Rust; most programmers would reach for enums as in the rewritten test.
You can call me a bad amateur Rust programmer, but I didn't know about ahash because std::collections::HashMap has been fine for me since I started writing Rust.
That's kinda the only part where I am not fully on board with the assessment. Quoting from the docs "While its performance is very competitive for medium sized keys, other hashing algorithms will outperform it for small keys such as integers as well as large keys such as long strings, though those algorithms will typically not protect against attacks such as HashDoS" - this does not read "omg it's slow" to me. But looking at the benchmark, it still seems ok.
For most real-world use cases, the standard hashmap is plenty fast. But when your application is meaningfully bottlenecked by hashing, or when you have a benchmark solely testing map insertions, it quickly becomes a test of the hash algorithm, rather than the surrounding language. Though the authors here don’t mention what hashing algorithm tov/typescript uses and whether it is intended to be cryptographically secure, which I think would be worth mentioning for completeness.
Yeah exactly. It does different things but probably could have been chosen better or documented better. I'm a bit confused why a 2nd (fast) one would only be on crates, not in std, but I guess this being the default is ok.
The standard library authors generally try to keep it small and lean on the crate ecosystem for non-default options. In the case of hashes, they probably wanted to keep a single, cryptographically secure default option, and anyone who wants to opt into a less secure option does it explicitly and intentionally by going to the crates ecosystem or writing their own, rather than choosing a less secure option from std.
To be clear, it’s not HashMap that’s slow[ish], it’s the default hash function (SipHash). IIRC HashMap uses a state of the art hash table algorithm (is it SwissTable?)
It is! Originally there was a crate called hashbrown - https://docs.rs/hashbrown/latest/hashbrown/ - that was a drop-in SwissTable replacement for std's HashMap, and then in 2019, Rust 1.36.0 adopted hashbrown's implementation as the new std HashMap: https://blog.rust-lang.org/2019/07/04/Rust-1.36.0/#a-new-hashmap-k-v-implementation
Yes. People often misunderstand the goals of Rust/C/C++/etc: They give you as many tools as possible to achieve bare metal performance, available to be used whenever they are needed. All that extra stuff kinda piles up and makes people confused.
Languages like Tov just try to make common use cases as fast as possible, but as soon as you leave that space, you soon run out of official tools to optimise things with and you end up working against the language. NGL it's super fun optimising things in spite of the tooling, but just not at work.