Announcing Zstandard in Rust
32 points by jmillikin
32 points by jmillikin
This is very welcome news! Just the other day I had to pull in libc-dev so that I could build zstd for one of my dependencies, and was wondering if anyone had tried to do a serious reimplementation in Rust.
I hope it sees wide adoption within the community.
Related - I'm currently building a project based off WireGuard in Rust and so I'm pulling various rust crypto libraries. They have the obvious advantage of being memory-safe but unlike long-standing C libraries, not all of them have security audits.
My question is basically: is rewriting these algorithms in Rust worth the overhead?
Absolutely. Cryptography is rarely broken by attacks on the algorithms themselves, but regularly gets bypassed due to flaws in the implementations. There's a lot of "boring" non-cryptographic code that has to handle parsing, protocol state, and buffer management correctly to keep the system secure. Attackers won't bother with advanced cryptanalysis or a timing side channel that reduces cracking from millennia to decades if they can just send you a magic packet that executes arbitrary code.
If your project's performance requirements aren't too strict and you're up for a bit of an FFI adventure then it's possible to use the Go cryptography stack from Rust. This has the advantage of being memory safe at both ends (with a narrow unsafe FFI waist), and the Go crypto libraries are more mature than what the Rust (or rather, crates.io) ecosystem currently has available.
By default decompression performance of our implementation is a few percent slower than the C reference implementation.
That's all you need to know about this project.
Could you elaborate on what this says to you?
For reference, immediately after the quoted text:
We believe we can justify this regression though, because with the
unsafe-performance-experimentalfeature flag enabled we match C performance. This feature flag disables 4 bounds checks where data from the input is used to index into a data structure. For most users a ~3% performance regression is likely an acceptable price to pay for increased memory safety. If you really do need that last bit of performance, you can enable the flag at your own risk. Its behavior in these four locations matches C which also does not check the bounds and appears to run just fine in many production systems.