Fearless SIMD v1.0 is here
42 points by ohrv
42 points by ohrv
I was checking the docs on Crates.io, and the usage doesn't seem straightforward. 3 things stand out; I'm not a Rust expert, so forgive me if the answers are obvious.
The first example has this code:
#[simd]
fn double_u32s<S: Simd>(_: S, values: &mut [u32]) {
for value in values {
*value = *value * 2;
}
}
Which takes an unused SIMD parameter; what I don't understand is why. If the parameter is not useful in simple cases, then why is it required?
I also see that compilation requires a call to a dispatch macro like so:
dispatch!(level, simd => double_u32s(simd, &mut values));
This feels unnatural; I see from the docs that it is used to select the appropriate SIMD level. I would expect to just write: double_u32s(Simd::Level::new(), &mut values); but perhaps Rust doesn't provide many ways to allow hooking into the compilation pipeline.
For context, fearless_simd works by creating implementations of the Simd trait for different SIMD levels, and then your function gets invoked as double_u32s(Avx512, values) or similar, using a different type for AVX-512 vs AVX-2 vs [...etc...].
With that in mind:
If the parameter is not useful in simple cases, then why is it required?
It's because #[simd] generates code that does use the parameter, but presumably changing the signature would be even more confusing (since then it's also affecting the perception to the callers). You can see the fearless_simd_macros docs for more info on this.
I also see that compilation requires a call to a dispatch macro [...] I see from the docs that it is used to select the appropriate SIMD level.
The important bit is that the Simd implementations are different types, specifically. If you had a single invocation to double_u32s(Something, &mut values), then...what is the type of Something? What does S end up inferred as? In a given call site, we can have only one type for S, but we inherently need to be able to dynamically determine what type to use. Hence, dispatch! will generate different invocations of double_u32s for each SIMD implementation type, branching into the correct invocation based on the detected SIMD level.
What if there were a SIMD type parameterised over type and length, with all SIMD operations defined on this type? double_u32s could just take this type and operate on it. This is close to how SIMD in Mojo works. It does mean that you need to write infra code to specialise to the best SIMD intrinsic available on the target platform, but the advantage is that common patterns are abstracted and packaged; in the examples, I see the vector values are always iterated over, with the SIMD operation done inside the loop. This is repetitive. With Mojo, the entire first example could be represented as:
def main():
var values: SIMD[Dtype.int32, 4] = [1, 2, 3, 4]
var result = values * 2
The multiplication is defined as an overloaded operator method inside the SIMD type.
Such a type doesn't tell you what SIMD instructions are available on the current machine, just what data widths you want to operate on. fearless_simd does in fact have types like this, albeit they don't make the width a generic parameter because const generics in rust kinda suck at the moment, so the types are u8x16, u16x32 and so on.
The Simd types passed to functions serve a different purpose: they's a sort of proof token that a particular set of instructions are available on the current machine. So at runtime if your function receives an Avx2 value (which is a type that implements the Simd trait), you know for certain that you can use AVX2 instructions without crashing or undefined behavior.
Generics are also the way you express to the rust compiler that it needs to generate multiple implementations of a function, one for each concrete type that gets used in the generic position. fearless_simd's core macros uses this to make the compiler generate a specialized implementation of the function for each SIMD level (i.e. one for SSE4.2, one for AVX2, ...). Ideally the compiler would have a builtin way of expressing optional instruction sets and runtime dispatch, but it doesn't. So you write a generic function once, and fearless_simd generates the glue that lets the compiler know what instructions it's allowed to use in each variant, and the glue to dispatch to the best available variant at runtime.
Admittedly this is a bit confusing for the simple example you posted, because this magic value isn't being used in the function body, its only purpose is to enable the glue in the caller that creates the specialized implementations.
In more complex examples that take explicit control over the vectorization instead of letting the compiler autovectorize, the value does get used to generically construct wide SIMD types like u8x64, which will get specialized implementations for the current SIMD instruction set. In those contexts it's also used to write generic code that needs to be aware of e.g. the native width of SIMD registers.
In a perfect world all this specialization for optional instruction sets and runtime dispatch would be a builtin language feature. But in the absence of that, this is pretty much the least intrusive way to make those happen using the tools the language does provide.
Hmm interesting! It doesn't handle any scalable SIMD architectures (aarch64 SVE or riscv's vector extension) which are far more challenging from an API point of view so I’d be interested to see how it handles those challenges (or if it can).
It’s also a little unclear to me when the function dispatching happens?
This says it's "runtime" but that could still mean "load time" (like ifunc's) as sometimes they are conflated.
Either way versioning like this is costly. It nearly always means at least an indirect function call, you lose any function inlining or outlining benefits, and the code size increase can hurt performance as you get worse cache locality. So I'm a little concerned about those overheads, especially for uses on loops with medium to low iteration counts.
For the same reasons I'm always a little wary of these "higher level" SIMD API's, as if you're at the level of optimisation where you want to start hand vectorizing things (rather than letting the compiler do it) then likely you should also care about these other low level details. But the abstraction level of the API introduces inherent costs.
It's actual runtime dispatch. Inside the dispatch! macro there's effectively a match on the detected SIMD level, with each arm calling the right specialized impl. This isn't exposed in the public API, so fearless_simd could technically switch up the dispatch strategy later.
But yes, there's definitely a small overhead for the runtime dispatch, and for a particular piece of code you have to figure out the right point at which to insert that dispatch from scalar to vector code to balance cache sadness and calling overhead.
That said, this isn't really anything new, and you can still extract massive gains from this style of SIMD programming. I've been tinkering with a column store implementation using fearless_simd, and I get near-linear speedups as I crank up the allowed SIMD level. I do operate on blocks of 1024 values in order to amortize the dispatch costs, I expect if I made blocks be 16/32/64 values then yes the dispatch cost would dominate.
IMO the big deal with libraries like fearless_simd is that it's just so easy to do compared to the "perfect" implementation that has no overhead. I don't think the choice in practice ends up being "fearless_simd or handcrafted zero overhead SIMD", it ends up being "fearless_simd or I just won't bother with SIMD and hope that's okay".
Yeah fair enough, it sounds like you’re in a situation where you know you have high loop counts where I agree that this is an effective solution, and those overheads can easily be amortised.
I guess I was thinking more about situations like strlen where the loop counts is very unknown and balancing performance of small loop counts and high loop counts becomes more challenging.
I also suppose in your situation you weren’t seeing any autovectorization before hand?
I ideologically prefer auto vectorisation of the scalar code, as that gives a path forwards for future enhancement and architectures without requiring rewrites/reoptimization. But autovectorization can be pretty limited and to some degree that’s inherent in its constraints.
But you’re right, this definitely has a place, and seems like it does its job pretty elegantly.