Hegel, a universal property-based testing protocol and family of PBT libraries
40 points by winter
40 points by winter
This is something my colleagues (+ me, a little bit) have been working on for some time now, and we're super excited to finally release it.
There's more of a walkthrough-y style overview here, but I felt that posting the docs might be better. Happy to change the link if this introduction is nicer to read, though!
I’m curious why the protocol works over IPC (Unix sockets) as opposed to some kind of FFI. Is that just easier to port to more languages, and the throughput is good enough? I think that’s the reason LSP works that way, but LSP sends a limited amount of data per UI interaction while PBT/fuzzing wants to generate as many examples as possible per unit time.
Part of the answer is that right now it's a lot easier and we're pretty far from the point where the IPC protocol is the bottleneck (that would be uh that we're running Python). Another part is that we want to be able to have the data supplied by an external fuzzer program (e.g. but not exclusively Antithesis), which is much easier in a protocol-based approach.
One of the open questions about the future of Hegel is whether we move to a model in which we ship a shared library and most of the time it uses that via an FFI in cases where we don't need the protocol. I think it's pretty likely we'll eventually move to that model, and we're confident that we've not made any design decisions that would stop us from doing that later.
I'm also curious about this — it seems as if PBT is a throughput-sensitive use-case. You could imagine targeting C FFI or WASM to achieve (hopefully) broad reach.
This was addressed in the article: this is the simplest way to expose the Hypothesis Python impl to any other language, and they're planning to build a Rust impl that exposes a proper FFI.
Well we're planning to build a rust implementation, and we're at least considering exposing a proper FFI, and those plans may or may not end up being linked.
I remember a while back that there was a Rust core being pulled out of Hypothesis, and it was used for the Ruby port. I wonder if this is a continuation of that, or if they rebuilt this from the ground up?
Either way, this is super exciting! Figuring out the different idiosyncrasies between different PBT implementations has been a time suck for those of us who won't work in the same stack continuously.
Spiritually sortof a continuation, but has no actual code in common. The Rust core and Ruby port mostly bitrotted and we formally abandoned them recently. It wasn't as easy as I hoped it would be at the time to get Python on the shared core, I wasn't personally particularly invested in Ruby (there was some paid work from Stripe for developing that, but only a bounded amount) and other projects ended up pulling me off it.
It was always a goal of mine to have a core of Hypothesis written in one language that could be used from any language. I originally thought about it as being C, then Rust, and well... right now it turns out that the answer is Python.
With my PyO3 maintainer hat on: if you’re ever interested in gradually reimplementing hypothesis into Rust, IMO PyO3 is pretty great for that. Happy to help out and answer questions.
Also, this is really cool! Now I know what Liam has been working on the past couple months!
Yup, big fan of PyO3, and it is definitely a big part of my sketch plan to try to get the hegel core ported over to Rust. With any luck we can do this in a way that lets us share a lot of the rust code with Hypothesis, but even if we can't we'll start from a working Python version and gradually turn it into a Rust version.
I think back when I was doing Hypothesis for Ruby, PyO3 existed but wasn't super usable yet, but it's also possible I just didn't know about it. I never really got very far into the part where I tried to make a Rust backend for Hypothesis, but now I'm much more motivated!
Hegel communicates with a Hypothesis server:
This is the core idea of Hegel: We run Hypothesis, and let it be the source of all generated data, wrapping it with a thin client library that turns it into values in your preferred target language. We get to implement the full feature-set of Hypothesis, because it’s all there for us already.
I see. I went back and found this too:
We’d like to drop the Python dependency for Hegel. As well as being kind of weird, it’s definitely the current performance limiter on running Hegel tests. Our current long-term plan is to implement a second Hegel server in Rust, but we’re not promising this will happen or committing to any timelines yet.
So they may replace this in time (which I think is a good idea because perf), but at least now they get to be feature-complete on day 1.
I remember a while back that there was a Rust core being pulled out of Hypothesis, and it was used for the Ruby port.
It appears that those were removed from the main Hypothesis repo back in January:
We haven't touched these in a long time, and any future cross-language efforts would be built on something different. I checked with <at>DRMacIver, who confirmed he doesn't have plans for these in the hypothesis codebase.
On a related note, DRMacIver discusses rewrites/ports to Rust in this comment on the HN submission:
It's on the agenda! We definitely want to rewrite the Hegel core server in rust, but not as much as we wanted to get it working well first.
My personal hope is that we can port most of the Hypothesis test suite to hegel-rust, then point Claude at all the relevant code and tell it to write us a hegel-core in rust with that as its test harness. Liam thinks this isn't going to work, I think it's like... 90% likely to get us close enough to working that we can carry it over the finish line. It's not a small project though. There are a lot of fiddly bits in Hypothesis, and the last time I tried to get Claude to port it to Rust the result was better than I expected but still not good enough to use.
Just wanted share how awesome hypotheses is! Everyone should be using it. I also like that (for now) for the first time Rust tools are being (re-) written in Python. Usually it is the other way around :D