Jepsen: ParticleDB 1.1.4
91 points by aphyr
91 points by aphyr
This is the most devastating Jepsen report I've ever read, and I will personally be hammering the entirety of section 4.5 ("LLM Tradeoffs") to our company doors. Too many good quotes to pull, but this from section 4.3 is perhaps the most emblematic:
Most databases have a test suite which exercises at least basic examples of each feature, and it is customary to run those tests before release, if not continuously. Where were those tests for ParticleDB? How does one add support for NOT NULL constraints without a test which tries to insert at least one NULL value? What development process adds a CHARACTER(10) type without trying to write a string of (e.g.) five characters?
It does raise the question of why Orkes engaged Jepsen in the first place; like hiring Michelangelo to paint your toilet. How could they be so utterly unaware of the most basic flaws in their product? The impression one takes away from the report is of an entire team deep in AI psychosis, experiencing their work (?) solely through agents, wholly out of touch with reality.
(Also great to see Kyle giving technical writing its due, another all too common victim of the mad dash for LLMs.)
Note that the previous Jepsen report last week was for different (less broken) software from the same supplier. Since ParticleDB is intended to support Conductor, it makes sense that a full examination of Conductor should also cover ParticleDB; and since Conductor can use alternative databases (Jepsen tested with Postgres), it makes sense to test ParticleDB separately.
AIs are usually eager to over test anything and everything. This feels less of a failure of AI, and more of a failure of a company deliberately omitting testing.
That's a disconnect I've seen a few times when a company engages external consultants before addressing known issues or asking internally for a review. Then again, from a purely business perspective... I'm not sure it's that terrible? If you want to do some specific kind of review, don't have the available hours internally, and don't want more permanent staff - maybe letting the consultants raise some basics/duplicates during the engagement isn't a bad outcome?
I had different thoughts about it in different situations. There's a fuzzy threshold there between targeted money spending and madness.
This wasn’t just “known issues”, the system didn’t work and hadn’t been tested; reading between the lines one could argue that it had only been run by the LLM. Taking this very public step with software in this state was terrible judgement and has put the product - and possibly the whole company - at risk.
I will add this entry to my memorable quotes register: "like hiring Michelangelo to paint your toilet". It does feel that software is getting a bit out of control and that some of us are just sailing through any kind of due diligence. We need to learn restraint and stop implementing features we don't test, or even use.
With this analysis’ “LLM Tradeoffs” in mind, I’m becoming increasingly wary of our decision at $DAYJOB to use a DBMS that’s recently seen a lot of LLM assistance in development (Datahike). I’m drawing some consolation from the facts that
But I’d love to see Jepsen test Datahike!
Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible.
That seems like the right conclusion, wow.
I have to wonder what leads a database developer to hiring Jepsen without first doing even fairly basic testing of their own database. Though I guess in this case it was probably "the LLM told them to", and they didn't even take the preliminary step of asking the LLM to do its own Jepsen-style analysis first (which obviously is nowhere near the same standard, but it sure sounds like it would have caught a lot of this stuff). Or maybe the LLM claimed that it had done so and they believed it?
Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work.
I asked about this too, hoping to discuss it in the report. FWIW, I ran the same Postgres Jepsen test against ParticleDB--it took a little bit to get it to set up the database and connect, but once it did, it started screaming about every possible kind of safety violation. I'm not sure how they got it to pass!
..."As with most Jepsen analyses, Orkes engineers offered detailed explanations of the causes and resolutions of the bugs we found in Conductor. However, Orkes seemed unable to explain the causes or resolutions of nearly every issue we found in ParticleDB."...😬
I am wondering to what extent can this be mitigated (bugs) by relying on DST and tools like Antithesis?
Naively, based on vibes from Twitter: DB + AI + DST + Formal Methods == Bugs Vanished. I am curious on how far can we push this?
Though at the end of the day, I would have a hard time on trusting a DB built this way.
Sure, DST would have been helpful in finding bugs. Even basic manual QA would have been helpful! Most of these issues just required starting a fresh node and typing a handful of SQL queries into the shell.
Understanding a bug is different than finding one. To understand the cause of a bug, you probably need to read some of the code involved, or at least have a solid mental model of the system. Until ~2025, clients generally needed to read code and understand a bug to fix it. With AI, you can just paste a bug report into Codex and say "fix".
Anecdotally, that does work! ParticleDB is much safer now than it was when we began. I don't think this kind of AI development is as robust a process as human understanding though. From working with a few clients who rely heavily on LLMs, bugs are often "fixed" but still happen, or a specific case would be resolved but the general pattern remained, or the fix would introduce even weirder behavior.
To what extent do the bugs pop back up because the clients aren’t using generative/simulation testing/fuzzing?
I’ve been fairly blindly asking LLMs to write simulation tests as well as non-simulated tests (golang) and that definitely finds issues, but at minimum I need to review the code for potential issues and then the clanker writes tests and finds more issues. It feels a little bit like I’ve made the fastest whackamole machine.
I would have thought that given a test suite, agents would probably not claim a bug was fixed when the test still failed. However, Orkes was able to run the generative test suite I was writing through this process, and we had to go back and forth several times on bugs which were supposedly fixed but weren't.
Part of that might just be normal human struggle to keep up with a high volume of complex bugs. I normally find a handful of serious bugs in a 4-8 week collaboration, and here I found 10-20 per week. Likewise, I normally test just a few versions, and here we'd have several builds to test each week. I'm still waiting on Orkes to publish the repo with the issues we found, but once that goes live you can see both of us made mistakes about what versions had which bugs, or exactly what reproductions look like.
Another challenge is that ParticleDB was so broken that the tests would fail for lots of reasons. To a human without intuition, or maybe to an LLM, it might be hard to tell whether a bug was fixed. Our most productive test, compare, also produced a fair bit of false positives--places where Postgres and ParticleDB behavior diverged in harmless ways. Easy for a human to sort through, but apparently really hard for LLMs to cope with.
Agents will also just... flat-out lie to you about what the tests say, so that doesn't help!
Thank you for your work and integrity. I find it unbelievable that they could have missed all this low-hanging fruit while being bold enough to make the claims quoted. Or to shill a database at all… I’m doubly surprised they paid for this analysis. I wonder how do you understand their motivations? Assuming it’s vibes from top to bottom why didn’t they have their LLM run a Jepsen style analysis? Clearly they knew enough to want one. And how is it possible they can make so many vague guarantees in their marketing with a straight face? Is it just inexperience and ignorance or more?
Strange!
I don't think it would be professional/ethical for Kyle to answer these questions, unfortunately. He was tasked with identifying flaws in the database, and he has quite clearly shown that these people cannot be trusted with your data (now and in the future):
Given the severity and extent of the safety errors we found, and the process which produced them, Jepsen recommends users halt further deployment of ParticleDB and migrate to other databases as soon as possible. Consider checking and/or rebuilding database state from more reliable sources, if available.
People have been clearly warned, without falling into speculation about Orkes' motivations.
Having said that... I'd love to know how shit like this could ever happen in the first place! Maybe it's just "AI bros thought they could vibe-code a database" and end of the story.
Dang. I've never seen Jepsen link to a Tiktok meme before.
...wow this is really appropriate to post on Halloween.