You can just choose how many bugs you want now
7 points by cgrinds
7 points by cgrinds
I'm not really sure if this is true, or rather if this is true, then it's equally true that we you could choose this when knowledge about SQLite and FoundationDB's testing techniques was popularized, or when CompCert/sel4 were created.
My experience is that, in a complex system, you can basically find as many bugs as you ask the agents for. If you get tired of tackling bugs in the PR itself, have no fear: the agent will also find plenty of preexisting bugs for you to spend time on. The question is just when you want to stop and call it “done.”
[...] But the finding of the bugs has become nearly free, and AI agents are also capable of finding very subtle, intricate bugs that otherwise could have flown under the radar for years.
I don't know what kind of systems the author worked on pre-LLMs, but for every system I've worked on, it has always been the case that (1) there were way more open bugs than the team had capacity/willingness/permission etc. to tackle (2) you could basically find various kinds of bugs pretty easily if you were paying attention or stressed the system slightly off the beaten path (this includes compilers!).
If you do an exercise of logging all the bugs you see in week like Dan Luu did (https://danluu.com/everything-is-broken/), if you're paying attention, hitting a few dozen should be fairly easy across all the software you use.
Author here. I've worked on a fairly wide range of open- and closed-source software: database libraries, Android apps, web apps, backend systems, etc. My experience is the same as yours: there are always more bugs than the team has time to work on.
What's changed IMO is that it's much easier to 1) identify the bugs, and 2) root-cause them. I've worked on some bugs where I had a repro and yet even after spending days on it, I couldn't find a valid solution because I couldn't wrap my head around the complexity of the problem. Whereas I've found that agents are spookily good at reasoning things like "if system A has a race condition where it fires after system B when system B was hit with an AbortSignal, then system C would see the state of the database as...", etc.
Maybe the outcome is that we just realize we're even more underwater than we thought, and no more or fewer bugs actually get fixed. But I'd like to think that agents could help us at least prioritize the bugs and fix the criticals/highs much faster.
WAT.
...and you couldn't before? Work harder; ship fewer bugs. Build your theory. Do software architecture. Make Peter Naur proud!