Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

I’m working in golang for a fairly standard graphql server backed by mostly Postgres. The challenges are the usual caching and auth correctness ones. A prompt that says at greater length than this sentence “look for bugs, use your adversarial review capabilities, you may write tests to find bugs, including generative tests, remember concurrency bugs exist” and specifying the parts of the code to look at (unfortunately I think I may not have saved this as a skill) finds about 40% as many issues as the other approach.

The other approach is defining what properties we want to hold for the component in question (in this case our Apache casbin backed auth system), asking the agent to build a deterministic simulation testing suite (based on the existing deterministic simulation testing apparatus I’d had an agent build previously, plus rapid go generative testing library) run that, then use its adversarial review capabilities to find additional bugs or areas where we should define additional tests.

As I recall about half the findings of the prompt-to-find-bugs exercise overlapped with those found by the generative and review process.

I also had a hybrid process using that one shot prompt and showing it the results of the generative process. That overlapped but, like the other two that still found some of its own findings.

What I didn’t attempt to do was score the importance or quality of the findings at this stage. Amalgamating the whole set of findings there were a bunch of findings that amounted to identifying that we have some potential panics, including explicitly coded ones (which LLMs seem to hate), through real belt and braces type findings about what incorrectly constructed rules could do to the system (not a problem in practice), through to real concurrency bugs and a bug in the underlying casbin library (which in fairness I had told the agents about the manifestation of that bug in production; I’ve yet to report it to the casbin project although I should) which required multiple steps to reproduce.

I did (a) ask two of the agents to propose a design solution to the bugs and to the performance problem that together with the one observed bug kicked off the exercise; and (b) send them on their way to get on with it. At this stage I did give them read access to our staging database. Both of the agents I asked proposed some variant of sharding the authentication per customer.

I’ve had both agents working in parallel, reviewing each other’s work, telling them to adopt what seems better from the other’s work and sending each of them the reviews of the other. I’ve also been having the agents split out their work into chunks that can reasonably be considered as a single stacked merge request (MR), and having them look specifically at their own and each other’s chunks, I’ve reviewed the converged version and made suggestions (in several cases quite significant as to structuring or finding what was a fairly obvious race condition).

It’s not in production, and the work is ongoing, because (a) reviewing the produced amount of code is slow (b) the agents work in the background but their work is also quite slow (c) my only colleague who would review this is not at all AI-pilled which means I really need to bring each MR up to the highest standard (d) I’m doing this in parallel with other work that can be delivered faster.

Obviously the further you read into my account above the less this sounds like a win for agentic engineering. Ruthlessly qualifying the importance of the findings would have been a huge improvement in the process. Secondly, I wasn’t particularly familiar with this part of the codebase, so reading it thoroughly no later than the finding qualification stage would almost certainly have been smart.