The benchmarkpocalypse

A recent example of this is a Claude powered benchmark (with accompanying youtube video) of a luatex engine that claims to be able to incrementally compile faster than Typst. It does look like they built an incremental luatex engine and editor with Claude, which is quite cool! But the benchmark doesn't actually test Typst fairly, and they don't seem to ever consider the soundness or correctness of incremental compilation outside of paragraphs (although the software still appears to be closed source). Of course, their website is straight slop and seems to promise every possible feature. It doesn't seem like the author is acting in bad faith, but they are certainly following AI confirmation bias instead of healthy curiosity.

Rather than point to someone's bad claim, I'll point to FRE, this regex engine I had an agent build

Man dropping a complete overfitted regex engine to prove benchmarking is hard is just so next level.