The Benchmarking Crisis

nk submissions on a hidden test set that participants never see, which makes overfitting on the public data pointless

I'm surprised this isn't done for database benchmarks, I guess there was no urgent need to fix this until LLM came along (there's always been microbenchmark gaming, but now it's 100x worse).

I guess the flipside is that the operational costs go up, since now the only person/org able to run the benchmark are the ones that created them instead of this being a distributed operation.

Has anyone thought about making a benchmark that generates queries (from prng seeds), rather than have fixed queries? Similar to how people do property-based testing and fuzzing and so on.

Then pick some random seeds every time a DB is added to the benchmark or every once in a while, average the results, and call it done. Given a u64 seed, you'll likely run out of ideas for new DBs before the seed combinations are exhausted even if you pick like a million seeds to average against.

In my mind these benchmarks really want to be more like fuzzers.