How do programming languages impact token efficiency and correctness?

This is actually quite surprising. I would've expected, like Dan, that on reasonably large coding evals like these the static languages would've easily outperformed the dynamic ones in terms of correctness, since the LLMs can use at least some ground truth to guide themselves. But Ruby and Python are generally just as good as, or even better than, languages like Rust, Go, and TypeScript even in terms of raw correctness.

Instructing agents not to cheat while having a holdout set of tests didn't prevent blatant cheating that scored extremely poorly on holdout tests, but telling agents that there was a holdout set of tests they were graded against seemed to reduce the score they achieved on the agent-visible tests while increasing the score they achieved against holdout tests (without telling them this, a number of agents achieved 100% on the Pandoc tests with uselessly brittle code; on telling them there's a holdout, no agent scored 100% after 1 turn on ultra, but the holdout scores were substantially better, indicating better generalization).

That's pretty funny. I'm guessing this kind of advice/lore will end up in a bunch of "skills" files even without there truly being holdout tests.

Clojure programs had test failures because byte conversion throws on 128–255 […] cargo repeatedly gets invoked with the wrong arguments […] the failures that caused the score are generally something idiosyncratic where it's not always obvious how much the issue generalizes across tasks or across setups. There's no way to look at the score on one eval or even five or ten evals and draw a conclusion about programming in general.

So even with the ability to automate research at this scale, More Research Needed™ due to all the noise from such idiosyncratic failures and it's really hard to conclude anything in general. ("Appendix: selected issues in ai-coding-lang-bench" shows even more such stupid noise leading to hard-to-trust results.)