The Mathocalypse
16 points by mseri
16 points by mseri
I've read a fair bit of Aaronson's work and generally enjoy it. This is the first time I'm reading about his thoughts on AI though. I'm largely against AI and he seems cautiously in favour. So, respecting his opinion, I read the article he linked to from another Scott, Scott Alexander, which is a rebuttal to Steven Pinker's open letter to Alexander.
I just can't get past the petty squabbling, from both Alexander and Pinker. I can just about make out their positions if I squint, but how can I take either of them seriously when they're clearly so acrimonious? And Aaronson thinks it's worth my time to go down that rabbit hole? How convenient for me, I can just feel vindicated about my anti-AI stance without having to engage with challenging ideas.
I want to know more about this quote from Alexander:
GPT-6 is more intelligent than GPT-3. You can split hairs, you can come up with some obscure sense in which that word is slightly off, but common-sensically it’s true. In the same way, GPT-9 will be more intelligent than GPT-6. At some point, this intelligence will exceed human intelligence. My argument requires nothing more complicated or philosophically dubious than this.
I sense the frustration in there, which makes me worry that he might be neglecting rigour. And I feel I must be missing some context, because that seems obviously philosophically dubious. Where is the consensus that human and artificial intelligence are of the same category? I think Aaronson assumes this too? In a conversation with Pinker in 2022 Aaronson writes:
I freely admit that I have no principled definition of “general intelligence,” let alone of “superintelligence.” To my mind, though, there’s a simple proof-of-principle that there’s something an AI could do that pretty much any of us would call “superintelligent.” Namely, it could say whatever Albert Einstein would say in a given situation, while thinking a thousand times faster. Feed the AI all the information about physics that the historical Einstein had in 1904, for example, and it would discover special relativity in a few hours, followed by general relativity a few days later. Give the AI a year, and it would think … well, whatever thoughts Einstein would’ve thought, if he’d had a millennium in peak mental condition to think them.
It's this bit, "it could say whatever Albert Einstein would say in a given situation". It seems profoundly reductionist. Aaronson has always seemed so carefully rigorous to me, but I just don't understand how he can think that. I guess this is just the common belief of rationalists?
I think part of the problem may be in trying to fit the disagreement between Alexander and Pinker into a "pro/anti AI" dichotomy. I think neither of them is at exactly one of the two poles of that. And yes, it's hard to read because their prose is insufferable, this is par for the course for rationalists.
Basically Pinker is saying we shouldn't be worrying about AI doom (human extinction or near-extinction) and Alexander is that we should. Pinker has his own anti-AI talking points relating to the more immediately obvious downsides of AI, and Alexander has some weirdly optimistic ideas about an amazing future where AI doom is avoided because we solve alignment.
imagine a hunter-gatherer who’s spent his entire life learning to survive deep in an unforgiving rainforest, then a giant resort hotel springs up right next to him with a helipad and heated pools and AirBnBs, and without missing a beat, the hunter-gatherer says: “alright fine, so now my new job is to run wilderness retreats for the tourists, or something.”
Oh I feel this about vibecoding as well - but maybe the new job pool cleaning or gardening or bouncer at the casino.
Is the end result of the mathocalypse just the model barfing out some stuff that technically works but the explanation is completely incomprehensible to humans? Buddy, I work in software, none of that is new to me.