The Llama 4 herd

I think @peterbourgon is making a distinction between social and statistical bias – that’s a little more evident if you quote their last statement in full:

if this is how you define left vs. right, then of course any LLM worth its salt is gonna have a left-wing bias. Because, by these rules – as has been said many times, by folks much wiser than me – reality itself has a left-wing bias.

A model that adequately reproduces statistical features of its training set isn’t biased in the statistical sense. That’s a feature, a statistical model that’s unresponsive to its population is bad by definition. There are populations for which bias is inherent and you can’t “sanitise” it out of the training data. For example, a model that would provide uniform responses, without regional or national bias, to a prompt asking for the name of the ten largest active volcanoes, would literally be hallucinating. You can’t sanitise the volcanoes out of the Pacific Ring of Fire.

We don’t perceive that type of statistical features as “bias”, with its negative connotation, because there’s no social or political controversy around that particular matter. There’s some taxonomic debate about “largeness” (by elevation? eruption volume?) but it’s hardly a social issue, and it’s trivially resolved by specifying the ordering criteria, in any case.

It’s a difficult discussion to have in more condensed terms because the terminology isn’t crystalised, either – I myself am using this one because it’s the one with most traction, but that’s not a given. Much of the literature on bias comes from the social sciences circle, so the two mechanisms are conflated.

That’s certainly not to say that training bias is universally good – a training set that’s unrepresentative of its population is just as bad as a statistical model that’s unresponsive to its population. So this is definitely a relevant discussion.

I don’t think the remark that this particular case “looks specifically designed to please the current US government” is unfair. The literature review in the last study you’ve cited points out evidence of this sort of bias going back four years ago – knowledge of left-wing/right-wing bias is literally older than Llama itself. Surely Meta isn’t just learning about it, it’s obviously a big deal for them judging from their press release, so waiting until the fourth major release to do something about it rings a little hollow. Besides, left-leaning vs. right-leaning is hardly the only type of social bias documented in LLMs. This one’s just the one that’s most politically significant at the moment, so it’s the one getting addressed.

That’s… by design, if you ask me, and IMHO not unfair, either. Meta is an American company, of course its policy and its software are shaped by American politics. This is how the software industry works. We may not like it, but it is political to some degree. I don’t want to debate the actual politics here but it would be wishful thinking to claim that politics has no bearing on the policy of large software companies.

The claim made was “it’s dubious that LLMs are biased”, I asserted that “no it isn’t dubious andthis is pretty well researched, it ain’t dubious”, and then you even agreed with my assertion that is wasn’t dubious

My point is that “bias” is not any kind of objective metric – especially not if the categorization of “bias” is assumed to be bimodal, i.e. left vs. right; and especially not if “bias” is assumed to be defined in terms of the current American political spectrum.

tl;dr: linked papers rely on subjective/arbitrary definitions of “bias” and in no way support conclusions like “LLMs are [politically] biased” in general