Unlawful by design: Exposing the human rights costs of generative AI
52 points by mccd
52 points by mccd
unlawful web scraping
Ugh. Let's not go banning web scraping as yet another side effect here
I don't think they're pushing to ban scraping at all. If you read the relevant section the concerns seem very specific and entirely reasonable. Companies are mass collecting personal data without consent, and then commercializing it in a way that can risk exposing that data.
Amnesty International is calling for a prohibition of such systems.
They are not messing around.
I wish they didn’t state it that way, because some readers will probably disregard it as untenable idealism (“you want us to un-open Pandora’s box?”).
But they make good points around e.g. the need for recourse in AI-driven decision making, observed bias in models, and of course the usual issues around scraping without consent, attribution, or compensation.
I wish they didn’t state it that way
I would say they didn’t. The full sentence is much more qualified than that:
The briefing concludes that standalone generative AI systems, based on unlawful web scraping, depend on mass invasions of privacy by design, and are fundamentally incompatible with IHRL. As such, Amnesty International is calling for a prohibition of such systems, including where such systems are identified as exacerbating existing inequalities or creating new forms of discrimination.
Which I could sum up thus:
The briefing concludes that [X] […] are fundamentally incompatible with [human rights]. As such, Amnesty International is calling for a prohibition of [X], […].
Calling for the prohibition of whatever's incompatible with human rights is kind of the bare minimum. Anyone advocating that we give up ("untenable idealism" is indeed something they might say), is at best a cynic, and at worst no better than a war criminal. They should be called out a such.
Of course this all hinges on the premise that the aforementioned AI systems are incompatible with human rights, but most accusations of idealism do accept the existence, and even scale, of the problem. Instead they assert, often without much justification, that it’s unsolvable. I don’t believe it is, though.
I'm not done reading this yet, but something jumped out at me. From section 5.6 (Conclusion), p. 35:
Amnesty International contacted DeepSeek, Google, OpenAI, Meta, Stability AI, Intel, VMware, Midjourney, Microsoft, and Amazon, for their responses to findings and concerns regarding standalone generative AI systems and their human rights risks.
Anthropic is conspicuously missing from this list.
I searched for all references of Anthropic in the report. There was just this:
In February 2026, reports emerged that the US military had used Anthropic’s AI model, Claude, as part of its joint strikes on Iran with Israel. This came after the US government ignored contractual limitations instated by Anthropic on the use of its product for “mass domestic surveillance”, and “fully autonomous weapons […] without oversight”. Anthropic’s products have been used by the US government and military since 2024, and reportedly the first “advanced AI company” with products deployed across government agencies engaged in classified work. Anthropic was labelled a “Supply Chain Risk” by the Pentagon on 6 March 2026, following their refusal to amend these limitations."
(pp. 22-23)
plus footnotes for the above section.
This feels strange. Not trying to accuse Amnesty of pro-Anthropic bias, but I'm confused why they wouldn't include them on that list.
edit: The report mentioned OpenAI, Google, Meta, and others several times, citing sources for specific reported harms, which is perhaps how they got that list. Maybe Amnesty didn't have similar sources for Anthropic? Hm. Still feels strange.
This briefing focuses on the “data pipeline” aspect of the supply chain of generative AI products, specifically, the stages related to data capture, analysis, and processing.
Amnesty International defines standalone generative AI tools as products that are developed, deployed and marketed for their generative AI capabilities solely and specifically, such as AI chatbots, image/video/audio/text generators, and so on. This does not include products where generative AI is an added feature or function in a larger suite of products, for example word processing software with optional generative AI features. Standalone generative AI products are, in other words, generative AI models that come with their own front-end for direct use that is fundamentally concerned with generating outputs