Forgery of C2PA on a Pixel 10
52 points by lvig
52 points by lvig
The mere idea of having to sign stuff to prove it’s not AI-generated sounds utterly terrible to me...
Image doctoring is far from new, it's just been made cheap enough to be done plausibly in large volumes by even the most mediocre scumbag :-/
On the other hand I also routinely see videos that are clips from fictional tv court shows with absurd fictional tv plots, multiple camera angles (because again: a tv show), etc filled with comments raging about how insane/bs the case is. Or random collections of video clips with a ai voice telling a clearly inciting story that has no correlation to the glued together clips, again comments filled with raging incompetence.
You don't need AI image or video generation when so many people will see something absolutely insane and then go insane themselves rather than going "this sounds insane, maybe I should do a quick search of the story to see if it exists anywhere other than the single 2 day old account that has 3 or 4 similar videos and 10s of thousands of followers".
It wasn't actually about AI or not, but because news companies wanted a unified way to be confident that images they broadcast/puvlished weren't photoshopped.
I think for photo/video provenance specifically some sort of unforgeable hardware attestation would be a wonderful thing to have. whether that can be reliably done, and whether people will try to sneak in a bunch of other surveillance stuff along with it, is a different matter.
I have a fear that this will somehow lead to an even bigger push for strict Play Integrity checks and other user-hostile, often superficial methods for device attestation, and ultimately punish systems like GrapheneOS (which does not allow root and goes further than stock at preventing exploits, but cannot pass strict Play Integrity because it is not an OEM-installed OS).
... but it might not!
Other than having an image sensor chip that can sign the raw image before it even goes to the CPU, I can't think of a way that you could stop someone with root access from signing images.
I believe that’s closer to the approach Apple is taking with their new Reference Image feature. Does anyone have thoughts on the differences between the Google and Apple solutions here? This is Apple’s writeup: https://security.apple.com/blog/apple-reference-image/
Both are flawed and it's generally a bad idea.
This can't be how we identify the truth because what if tomorrow, Apple starts refusing to sign images of things their regime doesn't approve of?
I agree that the concept in the abstract is flawed.
For the concern you mentioned, how would Apple even know what the images being signed are, let alone selectively refuse to sign some? If I understand the architecture correctly, it would require some kind of very theoretical power or RF side channel on the Private Cloud Compute servers that would probably be detectable.
I do think that the whole thing leaves a bad taste in my mouth, but so does AI image generation.
how would Apple even know what the images being signed are
One example would be indexing by Enhanced Visual Search - it is on by default and actively checks what's on all our photos.
Google's is obviously wrong. Apple's isn't obviously wrong.
I look forward to seeing some creative new attacks against the Apple version soon, once the hardware is in researchers’ hands! This is one of those things that practically has to be perfect to be useful at all, so hopefully Apple did their homework.
I have never understood how content provenance is supposed to work, in the real world. What is there to stop me from simply taking a picture of a slopped-out image?
The analogue hole seems quite insurmountable, and the practical effect of the C2PA organization seems to be increasing tech centralization. That sucks. I have no idea why people support them.
The LiDAR sensor in certain iPhones maybe could be used to encode depth information alongside the photo.
It's less of an issue for photos, because unlike audio, actually getting a camera to take a photo of a screen or printout, and having it look indistinguishable directly from the camera without editing, is near impossible.
I've already read several "C2PA is broken" stories on Lobsters, so to me, the really amazing thing about this story was the depth to which the Google rep didn't understand their own security model, or perhaps just security in general? Confused deputy attacks aren't new or novel.
I'm not going to pay for a Pixel 10 to do it, but man do I want to make a C2PA equivalent of @ForeshadowAaaS - here's my favorite one. For those who don't want to load Twitter since XCancel is gone, you'll miss the attached photo of the signature, but here's the text:
Here is your attestation that "Honest Achmed's Used Cars, Certificates, and Genuine Intel SGX Enclaves" is a genuine SGX enclave https://github.com/TeeAaas/ForeshadowAaaS/blob/master/2018_08_15_21_52_35_031150_50305faab8ed3ce46059be7af335eac8.py