PhotoDNA and Limitations(hackerfactor.com)
hackerfactor.com
PhotoDNA and Limitations
https://www.hackerfactor.com/blog/index.php?/archives/931-PhotoDNA-and-Limitations.html
18 comments
Apple's recent CSAM announcement does not use PhotoDNA. Apple is not using PhotoDNA's hashes. Apple is using a different system called NeuralHash.
NCMEC operates like a roach hotel: CSAM goes in, and it only goes out to law enforcement. I am 99% certain that Apple never received CSAM material from NCMEC for training. While Apple's people may have sat in NCMEC's building for meetings, I doubt they were ever shown any actual CSAM. I've been told that NCMEC used a tool by Apple to create hashes for NeuralHash. The hashes are based on some number of "the most common" CSAM content. (In the past, "the most common" was around 30,000 files.)
In effect, Apple may not know what is in their trained data set. However, researchers (not me) are already looking for ways to extract data from the libraries. I think it's only a matter of time before this becomes a much bigger problem.
NCMEC operates like a roach hotel: CSAM goes in, and it only goes out to law enforcement. I am 99% certain that Apple never received CSAM material from NCMEC for training. While Apple's people may have sat in NCMEC's building for meetings, I doubt they were ever shown any actual CSAM. I've been told that NCMEC used a tool by Apple to create hashes for NeuralHash. The hashes are based on some number of "the most common" CSAM content. (In the past, "the most common" was around 30,000 files.)
In effect, Apple may not know what is in their trained data set. However, researchers (not me) are already looking for ways to extract data from the libraries. I think it's only a matter of time before this becomes a much bigger problem.
> However, researchers (not me) are already looking for ways to extract data from the libraries. I think it's only a matter of time before this becomes a much bigger problem.
What's the use of the data, and what problems do you see?
What's the use of the data, and what problems do you see?
Well, according to this site:
> Perhaps there is a reason that they don't want really technical people looking at PhotoDNA. Microsoft says that the "PhotoDNA hash is not reversible". That's not true. PhotoDNA hashes can be projected into a 26x26 grayscale image that is only a little blurry. 26x26 is larger than most desktop icons; it's enough detail to recognize people and objects. Reversing a PhotoDNA hash is no more complicated than solving a 26x26 Sudoku puzzle; a task well-suited for computers.
> I have a whitepaper about PhotoDNA that I have privately circulated to NCMEC, ICMEC (NCMEC's international counterpart), a few ICACs, a few tech vendors, and Microsoft. The few who provided feedback were very concerned about PhotoDNA's limitations that the paper calls out. I have not made my whitepaper public because it describes how to reverse the algorithm (including pseudocode). If someone were to release code that reverses NCMEC hashes into pictures, then everyone in possession of NCMEC's PhotoDNA hashes would be in possession of child pornography.
https://www.hackerfactor.com/blog/index.php?/archives/929-On...
> Perhaps there is a reason that they don't want really technical people looking at PhotoDNA. Microsoft says that the "PhotoDNA hash is not reversible". That's not true. PhotoDNA hashes can be projected into a 26x26 grayscale image that is only a little blurry. 26x26 is larger than most desktop icons; it's enough detail to recognize people and objects. Reversing a PhotoDNA hash is no more complicated than solving a 26x26 Sudoku puzzle; a task well-suited for computers.
> I have a whitepaper about PhotoDNA that I have privately circulated to NCMEC, ICMEC (NCMEC's international counterpart), a few ICACs, a few tech vendors, and Microsoft. The few who provided feedback were very concerned about PhotoDNA's limitations that the paper calls out. I have not made my whitepaper public because it describes how to reverse the algorithm (including pseudocode). If someone were to release code that reverses NCMEC hashes into pictures, then everyone in possession of NCMEC's PhotoDNA hashes would be in possession of child pornography.
https://www.hackerfactor.com/blog/index.php?/archives/929-On...
Here's a greyscale 26x26 of Obama:
torso: https://imgur.com/a/Oz0pnyF
closeup of face: https://imgur.com/a/Hr1OS2P
I guess it has to be decided if wide access to blurry images like this is better or worse than finding full resolution images. This all assumes there's not some obfuscation layer, I suppose (is that possible?).
torso: https://imgur.com/a/Oz0pnyF
closeup of face: https://imgur.com/a/Hr1OS2P
I guess it has to be decided if wide access to blurry images like this is better or worse than finding full resolution images. This all assumes there's not some obfuscation layer, I suppose (is that possible?).
We might also see DL image upscaling used. Obtain similar-looking pictures (at 128x128 or even smaller), downscale to 26x26, train DL to do 128x128 or larger upscaling using the original picture as training data. Assuming most content in the actual DB is similar, you can probably obtain plausible results. A similar technique is used by Peacemaker Filmworks to upscale 4k to 8k: https://youtu.be/umyglbDr4IE?t=257
[deleted]
That will be an incredibly questionable dataset used to train it. At that point, there would probably be no need to upscale these images. Just downscale then use this questionable DL to re-upscale any image.
[deleted]
> 26x26 is larger than most desktop icons; it's enough detail to recognize people and objects.
This doesn’t seem to be true. These are 26×26 icons, for instance:
https://cdn.dribbble.com/users/975594/screenshots/6193663/ic...
This doesn’t seem to be true. These are 26×26 icons, for instance:
https://cdn.dribbble.com/users/975594/screenshots/6193663/ic...
Depends on what you define as "Desktop Icons". Many icons in Windows are 16x16. The icons that literally sit on the Windows desktop are 32x32 or bigger, though.
When the algorithm is known, adversaries can figure out how to minimally modify images to certainly avoid detection. Uncertainty about detection capabilities may have been a deterrent.
One could claim that knowing the algorithm exists, for all cloud photo services (scanning isn't just an Apple iCloud thing), means that these people just don't upload to cloud services and not worry about any of this. For the Apple implementation, avoiding detection is as simple as switching off iCloud sync in the settings. In the case where they've modified the images to enable iCloud sharing, that would be risky game since those tweaked images would most likely make their way back into the database, and any tweak to the algorithm would mean it's all detected. My naive assumption would be that the latest model would be required to interact with the iCloud service (seems like a good idea at least).
All perceptual hashes, including AI-based perceptual hashes, have a "projection" property. If you have the hashes, you can project them into some kind of image. Some hashes result in blurry blobs (pHash, wavelet hashes). Some result in silhouettes (aHash, dHash). And some show well-defined images (PhotoDNA).
We don't know how Apple's solution works. But if it can project recognizable images, then it means people can use Apple's hash system to regenerate child porn.
Without details and an actual code review, I'm not willing to accept Apple's assurance that an image projection isn't possible. We hear that same promise from Microsoft's PhotoDNA, and it turned out to be false.
Imagine the problems if every iPhone and every Mac was in possession of child porn because Apple put it there...
Yes, this is a lot of 'if's, but until it has been evaluated, it is in the very real realm of possible.
We don't know how Apple's solution works. But if it can project recognizable images, then it means people can use Apple's hash system to regenerate child porn.
Without details and an actual code review, I'm not willing to accept Apple's assurance that an image projection isn't possible. We hear that same promise from Microsoft's PhotoDNA, and it turned out to be false.
Imagine the problems if every iPhone and every Mac was in possession of child porn because Apple put it there...
Yes, this is a lot of 'if's, but until it has been evaluated, it is in the very real realm of possible.
> Apple is not using PhotoDNA's hashes.
Do you have a source for this, or is it an assumption you are making?
> Apple is using a different system called NeuralHash.
No, one of the systems Apple is using is NeuralHash. They also use a second, undisclosed system on the visual derivative before human review. While it’s possible they have invented two separate perceptual hashing systems, it’s also quite possible that they would use PhotoDNA for this.
Details from Apple:
> Once Apple's iCloud Photos servers decrypt a set of positive match vouchers for an account that exceeded the match threshold, the visual derivatives of the positively matching images are referred for review by Apple. First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.
— https://www.apple.com/child-safety/pdf/Security_Threat_Model...
Do you have a source for this, or is it an assumption you are making?
> Apple is using a different system called NeuralHash.
No, one of the systems Apple is using is NeuralHash. They also use a second, undisclosed system on the visual derivative before human review. While it’s possible they have invented two separate perceptual hashing systems, it’s also quite possible that they would use PhotoDNA for this.
Details from Apple:
> Once Apple's iCloud Photos servers decrypt a set of positive match vouchers for an account that exceeded the match threshold, the visual derivatives of the positively matching images are referred for review by Apple. First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.
— https://www.apple.com/child-safety/pdf/Security_Threat_Model...
> They also use a second, undisclosed system on the visual derivative before human review. While it’s possible they have invented two separate perceptual hashing systems, it’s also quite possible that they would use PhotoDNA for this.
Yep I probably didn't phrase my question well enough but that was exactly part of what I was wondering.
They claim it's a second independent hash as you pointed out, so I would assume this is PhotoDNA server side, unless they made two separate hash systems (one which was not initially disclosed, if not implemented, at announcement time) ?
I'm also wondering exactly how they trained their NeuralHash, but may have missed some parts of the discussion. Supposedly NCMEC is not sharing the database of the "original" content that one would use to train such a system.
Edit : Guess that's a moot point now, they postponed it!
Yep I probably didn't phrase my question well enough but that was exactly part of what I was wondering.
They claim it's a second independent hash as you pointed out, so I would assume this is PhotoDNA server side, unless they made two separate hash systems (one which was not initially disclosed, if not implemented, at announcement time) ?
I'm also wondering exactly how they trained their NeuralHash, but may have missed some parts of the discussion. Supposedly NCMEC is not sharing the database of the "original" content that one would use to train such a system.
Edit : Guess that's a moot point now, they postponed it!
It's somewhat curious that a google exact search for "photodna.dll" only pulls up seven results.
This seemed to be the most interesting one:
http://www.herdprotect.com/knowledgebase_signer.aspx?serial=...
This seemed to be the most interesting one:
http://www.herdprotect.com/knowledgebase_signer.aspx?serial=...
Side question : my initial assumption on Apple's announcement was that they (like everyone else) were using PhotoDNA.
Apple's own perceptual hash has been found in current iOS and macOS releases, do we actually know if it is matching the results of that hash to a PhotoDNA database server side or not ? If not, did they train their model on the raw database with NCMEC's help ?
There was so much hand waving on their "tech" papers (and the recent announcement that they have a server side check anyway as a response to collisions being found), that I have a hard time reconciling everything.