Apple plans to scan US iPhones for child abuse imagery(ft.com)
ft.com
Apple plans to scan US iPhones for child abuse imagery
https://www.ft.com/content/14440f81-d405-452f-97e2-a81458f5411f
387 comments
Don't worry, AI is very good at these things and never makes mistakes. It is so advanced it even understands nuance and complex communication mechanisms like sarcasm in text, even if it's only a single sentence to contrast! It will be able to know exactly the context of each image and it will affect only the sickos. We're also working on watching your face through the camera to see your true expression when viewing them. Oh, but if a few innocent people do happen get caught up in it and lives ruined, it's totally justifiable because this type of activity is the absolute worst. And false positives are exceeeeedingly rare. We'll apologize and adjust our algorithms. Sue us? How dare you? We're really trying to wipe out sickos. Are you a sicko or something? You must be. Only sicko's would disagree with protecting children.
There is zero AI classification involved, it’s a lookup vs hashes of known child porn images. You would need a hash collision, misidentified photo in the dataset, or legitimate use case to end up with a false positive.
And they aren't going to act on a single hash collision. People who have these images likely aren't going to have one or two.
People can argue against this approach for privacy reasons, but I think the false positive argument is a relatively weak one.
People can argue against this approach for privacy reasons, but I think the false positive argument is a relatively weak one.
There won’t be any collisions. It’s not a bit-for-bit Comparison. The algorithm uses machine learning to categorize images using perceptual match.
There will be many false positives, they will be reviewed by people. When there’s more than a few false positives, you will be investigated by the FBI.
There will be many false positives, they will be reviewed by people. When there’s more than a few false positives, you will be investigated by the FBI.
Zero AI involved? Simple hashes? Source needed. From what I read, you are making false claims.
The concern is less false positives, and more if the Chinese government tells Apple "find everyone who has this picture on their phone in Hong Kong"
[deleted]
The PhotoDNA platform doesn’t offer the participating organization raw photos, just hashes. The AI part is stitching photos with matching edges together, incase someone cropped them into different parts.
Again, Apple nor any company, have access to the source data, just hashes.
Again, Apple nor any company, have access to the source data, just hashes.
My point was that it can’t be “simple” hashes, as that is easy to defeat. I never said anything about Apple having the source images.
The article says that Apple calls this system neuralMatch. I'm not sure why that name gives you confidence that there is zero AI involved?
Also, we've seen more and more AI hardware being shipped built-in to phones.
I hope it's hash-based and not AI. Training that NN would be one of the worst jobs I could imagine, you're adding commercial value to child porn and all the misery it entailed.
They trained a neural net to perturb images. Not classify them.[1]
They did add a classifier to iMessage. But it's designed to prevent children seeing any sexually explicit images.[2] There wouldn't be a reason to train it on images of children specifically.
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[2] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...
They did add a classifier to iMessage. But it's designed to prevent children seeing any sexually explicit images.[2] There wouldn't be a reason to train it on images of children specifically.
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[2] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...
They are using AI techniques as part of perceptual hashing. In other words it’s trying to match files with specific images rather than abstract concepts.
I’m pretty sure you don’t know what you’re talking about. Perceptual matching is not image matching.
Perceptual matching is used to sort categories of images. A quick DuckDuckGo will turn up many results. No stretch of the imagination will turn this into a bit for a bit comparison. This is a machine learning algorithm used to categorize images. https://www.ibm.com/blogs/research/2019/10/learning-implicit...
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hashing”
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
Perceptual matching is used to sort categories of images. A quick DuckDuckGo will turn up many results. No stretch of the imagination will turn this into a bit for a bit comparison. This is a machine learning algorithm used to categorize images. https://www.ibm.com/blogs/research/2019/10/learning-implicit...
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hashing”
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
Perceptual hashing has a very specific meaning. It’s for verifying one file is directly related to a different file not an overall classification.
“Perceptual hashing is the use of an algorithm that produces a snippet or fingerprint of various forms of multimedia.[1][2] A perceptual hash is a type of locality-sensitive hash, which is analogous if features of the multimedia are similar.”
https://en.wikipedia.org/wiki/Perceptual_hashing
The goal is to verify a black and white copy of an image is identical to a colored original. Search algorithms want a similar thing so they can validate an image contains a blue car. However, a perceptual hashing algorithm must differentiate between different images containing a blue car while matching a photoshopped copy of the same image.
“Perceptual hashing is the use of an algorithm that produces a snippet or fingerprint of various forms of multimedia.[1][2] A perceptual hash is a type of locality-sensitive hash, which is analogous if features of the multimedia are similar.”
https://en.wikipedia.org/wiki/Perceptual_hashing
The goal is to verify a black and white copy of an image is identical to a colored original. Search algorithms want a similar thing so they can validate an image contains a blue car. However, a perceptual hashing algorithm must differentiate between different images containing a blue car while matching a photoshopped copy of the same image.
If they're using perceptual hashes, which they almost certainly are at least some of the time, then there's a huge space for false positives for each perceptual hash.
Don't worry, some poor traumatized souls will be tasked with reviewing visually all those positives. I hope they have good mental care.
Ok. Hashed-based image rec. Where are the hashes? Is the comparison done in my phone against a downloaded database, or is a hash of every image on my phone uploaded for comparison on a server? Could this program then be expanded to other classes of images? Animal abuse imagery? Terrorism? Hashes of known bomb-making instructions? How about people sharing illegal pdfs and text files? Will the
MPAA be allowed to submit hashes of their copyrighted material? In short: What other warrantless government inspections of my files will Apple allow?
It’s done on Apple’s servers using your iCloud backups. For good or ill this lets people opt out by disabling iCloud.
I would hope most privacy conscious people disable iCloud, but that’s another story.
I would hope most privacy conscious people disable iCloud, but that’s another story.
Most of the process is done on customer's devices.[1]
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
I was answering this: “Is the comparison done in my phone against a downloaded database, or is a hash of every image on my phone uploaded for comparison on a server?”
To be clear each image, the image’s NeuralHash, and a visual derivative are uploaded to iPhoto. This allows for the inspection of the NeuralHash algorithm used which I actually prefer.
The phone isn’t downloading the hash database.
To be clear each image, the image’s NeuralHash, and a visual derivative are uploaded to iPhoto. This allows for the inspection of the NeuralHash algorithm used which I actually prefer.
The phone isn’t downloading the hash database.
From that same page: "Apple further transforms this database into an unreadable set of hashes that is securely stored on users’ devices". The phone is downloading the hash database.
[deleted]
If apple really cares about security then those images in iCloud should be stored in an encrypted form that makes hash comparisons impossible. If they are hashing them then they have access to them in plaintext. If I used iCloud I would be more worried about the wider implications that creates.
iCloud needs to allow people to restore backups on a new iPhone after losing their old one.
You can setup secure encrypted backups, but the customer losing the key means losing the back so that’s not what consumer focused companies are going to do. In other words any backup service that doesn’t have big warnings that losing your key loses your backup means they can read your data.
You can setup secure encrypted backups, but the customer losing the key means losing the back so that’s not what consumer focused companies are going to do. In other words any backup service that doesn’t have big warnings that losing your key loses your backup means they can read your data.
https://blog.cryptographyengineering.com/2012/04/05/icloud-w...
"The mud puddle test: You don’t have to dig through Apple’s ToS to determine how they store their encryption keys. There’s a much simpler approach that I call the ‘mud puddle test’"
"The mud puddle test: You don’t have to dig through Apple’s ToS to determine how they store their encryption keys. There’s a much simpler approach that I call the ‘mud puddle test’"
[deleted]
You are wrong. It's AI.
Citation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
Citation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[deleted]
It’s perceptual hashing not AI classification. “NeuralHash is a perceptual hashing function that maps images to numbers.”
The perceptual hashing is based on AI techniques. “The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process.”
The difference is AI classification is based on defining something as say a “Cat” and then the AI spits some association with how cat like the image is. This extracts features from an image then compares lists of features to specific images.
The perceptual hashing is based on AI techniques. “The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process.”
The difference is AI classification is based on defining something as say a “Cat” and then the AI spits some association with how cat like the image is. This extracts features from an image then compares lists of features to specific images.
It sure sounds a lot like AI to me. Saying its just hashes seems misleading to me.
From the PDF:
"The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements.
The neural network that generates the descriptor is trained through a self-supervised training scheme. Images are perturbed with transformations that keep them perceptually identical to the original, creating an original/perturbed pair. The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. "
From the PDF:
"The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements.
The neural network that generates the descriptor is trained through a self-supervised training scheme. Images are perturbed with transformations that keep them perceptually identical to the original, creating an original/perturbed pair. The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. "
It’s closely related to AI, but it’s not image classification. The important thing is your trying to match specific files to hashes rather than features to an image so the output doesn’t need to be human readable.
Image classification on the other hand cares about if the image contains say a stop sign or a trash can. That’s useful for self driving cars etc.
Aka classification you might want to match two different bands playing the same song as identical. Where perception hashing would want them to be classified differently.
Image classification on the other hand cares about if the image contains say a stop sign or a trash can. That’s useful for self driving cars etc.
Aka classification you might want to match two different bands playing the same song as identical. Where perception hashing would want them to be classified differently.
You are wrong. It is not a numerical Hash of the exact image.
It is a perceptual Hash of the images characteristics and perceived continent by the algorithm, that’s the “neural” in neuralmatch.
Apple, for yours has been building-in machine learning dedicated chips into their builds so this shouldn’t affect battery life.
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hasing"
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
It is a perceptual Hash of the images characteristics and perceived continent by the algorithm, that’s the “neural” in neuralmatch.
Apple, for yours has been building-in machine learning dedicated chips into their builds so this shouldn’t affect battery life.
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hasing"
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
Yep. And if you dont like it, build your own iPhone.
And be immediately labelled by society as a CP holder
This is painful to read with the arrogant undertone while you don't actually know how CSAM scanning works.
It's fuzzy hash, not ""AI"", based. Cloudflare uses it too, and last time i checked the web is still functional.
https://support.cloudflare.com/hc/en-us/articles/36004610611...
How about we turn the tables and have the complainers suggest a solution. Because every single time an approach to targeted child porn takedowns has been suggested, such as datacenter raids which would not affect as many people, someone is screaming about their privacy.
Child abusers evolve and are very happy if law enforcement doesn't.
It's fuzzy hash, not ""AI"", based. Cloudflare uses it too, and last time i checked the web is still functional.
https://support.cloudflare.com/hc/en-us/articles/36004610611...
How about we turn the tables and have the complainers suggest a solution. Because every single time an approach to targeted child porn takedowns has been suggested, such as datacenter raids which would not affect as many people, someone is screaming about their privacy.
Child abusers evolve and are very happy if law enforcement doesn't.
> Child abusers evolve and are very happy if law enforcement doesn't.
Yes. And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system. The only effect this will have is that moral panickers like you will have got everybody's privacy invaded over your moral panic of the day.
Yes. And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system. The only effect this will have is that moral panickers like you will have got everybody's privacy invaded over your moral panic of the day.
> And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system
They don't use simple file hashes to match images, but perceptual hashes. That way they can find modified derivatives of a source image. The problem with this approach, though, is that this is ripe for false positives. Two completely unrelated images can have similar hashes.
They don't use simple file hashes to match images, but perceptual hashes. That way they can find modified derivatives of a source image. The problem with this approach, though, is that this is ripe for false positives. Two completely unrelated images can have similar hashes.
could you use multiple perceptual hash functions with different salts, so that collisions would be less likely while allowing derivatives to be detected?
That reduces to just inventing a “fancier” single hash function. This adds no value or security in cryptography; it just makes things slower. I expect the same is true of perceptual hashes.
They aren't just matching exact hash hits, but are using a metric like the hamming distance between hashes to determine if one image is the same as , or a derivative of, another. The data structures that allow for efficient lookups rely on that metric, or another metric, for matching.
Again, you don't appear to know how this works. Look up fuzzy hash, i even mentioned it in the comment.
I have a hard time believing that this algorithm will be able to resist simple image manipulations while still being sensitive enough to avoid false positives.
What algorithm are they using?
I don't know. If they made that public it would be completely ineffective.
I have similar qualms with this, because this is increasing the number of photos scanned by 2-3 orders of magnitude, and the number of false positives presumably also increases correspondingly.
Just saying "fuzzy hash" doesn't begin to explain how this would work. There are an infinite variety of algorithms along with arbitrary tolerances configurable. "fuzzy hash" just isn't helpful no matter how many times you repeat it.
I was a kid in the 80s and teenager in the 90s. My favorite thing during that time was pirating video games. A game would come out, and it was cracked, usually within hours and often before the game was even released to be sold. That's when "zero day" had a different meaning. All the "warez" ftp sites had a section for 0 day warez. The cryptologists and math brains would come up with new protection methods to protect their IP from being copied. Spend millions, probably billions for all of these projects. Yet, some kid in their basement with a commodore 64 was always able to crack them. Sometimes it would take longer. There were a few that took years, but once figured out, unlocked hundreds of titles previously secured.
This is, and always is, a game of cat and mouse. Law enforcement is always catching up. They are the cryptologists here. They are never ahead, always behind, because they don't know the new protections peddlers are using until they have been in use and later discovered.
No matter what vector you plug, they will use another, and the game continues (sick game). Maybe divide the image into 32 different quadrants and rearrange them, then put them back in the correct order when viewing through a specific image viewer. I'm sure that would bypass whatever detections they've come up in their fuzzy fingerprinting with as the entire image is now different. By the time they catch someone using this, they'll have already moved on to something different, as they always do.
I will never be ok with warrantless searches of my personal property, no matter the reason or justification or subject, and no matter who it is done by (government or private company). And I say that as a survivor of some pretty horrific shit as a kid to the point I fucking tremble with absolute rage when thinking about it 35+ years later. I would be banned from everything for life if I were to honestly state what I would do with these types of people. The movie "Saw" is tame in comparison. I have no compassion or sympathy for these sickos. But when reading world history, I can absolutely see the importance of "innocent until proven guilty" and Blackstone's Ratio "It is better that ten guilty persons escape than that one innocent suffer." Most of human history was the opposite, and it was brutal and full of literal witch hunts. Are we progressing as a species, or regressing in terms of human rights when it comes to technology?
This is, and always is, a game of cat and mouse. Law enforcement is always catching up. They are the cryptologists here. They are never ahead, always behind, because they don't know the new protections peddlers are using until they have been in use and later discovered.
No matter what vector you plug, they will use another, and the game continues (sick game). Maybe divide the image into 32 different quadrants and rearrange them, then put them back in the correct order when viewing through a specific image viewer. I'm sure that would bypass whatever detections they've come up in their fuzzy fingerprinting with as the entire image is now different. By the time they catch someone using this, they'll have already moved on to something different, as they always do.
I will never be ok with warrantless searches of my personal property, no matter the reason or justification or subject, and no matter who it is done by (government or private company). And I say that as a survivor of some pretty horrific shit as a kid to the point I fucking tremble with absolute rage when thinking about it 35+ years later. I would be banned from everything for life if I were to honestly state what I would do with these types of people. The movie "Saw" is tame in comparison. I have no compassion or sympathy for these sickos. But when reading world history, I can absolutely see the importance of "innocent until proven guilty" and Blackstone's Ratio "It is better that ten guilty persons escape than that one innocent suffer." Most of human history was the opposite, and it was brutal and full of literal witch hunts. Are we progressing as a species, or regressing in terms of human rights when it comes to technology?
You're spot on about the 0-day comparison. As always, something will arise to let people with the motivation hide illegal images. The problem is how it will be used against everyday citizens, who don't have sophisticated tools and maybe just share images of Hong Kong freedom protests, or books, or anything anti-totalitarian. The emotional appeals about this being about child abuse are absurd on their face because of how easily those people will hide. It's a good thing that some people are able to see through that as a ploy. We shouldn't have to go there and prove our bonafide hatred of abusers every time we justify our right to secure encryption or freedom from surveillance. Doing so almost validates the government's position. Just like in China you would have to say "of COURSE I hate the democracy protesters! I just think..." No, you shouldn't have to take a deep emotional dive into a history of abuse to justify your human right to privacy.
So this means it is checking if you are sharing known CP images? That does seem to be much less invasive and problematic as there is likely no good reason to be sharing these images.
Yes, that's exactly it. It uses a database compiled by NGOs and specialized firms comprising of file hashes matching child porn. These lists are handled by humans.
Fuzzy means that it takes compression and the like into account, because even if just one pixel out of 20 thousand is different, the hash is different too. Fuzzy hash still recognizes it as the same image, so using an algorithm to alter the color etc. won't work.
Fuzzy means that it takes compression and the like into account, because even if just one pixel out of 20 thousand is different, the hash is different too. Fuzzy hash still recognizes it as the same image, so using an algorithm to alter the color etc. won't work.
> These lists are handled by humans.
That's also true for the no-fly list and the Terrorist Screening Database,[1] yet those are full of false positives. And unlike those lists, CSAM databases cannot be independently verified. To do so would require having the original images, which is illegal.
1. https://en.wikipedia.org/wiki/Terrorist_Screening_Database
That's also true for the no-fly list and the Terrorist Screening Database,[1] yet those are full of false positives. And unlike those lists, CSAM databases cannot be independently verified. To do so would require having the original images, which is illegal.
1. https://en.wikipedia.org/wiki/Terrorist_Screening_Database
The no-fly list and the terrorist screening database aren't used in a court of law. The Confrontation Clause of the Sixth Amendment guarantees you access to all the evidence presented against you. You also don't need the original images to defend yourself, though apparently CP can be presented to a (traumatized) jury [0].
So if you're charged on the basis of a fuzzy hash matching, you'd subpoena Apple for the photo in your backup that matched, present it to the court (since it doesn't actually matter if it's CP or not to be admissible), and you win the case.
0. https://www.johntfloyd.com/the-difficulty-with-criminal-evid...
So if you're charged on the basis of a fuzzy hash matching, you'd subpoena Apple for the photo in your backup that matched, present it to the court (since it doesn't actually matter if it's CP or not to be admissible), and you win the case.
0. https://www.johntfloyd.com/the-difficulty-with-criminal-evid...
It’s not just checking files you are sharing, it’s also scanning files that exist on your device. The worry, or slippery slope argument, is that it’s one step away from scanning your device for other types of content, like memes critical of the government or just general wrongthink.
A lot of governments would be very interested in this feature.
There actually are some edge cases even for matching against image blacklists. Google has experience with hitting them because it's used this type of image simhash for years (for shared cloud files at least).
The definition of child porn varies around the world. These systems use the US definition. This is not entirely what you might expect. For example, in the USA the courts have decided that cartoons can be child porn even though no actual children are in the picture. Most of the world does not agree with this, meaning an image can be CP in one place but not another. Is Apple going to enforce the US definitions or the ones where the user actually lives?
In the USA, photos an under-age person takes of themselves can also be considered CP.
What counts as a "child" for sexual purposes also varies around the world. Some countries have a lower age of consent than other places. In some parts of the world the age of consent and the age at which a child stops being a child for CP purposes are different, meaning that a teenager can have sex legally but if they take a photo of themselves doing it, they are trafficking in CP.
Finally, what is actually on these image blacklists? Hardly anyone actually knows because of the third rail nature of CP. Tech firms are often delivered image hashes, not even the images themselves, by third party 'charities' of various kinds and tech workers are - for obvious reasons - not normally given access to the actual pixels. Additionally, appeals from users are invariably ignored because people say "legal issues, it's complicated" and so everyone clams up. If FPs occur there is no way to resolve it and the people who see your appeal, if there even is one, won't be willing to actually look at the image to find out what it was.
It should be obvious how much potential for abuse this hands the people who actually manage these CP databases. Literally any image can be made verboten immediately, without any recourse, and basically nobody will ever find out including the people who shut down the affected users.
The definition of child porn varies around the world. These systems use the US definition. This is not entirely what you might expect. For example, in the USA the courts have decided that cartoons can be child porn even though no actual children are in the picture. Most of the world does not agree with this, meaning an image can be CP in one place but not another. Is Apple going to enforce the US definitions or the ones where the user actually lives?
In the USA, photos an under-age person takes of themselves can also be considered CP.
What counts as a "child" for sexual purposes also varies around the world. Some countries have a lower age of consent than other places. In some parts of the world the age of consent and the age at which a child stops being a child for CP purposes are different, meaning that a teenager can have sex legally but if they take a photo of themselves doing it, they are trafficking in CP.
Finally, what is actually on these image blacklists? Hardly anyone actually knows because of the third rail nature of CP. Tech firms are often delivered image hashes, not even the images themselves, by third party 'charities' of various kinds and tech workers are - for obvious reasons - not normally given access to the actual pixels. Additionally, appeals from users are invariably ignored because people say "legal issues, it's complicated" and so everyone clams up. If FPs occur there is no way to resolve it and the people who see your appeal, if there even is one, won't be willing to actually look at the image to find out what it was.
It should be obvious how much potential for abuse this hands the people who actually manage these CP databases. Literally any image can be made verboten immediately, without any recourse, and basically nobody will ever find out including the people who shut down the affected users.
> So this means it is checking if you are sharing known CP images?
No. NeuralMatch was “trained” using 200,000 CP images. “Neural“ is likely a reference to the perceptual matching that it uses. It is not a bit for bit match.
Perceptual matching is a technique used for categorizing images based on characteristics and content.
The algorithm will scan your library containing new information and compare it to what it understands as CP.
No. NeuralMatch was “trained” using 200,000 CP images. “Neural“ is likely a reference to the perceptual matching that it uses. It is not a bit for bit match.
Perceptual matching is a technique used for categorizing images based on characteristics and content.
The algorithm will scan your library containing new information and compare it to what it understands as CP.
Microsoft has been using PhotoDNA [0] to scan OneDrive content for quite some time. The news is that a device manufacturer is doing it on your device with local data. There's some MS research to use machine learning on metadata to identify offending material [1].
[0] https://en.wikipedia.org/wiki/PhotoDNA
[1] https://arxiv.org/pdf/2010.02387.pdf
[0] https://en.wikipedia.org/wiki/PhotoDNA
[1] https://arxiv.org/pdf/2010.02387.pdf
Not a single joke was made about child abuse.
The specific subject of child abuse is irrelevant in my commentary. It was a commentary on the general category of AI, used all over, for many things, and more and more every day, but nice try.
Edit: You edited out what I was referring to as I was replying.
The specific subject of child abuse is irrelevant in my commentary. It was a commentary on the general category of AI, used all over, for many things, and more and more every day, but nice try.
Edit: You edited out what I was referring to as I was replying.
>"have the complainers suggest a solution"
Being worried about privacy, establishing a precedent for scanning my data against a government database, and the risk of false positives with such an insanely emotionally charged crime is more than mere complaining.
The onus should not be on me to justify why this shouldn't be done. This is something new and it is perfectly fine to argue against it without needing to provide an alternative.
That being said, my solution is to continue to follow the process that law enforcement is currently using.
Being worried about privacy, establishing a precedent for scanning my data against a government database, and the risk of false positives with such an insanely emotionally charged crime is more than mere complaining.
The onus should not be on me to justify why this shouldn't be done. This is something new and it is perfectly fine to argue against it without needing to provide an alternative.
That being said, my solution is to continue to follow the process that law enforcement is currently using.
>That being said, my solution is to continue to follow the process that law enforcement is currently using.
If you knew how they approach this you wouldn't be satisfied either
If you knew how they approach this you wouldn't be satisfied either
This all boils down to the classic "Liberty vs. Security" dilemma. While I obviously want to see more prosecutions for people who commit these crimes, a value judgement must be made about what it takes to achieve that.
[deleted]
> It's fuzzy hash, not ""AI"", based. Cloudflare uses it too, and last time i checked the web is still functional.
If they're using fuzzy matching with perceptual hashes, then the space that false positives can exist in for each perceptual hash is huge.
If they're using fuzzy matching with perceptual hashes, then the space that false positives can exist in for each perceptual hash is huge.
It makes sense for website owners to scan what's being uploaded to their servers, but that is totally different from scanning what's stored locally on people's devices.
There is no reference to fuzzy anything in any of the write ups. I don’t know how you can proclaim that this is a fuzzy hash. Where is your source?
> perfectly innocent photos on someone's phone of their own children
They are using a library of human verified images to compare the hashes. Probably similar to PhotoDNA[0]. There can be false positives, but AFAIK the algorithms are not trying to identify naked children, but comparing 2 similar image hashes.
[0] - https://en.wikipedia.org/wiki/PhotoDNA
They are using a library of human verified images to compare the hashes. Probably similar to PhotoDNA[0]. There can be false positives, but AFAIK the algorithms are not trying to identify naked children, but comparing 2 similar image hashes.
[0] - https://en.wikipedia.org/wiki/PhotoDNA
> They are using a library of human verified images to compare the hashes
I didn't get that from the article, could you point to a source on this?
I didn't get that from the article, could you point to a source on this?
Images are submitted by over 1400 organizations[0] like the FBI and NCMEC, although the inner-workings of the database aren’t public and I can’t find a source or confirmation of every image actually being human-verified. The closest is how the "Child Victim Identification Program" helps identify victims in these images.
0: https://www.missingkids.org/theissues/csam#:~:text=Electroni... (ctrl-f "1,400")
0: https://www.missingkids.org/theissues/csam#:~:text=Electroni... (ctrl-f "1,400")
[deleted]
Of all of the discussion, my brain has only keyed in on the child porn aspect. It just now clicked that if a missing kid is in a photo (even if not porn), it could help ID the missing kid. Some runaway at the age of 13 gets photographed in a group of other kids just doing teenager things.
So even if not running hashes against known porn, it could still be doing facial recognition. Wonder what kind of implications that would have for people in witness protection or battered spouses and other legitimately "disappeared" people.
So even if not running hashes against known porn, it could still be doing facial recognition. Wonder what kind of implications that would have for people in witness protection or battered spouses and other legitimately "disappeared" people.
Think of the children will erode everything. It is magic.
Appeal to emotion: https://en.wikipedia.org/wiki/Appeal_to_emotion
Directly said in the article:
“Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.”
“Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.”
> The system has been trained on 200,000 sex abuse images collected by the US non-profit National Center for Missing and Exploited Children.
Also from the article.
‘Trained’ being important.
Also from the article.
‘Trained’ being important.
It's still bad. I used to scrape pictures from many legal pornography sites. I still have hundreds of thousands of pictures. Since many are user uploaded, I imagine a small number could be real CP.
Thank God, I don't use Apple.
(I should probably delete them, since I've basically never used them... although they could be useful for one of those end-of-world Raspberry Pi builds: https://back7.co/home/raspberry-pi-quick-kit-one Edit: the non-bad ones, in case this gets misconstrued)
Edit: I've deleted my archive.
Thank God, I don't use Apple.
(I should probably delete them, since I've basically never used them... although they could be useful for one of those end-of-world Raspberry Pi builds: https://back7.co/home/raspberry-pi-quick-kit-one Edit: the non-bad ones, in case this gets misconstrued)
Edit: I've deleted my archive.
Yeah, might want to consider deleting them. That could turn into a nightmare.
I’ve never once wanted GIF support on HN comments until this moment. Good god man what is wrong with you
> Since many are user uploaded, I imagine a small number must be real CP.
I mean, that is actually illegal. The problem here is the possession of child pornography.
I mean, that is actually illegal. The problem here is the possession of child pornography.
[deleted]
get52(1)
If you think there's a reasonable chance that you have CP on your computer why wouldn't you delete all the stuff? This is not complicated.
cat /dev/urandom and it also has CP, oh noes! quick delete everything.
Hoarding porn is kind of useless nowadays, especially without any kind of selection and categorization, but maybe someone does it because it's their hobby (or obsession), who knows.
Also if a lot of people has a bunch of porn images scraped from forums it's quite possible that eventually some of that might become illegal as the laws change here or there around the world. Should they proactively delete everything? Sure, they could. To be safe.
But this kind of safety is absolutely the bad kind on the long term. (Though of course it's possible to advocate for reform while not storing content with uncertain legality.)
Hoarding porn is kind of useless nowadays, especially without any kind of selection and categorization, but maybe someone does it because it's their hobby (or obsession), who knows.
Also if a lot of people has a bunch of porn images scraped from forums it's quite possible that eventually some of that might become illegal as the laws change here or there around the world. Should they proactively delete everything? Sure, they could. To be safe.
But this kind of safety is absolutely the bad kind on the long term. (Though of course it's possible to advocate for reform while not storing content with uncertain legality.)
Dude, this is all great in theory but in a court of law you'll get completely horsefucked if you have child porn on your hard drive. The judge and jury won't care about /dev/random or a fall to reform
I bet(or know...) the people who verify those images don't get paid anywhere near enough.
I believe that US federal agents tasked with investigating these kinds of crimes typically only do so for a limited time.
The conventional wisdom for people integrating PhotoDNA is that false positives are vanishingly rare. If you run it against your data and you get a hit then you should call the FBI.
It would be better if the false positive rate was known to be nonzero.
And what, in this context, is "your data?" Is a photo on an iPhone Apple's data?
I'd argue that the thing you should do if you get a hit is to delete your data, unless you want to spend enormous amounts of time and money defending yourself from prosecution by people who believe that false positives are vanishingly rare.
And what, in this context, is "your data?" Is a photo on an iPhone Apple's data?
I'd argue that the thing you should do if you get a hit is to delete your data, unless you want to spend enormous amounts of time and money defending yourself from prosecution by people who believe that false positives are vanishingly rare.
I'm talking from the perspective of a service-owner who handles user-generated content. You typically set up PhotoDNA on two parts of your infrastructure. You scan client-side and refuse uploads to prevent the images from getting on your servers in the first place and then you scan on the backend to catch anything that slipped through. You do the first part so you don't have to call the FBI.
I really don't care about the semantics of data ownership. If Apple wants to scan photos you upload to iCloud so they don't run into a scandal years down the road that "iCloud is being used to distribute CP" then that's their call.
And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
I really don't care about the semantics of data ownership. If Apple wants to scan photos you upload to iCloud so they don't run into a scandal years down the road that "iCloud is being used to distribute CP" then that's their call.
And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
In the real world the system delete the data, deletes the associated user, bans the user and no human will do anything about it. Most likely there isn't even a report made to a human it just goes under ToS violation or something so no one inside the companies aks questions.
No FBI is called. Imagine the financial damage this would do if specialist come over and try to collect evidence and the company is forced to cooperated and work for them, for free ofc. Only the big tech does this. They have whole teams for this and automated system and what not because surely the FBI wont walk into googled data center ever to collect evidence. But all the smaller companies they cant afford to do this and the detection system is actually to protect them form going bankrupt because they accidentally stored some illegal data from a user.
>And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
Yes such laws exist. So they use filter software from third parties. They intentionally offer some generic filter that includes "unwanted content" and doesn't specify what it is when it find something so the companies can wave away the liability as they have no way to know if the image was valuable evidence of a crime or just some LiveLeak gore. They can justify that no human watches it because that's to expensive.
No FBI is called. Imagine the financial damage this would do if specialist come over and try to collect evidence and the company is forced to cooperated and work for them, for free ofc. Only the big tech does this. They have whole teams for this and automated system and what not because surely the FBI wont walk into googled data center ever to collect evidence. But all the smaller companies they cant afford to do this and the detection system is actually to protect them form going bankrupt because they accidentally stored some illegal data from a user.
>And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
Yes such laws exist. So they use filter software from third parties. They intentionally offer some generic filter that includes "unwanted content" and doesn't specify what it is when it find something so the companies can wave away the liability as they have no way to know if the image was valuable evidence of a crime or just some LiveLeak gore. They can justify that no human watches it because that's to expensive.
[deleted]
[deleted]
Changing the size or file type of the image would easily change the hash.
Microsoft implemented this a decade ago.
>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)
https://www.nbcnews.com/technolog/your-cloud-drive-really-pr...
There was another round of outrage when the Windows 10 TOS extended this to local storage.
It also has the use case of scanning for DRM violations. (ripped mp3 files or movies)
>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)
https://www.nbcnews.com/technolog/your-cloud-drive-really-pr...
There was another round of outrage when the Windows 10 TOS extended this to local storage.
It also has the use case of scanning for DRM violations. (ripped mp3 files or movies)
Again, that is not the same as scanning a local machine for it. That is all things that are on Microsoft's own servers or in their search results. Microsoft, Google, Cloudflare all have features like this. It's not the same as scanning local machines.
What will happen is that criminals who actually had bad intent will move off of the platform and the ones who get the brunt of the blame are the innocent who had no ill intent.
The process is described above, but it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
And the way it’s being done (hashes), a collision is highly unlikely. If it does occur it doesn’t mean it’s similar in nature (e.g. innocent picture of own child in bath). The hash isn’t looking at the image content in the sense of “what’s in the picture”, just the bits of the file. So it’s highly, highly unlikely, even if a collision occurs, that the collision would be an image that happens to be another child innocently bathing.
And the way it’s being done (hashes), a collision is highly unlikely. If it does occur it doesn’t mean it’s similar in nature (e.g. innocent picture of own child in bath). The hash isn’t looking at the image content in the sense of “what’s in the picture”, just the bits of the file. So it’s highly, highly unlikely, even if a collision occurs, that the collision would be an image that happens to be another child innocently bathing.
> The process is described above, but it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
Actually, it's very easy to end up with an image that has a similar perceptual hash to an illegal image.
They are not doing MD5 hashing, they're taking perceptual hashes and then using something like the hamming distance or Levenshtein distance to make a fuzzy match with hashes from illegal images.
I've built products using these methods, and it is incredibly easy to make a fuzzy match based on perceptual hashes from two images that have nothing to do with each other.
Actually, it's very easy to end up with an image that has a similar perceptual hash to an illegal image.
They are not doing MD5 hashing, they're taking perceptual hashes and then using something like the hamming distance or Levenshtein distance to make a fuzzy match with hashes from illegal images.
I've built products using these methods, and it is incredibly easy to make a fuzzy match based on perceptual hashes from two images that have nothing to do with each other.
> but it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
Are you sure, given how many iPhone takeover/jailbreak bugs regularly exist, including the recent 0-click iMessage bug? Would you like to dare a hacking group, domestic or foreign, to land a single verboten image on your iPhone?
> And the way it’s being done (hashes), a collision is highly unlikely.
The limited details so far are suggestive that its using perceptual hashes, which are more susceptible to collision-engineering/false-positives than cryptographically-secure hashes.
Are you sure, given how many iPhone takeover/jailbreak bugs regularly exist, including the recent 0-click iMessage bug? Would you like to dare a hacking group, domestic or foreign, to land a single verboten image on your iPhone?
> And the way it’s being done (hashes), a collision is highly unlikely.
The limited details so far are suggestive that its using perceptual hashes, which are more susceptible to collision-engineering/false-positives than cryptographically-secure hashes.
What happens if someone spams CP via iMessage to their enemies?
Additionally, this isn't just a hash of the file but a perceptual hash on the image content. So e.g. changing a single bit in the image would create a different cryptographic hash, but generally not a different perceptual hash.
Additionally, this isn't just a hash of the file but a perceptual hash on the image content. So e.g. changing a single bit in the image would create a different cryptographic hash, but generally not a different perceptual hash.
Presumably the verification that these false matches are not problematic is manual?
That's not good enough given how our media currently works. Imagine articles published when information about such a check leaks that say "celebrity X's phone checked by police for suspected CSAM." While that is the truth ("suspected") no one cares about that nuance and such a person would get cancelled very quickly, even if there was no evidence of wrongdoing.
That's not good enough given how our media currently works. Imagine articles published when information about such a check leaks that say "celebrity X's phone checked by police for suspected CSAM." While that is the truth ("suspected") no one cares about that nuance and such a person would get cancelled very quickly, even if there was no evidence of wrongdoing.
> it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
Of course, there's no telling what images will be in the database when this inevitably expands beyond CP.
Of course, there's no telling what images will be in the database when this inevitably expands beyond CP.
Presuming that is what they're doing, it won't stay that way for long.
It's absurdly easy to change the hash of a file. In the case of something like JPEG, you don't even have to change the file itself, you can just change the metadata. Apple could presumably only hash the image itself, but again, all you have to do is make tiny, imperceptible to humans changes to the image and the hash is totally different.
Long story short, this is either nearly pointless and privacy invasive, or it's about to get drastically more invasive to be effective.
It's absurdly easy to change the hash of a file. In the case of something like JPEG, you don't even have to change the file itself, you can just change the metadata. Apple could presumably only hash the image itself, but again, all you have to do is make tiny, imperceptible to humans changes to the image and the hash is totally different.
Long story short, this is either nearly pointless and privacy invasive, or it's about to get drastically more invasive to be effective.
It would be trivial for a bad intentioned actor to send those images to a target that he wanted to incriminate if the target is not tech savvy.
Furthermore, "highly unlikely" is an understatement. Using SHA256 there are 2^256 hashes.
That's 115792089237316195423570985008687907853269984665640564039457584007913129639936. Probability of collision (two different inputs generating the same hash) is infinitesimal.
That's 115792089237316195423570985008687907853269984665640564039457584007913129639936. Probability of collision (two different inputs generating the same hash) is infinitesimal.
Fairly sure criminals can't prove the efficient market hypothesis either. You still want to catch the dumb ones.
Just lobby Congress to make it illegal to switch to Android. Problem solved.
I actually do want a feature where I will get a notification on my phone if someone nearby has snagged a picture of me or my kids. A creepy shopper at Costco once took a pic of my 2-year-old. They pretended to be working up a conversation with my little kid meanwhile they had their phone at hip-level to snag a shot. I even confronted them about it. The Costco employees couldn't do a thing. Phones with cameras are useful but have also become a nuisance.
Edit: It makes sense that such a notification system would be impossible to implement. Perhaps, registered offenders should not only have their location placed on a map but their phones have certain privileges disabled or monitored.
Edit: It makes sense that such a notification system would be impossible to implement. Perhaps, registered offenders should not only have their location placed on a map but their phones have certain privileges disabled or monitored.
That's silly. You're in public. Your photo (and your kids') are being taken all the time.
It's not the case when you're inside Costco. I'd also argue that it's not open to the public because you're required to have a membership to go in.
The point is, Costco is taking many pictures of you, including possibly correlating with events or actions.
We accept Costco taking pictures of you as a condition of entering the store and securing a membership.
The same does not go for random creeps wanting to take close up pictures of a child.
The same does not go for random creeps wanting to take close up pictures of a child.
In this specific example, sure. You have no expectation of privacy in public and you are photographed, repeatedly.
The creep behavior is the problem. The photograph is a symptom of it.
The creep behavior is the problem. The photograph is a symptom of it.
Being on a security camera is very different from covertly photographing a kid from only a foot or two away. Why make such specious comparisons?
It may not be illegal in the US, but I wouldn't call it silly to not want strangers to take photos of you and your kid in public.
Or when an internet troll gets your email address and/or phone number and starts spamming you with child porn pictures, which your phone immediately identifies and notifies authorities.
And to get cleared you’re going to have to let law enforcement clone and inspect the content of your phone.
A completely new way of swating someone
Yeah if you ran the giant box of photos in my mom's closet through an AI you'd totally get hits from photos of baby me being bathed in the sink, baby me running around the back yard naked, etc. This can't end well.
If this even comes close to doing any “recognition” on local content and isn’t just (as the article says) a simple lookup of known images through hashes, then that would be a massive scandal. So I’m giving them the benefit of the doubt until it’s clarified, as the only confirmed way it works is the image hash lookup.
There are words like “neural” and “trained” here, but no clear sign that there is actually any kind of image recognition that would risk false positives other than hash collisions.
There are words like “neural” and “trained” here, but no clear sign that there is actually any kind of image recognition that would risk false positives other than hash collisions.
I think they are probably just going to match the hashes to known hashes of "bad" images.
Plain file hashing for this kind of thing has not been used for years (or even decades), Microsoft came out with Photo DNA along time ago, and I am sure now days they have something ML powered to do it.
I am not sure the accuracy of these systems but even if they are pretty accurate there is something off putting about this trend.
This very much feels like a Guilty until proven innocent type of program... "if you have nothing to hide you have nothing to fear"...
Neither of those normally work out really well.
I am not sure the accuracy of these systems but even if they are pretty accurate there is something off putting about this trend.
This very much feels like a Guilty until proven innocent type of program... "if you have nothing to hide you have nothing to fear"...
Neither of those normally work out really well.
> Guilty until proven innocent type of program
But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I agree though, the scary part might be how to clear your name after an overreaction occurs, similar to the effect swatting has on innocents or other kinds of false accusations. But those would happen even without such a system - we’ve heard about such stories before. “Man held for child porn charges, but it turned out they were innocent,” - it doesn’t take this kind of system to have police look over your phone for evidence and find a different crime than expected.
That all said… child porn is really, really terrible and I’m actually in favour of this. As a society, we can decide on what’s acceptable. As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes. I do agree though, all these systems might make it harder to catch perpetrators if it pushes them offline. But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I suppose the fundamental problem is that having more evidence doesn’t necessarily lead to justice if it paints a false picture.
But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I agree though, the scary part might be how to clear your name after an overreaction occurs, similar to the effect swatting has on innocents or other kinds of false accusations. But those would happen even without such a system - we’ve heard about such stories before. “Man held for child porn charges, but it turned out they were innocent,” - it doesn’t take this kind of system to have police look over your phone for evidence and find a different crime than expected.
That all said… child porn is really, really terrible and I’m actually in favour of this. As a society, we can decide on what’s acceptable. As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes. I do agree though, all these systems might make it harder to catch perpetrators if it pushes them offline. But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I suppose the fundamental problem is that having more evidence doesn’t necessarily lead to justice if it paints a false picture.
I see it as an unwarranted search, which makes me feel like the state views me as inherently guilty until innocence is proven by sifting through my photos.
It’s not the state, it’s Apple. And they are making the decision that they do not want to put out a product that makes it easy for CP to be shared or viewed. As a huge player in the market this is great.
This also limits the possible avenues for future exposure - if one has less ways to distribute or get CP or the overall amount of it is reduced, they may never get into it in the first place and the market for it will shrink. Thereby reducing the amount produced over time.
This also limits the possible avenues for future exposure - if one has less ways to distribute or get CP or the overall amount of it is reduced, they may never get into it in the first place and the market for it will shrink. Thereby reducing the amount produced over time.
According to the article, Apple is responding to government and law enforcement pressure to implement the scanning. The state is just laundering its desires through private corporations; there must be a word for this type of system.
Today they say it’s for CP, tomorrow they will say it’s for pirated content, or anything deemed illegal by the government.
Apple will be scanning the photos on your phone and cross-referencing them against photoDNA for a government database. You don't see how that's not a huge potential for abuse right there? The government decides the images. And you best believe they're not going to release the images publicly to verify what's in the db, for [many] obvious reasons.
This is a horrible idea, a massive regression in Apple (and therefore the broader industry's) stance on security/privacy, and the fact that you are dutifully swallowing the "child porn" justification is so incredibly concerning to me.
This is a horrible idea, a massive regression in Apple (and therefore the broader industry's) stance on security/privacy, and the fact that you are dutifully swallowing the "child porn" justification is so incredibly concerning to me.
The article says Apple will have your phone generate safety vouchers for images you upload into iCloud, and when enough of those vouchers are for photos that match a government-supplied blacklist, they will access the photos you’ve given them. How is this worse than what Google Photos and Facebook already do?
You think Facebook and Google Photos should be the goal for privacy standards? You and I are practically in different universes then, philosophically.
You're also being naive by swallowing the "it's only iCloud" justification. There is a clear intent to progress to scanning all local photos. After all, if photos must be match those of a government child abuse database, and any matches trigger manual review, what would an innocent person possibly have to hide? By that logic it'd be absurd to not scan just because it's only locally. After all, you're only catching bad guys! This is about protecting the children after all! /s
I oppose government-supplied blacklists. Doubly so when we can't even see what's in the database, but I would oppose them either way.
You're also being naive by swallowing the "it's only iCloud" justification. There is a clear intent to progress to scanning all local photos. After all, if photos must be match those of a government child abuse database, and any matches trigger manual review, what would an innocent person possibly have to hide? By that logic it'd be absurd to not scan just because it's only locally. After all, you're only catching bad guys! This is about protecting the children after all! /s
I oppose government-supplied blacklists. Doubly so when we can't even see what's in the database, but I would oppose them either way.
What advice would you give Apple if they’re faced with a legitimate government asking it to comply with the laws they have instituted in the society Apple wants to do business?
I don’t appreciate you calling me naive. There isn’t clear intent to scanning all local photos. In fact, it seems like the system was deliberately designed to only work with photos that are being uploaded to iCloud.
I don’t appreciate you calling me naive. There isn’t clear intent to scanning all local photos. In fact, it seems like the system was deliberately designed to only work with photos that are being uploaded to iCloud.
> What advice would you give Apple if they’re faced with a legitimate government asking it to comply with the laws they have instituted in the society Apple wants to do business?
When someone is trying to rob you, generally the best thing to do is to hand over your wallet. If a law passes you should obey it for your own sake. That's an entirely orthogonal question to whether we as the users should be outraged or not.
Also while obeying an injust law, a company has a responsibility to legally challenge it in court with their resources, if for no other reason than to avoid the profit loss of having to comply with invasive regulations
Furthermore, it's not clear that this is the direct result of some federal law. It reads more like pro-actively appeasing their federal overlords.
When someone is trying to rob you, generally the best thing to do is to hand over your wallet. If a law passes you should obey it for your own sake. That's an entirely orthogonal question to whether we as the users should be outraged or not.
Also while obeying an injust law, a company has a responsibility to legally challenge it in court with their resources, if for no other reason than to avoid the profit loss of having to comply with invasive regulations
Furthermore, it's not clear that this is the direct result of some federal law. It reads more like pro-actively appeasing their federal overlords.
> What advice would you give Apple if they’re faced with a legitimate government asking it to comply with the laws they have instituted in the society Apple wants to do business?
It is good advice not to carry around huge amounts of money on your wallet. Someone might want it and have the power to get it.
Same with megacorps constructing and bragging about tools for oppression:
Better not do it.
Otherwise just be prepared that China will demand that you also upload their hashes of extremely bad photos - and one or two guys they want to get hold of.
It is kind of a Manhattan project for digital privacy - it is meant to bring a quick end to a actual real problem we have - but it has the potential to become a shadow over society for decades if not centuries to come.
It is good advice not to carry around huge amounts of money on your wallet. Someone might want it and have the power to get it.
Same with megacorps constructing and bragging about tools for oppression:
Better not do it.
Otherwise just be prepared that China will demand that you also upload their hashes of extremely bad photos - and one or two guys they want to get hold of.
It is kind of a Manhattan project for digital privacy - it is meant to bring a quick end to a actual real problem we have - but it has the potential to become a shadow over society for decades if not centuries to come.
> But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I doubt they need, or want to match more than one. They'll use the reasoning that it only catches already-known child pornography, and that that person might have a whole cache of novel child pornography. Just look at the drug wars and how many people get arrested for trivial quantities of marijuana. And we hate child pornography way more than drugs.
> As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes.
The judicial system has a presumption of innocence. No one else does, or has any requirement to. The court of public opinion will ruin your life regardless of whether it's true or not.
That would be far less concerning if active cases were sealed, and only published if the defendant was ruled guilty.
> That all said… child porn is really, really terrible and I’m actually in favour of this.
I have a slight disagreement with this. The production of child pornography is obviously abhorrent. This system does nothing to stop that though, because it can only flag already-known child pornography.
This system stops the consumption of already produced and documented child pornography. While it's far outside my own interests, I can see some validity to the argument that consumption is of far less concern because it doesn't directly involve any abuse. The abuse has already happened, and no amount of tracking down pedophiles is going to reverse that. There are second order effects, like potentially reducing the demand for novel child pornography, though. Which leads me to this:
> But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I would imagine this is far more likely to affect consumers than producers. In the drug market, stop and frisk is far more likely to net you users than producers or dealers. For one, there are more users than dealers/producers. For two, dealers, but particularly producers, are typically far more cautious than users.
That's a long-winded way of saying again that this doesn't seem likely to do much in the way of stopping people from abusing children.
But to directly answer your question: doing nothing is absolutely a valid option here. We do that for lots of bad things that happen when the costs of the solution are unbearable. It's what we're currently doing for famine in other parts of the world, which almost surely impacts more children than child pornography could even dream of doing. It's what we do for cars, which again, almost certainly impacts more children than child pornography does.
We wait until we can find a method with benefits commensurate to its costs. A 2016 DoJ report said that since 2002, 10,500 victims have been located and identified by law enforcement. That's 750 per year. There are 74 million children under 18 in the US, so that's 0.001% of children per year. 10,500 people is only 200 people higher than died from "accidental suffocation and strangling in bed" over the same period according to the CDC. The scale of the problem is the same, so should we go ahead and let Amazon send pings to the police if your Alexa thinks your suffocating in bed too?
I doubt they need, or want to match more than one. They'll use the reasoning that it only catches already-known child pornography, and that that person might have a whole cache of novel child pornography. Just look at the drug wars and how many people get arrested for trivial quantities of marijuana. And we hate child pornography way more than drugs.
> As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes.
The judicial system has a presumption of innocence. No one else does, or has any requirement to. The court of public opinion will ruin your life regardless of whether it's true or not.
That would be far less concerning if active cases were sealed, and only published if the defendant was ruled guilty.
> That all said… child porn is really, really terrible and I’m actually in favour of this.
I have a slight disagreement with this. The production of child pornography is obviously abhorrent. This system does nothing to stop that though, because it can only flag already-known child pornography.
This system stops the consumption of already produced and documented child pornography. While it's far outside my own interests, I can see some validity to the argument that consumption is of far less concern because it doesn't directly involve any abuse. The abuse has already happened, and no amount of tracking down pedophiles is going to reverse that. There are second order effects, like potentially reducing the demand for novel child pornography, though. Which leads me to this:
> But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I would imagine this is far more likely to affect consumers than producers. In the drug market, stop and frisk is far more likely to net you users than producers or dealers. For one, there are more users than dealers/producers. For two, dealers, but particularly producers, are typically far more cautious than users.
That's a long-winded way of saying again that this doesn't seem likely to do much in the way of stopping people from abusing children.
But to directly answer your question: doing nothing is absolutely a valid option here. We do that for lots of bad things that happen when the costs of the solution are unbearable. It's what we're currently doing for famine in other parts of the world, which almost surely impacts more children than child pornography could even dream of doing. It's what we do for cars, which again, almost certainly impacts more children than child pornography does.
We wait until we can find a method with benefits commensurate to its costs. A 2016 DoJ report said that since 2002, 10,500 victims have been located and identified by law enforcement. That's 750 per year. There are 74 million children under 18 in the US, so that's 0.001% of children per year. 10,500 people is only 200 people higher than died from "accidental suffocation and strangling in bed" over the same period according to the CDC. The scale of the problem is the same, so should we go ahead and let Amazon send pings to the police if your Alexa thinks your suffocating in bed too?
So are you in Favor of installing Camera's in every room in your home, that you do not control, and that the police and review anytime?
If in the US Do you like the concept behind 4th amendment? being free from unreasonable searches unless probable cause can be articulated that you have committed a crime?
It is very scary how effect the "think of the children" narrative is to get people to willing give up their liberty
If in the US Do you like the concept behind 4th amendment? being free from unreasonable searches unless probable cause can be articulated that you have committed a crime?
It is very scary how effect the "think of the children" narrative is to get people to willing give up their liberty
They work out very well. Any significant storage provider has been using them for years.
> I really don't see how this is going to end well, there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things that kids do. Like a kid running butt naked around the house, or a photo of something like a rash that is sent to a nurse friend for advice on what it is etc.
These are still illegal in the US right now right? It's been a major criticism of these laws that they can hurt people accidentally.
These are still illegal in the US right now right? It's been a major criticism of these laws that they can hurt people accidentally.
I don't think just nudity was illegal ever. But just suspicion can hurt people.
I am sure, presumbly in a sane world, computer scanning is only the early stage of the process, human will be invovled to make final judgements and decisions.
Remember the US No Fly List? https://en.wikipedia.org/wiki/No_Fly_List
The list ballooned to over a million names, then required an ACLU lawsuit until humans became involved and a review process was initiated.
As long as the majority of Americans dont feel burdened by something, dont expect anything logical to happen.
The list ballooned to over a million names, then required an ACLU lawsuit until humans became involved and a review process was initiated.
As long as the majority of Americans dont feel burdened by something, dont expect anything logical to happen.
I imagine the biggest catch of this will be teens taking inappropriate photos of themselves because teens often do things like this in their awkward years when sexuality and body consciousness kicks in. With this we'll be seeing warrants for those images and a big fuss made on this most suicidal age group who thought they were doing something completely private. This isn't good.
Yes, not no to mention the gulf that exist between american culture of what is innocent and acceptable and the rest of the world.
If we apply the american paranoid standards to the rest of the world, i am pretty sure we will have plenty of false positives.
What I don't understand is how is it even constitutional in the US. Isn't it a warrant-less search? It's not like scanning content on a cloud. The cloud owns the device on which you store your data. But you own your mobile.
Not if you give permission, which you do by your purchase agreement.
You're correct, but I would love to see a TOS that gives Apple permission to exercise cruel and unusual punishment for violating their terms. The eighth amendment only applies to the government, after all! /s?
The constitution constrains what the Federal government can do, not what private companies can do.
Not that I am qualified to hold a legal debate, but as far as I know, if Apple implements this under the pressure from the state, and that it reports what it findings to the state, it is effectively an agent of the state and it should apply.
As far as I’m aware, the laws which require cloud service operators to scan for CSAM have been on the books for a long time, and none of the usual orgs have raised a constitutional challenge to them. I’m not a legal expert either, though.
But that's the big difference, cloud providers are scanning the data they hold on their device. Your iphone is not Apple's device. Cloud providers have a liability if they knowingly store illegal content, the data is stored on their property, Apple does not, it is your property. It's like if Ford was going to scan what you store in the trunk of your car.
"Your iphone is not Apple's device". Well, is it not ? When you "jailbreak" it, who has the key of the cage ? How many network exchange does your phone with Apple do every day, every hour, every second ?
Oh, the irony. Apple users will finally demand that their phone be "their" device and won't say that they are perfectly happy with Apple making decisions for them because they absolutely trust Apple
The headline is misleading. The rumored change only impacts photos your device is uploading to iCloud, and it seems like Apple servers are a necessary part of the protocol.
Doesn't apple upload all your photos to icloud by default?
It’s been a long time since I’ve set up my phone, so I don’t remember the defaults. If I recall correctly, photo stream (last ~100? photos) was on by default but not the entire photo library.
Under orders yes. Under pressure no.
The Constitution applies only to the government. A user's relationship with Apple is not in any way governed by the Constitution, just like how Twitter banning people who violate the terms of service is not a violation of the first amendment to the Constitution.
Well that is under the false assumption of you own your mobile.
Which isn't true for Apple. Not too sure about Android. But everything you do on your Apple Devices, including its Hardware, Software and its Services, works more like Apple lend it to you for a price.
And as other mention, only the government and not a private company.
Which isn't true for Apple. Not too sure about Android. But everything you do on your Apple Devices, including its Hardware, Software and its Services, works more like Apple lend it to you for a price.
And as other mention, only the government and not a private company.
They need to stop calling it an iPhone. It's a govPhone. It now belongs to the government with "In-built Citizen Scanning" as free civic service. Thank you for serving the nation!
there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things
How would that match a hash list of known CP imagery though?
How would that match a hash list of known CP imagery though?
Apparently, there's a Threshold Secret Sharing (TSI) technology in place to eliminate false positive. Besides, NeuralHash will basically match your photos against a database provided by NCMEC.
[deleted]
And you can imagine what's on a dermatologist or urologist phone!
[deleted]
We really shouldn't be moving the conversation window into the technicalities just yet without discussing if this is even ok or reasonable in the first place.
If the photo of your kid running around the house ends up in a CP database, you messed up before Apple did.
> Like a kid running butt naked around the house...
Thought it was buck naked — but that works too. :-)
Thought it was buck naked — but that works too. :-)
I would suggest that if it is widely used, it isn't an error, regardless of what Professor Brians thinks.
This system isn't meant for child abuse.
It's going to be used in China against enemies of the state.
Then it will be used domestically for the same.
Apple is a horrific company that you should stop using.
It's going to be used in China against enemies of the state.
Then it will be used domestically for the same.
Apple is a horrific company that you should stop using.
https://towardsdatascience.com/black-box-attacks-on-perceptu...
I mean, who's to say that my hash is actually what they think it is...
...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone?
"AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!"
"DEFENDANT: I'm not going to give you my phone's password."
"THE COURT: Bonk: Stay in jail for contempt."
I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
I mean, who's to say that my hash is actually what they think it is...
...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone?
"AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!"
"DEFENDANT: I'm not going to give you my phone's password."
"THE COURT: Bonk: Stay in jail for contempt."
I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
I have always been concerned that this system could be weaponized as a way gain access to someone's account. For example:
- Add the hash of a non-pornographic image to the database
- Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it.
- The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
- Add the hash of a non-pornographic image to the database
- Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it.
- The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
This is how the police gets warrantless searches "for drugs" - having identified the subject, call for a dog and have it "alert" on the subject (or the vehicle belonging to the subject), after which there's a probable cause for a search and everything that is found can be used against the subject. The fact that there was no drugs is irrelevant.
Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you.
But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you.
But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
EDIT: It appears they are using perceptual hashes, which are likely much more prone to collisions. The following math does NOT apply to the tech Apple is using.
>> I mean, who's to say that my hash is actually what they think it is...
Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1]
"The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate.
And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches)
1. https://en.wikipedia.org/wiki/Universally_unique_identifier#...
>> I mean, who's to say that my hash is actually what they think it is...
Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1]
"The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate.
And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches)
1. https://en.wikipedia.org/wiki/Universally_unique_identifier#...
> "The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years."
If we're talking about random files, sure.
What if the files are intentionally created to match the desired hash? The malicious actor doing this could be a private party intending to disrupt your life, but it could also be law enforcement as a means of gaining access to your device when it would otherwise require a warrant.
If we're talking about random files, sure.
What if the files are intentionally created to match the desired hash? The malicious actor doing this could be a private party intending to disrupt your life, but it could also be law enforcement as a means of gaining access to your device when it would otherwise require a warrant.
Couldn't someone that motivated come up with a dozen other ways to frame someone?
This seems a little alarmist to me.
This seems a little alarmist to me.
They could come up with a dozen other ways apart from swatting too, but swatting still happens and we generally agree that it's a problem where actions should be taken. Political debates over whether that means the police should act in a less militarised fashion, or harsher sentences for culprits are ongoing, but cutting down on avenues for life ruining abuse is generally supported.
So why would we open up new ones?
So why would we open up new ones?
SWATing is actually the absolute cheapest option - if you want to ruin someone's week from across the nation (or across the ocean), your best bet is a burner phone calling the person's local 911 with a threat that warrants a SWAT team response (I am of course not advocating for anyone to do this, just explaining the process malicious actors currently use to trigger these). Outside of that, you'd need to pay a lot of money for something like a PI or obfuscated malware to steal PII or credentials from the victim.
I hope you don't ever run afoul of an internet mob from somewhere like 4chan! They might SWAT you while DDOSing your IP and doxing you then emailing your iphone dozens of photos that collide with CSAM perceptual hashes.
LEO have never, ever planted a baggie of weed in someone's car. Likewise, LEO would never, ever ensure that a picture that matched a known phash gets on a target's phone.
Once someone develops the tool for that, it WILL become available in the wild. Then, any script kid will be able to "swat" you in a even more insidious way.
> What if the files are intentionally created to match the desired hash?
That's called a second-preimage attack and is similarly infeasible to achieve against a cryptographically secure hash function.
There are good arguments to be made for why this is a bad idea, but hash collisions aren't one of them.
That's called a second-preimage attack and is similarly infeasible to achieve against a cryptographically secure hash function.
There are good arguments to be made for why this is a bad idea, but hash collisions aren't one of them.
But this isn't your average hash function.
This is a function that is created to be robust against modifications to the image.
A perfectly robust function would - in my opinion - be one that could detect even a recreation of the original setup.
I don't know where on this scale the function law enforcement use is, but if it is any good I struggle to see how it cannot detect similar legal imagesm
This is a function that is created to be robust against modifications to the image.
A perfectly robust function would - in my opinion - be one that could detect even a recreation of the original setup.
I don't know where on this scale the function law enforcement use is, but if it is any good I struggle to see how it cannot detect similar legal imagesm
Or what if you're a researcher who has downloaded a collection of these assets? Wouldn't you be a very high signal hit, beyond even actual perpetrators of crime? Would you need the protection of an umbrella organization to do research? Or register with the government perhaps?
I'm as against this as anyone,but to be blunt:
This is digital biowarfare material of the worst kind and should be handled as such.
Any researcher downloading this material to anything but lab machines should probably receive a visit from the relevant authorities to make sure they understand the seriousness of it.
This is digital biowarfare material of the worst kind and should be handled as such.
Any researcher downloading this material to anything but lab machines should probably receive a visit from the relevant authorities to make sure they understand the seriousness of it.
The law doesn't have an exception for independent research.
They're doing perceptible hashes, not file hashes. This was covered earlier in a different article posted here.
What if an attacker has intentionally produced an image with the same hash as a well known blacklisted image?
Then they could send you the new image and fuck up your life.
Then they could send you the new image and fuck up your life.
Hell, what if you were browsing a site like Facebook or Imgur or Reddit, saw an abuse image, reported it, but it still lives in your phone's browser cache?
But compare practice to DMCA Takedown notices. Lots of false positives, because there's no penalty for mistakes.
Fifth amendment holds that the state can't compel someone to give up their password.
Except 100 miles from the border apparently
https://www.aclu.org/other/constitution-100-mile-border-zone
https://www.aclu.org/other/constitution-100-mile-border-zone
The 100 mile zone is clearly fucked up, but I haven't seen any cases of it being used to force anyone to give up their password. Even with a warrant US citizens do not have to give up their password. The only exception I've seen is a case where the defendant shared some info on their phone but refused to share everything. The judge ruled the defendant waived his fifth amendment rights in that case. I think it's clear that that was the wrong decision, but the solution is clear and easily done, don't give cops any information.
But you can be jailed for 18 months for not giving it up
Not if your lawyer is any good.
https://arstechnica.com/tech-policy/2020/02/man-who-refused-...
Nope, federal law states they can hold you for 18 months. So long as they can claim to already know what's on the device (which they will, because Apple pinged them about it), they can hold you for 18 months.
Nope, federal law states they can hold you for 18 months. So long as they can claim to already know what's on the device (which they will, because Apple pinged them about it), they can hold you for 18 months.
Alright, after further investigation it looks like the Supreme Court has so far declined to rule on this issue so it depends on the state. It sounds like most states supreme courts have ruled that you can't be forced to unlock a device, but some, like pennsylvania say that if the state can prove you know the password they can force you to unlock the device.
Definitely a pretty thorny situation.
Definitely a pretty thorny situation.
But that hasn't stopped judges from demanding it and keeping people in jail for contempt of court when they refuse.
I understand the danger of the false positive but I suspect someone with actual child pornography on their device would trigger a dozen or more matches. Clever software would ignore a single match.
If it matches, am I guilty until found innocent?
I have no illicit images, but false positives are always going to be a problem and even at sub-1% rates if you're scanning literally every image stored on an iOS devices that could still be thousands of wrong matches. If I lose the false-positive lottery, am I going to have the police calling and have my mugshot in the evening news for "CP on their device" in particular in my state's months long backlog for digital forensics. Law enforcement aren't exactly known for their understanding of technical evidence like hash-matching.
I'm very sympathetic/supportive of anti-CP initiatives in general, but this still seems problematic.
PS - And before someone replies with "they aren't contacting the police," don't forget the "yet." Once this is in place they'll immediately get pressured by politicians and well-intentioned people alike to flag any positive detections to authorities who will overreact (as law enforcement has a history of doing, particularly around this topic).
I have no illicit images, but false positives are always going to be a problem and even at sub-1% rates if you're scanning literally every image stored on an iOS devices that could still be thousands of wrong matches. If I lose the false-positive lottery, am I going to have the police calling and have my mugshot in the evening news for "CP on their device" in particular in my state's months long backlog for digital forensics. Law enforcement aren't exactly known for their understanding of technical evidence like hash-matching.
I'm very sympathetic/supportive of anti-CP initiatives in general, but this still seems problematic.
PS - And before someone replies with "they aren't contacting the police," don't forget the "yet." Once this is in place they'll immediately get pressured by politicians and well-intentioned people alike to flag any positive detections to authorities who will overreact (as law enforcement has a history of doing, particularly around this topic).
"I have no illicit images, but false positives are always going to be a problem"
Or someone that doesn't like you can MMS/iMessage/e-mail you images that match PhotoDNA image hashes. Maybe even in a way where the images don't appear to be anything alarming.
Or someone that doesn't like you can MMS/iMessage/e-mail you images that match PhotoDNA image hashes. Maybe even in a way where the images don't appear to be anything alarming.
Yep, Google can break SHA1, so what hash are they using? Proprietary information I assume.
They aren't using file hashes like SHA1, but perceptual hashes and then a fuzzy metric like hamming distance to make a fuzzy match. It's entirely possible for two images that have nothing to do with each other to have similar hashes.
Wow. That's much worse.
Yes, that's what I was alluding to. It's a perceptual hash, so you could probably take a flagged image and superficially alter it (contrast/color/crop/whatever) until it wasn't anything visually objectionable on it's own. But it would still match the perceptual hash.
The poor souls that do the work of entering the images into the database select identification blocks, not showing the victim (a lamp, wall paper, etc), that reviewers on the “client” side check against
In the United States, the process of reporting potential images of CSAM is defined by the government, and a technology company’s active responsibility ends with a report to the National Center for Missing and Exploited Children’s CyberTipline: https://www.missingkids.org/gethelpnow/cybertipline#whathapp...
From the article:
> The automated system would proactively alert a team of human reviewers if it believes illegal imagery is detected, who would then contact law enforcement if the material can be verified. The scheme will initially roll out only in the US.
This is dystopian and needs to be opposed, not applauded.
> The automated system would proactively alert a team of human reviewers if it believes illegal imagery is detected, who would then contact law enforcement if the material can be verified. The scheme will initially roll out only in the US.
This is dystopian and needs to be opposed, not applauded.
This is how every single cloud service has worked for a decade. You’ll need to elaborate on how, exactly, you believe it’s dystopian. The article is clearly wrong, because Apple cannot know which jurisdiction to forward a report to. That is NCMEC’s responsibility.
This is on your device, not hosted by a third party (and therefore not a third party liability). Kind of like if your microwave was checking that there wasn't any illegal activity happening in your home and phoning home if it has a suspicion.
As others mentioned, privacy erosion only goes one way. This is a frontier being breached, i.e. searching your physical device without a warrant, and as usual in the name of CP (the alternative typical pretext being terrorism). It won't take very long before this gets extended to any illegal activity. Gay dating was an illegal activity until not so long ago. This sort of chinese-style monitoring is not desirable in any way, and I find it appalling that Apple seems to want to be the sponsor.
As others mentioned, privacy erosion only goes one way. This is a frontier being breached, i.e. searching your physical device without a warrant, and as usual in the name of CP (the alternative typical pretext being terrorism). It won't take very long before this gets extended to any illegal activity. Gay dating was an illegal activity until not so long ago. This sort of chinese-style monitoring is not desirable in any way, and I find it appalling that Apple seems to want to be the sponsor.
No it’s not. Read the article: the proposed mechanism is implemented as part of photo upload to iCloud.
The title says "scan US phones", the article is behind a paywall.
Thanks. The article confirms the title, this is not about content uploaded to the cloud:
Apple’s system is less invasive in that the screening is done on the phone, and “only if there is a match is notification sent back to those searching”, said Alan Woodward, a computer security professor at the University of Surrey. “This decentralised approach is about the best approach you could adopt if you do go down this route.”
Apple’s system is less invasive in that the screening is done on the phone, and “only if there is a match is notification sent back to those searching”, said Alan Woodward, a computer security professor at the University of Surrey. “This decentralised approach is about the best approach you could adopt if you do go down this route.”
https://twitter.com/matthew_d_green/status/14231090022805135... and https://twitter.com/matthew_d_green/status/14231094582936289... kind of sums it up. Clearly there is an intent to progress to more than just what's been shared with iCloud. Otherwise there'd be no need for software on the devices themselves at all.
That is not the only explanation. Involving the device could be considered better because it doesn’t require Apple to build infrastructure to scan all your photos in the cloud, which could be more easily repurposed without user consent.
They scan photos in the cloud now. And do you think Apple will announce and let users opt out of client side scanning changes?
The slant I took from that thread is this is the way to comply with known-bad-hash matching requirement while preserving E2E encryption for cloud-stored content: checking hash at an end.
Don’t need to MITM when you put an M at an end.
Don’t need to MITM when you put an M at an end.
iCloud photos aren't E2E encrypted now. This could be a way to add E2E encryption while preserving the current level of surveillance. Except the article said Apple can decrypt suspect photos. And it isn't a legal requirement apparently.[1]
[1] https://news.ycombinator.com/item?id=28077619
[1] https://news.ycombinator.com/item?id=28077619
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
AWS doesn't do any automatic fingerprinting of files being uploaded to S3 and never has. The only time I've heard of anything like this was briefly when Microsoft was doing it, and I don't know if that was shutdown long ago due to outrage or not.
AWS doesn’t have to because the entity which writes those files is obligated to do this scanning if there is reason to believe the files contain CSAM.
I guess I should clear up: Amazon isn't doing any type of automatic file-scanning for anything being uploaded to S3, or feeding md5/sha1/sha256sums to some 3rd party service automatically for everything that touches S3.
If they get a subpoena and then maybe send some kind of file list/checksums they've been required to hand over to some 3rd party / government? That's possible, but an entirely different situation.
If they get a subpoena and then maybe send some kind of file list/checksums they've been required to hand over to some 3rd party / government? That's possible, but an entirely different situation.
Sure, but whoever owns the account which wrote those files in is, in the United States, responsible for doing some version of what Apple has announced they will do with iCloud Photos today.
It's dystopian because a private company, who in the past has been committed to maintaining user privacy even when it inconveniences the government, is now regressing and going to scan the "private" contents of peoples' phones to see if they have any matches for images that have been added to a government-supplied blacklist. This blacklist is completely opaque to us as citizens; there is no way to audit or review the contents. Additionally because it's using photoDNA, if someone ever did get ahold of even a single entry in the database, they could craft malicious images that fool the perceptual hash algorithm into flagging an innocent image. (Yes, I get there will be manual review, etc).
Furthermore in principle there's no reason this has to be, or will be, just CP. First it'll expand to blatant terroristic images, say ISIS beheading type videos. Then it will progress to subversive material: COVID "misinformation", "domestic terrorism" (read: anyone the feds don't like), the list goes on. This tool and its successors will be used to completely abolish the concept of device privacy and will be used to ruin the lives of anyone who sufficiently pisses off the feds. And that's just in America, god knows what they're going to do with this system in China.
Furthermore in principle there's no reason this has to be, or will be, just CP. First it'll expand to blatant terroristic images, say ISIS beheading type videos. Then it will progress to subversive material: COVID "misinformation", "domestic terrorism" (read: anyone the feds don't like), the list goes on. This tool and its successors will be used to completely abolish the concept of device privacy and will be used to ruin the lives of anyone who sufficiently pisses off the feds. And that's just in America, god knows what they're going to do with this system in China.
1. Apple has only stated a commitment to protecting user privacy to the extent it is legal. They make a lot of suspicious operational changes to how they operate in China, for example. Their opposition to the FBI case was because they believed the request placed on them was illegal.
2. This article only provides evidence to say Apple will scan photos you upload into iCloud. Scanning local photos is your speculation.
3. The operation of NCMEC and the blacklist is defined by the Federal government. American (citizens) have some mechanism to oppose this design. Apple could perhaps lobby against it, but their lobbying operations are famously minimal.
4. In principle, Apple could do anything. That doesn’t really inform what they’re likely to do. Just like in principle, anyone could slip a malicious content scanning patch into the Linux kernel (and this already has a POC!).
2. This article only provides evidence to say Apple will scan photos you upload into iCloud. Scanning local photos is your speculation.
3. The operation of NCMEC and the blacklist is defined by the Federal government. American (citizens) have some mechanism to oppose this design. Apple could perhaps lobby against it, but their lobbying operations are famously minimal.
4. In principle, Apple could do anything. That doesn’t really inform what they’re likely to do. Just like in principle, anyone could slip a malicious content scanning patch into the Linux kernel (and this already has a POC!).
> will initially roll out only in the US.
Ok so they’re planning to eventually roll out this spyware worldwide. I do not know how Americans can think China is bad in this while allowing this to happen.
Ok so they’re planning to eventually roll out this spyware worldwide. I do not know how Americans can think China is bad in this while allowing this to happen.
Never ceases to amaze how people believe that China is somehow more oppressive than the the country with world highest incarceration rate.
The US has a high incarceration rate because it has a high crime rate. If EU countries had US levels of crime, they'd have US levels of incarceration.
Also China doesn't count Uyghurs as incarcerated despite the fact that the PRC is committing genocide against them.[1] The PRC imprisons almost 2 million of Uyghurs a year. Of those, around a quarter million die. Oh and there's the forced sterilizations, destruction of mosques, brainwashing, banning of muslim-sounding names[2], forced labor, organ harvesting, and mandatory quartering of government agents.[3]
1. https://en.wikipedia.org/wiki/Uyghur_genocide
2. https://en.wikipedia.org/wiki/Naming_laws_in_China
3. https://en.wikipedia.org/wiki/Civil_Servant-Family_Pair_Up
Also China doesn't count Uyghurs as incarcerated despite the fact that the PRC is committing genocide against them.[1] The PRC imprisons almost 2 million of Uyghurs a year. Of those, around a quarter million die. Oh and there's the forced sterilizations, destruction of mosques, brainwashing, banning of muslim-sounding names[2], forced labor, organ harvesting, and mandatory quartering of government agents.[3]
1. https://en.wikipedia.org/wiki/Uyghur_genocide
2. https://en.wikipedia.org/wiki/Naming_laws_in_China
3. https://en.wikipedia.org/wiki/Civil_Servant-Family_Pair_Up
US has a high crime rate, because it has laws introduced specifically to increase the crime rate in specific racial groups; that’s where the War on Drugs came from.
China doesn’t count Uyghur extremists as incarcerated for the same reason the US doesn’t count their brethren it kills en masse in Middle East as death sentences. It’s just a different thing.
As for the rest of the list - most of it is just US state propaganda calling resocialization programs “brainwashing”, or leveling commercial buildings built without building permit “destruction of churches”. Some of that works exactly the same in US (forced labor), or is just entirely made up (“quarter million die”).
Now, some of the stuff is a real problem of course - death sentences and organ harvesting - but throwing it together with the usual US propaganda doesn’t help the cause.
China doesn’t count Uyghur extremists as incarcerated for the same reason the US doesn’t count their brethren it kills en masse in Middle East as death sentences. It’s just a different thing.
As for the rest of the list - most of it is just US state propaganda calling resocialization programs “brainwashing”, or leveling commercial buildings built without building permit “destruction of churches”. Some of that works exactly the same in US (forced labor), or is just entirely made up (“quarter million die”).
Now, some of the stuff is a real problem of course - death sentences and organ harvesting - but throwing it together with the usual US propaganda doesn’t help the cause.
> US has a high crime rate, because it has laws introduced specifically to increase the crime rate in specific racial groups; that’s where the War on Drugs came from.
That's not true. If you released everyone convicted of a drug crime (including manufacturing, selling, or distributing), you'd decrease the US prison population by 20%.[1] I'm all in favor of ending the drug war, but that won't do much to the incarceration rate. Also, the drugs that are illegal in the US are just as illegal in other developed countries (if not moreso). To use one example: in the UK, you can get up to 7 years in prison for simply possessing LSD or shrooms, and life in prison for for supplying or producing them.[2] No european country has legalized recreational marijuana[3], while the state I live in legalized it a decade ago. Now look at China's drug laws. While in China I've been solicited for all kinds of illegal goods and services, but never drugs. Stolen or fake merchandise? Sure. Prostitution? Of course. But drugs? Not a chance. The reason for this is that China executes people for drug possession. They randomly test foreigners entering the country and prosecute them even if they have no drugs on them.[4] The government raids nightclubs and residences, then forces everyone to take drug tests and prosecutes them if they test positive.
But please go on about how oppressive the US is.
I've seen you comment on China several times before and it's always in favor of the PRC. This confuses the hell out of me. Consider this: We could not have this discussion in China. To contradict the official narrative there is to risk your career and your freedom. Yes, the US and other countries have big problems, and it's good to try and fix them. But the level of surveillance and oppression in China is unlike anything in the west.
1. https://www.prisonpolicy.org/reports/pie2020.html#firstmyth
2. https://www.gov.uk/penalties-drug-possession-dealing
3. https://commons.wikimedia.org/wiki/File:Map-of-world-cannabi...
4. https://www.gov.uk/foreign-travel-advice/china/local-laws-an...
That's not true. If you released everyone convicted of a drug crime (including manufacturing, selling, or distributing), you'd decrease the US prison population by 20%.[1] I'm all in favor of ending the drug war, but that won't do much to the incarceration rate. Also, the drugs that are illegal in the US are just as illegal in other developed countries (if not moreso). To use one example: in the UK, you can get up to 7 years in prison for simply possessing LSD or shrooms, and life in prison for for supplying or producing them.[2] No european country has legalized recreational marijuana[3], while the state I live in legalized it a decade ago. Now look at China's drug laws. While in China I've been solicited for all kinds of illegal goods and services, but never drugs. Stolen or fake merchandise? Sure. Prostitution? Of course. But drugs? Not a chance. The reason for this is that China executes people for drug possession. They randomly test foreigners entering the country and prosecute them even if they have no drugs on them.[4] The government raids nightclubs and residences, then forces everyone to take drug tests and prosecutes them if they test positive.
But please go on about how oppressive the US is.
I've seen you comment on China several times before and it's always in favor of the PRC. This confuses the hell out of me. Consider this: We could not have this discussion in China. To contradict the official narrative there is to risk your career and your freedom. Yes, the US and other countries have big problems, and it's good to try and fix them. But the level of surveillance and oppression in China is unlike anything in the west.
1. https://www.prisonpolicy.org/reports/pie2020.html#firstmyth
2. https://www.gov.uk/penalties-drug-possession-dealing
3. https://commons.wikimedia.org/wiki/File:Map-of-world-cannabi...
4. https://www.gov.uk/foreign-travel-advice/china/local-laws-an...
Sentencing for drug possession has secondary effects - it pushes large number of people into other kinds of crime. Thus, you cannot count only the direct 20%.
Regarding other countries - what would happen (and happened) to countries that refused to join War on Drugs?
Regarding possession - in UK essentially nobody cares, unless you’re distributing industrial quantities and/or are really obnoxious about it, eg walk into Constabulary smoking a joint, or by putting up advertisements. There are regular news about eg police finding a suspect car parked for six hours with engine on, filled with smoke (actual example I remembered because it’s funny) and they don’t result in charges. I suppose the law could be used if you also committed other offenses.
As for not being able to talk politics in China: where all the policy and law changes come from then? And if you can freely talk in the US, why so many people have many throwaway accounts here on HN?
Anyway: yes, China is oppressive in a number of ways. It’s about one generation behind the US, except for racism, where it’s ahead. US itself is perhaps a generation behind Western Europe. But everyone knows what’s bad about China, and everyone knows what’s good in the US. Anti-Chinese propaganda is everywhere, and I find it quite annoying. Particularly so because it detracts attention from real problems - if American propaganda is lying about genocides and missile silos, why assume other parts are true? Secondary: all the bad stuff in China stays in China. They don’t routinely exterminate hundreds of thousands people in other countries for financial or racial reasons. They don’t intentionally ruin other nations’ economy. The US does. And I feel like this perspective is entirely lost among western population.
Regarding other countries - what would happen (and happened) to countries that refused to join War on Drugs?
Regarding possession - in UK essentially nobody cares, unless you’re distributing industrial quantities and/or are really obnoxious about it, eg walk into Constabulary smoking a joint, or by putting up advertisements. There are regular news about eg police finding a suspect car parked for six hours with engine on, filled with smoke (actual example I remembered because it’s funny) and they don’t result in charges. I suppose the law could be used if you also committed other offenses.
As for not being able to talk politics in China: where all the policy and law changes come from then? And if you can freely talk in the US, why so many people have many throwaway accounts here on HN?
Anyway: yes, China is oppressive in a number of ways. It’s about one generation behind the US, except for racism, where it’s ahead. US itself is perhaps a generation behind Western Europe. But everyone knows what’s bad about China, and everyone knows what’s good in the US. Anti-Chinese propaganda is everywhere, and I find it quite annoying. Particularly so because it detracts attention from real problems - if American propaganda is lying about genocides and missile silos, why assume other parts are true? Secondary: all the bad stuff in China stays in China. They don’t routinely exterminate hundreds of thousands people in other countries for financial or racial reasons. They don’t intentionally ruin other nations’ economy. The US does. And I feel like this perspective is entirely lost among western population.
You forget that death sentences are carried out much more swiftly in China.
Also, while there's a lot to be said about court systems, public defenders etc in US I wouldn't compare the two.
Also, while there's a lot to be said about court systems, public defenders etc in US I wouldn't compare the two.
China have actual slaves. You can't even know the real rate of incarceration in China, and trying to figure it out is almost a sure way to get arrested.
[deleted]
This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything really) on uploaded user content since it is actually voluntary.
Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
The article states that the scanning is implemented as some kind of mutual operation between device and server when the device tries to upload photos into Apple’s cloud services. Presumably this is better than solely scanning photos on the server, because a client has even less control over what types of scanning Apple deploys there.
> All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
Even better -- someone could remotely exploit their adversary with malware like Pegasus, implant actual CP on the phone, hide malware traces, and then the Apple's system would alert the authorities. Who would believe the protests of innocence when there is the evidence on the phone, and the infraction is so horrific? A Perfect crime.
Pardon my ignorance on the subject, but is it feasible to create a false-positive hash match? Isn't a hash directly correlated to the file content? So if the file content is changed at all, the hash should be completely different, should it not?
You have two primary misunderstandings AFAICT.
(1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes
(2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
(1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes
(2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
irrc a cryptosecure hash is computationally infeasible to generate a new sample that has the same hash.
that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
Agreed. From my comment:
> although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
> although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
Repeating that this process doesn't appear to be using cryptographic hash functions in the first place, so this conversation isn't relevant, but:
Only in the sense that "we don't know of a way to generate a new sample that has the same hash, yet". Unless we can prove P!=NP, we can't even prove that there exist hash functions that are actually infeasible to generate a sample with the same hash - so far we are no where close to knowing whether P!=NP or P=NP.
Even if we assume P!=NP, that doesn't mean that it is infeasible to generate collisions for our current cryptographic hash functions. They're not backed by some mathematical proof of correctness assuming <assumptions>, just the fact that so far no one has publicly figured out a way to break them, and the people who spend time trying think that their design is one that is unlikely to be broken in the near future.
We have managed to find collisions against hash functions that we previously considered cryptographically secure, that were designed with the same sort of standard in mind: E.g. md5 (https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities) and sha1 (https://en.wikipedia.org/wiki/SHA-1#Birthday-Near-Collision_...).
There's no great reason to think that private groups could not have an attack against current hash functions [1], and there's definitely no reason to think that they won't find one in the future.
[1] There sort of is for SHA2, and that's that bitcoin has effectively created a giant bounty for breaking it.
Only in the sense that "we don't know of a way to generate a new sample that has the same hash, yet". Unless we can prove P!=NP, we can't even prove that there exist hash functions that are actually infeasible to generate a sample with the same hash - so far we are no where close to knowing whether P!=NP or P=NP.
Even if we assume P!=NP, that doesn't mean that it is infeasible to generate collisions for our current cryptographic hash functions. They're not backed by some mathematical proof of correctness assuming <assumptions>, just the fact that so far no one has publicly figured out a way to break them, and the people who spend time trying think that their design is one that is unlikely to be broken in the near future.
We have managed to find collisions against hash functions that we previously considered cryptographically secure, that were designed with the same sort of standard in mind: E.g. md5 (https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities) and sha1 (https://en.wikipedia.org/wiki/SHA-1#Birthday-Near-Collision_...).
There's no great reason to think that private groups could not have an attack against current hash functions [1], and there's definitely no reason to think that they won't find one in the future.
[1] There sort of is for SHA2, and that's that bitcoin has effectively created a giant bounty for breaking it.
A high rate of false positives will create distrust in the system, and ultimately make it less effective since no one will take it seriously.
As in an activist might distribute them far and wide to sabotage the system, yeah.
If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them".
But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them".
But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
Could be used to target specific individuals and discredit them (e.g. critics of government).
Like swatting, but worse.
Reminder that even accusations of abuse imagery will get you in cuffs and your children taken away (see article at the end)
Linux truly is the last bastion of sanity. I am so glad that linux desktops took off and work as well as they do. Last thing I should be worrying about is if an employee at Microsoft or Apple decides their "automated system with no errors" decides that a hash collision or a picture of a baby is close enough to actual abuse imagery. There are plenty of people that believe even a picture of a naked baby in a bathtub is abuse imagery. Anyone remember the case of Walmart? They took their kids away "pending investigation".
https://jonathanturley.org/2009/09/18/arizona-couple-sues-wa...
Linux truly is the last bastion of sanity. I am so glad that linux desktops took off and work as well as they do. Last thing I should be worrying about is if an employee at Microsoft or Apple decides their "automated system with no errors" decides that a hash collision or a picture of a baby is close enough to actual abuse imagery. There are plenty of people that believe even a picture of a naked baby in a bathtub is abuse imagery. Anyone remember the case of Walmart? They took their kids away "pending investigation".
https://jonathanturley.org/2009/09/18/arizona-couple-sues-wa...
[deleted]
Ah, the classic 'think of the children' approach to increasing surveillance. If you're opposed to it, what are you, a nonce?
In a world where all police were incorruptible and all laws were just, maybe I'd be ok with this. But since we live in this world, I feel like it's only a matter of time until the "think of the children" technology is coopted to scan for other "bad" images.
I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
Forget just the government. Imagine if corporations could abuse regulatory structure to enforce copyright on private images?
>I wonder how long it'll take for China...
All iCloud Data in China are already within their control. The whole DataCenter are done with "partnership" agreement.
HK user have their Data in US but I believe there could be a temporary server in HK DC before it is sent to US.
All iCloud Data in China are already within their control. The whole DataCenter are done with "partnership" agreement.
HK user have their Data in US but I believe there could be a temporary server in HK DC before it is sent to US.
Scan for child abuse today; scan for wrong-think tomorrow. I'm liking the idea of Graphene OS more and more each day... or just going phone-free entirely.
We will not be able to have nice things until we get better privacy laws or come up with novel ways to prevent unwarranted snooping. It's sad that it's come to this.
> ...ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography.
I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.
I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.
It's also indicative of the unhealthy relationship these giant semi-monopolistic companies have with the government(s), that they implement the government's supposed will out of the fear of anti-trust actions/legislation, without any need for the government to pass legislation. Same goes for the demands to curb "misinformation" of various kinds, which basically constitutes government mandated speech regulation, even though the standing jurisprudence clearly forbids it.
When it comes to Apple and privacy, it's important to differentiate two different types of privacy: (1) privacy against snooping by the state, and (2) privacy against snooping by all others. As we had seen with Apple and China, it's only the second type of privacy that Apple cares about. Sharing your information with the state is not out of character for Apple. In fact, had they done the opposite, it would have been out of character.
Apple cares about #1 when they feel the requests are not legal. https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute
Apple has to comply with the law, however.
Apple has to comply with the law, however.
What law prompted this new feature we are discussing here?
I don’t know, and I’m certainly troubled by the story.
There is a federal law requiring electronic service providers to notify if they spot such material. Apple may be subject to that. See the “Federal CSAM Law” section here:
https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-b...
There is a federal law requiring electronic service providers to notify if they spot such material. Apple may be subject to that. See the “Federal CSAM Law” section here:
https://cyberlaw.stanford.edu/blog/2020/01/earn-it-act-how-b...
Wow. Okay, so my theory about them trying to stave off future legislation is likely correct.
I assume 18 USC 2258A, but they seem to be going above and beyond. Under current law, they are only required to report, they are not required to prevent[1].
Presumably, they are doing this voluntarily now to stave off future legislation around this that might require more invasive scanning.
1. 18 USC 2258C(c)
Presumably, they are doing this voluntarily now to stave off future legislation around this that might require more invasive scanning.
1. 18 USC 2258C(c)
It doesn’t make sense to expect or hope any company will protect us from the state.
A big, rich company like Apple will have some leverage, but if the state is determined that won’t matter.
Our controls on the state will need to be outside of private companies.
A big, rich company like Apple will have some leverage, but if the state is determined that won’t matter.
Our controls on the state will need to be outside of private companies.
Apple should perhaps dogfood their system on their own employees first: https://www.losaltosonline.com/news/hills-man-arrested-on-ch...
Not employees but all of their board and all of their investors.
Also lets draft on all the children of senate and congress
https://quotepark.com/quotes/1911588-immortal-technique-the-...
https://quotepark.com/quotes/1911588-immortal-technique-the-...
> Apple intends to install software on American iPhones to scan for child abuse imagery
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system
Why is there any need for Apple to install software on the iPhone if they are isolating the algorithm to run only on cloud storage, not local images? Not a programmer, so maybe a simple explanation.
Also, if this is isolated to iCloud, won't criminals just start using a different cloud backup provider?
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system
Why is there any need for Apple to install software on the iPhone if they are isolating the algorithm to run only on cloud storage, not local images? Not a programmer, so maybe a simple explanation.
Also, if this is isolated to iCloud, won't criminals just start using a different cloud backup provider?
Doesn't really answer your question, but the article says this:
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
They are scanning client and server-side photos.
As any of the lawyers here can attest, the legal definitions here open to huge debates. Some material is unambiguous, other images reside extensive grey areas. The legality of material can even change based on context. Something that is perfectly legal in one area can be absolutely illegal in another. Example: If I had to list the most iconic photographs in world history, one of them is of a totally nude child (Vietnam). That isn't an illegal image, but try explaining that to a computer. At the moment these nuances are handled by cops, prosecutors and in extreme cases the courts. Is Apple going to put a marker down on a particular universal definition? Will that definition become the de facto world standard?
Who would add “gray area” pictures to the database of CP images that are matched?
Material that is deemed illegal in jurisdiction A but not in B. Material for which someone was once convicted but if found on someone else's device would not be illegal. Material isn't added to these databases after sober legal deliberation. Police agencies can add basically anything they see fit.
This is a system for just one jurisdiction though? Is there talk about doing this outside the US? For multiple jurisdictions you’d need different sets of known hashes.
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.
It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"?
If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives.
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities
Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"?
If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives.
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities
Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
The name is certainly concerning. It's one thing to check for specific hashes. It's another entirely to do approximate hashing. Locality sensitive hashing even in its best implementations has tons of false positives.
It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.
It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.
I bet Apple understands this is a bad idea (dilutes their "privacy" brand image) I wonder what forces have pushed Apple to this point.
>I bet Apple understands this is a bad idea
That would be Steve Job's Apple. Tim Cook's Apple will think they are doing it for the greater good.
The road to hell is paved with good intentions
That would be Steve Job's Apple. Tim Cook's Apple will think they are doing it for the greater good.
The road to hell is paved with good intentions
perhaps they're being required by law and gagged for indicating as such
https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...
https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...
Privacy was only a talking point to get them some advantage over the Eye of Sauron (Google). I have no doubt that they were ready to jettison their privacy protection marketing angle at a moment's notice. You can't have access to people's data in the cloud and on their phones, and not use it in certain ways. To do so would be to leave money, and the good will of the government, on the table.
Things that would reduce child sexual abuse more than scanning everyone's phones on a daily basis:
- Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household
- Fund undercover police work and informants, which is how most large busts already happen now. Ideally highly independent from existing courts and law enforcement to deal with Epstein- or Dutroux-type situations. The worst offenders are not swapping child porn on Facebook or Dropbox (or now iMessage and Apple Photos)
- Enforce obscenity laws against incest and other extreme forms of pornography
- Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly
Since these aren't under consideration, we can tell this isn't about reducing child abuse. It's about setting up a permanent and ever-expanding digital system for political control.
- Encourage marriage and remove benefits penalties for two-parent households, as children are substantially more likely to be abused when their biological father is not in the household
- Fund undercover police work and informants, which is how most large busts already happen now. Ideally highly independent from existing courts and law enforcement to deal with Epstein- or Dutroux-type situations. The worst offenders are not swapping child porn on Facebook or Dropbox (or now iMessage and Apple Photos)
- Enforce obscenity laws against incest and other extreme forms of pornography
- Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly
Since these aren't under consideration, we can tell this isn't about reducing child abuse. It's about setting up a permanent and ever-expanding digital system for political control.
Enforce obscenity laws against incest and other extreme forms of pornography
How would you go about enforcing this, and how does that differ from existing proposals about child porn?
Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly
All the worries about people being framed or wrongly convicted apply equally well to that proposal.
How would you go about enforcing this, and how does that differ from existing proposals about child porn?
Institute the death penalty for child sexual abuse and make sure it is performed swiftly and publicly
All the worries about people being framed or wrongly convicted apply equally well to that proposal.
https://news.ycombinator.com/item?id=28068741
This topic already made it to the current frontpage.
This topic already made it to the current frontpage.
> Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
> Apple themselves can't decrypt it no matter what?
Apple has root on every iPhone. They can do whatever they want by pushing whatever code they like to any iphone.
Apple has root on every iPhone. They can do whatever they want by pushing whatever code they like to any iphone.
iCloud backups are not encrypted, this is a major knowledge point.
Isn't Apple going to start auto scanning the text in your images too for searching/easy copying? How many years until the hash of text in your images is compared to a database of... god knows what. Sensitive information? Dangerous text? I honestly can't think where it's most likely headed but I don't like it.
I have yet to deep-dive into the techical details of Pegasus so this is pure speculation, but if Pegasus demonstrates the feasibility of deploying and executing arbitrary payloads on remote devices without the owner's knowledge or consent wouldn't that raise a blackmail vector with this new program? Again I'm not familiar with the specific exploits used by Pegasus, but is there anything stopping another malicious actor from relying on the same exploits to say install a bootloader that downloads images that would trigger a positive scan?
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.
This doesn't really make any sense. So it's not a Neural Engine thing but merely hash matching, and it also only applies to iCloud Backup? That is already non-e2e encrypted, so Apple can already access those files whenever it wants.
This doesn't really make any sense. So it's not a Neural Engine thing but merely hash matching, and it also only applies to iCloud Backup? That is already non-e2e encrypted, so Apple can already access those files whenever it wants.
The article is a mess, it could be anything. Locally? On iCloud? Hashed? Content detection? Who knows, the article has all of these keywords.
This will enable Apple to send hashes of matching fingerprint and face scans to their online database. This also completely disabled their argument that their product is safe because the data remains encrypted on your device. Instead of passing the data to the spy service they are sending a report on data to the cloud which will allowed to be much more opaque and misleading.
I just ordered a Iphone 12 Mini to replace an Iphone SE, but now I’m going to return it. No amount of cool tech in a product is worth enabling this slippery slope against human privacy rights.
I just ordered a Iphone 12 Mini to replace an Iphone SE, but now I’m going to return it. No amount of cool tech in a product is worth enabling this slippery slope against human privacy rights.
This just means child molesters will seek out PinePhones, Purism Librems, etc. And those phones will acquire reputations as "pedophones", and anyone buying or using one will be immediately considered sus. People with nothing to hide won't mind an OS with surveillance features.
Until few high profile cases of people with nothing to hide getting swatted because of false positive / image crafted by a GAN to create false positive.
That is most likely to happen when the FBI "discovers" child abuse material on the phone of an uncooperative potential informant. In which case they are going to supermax prison where they will be ventilated by the inmates and no one will miss them.
New malware wont be inserting CP, but running GAN to create CP inside the target.
I'm too old for this, but I really wonder what it's going to be like for high-school kids and nudes 10 years on.
I mean, if you get a really good nude from someone you're going to want to keep it. But for how long? Is there now a social rule of how long to keep nudes for? Is 21 too old, and you should delete them? What if you're still with the person?
Personally, I'm glad I don't have to figure it out, but teens sending each-other nudes just has all sorts of confusing ethical quandries to it.
I mean, if you get a really good nude from someone you're going to want to keep it. But for how long? Is there now a social rule of how long to keep nudes for? Is 21 too old, and you should delete them? What if you're still with the person?
Personally, I'm glad I don't have to figure it out, but teens sending each-other nudes just has all sorts of confusing ethical quandries to it.
While they are at it they should up the ante and also add COVID misinformation filters, so that a phone refuses to save it and/or doesn't send it anywhere. Would be very topical. There is also racism and other almost universally recognized wrongthink. If you decide to police offline activity as if it's a cloud platform, why not implement all the online platform's automatic content filtering/censorship? /s
COVID "misinformation", anti-lockdown protests, any movements that are commonly labelled as domestic terrorism (boog bois, antifa, maybe even BLM; in China it will be anything that speaks out against the CCP, etc) - the potential for abuse is massive.
Anyone that trusts the feds to do the right thing here is incredibly naive. They get to operate a black-box database; if your phone has any photos that match an entry in the db you will be reported to the authorities. I can't believe on HN of all places there's not 10x more of an outcry.
Anyone that trusts the feds to do the right thing here is incredibly naive. They get to operate a black-box database; if your phone has any photos that match an entry in the db you will be reported to the authorities. I can't believe on HN of all places there's not 10x more of an outcry.
>if your phone has any photos that match an entry in the db
I believe it's not "an entry in the db" but a neural classifier. The article calls it "neuralMatch". So it's much more general than a limited number of known images.
I believe it's not "an entry in the db" but a neural classifier. The article calls it "neuralMatch". So it's much more general than a limited number of known images.
I had a very sick 8 month old baby who I obviously had a million photos of. Unfortunately, my lil girl had 9 surgeries and her little body had gone through more than people can think of. She passed in May, and it's really difficult to look back at old pictures because of how much stuff was going on, various levels of critical ICU care, pre/post surgery, etc. But it wasn't until a few months after she was born I had gone to do something in Google and realised my little girl's very severe diaper rash, open sternum, chest tubes, etc, etc, etc photos were backed up and tagged in Photos.
This is sort of the premise to why I've de-Googled/de-Cloud myself because I didn't want my little girls pain and suffering to train some computer's image detection AI.
How would this work for a parent of a sick child who sends their doctor, S/O, possibly graphic in nature and personal photos of their own child? I was in close contact with my daughter's doctor, and would send him all sorts, I can only imagine the same is true for most parents of chronic children.
This is sort of the premise to why I've de-Googled/de-Cloud myself because I didn't want my little girls pain and suffering to train some computer's image detection AI.
How would this work for a parent of a sick child who sends their doctor, S/O, possibly graphic in nature and personal photos of their own child? I was in close contact with my daughter's doctor, and would send him all sorts, I can only imagine the same is true for most parents of chronic children.
This isn't scanning for naked children, it's known CSAM photos that are already circulating online.
Hopefully they are using phash to match the phone's images to known child abuse image hashes, because if they are using ML, I can only imagine what a false positive will do to someone's life.
Phashes can have collisions, and the false positive rate increases exponentially when you use a fuzzy metric like hamming distance to find derivatives of a source image.
On one hand, I applaud this - CASM is a massive issue, and the internet community has never taken it as seriously as it should have.
As many others have noted - the potential for misuse is rife. Also, realize that Apple has a single set of infrastructure for their phones and macs, especially now that M1 exists. I've read accounts in the person where disaffected people "planted" porn on other peoples system, then called the police.
Imagine a web server with a page and a hash map matching image, with a frame-size of 1px. It's now in your cache.
Also, it's almost inevitable that this is a slippery slope. Today is may be pre-hashed images. Tommorow it may be AI detection. The day after it may be vaccine falsehoods. The day after it may be other bad government speach.
This is needed. This is a major major problem. But I hope someone is thinking of how to mitigate the potential for abuse.
As many others have noted - the potential for misuse is rife. Also, realize that Apple has a single set of infrastructure for their phones and macs, especially now that M1 exists. I've read accounts in the person where disaffected people "planted" porn on other peoples system, then called the police.
Imagine a web server with a page and a hash map matching image, with a frame-size of 1px. It's now in your cache.
Also, it's almost inevitable that this is a slippery slope. Today is may be pre-hashed images. Tommorow it may be AI detection. The day after it may be vaccine falsehoods. The day after it may be other bad government speach.
This is needed. This is a major major problem. But I hope someone is thinking of how to mitigate the potential for abuse.
Imagine if Microsoft does this with Windows, lots of people would get arrested rightly so.
But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
No need to imagine, it's already done in OneDrive which comes with Windows 10 by default:
https://en.wikipedia.org/wiki/PhotoDNA
Do we know why did Apple put the effort to create this? Who forced them to do it? Like it is a move that is bad for PR and also does not make them money, I suspect some big government is demanding it but maybe I am to simple in my thinking and this makes sense.
so you’ll be able to ruin someone’s life by sending them a live photo where one frame has child abuse. i find the law enforcement part of these well intentioned anti child abuse initiatives to be absolutely terrifying because the standard is not that you were proven to have been a party to child abuse but that an offending file was found on your device.
Along with this: https://www.reuters.com/technology/exclusive-facebook-tech-g...
Just like the Patriot Act and the no fly list, it's only a very short matter of time until this is expanded to Android and more importantly, non-CP content. The only difference is its the other party doing it this time, in both cases its "to catch bad guys" and "for your own good".
Just like the Patriot Act and the no fly list, it's only a very short matter of time until this is expanded to Android and more importantly, non-CP content. The only difference is its the other party doing it this time, in both cases its "to catch bad guys" and "for your own good".
Today kiddy porn, tomorrow wrongthink.
Isn't PayPal already investigating users' purchases to help the ADL put pressure on transactions which they claim to be funding hate speech?
Yes they are!
Like having pictures of Tim Cook eating ice cream?
Wrongthink different.
Better not take any more pictures of the kids splashing in the bath for granny then.
This article raises serious privacy concerns. Just building the infrastructure for on-device scanning and reporting is extremely troubling. A slippery slope, or a break in the dam, as others have said. My view is that we've been on that slope for a long time, and this changes little. We just have to trust Apple, as has always been the case.
When I look at the technical details, it seems to me to be a reasonable compromise. It allows Apple and government to do something about the worst offenders, whereas it has no impact on anyone else.
Two technical reasons for this:
- The "neuralMatch" algorithm suggests some sort of CV, whereas the article talks about matching against a hash of know images. My guess is that the actual technology is something like Microsoft's PhotoDNA (https://news.microsoft.com/on-the-issues/2018/09/12/how-phot...). Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.
- As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow.
- It sounds like it doesn't flag a single image, but requires multiple hash hits.
So, if you want to feel better about this, understand that it is a system that will flag people who are downloading storing known child exploitation images on their devices and naively backing those up to iCloud.
This is the sort of privacy compromise that works in practice. Serious offenders are either caught or diverted to other channels. Minor offenders are probably not caught. The risk to non-offenders is zero or close to it.
Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
When I look at the technical details, it seems to me to be a reasonable compromise. It allows Apple and government to do something about the worst offenders, whereas it has no impact on anyone else.
Two technical reasons for this:
- The "neuralMatch" algorithm suggests some sort of CV, whereas the article talks about matching against a hash of know images. My guess is that the actual technology is something like Microsoft's PhotoDNA (https://news.microsoft.com/on-the-issues/2018/09/12/how-phot...). Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.
- As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow.
- It sounds like it doesn't flag a single image, but requires multiple hash hits.
So, if you want to feel better about this, understand that it is a system that will flag people who are downloading storing known child exploitation images on their devices and naively backing those up to iCloud.
This is the sort of privacy compromise that works in practice. Serious offenders are either caught or diverted to other channels. Minor offenders are probably not caught. The risk to non-offenders is zero or close to it.
Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
> As described here, the scanning only applies to images also uploaded to iCloud ("[the] algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system"). While we don't know whether this description is accurate, it suggests that if you don't back up images to iCloud your device won't do any scanning locally. Apple already has the keys to your iCloud backups, so if you value privacy you're probably not backing up to iCloud anyhow.
If we had a way to guarantee that it would stay that way, it'd be less concerning. As it stands, clearly there are plans to scan on the device itself, otherwise there would be no need to roll out software on iPhones at all.
> Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.
Agreed, but the issue is the government can stick whatever photos in the DB they want. And we have no way to verify what's in there, since they're not gonna release thousands of CP images for us to audit them.
> Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
It changes plenty. Currently Apple more or less promises not to spy on you, and so they have to actually be lying to do so. Is lying an option for them? Sure, but that's different than them just announcing in advance "hey we're gonna start scanning every photo you have and crossreferencing against a somewhat-transformation-invariant secret government database". I'd prefer that they have to lie to me to spy on me, since that at least has the potential for backlash if discovered. If they get this feature rolled out we're all fucked.
If we had a way to guarantee that it would stay that way, it'd be less concerning. As it stands, clearly there are plans to scan on the device itself, otherwise there would be no need to roll out software on iPhones at all.
> Hash collisions aside, this should only produce matches against images that are already in a government database. It will match manipulated images (e.g., rotated or cropped), but it won't match new images.
Agreed, but the issue is the government can stick whatever photos in the DB they want. And we have no way to verify what's in there, since they're not gonna release thousands of CP images for us to audit them.
> Apple has the control to do all sorts of invasive things to our privacy. They could be scanning and reporting all kinds of things already, and we might not even know. Or they could start doing so tomorrow. From this point of view we're already trusting them to do right by us as their customers, and this feature doesn't change that.
It changes plenty. Currently Apple more or less promises not to spy on you, and so they have to actually be lying to do so. Is lying an option for them? Sure, but that's different than them just announcing in advance "hey we're gonna start scanning every photo you have and crossreferencing against a somewhat-transformation-invariant secret government database". I'd prefer that they have to lie to me to spy on me, since that at least has the potential for backlash if discovered. If they get this feature rolled out we're all fucked.
Non pay-walled site.
https://www.msn.com/en-gb/money/technology/apple-plans-to-sc...
To everyone here who wishes we do nothing about this: you should rethink your position. There is a difference between “let’s be careful and design the right solution” and “let’s do nothing to maximize individual liberty”.
In 2017 a 369% increase in CSAM in terms of reports happened. The only explanation is that platforms now facilitate the sharing and a growing problem continues to grow.
In raw percentages the data clear: too many children are being abused and the numbers are growing.
https://storage.googleapis.com/pub-tools-public-publication-...
In 2017 a 369% increase in CSAM in terms of reports happened. The only explanation is that platforms now facilitate the sharing and a growing problem continues to grow.
In raw percentages the data clear: too many children are being abused and the numbers are growing.
https://storage.googleapis.com/pub-tools-public-publication-...
Related to this, I love Canva.com but then I noticed they were scanning my designs. Some warning popped up.
This idea of perpetually renting services than monitoring the use is something that should be criticized.
This idea of perpetually renting services than monitoring the use is something that should be criticized.
To laugh, I just imagine the society of 30 years ago and what would have happened if someone would have tried to suggest to do something like that:
ie, have private company give themselves the right to go have a look through your belonging and in your home because you might be a criminal.
It is now well known that most citizens of most countries are criminals, so no one deserve privacy anymore...
But sadly, it looks like that intelligence and common sense has generally decreased in the population.
Btw, the appleTV and iphone, that are always listening to your conversations in your home to detect some bad keywords and then denounce you to the police is not yet ready? Nazi collaborators, from their tombs, are still waiting for this nice feature!
Anyway, iphone owners, let me tell you that you did a good job by stupidly giving your money and power to Apple!
ie, have private company give themselves the right to go have a look through your belonging and in your home because you might be a criminal.
It is now well known that most citizens of most countries are criminals, so no one deserve privacy anymore...
But sadly, it looks like that intelligence and common sense has generally decreased in the population.
Btw, the appleTV and iphone, that are always listening to your conversations in your home to detect some bad keywords and then denounce you to the police is not yet ready? Nazi collaborators, from their tombs, are still waiting for this nice feature!
Anyway, iphone owners, let me tell you that you did a good job by stupidly giving your money and power to Apple!
What if under a secret order Apple is instructed to insert non-CSAM photos, to match whatever a relevant juridiction wants to match? Effectively, you now have a black box operating on your own computer, this is deeply disturbing. This is not going to end well. And by the way this is like all the DNA stuff, you're not only at risk from your own computer but also the all the libraries your picture happened to be in.
Standard "think of the children" logic. Plenty of countries will put up hashes of LGBT content, "incorrect" religious content, etc
> The proposals are Apple’s attempt to find a compromise between its own promise to protect customers’ privacy and ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography.
The 1st and 4th amendments tell the fed to go pound sand, Apple should follow suit.
The 1st and 4th amendments tell the fed to go pound sand, Apple should follow suit.
So, how long before the same technology is used to fight "misinformation" and "domestic terrorism" (e.g. memes that disagree with official point of view)? How long until it's used to ban "hateful speech", or any speech unapproved by the gov^H^H^HReal Government - the Almighty Apple?
Wow, this seems like a terrible, terrible idea on so many counts. It would certainly keep me from buying an iPhone -- the last thing I need is to get a visit from the cops because of false positives.
It also means that you can't have any trust about your data safety even if you never put it in the cloud.
It also means that you can't have any trust about your data safety even if you never put it in the cloud.
Will Google by some miracle be the company that offers freedom from forced snooping?
It seems not, so we have a choice between this and Google's always-track-you setup.
I'm sure no one here has any illegal pictures, but the precedent is terrifying. What about copyrighted images or downloaded videos without DRM ?
It seems not, so we have a choice between this and Google's always-track-you setup.
I'm sure no one here has any illegal pictures, but the precedent is terrifying. What about copyrighted images or downloaded videos without DRM ?
This is extremely alarming, I don’t care what the reason, I don’t want apple or anyone looking through my personal photos. It’s none of their business, and the sheer chance of false positives and ruining peoples lives is far too great to let this go forward.
Not disagreeing with you - but this reminds me of my college job spending all day looking through personal photos.
I put myself through school working at a one-hour photo shop in a drug store. It was my job to feed the machine, keep the chemicals balanced, paper loaded and perform white-balancing and cropping to make sure the end result matched customers' expectations. It was a relatively small town, and I lived in the same neighborhood that I worked in - so I basically saw 80% of my neighbor's photos during the development, cropping, and packing process. I saw hundreds of baby's in bath tubs, and bare a* toddlers on the run, in the mud, etc. Also saw a lot more bedroom photos than you (or at least 17yo me) might expect.
In the three years of work, I only had one roll of film for which I called the police. It was obvious child abuse. I stopped the machine, called the police and they collected the film directly into evidence and made the arrest when the person came to pick it up. The guy in question thought that the 1hour process was "all automated" and he was pissed that somebody was looking at all his photos.
Not sure that has anything to do with the iPhone AI scanning - but it might be an anecdote of interest to those that don't remember a time when somebody looked at all your photos as part of the creation process.
I put myself through school working at a one-hour photo shop in a drug store. It was my job to feed the machine, keep the chemicals balanced, paper loaded and perform white-balancing and cropping to make sure the end result matched customers' expectations. It was a relatively small town, and I lived in the same neighborhood that I worked in - so I basically saw 80% of my neighbor's photos during the development, cropping, and packing process. I saw hundreds of baby's in bath tubs, and bare a* toddlers on the run, in the mud, etc. Also saw a lot more bedroom photos than you (or at least 17yo me) might expect.
In the three years of work, I only had one roll of film for which I called the police. It was obvious child abuse. I stopped the machine, called the police and they collected the film directly into evidence and made the arrest when the person came to pick it up. The guy in question thought that the 1hour process was "all automated" and he was pissed that somebody was looking at all his photos.
Not sure that has anything to do with the iPhone AI scanning - but it might be an anecdote of interest to those that don't remember a time when somebody looked at all your photos as part of the creation process.
The dumbest idea ever -so glad that I am an Android user.
So is the phash code actually going to run locally (I.e. can dump it and figure out the function) or is it only running on icloud?
IIRC they've already been running it on icloud photos for a year or two. Hashing local files is the step they're taking now
Next, something that scans for pictures of cops, to detect people taking pictures of law enforcement.
So now anyone can frame you by sending unsolicited child porn to your iPhone.
> the screening is done on the phone.
Where is my right to refuse to dedicate my devices cpu, battery time and network bandwidth for this activity?
Did I agree to this already?
(Sorry for asking a question! I'll try to simply trust Apple at all times from now on...)
Where is my right to refuse to dedicate my devices cpu, battery time and network bandwidth for this activity?
Did I agree to this already?
(Sorry for asking a question! I'll try to simply trust Apple at all times from now on...)
So everyone here just has an FT subscription? I don't see anything but a paywall.
Privacy, except for thoughtcrimals.
Are they going to compensate the owners for the CPU time, electricity and bandwidth this will cost?
I asked a similar question in both threads here and both times I got downvoted by some courageous folks without an answer.
I also would like to know if I already agreed to this and where can I read the agreement that I signed?
I also would like to know if I already agreed to this and where can I read the agreement that I signed?
Does this paywalled article have any further details? Aside from a cryptic twitter post that mentions CSAM, are there any further details?
CSAM also means Cyber Security Asset Management.
CSAM also means Cyber Security Asset Management.
I’ve been meaning to start a home cloud for pictures on top of Raspberry Pi. Looks like that will get done sooner than I planned
Whose to say what is abuse? Are Caesar Milan tactics on a child abuse? This is outright dangerous beyond belief. Nobody is gonna wanna be a parent if you have to worry about your kid being mad at you and then posts something like that.
IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
That would be false testimony, not hearsay. Hearsay ("John Smith told me he heard MeinBlut kidnapped the Lindbergh baby") is not allowed. Witnesses lying about what they saw and or participated in is allowed, although it can be lies.
Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
I don't think society can function without harsh penalties for child abuse, and I think thats a widely held belief everyone agrees with.
I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
Huh, I specifically said "concerns about collisions" is a real concern. I said that "mass surveillance" is a real concern.
What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.
What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.
Doesn't seem to be working very well from here; risk of penalty isn't going to stop most of these people, more likely make their game even more exciting.
Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
Apple: An Apple NeuralMatch Agent has positively matched images on device "Apple iPhone 12 (Red™)" geotagged at the coup attempt against Prime Minister Trump occurring on January 30th, 2024. Images and documents have been submitted to the Thump Bureau of Investigation for further evaluation. Your Apple account has been permanently suspended.
You can request a review of Apple's actions, however we are receiving higher than normal requests at this time. This means we may be unable to review your account.
Sorry for any inconvenience.
You can request a review of Apple's actions, however we are receiving higher than normal requests at this time. This means we may be unable to review your account.
Sorry for any inconvenience.
It’s 2035, a major terrorist attack occurs. Apple scans all phones for location and images near the area over the last 12 months, sends the list over since the government declared Marshall law.
You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are.
Edit:
Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this? ____________
Second Edit:
Wanted to reply to another comment but being rate limited:
As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities.
The number of Muslims in America:
A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population.
Lol. Jesus Christ, you do the math.
You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are.
Edit:
Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this? ____________
Second Edit:
Wanted to reply to another comment but being rate limited:
As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities.
The number of Muslims in America:
A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population.
Lol. Jesus Christ, you do the math.
> isn’t the government indirectly committing unlawful search?
No. That's the beauty of the third-party doctrine. Your rights disappear in a puff of twisted logic because you agreed to the fine print.
No. That's the beauty of the third-party doctrine. Your rights disappear in a puff of twisted logic because you agreed to the fine print.
I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.