Apple’s New ‘Child Safety’ Initiatives, and the Slippery Slope(daringfireball.net)
daringfireball.net
Apple’s New ‘Child Safety’ Initiatives, and the Slippery Slope
https://daringfireball.net/2021/08/apple_child_safety_initiatives_slippery_slope
32 comments
This is by far the best reporting on this issue I’ve seen on HN or elsewhere.
If you skipped the article to read the HN comments, I wouldn’t bother. The article’s discussion is much better - just read that.
If you skipped the article to read the HN comments, I wouldn’t bother. The article’s discussion is much better - just read that.
It's widely reported that Jeffrey Epstein and Ghislaine Maxwell were using their whole shenanigans to blackmail powerful people. This Apple "tech" will eventually fall in such people's hands and make it even easier for them to get blackmail material.
We have already seen plenty of examples of SWATTING gone wrong where online trolls call fake threats to the police so law enforcement responds and ends up killing innocent people. What's to say this won't happen with this privacy violation too where someone pranks others?
This along with the obvious abuse potential by governments to silence protests, political and cultural movements is a very dangerous edge of the sword. It's very similar to the censorship debate. When the wind blows in the wrong direction (and it eventually will), people will complain. But by then, it will be too late. Just like the patriot act.
Also I find it disgusting how pointing out the obvious dangerous precedent of such privacy/free speech/spying violations gets labelled as "child porn sympathizer" / "racism sympathizer" / "terrorist sympathizer".
I would expect such privacy violating tech to come out of likes of Facebook or Google before Apple which claims to market itself as the privacy pioneer. How did this get through top people at Apple without someone stopping to say "Hey, this might not be a good idea" is beyond me.
Their site still says:
> "What you share from those experiences, and who you share it with, should be up to you."
https://www.apple.com/privacy/
We have already seen plenty of examples of SWATTING gone wrong where online trolls call fake threats to the police so law enforcement responds and ends up killing innocent people. What's to say this won't happen with this privacy violation too where someone pranks others?
This along with the obvious abuse potential by governments to silence protests, political and cultural movements is a very dangerous edge of the sword. It's very similar to the censorship debate. When the wind blows in the wrong direction (and it eventually will), people will complain. But by then, it will be too late. Just like the patriot act.
Also I find it disgusting how pointing out the obvious dangerous precedent of such privacy/free speech/spying violations gets labelled as "child porn sympathizer" / "racism sympathizer" / "terrorist sympathizer".
I would expect such privacy violating tech to come out of likes of Facebook or Google before Apple which claims to market itself as the privacy pioneer. How did this get through top people at Apple without someone stopping to say "Hey, this might not be a good idea" is beyond me.
Their site still says:
> "What you share from those experiences, and who you share it with, should be up to you."
https://www.apple.com/privacy/
Without knowing how neuralHash works, there are significant claims in this article about fingerprinting that can’t be verified.
“ Fingerprints from images of similar content are not themselves similar. Two photographs of the same subject should produce entirely dissimilar fingerprints. The fingerprints of your own photos of your kids are no more likely to match the fingerprint of an image in NCMEC’s CSAM database than is a photo of a sunset or a fish.”
You and I can’t know this is true!
“ Fingerprints from images of similar content are not themselves similar. Two photographs of the same subject should produce entirely dissimilar fingerprints. The fingerprints of your own photos of your kids are no more likely to match the fingerprint of an image in NCMEC’s CSAM database than is a photo of a sunset or a fish.”
You and I can’t know this is true!
If you have worked on perceptual hashing, you know this just isn't true. It's like someone saying TLS prevents all eavesdropping. No, it doesn't!
On a population level scale, everyday people will have their photos analysed by a human working for Apple, and potentially arrested and charged for false positives.
And on a world scale, this will be a tool mandated by governments via legislation to silence dissents, trace the origins of leaks, and can even make possession of things like the Snowden documents illegal.
The NSA is probably thinking about how they give hashes of their top secret documents so they arrest the next Snowden before he/she gets the word out.
On a population level scale, everyday people will have their photos analysed by a human working for Apple, and potentially arrested and charged for false positives.
And on a world scale, this will be a tool mandated by governments via legislation to silence dissents, trace the origins of leaks, and can even make possession of things like the Snowden documents illegal.
The NSA is probably thinking about how they give hashes of their top secret documents so they arrest the next Snowden before he/she gets the word out.
That is a very important point. I was ambivalent about Apple's ecosystem and thought about about making a buy in due to their excellent UI integration. I believe I will wait and have a good look at alternatives again.
And we know how this has went in the past: wooops, our system has made a mistake and ruined your like, we apologize and won't do it again. No consequences and they'll do it again.
This is a very reasonable assessment of the current controversy. It didn't change my mind (I'm with Snowden on this) but it is well written, clear and well-balanced. Recommended reading up to the very end.
It's reasonable but it misses some omissions. Specifically:
1. What if the NSA/FBI, through the Patriot Act, demand Apple provide bulk interception safety vouchers ('metadata') for "counter-terrorism" purposes? Much like we learned from Snowden that this happens for all cell phone records for decades?
This is not a strawman argument: it is a question concerning applications of laws in force today, that have been applied for mass surveillance already in other contexts.
2. The whole on-device spyware is a single feature toggle away from working on local files too. We're software engineers. We know that "if (iCloudPhotosEnabled) {" is a simple boolean comparison.
Combine the two, and you realise Apple has just built the most intrusive mass surveillance tool in history, disguised as child safety.
Apple's choice of announcing all 3 features together isn't a mistake. This is one of the best PR department in the world. Logic requires us to dig deeper.
1. What if the NSA/FBI, through the Patriot Act, demand Apple provide bulk interception safety vouchers ('metadata') for "counter-terrorism" purposes? Much like we learned from Snowden that this happens for all cell phone records for decades?
This is not a strawman argument: it is a question concerning applications of laws in force today, that have been applied for mass surveillance already in other contexts.
2. The whole on-device spyware is a single feature toggle away from working on local files too. We're software engineers. We know that "if (iCloudPhotosEnabled) {" is a simple boolean comparison.
Combine the two, and you realise Apple has just built the most intrusive mass surveillance tool in history, disguised as child safety.
Apple's choice of announcing all 3 features together isn't a mistake. This is one of the best PR department in the world. Logic requires us to dig deeper.
> Apple's choice of announcing all 3 features together isn't a mistake. This is one of the best PR department in the world. Logic requires us to dig deeper.
While I think your other critiques are reasonable (with an asterisk I'll get back to), this is a little conspiratorial-sounding -- and I would submit that we have a lot of circumstantial evidence from the last ~36 hours of reporting that the PR generated around this has not exactly been positive. I don't see how announcing all of these features together benefits Apple. (And at risk of being a little conspiratorial-sounding myself, I suspect there are other more intrusive mass surveillance tools already built and in use by other corporations and governments.)
As for that asterisk: assuming the service works the way Apple describes it, I don't think it's comparable to cell phone call metadata. You could obviously use the system to search for other known photos besides CSAM and also search photos that are local to the device, but other use cases still need to fit "this photo matches the hash of this other photo" (I'm not even sure if you can abstract "photo" to "other media file"). Photos, of course, have metadata that would be interesting to law enforcement agents, but if this does anything with that metadata then the service doesn't work the way Apple describes it -- and if we assume that Apple is starting out lying about the way the service works, then all bets are off anyway.
While I think your other critiques are reasonable (with an asterisk I'll get back to), this is a little conspiratorial-sounding -- and I would submit that we have a lot of circumstantial evidence from the last ~36 hours of reporting that the PR generated around this has not exactly been positive. I don't see how announcing all of these features together benefits Apple. (And at risk of being a little conspiratorial-sounding myself, I suspect there are other more intrusive mass surveillance tools already built and in use by other corporations and governments.)
As for that asterisk: assuming the service works the way Apple describes it, I don't think it's comparable to cell phone call metadata. You could obviously use the system to search for other known photos besides CSAM and also search photos that are local to the device, but other use cases still need to fit "this photo matches the hash of this other photo" (I'm not even sure if you can abstract "photo" to "other media file"). Photos, of course, have metadata that would be interesting to law enforcement agents, but if this does anything with that metadata then the service doesn't work the way Apple describes it -- and if we assume that Apple is starting out lying about the way the service works, then all bets are off anyway.
Right?
I really enjoyed the clarity. Helped me understand what was going on without trying to inject a final conclusion for me.
I really enjoyed the clarity. Helped me understand what was going on without trying to inject a final conclusion for me.
> Will Apple actually flatly refuse any and all such demands? If they do, it’s all good.
I don't understand how he can write this when there is an established record of Apple folding to such demands for access to lucrative markets.
I don't understand how he can write this when there is an established record of Apple folding to such demands for access to lucrative markets.
Because Gruber is such an Apple fanboy, it's near impossible for him to accept that Apple just gave him the finger.
Not to mention even if we could somehow trust them now, there is an infinite variables on the futures for this to change.
You're pulling that quote out as if the article is asserting the obvious answer to that question is yes, but the article's entire thesis is "the slippery-slope argument is a legitimate concern" (that's in quotes because, well, it's literally a quote). Literally the next sentence after the one you quoted:
> If they don’t, and these features creep into surveillance for things like political dissent, copyright infringement, LGBT imagery, or adult pornography — anything at all beyond irrefutable CSAM — it’ll prove disastrous to Apple’s reputation for privacy protection.
> If they don’t, and these features creep into surveillance for things like political dissent, copyright infringement, LGBT imagery, or adult pornography — anything at all beyond irrefutable CSAM — it’ll prove disastrous to Apple’s reputation for privacy protection.
I'm pulling the quote out to highlight that he's willing to believe the answer could be yes, which is absurd.
It’s a massive backdoor.
Take any person’s set of photos that they have ever put online, anywhere (publicly, via email, etc.), feed it to this back door and you can have their identity and GPS coordinates in fractions of a second.
It’s a horrifying, dystopian tool that should not exist. In any form, for any reason. Nobody can be trusted to be a gatekeeper for something like this.
Take any person’s set of photos that they have ever put online, anywhere (publicly, via email, etc.), feed it to this back door and you can have their identity and GPS coordinates in fractions of a second.
It’s a horrifying, dystopian tool that should not exist. In any form, for any reason. Nobody can be trusted to be a gatekeeper for something like this.
How many other companies are guilty of much worse?
Apple could already do this server side if they wanted to, right?
They could by implementing the photo hashing on uploaded photos. There probably is a debate to be had on the merits of your phone being used for purposes like this but I think the implementation details pale in comparison to the problems with the mass surveillance tool being created here.
It’s clear that the correct solution is to blindly encrypt photos on the user’s phone before uploading them to iCloud. Full stop.
It’s clear that the correct solution is to blindly encrypt photos on the user’s phone before uploading them to iCloud. Full stop.
If the fingerprints between an actual CSAM photo and a cropped/resized/grey-scaled version can match, it implies that the fingerprint is essentially composed of smaller fingerprints of tiny squares within a picture, as well as of fingerprints of the entire image in both its original as well as grey-scale form. It probably will also need to allow for rotation.
With all of these possibilities allowed, either the matching technology will miss most slightly-deformed matching images (criminals will quickly find effective minimal distortions), to avoid too many false positives, or will end up matching too many of people’s personal images that consist of their young kids in a state of undress (whose parents don’t have a picture or 2 of them in such a state from their childhood?).
So, either this technology is ineffective and not worth the potential misuse, or it is a grave threat to privacy.
With all of these possibilities allowed, either the matching technology will miss most slightly-deformed matching images (criminals will quickly find effective minimal distortions), to avoid too many false positives, or will end up matching too many of people’s personal images that consist of their young kids in a state of undress (whose parents don’t have a picture or 2 of them in such a state from their childhood?).
So, either this technology is ineffective and not worth the potential misuse, or it is a grave threat to privacy.
Actually it’s both without question.
I would recommend looking at how hashing of images works (there are special algorithm and concepts for it). Good reference e.g. https://github.com/JohannesBuchner/imagehash#references
of course Gruber is defending Apple.
He provides facts about how the system works, and then provides his own thoughts.
Do you disagree with the authenticity of the facts provided?
Do you disagree with the authenticity of the facts provided?
It does justify why the phone itself have to do it. Actually according to graphics from internal memo it's also unclear if the hashing is happening on every image on phone (even if it is not uploaded). https://9to5mac.com/2021/08/06/apple-internal-memo-icloud-ph...
Why put this system on the phone unless you want to expand it in the future? Cloud encryption with Apple being able to decrypt it could not be the reason. System can be also easily changed anytime from Apple to report on offline images.
Why put this system on the phone unless you want to expand it in the future? Cloud encryption with Apple being able to decrypt it could not be the reason. System can be also easily changed anytime from Apple to report on offline images.
Gruber has yet to see a real Apple stance he does not support. [1]
[1] Credit to "The Economist has yet to see a war it does not like" https://www.prospectmagazine.co.uk/magazine/what-the-economi...
[1] Credit to "The Economist has yet to see a war it does not like" https://www.prospectmagazine.co.uk/magazine/what-the-economi...
His article spits out facts on what the system is doing, then he puts out his thoughts.
Do you believe the facts he’s provided about how the system works are untrue?
Do you believe the facts he’s provided about how the system works are untrue?
> Albergotti’s alarmist lead makes it sound like all content for all users in Messages will be “scanned”, whereas in fact nothing sent to or from an adult user in Messages will ever be “scanned” — unless an image is saved from Messages to Photos and iCloud Photo Library is enabled.
That's very cold comfort. Gruber himself says that this applies to most customers.
That's very cold comfort. Gruber himself says that this applies to most customers.
A very long article of technical details skipping over the obvious fundamental principal that it’s not ok for apple to scan files on your device and trying to be part of the normalization of that abomination of a concept.
https://www.apple.com/child-safety/pdf/Apple_PSI_System_Secu...