Security Threat Model Review of the Apple Child Safety Features [pdf](apple.com)
apple.com
Security Threat Model Review of the Apple Child Safety Features [pdf]
https://www.apple.com/child-safety/pdf/Security_Threat_Model_Review_of_Apple_Child_Safety_Features.pdf
388 comments
> there's a legitimate "slippery slope" argument
The slippery slope argument is the only useful argument here.
The fundamental issue with their PSI/CSAM system is that they already were scanning iCloud content [1] and that they're seemingly not removing the ability to do that. If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in their users best interests.
But it wasn't, so as it stands there's no obvious user benefit. It then becomes a question of trust. Apple are clearly willing to add functionality at the request of governments, on device. By doing this they lose user trust (in my opinion).
> In the long run, we solve this, at least in liberal democracies, by voting people into office who understand technology, understand the value of personal encryption
But this is clearly not the way it happens in practice. At least in part, we vote with our wallets and give money to companies willing to push back on governmental over-reach. Until now, Apple was one such company [2].
I realize that Apple likely don't "care" about privacy (it's a company, not a individual human). But in a purely cynical sense, positioning themselves as caring about privacy, and pushing back against governmental over-reach on users behalf was useful. And while it's "just marketing" it benefits users.
By implementing this functionality, they've lost this "marketing benefit". Users can't buy devices believing they're supporting a company willing to defend their privacy.
[1] https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
[2] https://epic.org/amicus/crypto/apple/
The slippery slope argument is the only useful argument here.
The fundamental issue with their PSI/CSAM system is that they already were scanning iCloud content [1] and that they're seemingly not removing the ability to do that. If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in their users best interests.
But it wasn't, so as it stands there's no obvious user benefit. It then becomes a question of trust. Apple are clearly willing to add functionality at the request of governments, on device. By doing this they lose user trust (in my opinion).
> In the long run, we solve this, at least in liberal democracies, by voting people into office who understand technology, understand the value of personal encryption
But this is clearly not the way it happens in practice. At least in part, we vote with our wallets and give money to companies willing to push back on governmental over-reach. Until now, Apple was one such company [2].
I realize that Apple likely don't "care" about privacy (it's a company, not a individual human). But in a purely cynical sense, positioning themselves as caring about privacy, and pushing back against governmental over-reach on users behalf was useful. And while it's "just marketing" it benefits users.
By implementing this functionality, they've lost this "marketing benefit". Users can't buy devices believing they're supporting a company willing to defend their privacy.
[1] https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
[2] https://epic.org/amicus/crypto/apple/
> If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in their users best interests.
Absolutely. I don't know whether there's a reason for this timing (that is, if they are planning E2E encryption, why they announced this first), but this is probably the biggest PR bungle Apple has had since "you're holding it wrong," if not ever.
> Apple are clearly willing to add functionality at the request of governments, on device.
Maybe? I'm not as willing to state that quite as definitively, given the pushback Apple gave in the San Bernardino shooter case. Some of what their Privacy Engineering head said in that TechCrunch article suggests that Apple has engineered this to be strategically awkward, e.g., generating the hashes by using ML trained on the CSAM data set (so the hashing system isn't as effective on other data sets) and making the on-device hashing component part of the operating system itself rather than a separately updatable data set. That in turn suggests to me Apple is still looking for an engineering way to say "no" if they're asked "hey, can you just add these other images to your data set." (Of course, my contention that this is not ultimately an engineering problem applies here, too: even if I'm right about Apple playing an engineering shell game here, I'm not convinced it's enough if a government is sufficiently insistent.)
A minor interesting tidbit: your linked Sophos story is based on a Telegraph UK story that has this disclaimer at the bottom:
> This story originally said Apple screens photos when they are uploaded to iCloud, Apple's cloud storage service. Ms Horvath and Apple's disclaimer did not mention iCloud, and the company has not specified how it screens material, saying this information could help criminals.
It's hard to say what they were actually doing, but it's reasonable to suspect it's an earlier, perhaps entirely cloud-based rather than partially cloud-based, version of NeuralHash.
Absolutely. I don't know whether there's a reason for this timing (that is, if they are planning E2E encryption, why they announced this first), but this is probably the biggest PR bungle Apple has had since "you're holding it wrong," if not ever.
> Apple are clearly willing to add functionality at the request of governments, on device.
Maybe? I'm not as willing to state that quite as definitively, given the pushback Apple gave in the San Bernardino shooter case. Some of what their Privacy Engineering head said in that TechCrunch article suggests that Apple has engineered this to be strategically awkward, e.g., generating the hashes by using ML trained on the CSAM data set (so the hashing system isn't as effective on other data sets) and making the on-device hashing component part of the operating system itself rather than a separately updatable data set. That in turn suggests to me Apple is still looking for an engineering way to say "no" if they're asked "hey, can you just add these other images to your data set." (Of course, my contention that this is not ultimately an engineering problem applies here, too: even if I'm right about Apple playing an engineering shell game here, I'm not convinced it's enough if a government is sufficiently insistent.)
A minor interesting tidbit: your linked Sophos story is based on a Telegraph UK story that has this disclaimer at the bottom:
> This story originally said Apple screens photos when they are uploaded to iCloud, Apple's cloud storage service. Ms Horvath and Apple's disclaimer did not mention iCloud, and the company has not specified how it screens material, saying this information could help criminals.
It's hard to say what they were actually doing, but it's reasonable to suspect it's an earlier, perhaps entirely cloud-based rather than partially cloud-based, version of NeuralHash.
Right, and in the interview linked [1] above they state:
> The voucher generation is actually exactly what enables us not to have to begin processing all users’ content on our servers, which we’ve never done for iCloud Photos.
But they do appear to do "something" server-side. It's possible that all data in scanned as it is ingested for example. I dislike this statement, because it's probably technically correct but doesn't help clarify the situation in a helpful way. It makes me trust Apple less.
[1] https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
> The voucher generation is actually exactly what enables us not to have to begin processing all users’ content on our servers, which we’ve never done for iCloud Photos.
But they do appear to do "something" server-side. It's possible that all data in scanned as it is ingested for example. I dislike this statement, because it's probably technically correct but doesn't help clarify the situation in a helpful way. It makes me trust Apple less.
[1] https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
What I don’t understand is that if they announced they would do that scanning server side, the only eyebrows that would be raised is of people who thought they were doing it already. It’s not like if those pictures were e2e encrypted. I still haven’t seen any convincing argument about why searching client side provide any benefit to end users, while being a massive step in the direction of privacy invasion.
> But they do appear to do "something" server-side. It's possible that all data in scanned as it is ingested for example. I dislike this statement, because it's probably technically correct but doesn't help clarify the situation in a helpful way. It makes me trust Apple less.
The qualifier is "Photos" - different services have different security properties.
Email transport is not E2E encrypted because there are no interoperable technologies for that.
Other systems are encrypted but apple has a separate key escrow system outside the cloud hosting for law enforcement requests and other court orders (such as a heir/estate wanting access).
Some like iCloud Keychain use more E2E approach where access can't be restored if you lose all your devices and paper recovery key.
iCloud Photo Sharing normally only works between AppleID accounts, with the album keys being encrypted to the account. However, you can choose to publicly share an album, at which point it becomes accessible via a browser on icloud.com. I have not heard Apple talking about whether they scan photos today once they are marked public (going forward, there would be no need).
FWIW this is all publicly documented, as well as what information Apple can and can't provide to law enforcement.
The qualifier is "Photos" - different services have different security properties.
Email transport is not E2E encrypted because there are no interoperable technologies for that.
Other systems are encrypted but apple has a separate key escrow system outside the cloud hosting for law enforcement requests and other court orders (such as a heir/estate wanting access).
Some like iCloud Keychain use more E2E approach where access can't be restored if you lose all your devices and paper recovery key.
iCloud Photo Sharing normally only works between AppleID accounts, with the album keys being encrypted to the account. However, you can choose to publicly share an album, at which point it becomes accessible via a browser on icloud.com. I have not heard Apple talking about whether they scan photos today once they are marked public (going forward, there would be no need).
FWIW this is all publicly documented, as well as what information Apple can and can't provide to law enforcement.
Even without publicly sharing your iCloud photos, they are accessible on iCloud.com (e.g. you can see your camera role there).
> This is an area we’ve been looking at for some time, including current state of the art techniques which mostly involves scanning through entire contents of users’ libraries on cloud services that — as you point out — isn’t something that we’ve ever done; to look through users’ iCloud Photos.
https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
> This moment calls for public discussion, and we want our customers and people around the country to understand what is at stake.
- Tim Cook, Apple
At what point does the fact that half a decade has passed since those words were written yet the hacker community has made little contribution to that discourse about the importance of privacy start implicating us in the collective failure to act?
https://techcrunch.com/2021/08/10/interview-apples-head-of-p...
> This moment calls for public discussion, and we want our customers and people around the country to understand what is at stake.
- Tim Cook, Apple
At what point does the fact that half a decade has passed since those words were written yet the hacker community has made little contribution to that discourse about the importance of privacy start implicating us in the collective failure to act?
And yet we can go e.g. on Twitter and observe comments from relevant security researchers that appear to describe a chilling atmosphere surrounding privacy research, including resistance to even consider white paper.
That’s not my area of expertise and don’t know how to fix that, but that should be an important consideration.
That’s not my area of expertise and don’t know how to fix that, but that should be an important consideration.
> The voucher generation is actually exactly what enables us not to have to begin processing all users’ content on our servers, which we’ve never done for iCloud Photos.
I really dislike this statement. It's likely designed to be "technically true". But it's been reported elsewhere that they do scan iCloud content:
https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
Perhaps they scan as the data is being ingested. Perhaps it's scanned on a third party server. But it seems clear that it is being scanned.
I really dislike this statement. It's likely designed to be "technically true". But it's been reported elsewhere that they do scan iCloud content:
https://nakedsecurity.sophos.com/2020/01/09/apples-scanning-...
Perhaps they scan as the data is being ingested. Perhaps it's scanned on a third party server. But it seems clear that it is being scanned.
https://www.forbes.com/sites/thomasbrewster/2020/02/11/how-a...
My interpretation is Sophos got it wrong (they don't give a quote from the Apple Officer involved and manage to have a typo in the headline).
Apple does scanning of data which is not encrypted, such as received and sent email over SMTP. They presumably at that time were using PhotoDNA to scan attachments by hash. This is likely what Apple was actually talking about back at CES 2020.
They may have been also scanning public iCloud photo albums, but I haven't seen anyone discuss that one way or another.
My interpretation is Sophos got it wrong (they don't give a quote from the Apple Officer involved and manage to have a typo in the headline).
Apple does scanning of data which is not encrypted, such as received and sent email over SMTP. They presumably at that time were using PhotoDNA to scan attachments by hash. This is likely what Apple was actually talking about back at CES 2020.
They may have been also scanning public iCloud photo albums, but I haven't seen anyone discuss that one way or another.
That link doesn’t confirm that they have already been doing it. Just that they changed an EULA.
My understanding based on piecing together the various poorly cited news stories is that Apple used to scan iCloud Mail for this material, and that’s it.
If you have references to also help me piece this together I'd find that really helpful.
Forbes had the only evidence based reporting where they cited a court case where Apple automatically detected known CSAM in attachments to iCloud Mail: https://www.forbes.com/sites/thomasbrewster/2020/02/11/how-a...
Everyone else is making inferences from a chance to their privacy policy and a statement made by a company lawyer.
Everyone else is making inferences from a chance to their privacy policy and a statement made by a company lawyer.
> Last year, for instance, Apple reported 265 cases to the National Center for Missing & Exploited Children, while Facebook reported 20.3 million
According to [1] it does seem like Apple didn't do any wide scale scanning of iCloud Data.
[1] https://www.nytimes.com/2021/08/05/technology/apple-iphones-...
According to [1] it does seem like Apple didn't do any wide scale scanning of iCloud Data.
[1] https://www.nytimes.com/2021/08/05/technology/apple-iphones-...
If they weren't doing any scanning why would they find any to report? The data is encrypted as rest so... why would they find any to report. This clearly doesn't include search requests [1].
iCloud has perhaps 25% of the users of Facebook. Of that 25% it's not clear how many actively use the platform of backups/photos. iCloud is not a platform for sharing content like Facebook. So how many reports should we expect to see from Apple? It's unclear to me.
So, I'm not saying the number isn't suspiciously low. But it doesn't really clarify what's going on to me...
[1] https://www.apple.com/legal/transparency/pdf/requests-2018-H...
iCloud has perhaps 25% of the users of Facebook. Of that 25% it's not clear how many actively use the platform of backups/photos. iCloud is not a platform for sharing content like Facebook. So how many reports should we expect to see from Apple? It's unclear to me.
So, I'm not saying the number isn't suspiciously low. But it doesn't really clarify what's going on to me...
[1] https://www.apple.com/legal/transparency/pdf/requests-2018-H...
> The fundamental issue with their PSI/CSAM system is that they already were scanning iCloud content
This scanning was of email attachments being sent through an iCloud-hosted account, not of other iCloud hosted data (which is encrypted during operation.)
This scanning was of email attachments being sent through an iCloud-hosted account, not of other iCloud hosted data (which is encrypted during operation.)
I don’t think that photos are encrypted as you can view them from the iCloud website.
If it’s encrypted but apple has the key, it’s not encrypted to them.
Do you have a public reference for this?
> If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in their users best interests.
As I understand it, this was more "leaked" than "announced". That is, it wasn't part of Apple's planned rollout strategy.
My thought was that they'll announce that soon, but then again, I'm shocked soon wasn't last week. I have no idea. Maybe there is some other limitation (legal) on he iCloud E2E backups they needed to solve first?
As I understand it, this was more "leaked" than "announced". That is, it wasn't part of Apple's planned rollout strategy.
My thought was that they'll announce that soon, but then again, I'm shocked soon wasn't last week. I have no idea. Maybe there is some other limitation (legal) on he iCloud E2E backups they needed to solve first?
> If the PSI/CSAM system had been announced along side E2E encryption for iCloud backups, it would be clear that they were attempting to act in their users best interests.
Most likely because the only way this announcement makes sense, is that they tried to respond for misleading leaks. We'll see on September probably some E2EE announcements, since iOS 15 beta supports tokens for backup recovery. At least, let's hope so.
Most likely because the only way this announcement makes sense, is that they tried to respond for misleading leaks. We'll see on September probably some E2EE announcements, since iOS 15 beta supports tokens for backup recovery. At least, let's hope so.
Lots of people have made policy arguments. No US law requires client side scanning. No US law forbids E2E encryption. US courts don't let law enforcement agencies just demand everything they want from companies. Apple relied on that 5 years ago successfully.[1] And capitulating preemptively is bad strategy usually.
What Neuenschwander said doesn't establish it isn't just an arbitrary limitation.
[1] https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute
Where the hashes get checked is relevant to the policy problem of what
What Neuenschwander said doesn't establish it isn't just an arbitrary limitation.
[1] https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute
Where the hashes get checked is relevant to the policy problem of what
That’s different. The FBI can legally require Apple or any other US company to search for specific files it has access to on it’s own servers because nothing currently shields backup providers. They could and did force Apple to aid in unlocking iPhones when Apple had that capacity. What they couldn’t do was “These orders would compel Apple to write new software that would let the government bypass these devices' security and unlock the phones.”
Forcing companies to create back doors in their own is legally a very different situation. As to why iCloud is accessible by Apple, the point is to backup a phone someone lost. Forcing people to keep some sort of key fob with a secure private key safe in order to actually have access to their backups simply isn’t tenable.
Forcing companies to create back doors in their own is legally a very different situation. As to why iCloud is accessible by Apple, the point is to backup a phone someone lost. Forcing people to keep some sort of key fob with a secure private key safe in order to actually have access to their backups simply isn’t tenable.
Apple is a trillion dollar company with a lot of smart people. You could probably get them to design a system of N of M parts for recovery, or an apple branded key holder that you can store in your bank vault and friends houses. If they wanted to they'd do it.
More than that: Apple already designed and partially implemented such a "trust circles" system.
Apple legal killed the feature, because of pressure from the US government.
https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
They also run iCloud (mostly not e2e) on CCP-controlled servers for users in China.
They can decrypt ~100% of iMessages in real-time due to the way iCloud Backup (on by default, not e2e) escrows iMessage sync keys.
Apple does not protect your mprivacy from Apple, or, by extension, the governments that ultimately exert control over Apple: China and the USA.
Apple legal killed the feature, because of pressure from the US government.
https://www.reuters.com/article/us-apple-fbi-icloud-exclusiv...
They also run iCloud (mostly not e2e) on CCP-controlled servers for users in China.
They can decrypt ~100% of iMessages in real-time due to the way iCloud Backup (on by default, not e2e) escrows iMessage sync keys.
Apple does not protect your mprivacy from Apple, or, by extension, the governments that ultimately exert control over Apple: China and the USA.
That’s not what the article says, they simply don’t know.
“However, a former Apple employee said it was possible the encryption project was dropped for other reasons, such as concern that more customers would find themselves locked out of their data more often.”
“However, a former Apple employee said it was possible the encryption project was dropped for other reasons, such as concern that more customers would find themselves locked out of their data more often.”
Each year, Apple gives up customer data on over 150,000 users based on US government data requests, and NSL and FISA requests[1].
The idea that Apple would fight this is a farce, as they regularly give up customers' data without a fight when the government requests it.
[1] https://www.apple.com/legal/transparency/us.html
The idea that Apple would fight this is a farce, as they regularly give up customers' data without a fight when the government requests it.
[1] https://www.apple.com/legal/transparency/us.html
The idea that Apple would fight this is a farce, as they regularly give up customers' data without a fight when the government requests it.
There are laws regarding this, so they don't have a choice. If they get a subpoena from a FISA court, there's not much they can do, but that goes for every US-based company.
Whatever fighting is going on is behind the scenes, so we wouldn't know about it.
There are laws regarding this, so they don't have a choice. If they get a subpoena from a FISA court, there's not much they can do, but that goes for every US-based company.
Whatever fighting is going on is behind the scenes, so we wouldn't know about it.
None of the laws do yet. My observation isn't about the laws as they necessarily exist now, just as the worry about how this could be abused isn't about Apple's policy as it exists now.
If we trust US courts to stop law enforcement agencies from demanding everything they want from companies, they they can stop law enforcement agencies from demanding Apple add non-CSAM data to the NeuralHash set. If we don't trust the courts to do that, then we're kind of back at square one, right?
If we trust US courts to stop law enforcement agencies from demanding everything they want from companies, they they can stop law enforcement agencies from demanding Apple add non-CSAM data to the NeuralHash set. If we don't trust the courts to do that, then we're kind of back at square one, right?
I'm not American, but my understanding is that as soon as Government is forcing Apple to search our devices for something, 4th Amendment protections apply. (Unless they hold a search warrant for that specific person, of course.) Is this not correct?
No. The 4A protections don't apply to third parties. This is part of why the US has nearly nonexistent data protection laws.
> The 4A protections don't apply to third parties.
The government can't pay someone to break into your house and steal evidence they want without a warrant. I mean, they can, but the evidence wouldn't be admissible in court.
The government can't pay someone to break into your house and steal evidence they want without a warrant. I mean, they can, but the evidence wouldn't be admissible in court.
They don't need a warrant. You gave data to someone else. That someone isn't bound to keep it secret. They can demand a warrant if they are motivated by ethical principles but that is optional and potentially overruled by other laws.
But if they're looking for incriminating evidence on your private property (i.e. on-device scanning) then they do need a warrant. It doesn't matter if a copy of it was also given to a third party (i.e. uploaded to iCloud) what matters is where the actual search takes place.
The authorities aren't doing the scanning. You will be made to agree in the fine print to let Apple do it when iCloud sync is enabled. If they run across evidence of a crime then c'est la vie.
If Apple was performing scans on their cloud servers, you'd be absolutely right. But if the scanning is being done on the individual's device, I'm not sure it's that straightforward. The third party doctrine surely cannot apply if the scanning is performed prior to the material being in third party hands.
Therefore if the Government forces Apple to change the search parameters contained within private devices, I cannot see how this would work around the 4th Amendment.
If this is correct, it might be possible to argue that Apple's approach has (for Americans) constitutional safeguards which do not exist for on-cloud scanning performed by Google or Microsoft.
Therefore if the Government forces Apple to change the search parameters contained within private devices, I cannot see how this would work around the 4th Amendment.
If this is correct, it might be possible to argue that Apple's approach has (for Americans) constitutional safeguards which do not exist for on-cloud scanning performed by Google or Microsoft.
This point was made in the economist today.
https://www.economist.com/united-states/2021/08/12/a-38-year...
https://www.economist.com/united-states/2021/08/12/a-38-year...
I read the article; I don't think they highlighted this specific point that on-device scanning has a potential, hypothetical constitutional advantage in comparison to Google, Microsoft and Facebook who scan exclusively in the cloud.
They don’t draw out the comparison, but they do mention the protection.
You are right. But I think the distinction is an important one to make. People might not be aware about this critical distinction.
I did find someone in the media making this point, so it turns out I'm not being completely original: https://9to5mac.com/2021/08/10/misusing-csam-scanning-in-us-...
I did find someone in the media making this point, so it turns out I'm not being completely original: https://9to5mac.com/2021/08/10/misusing-csam-scanning-in-us-...
The sort of questions about 4A protections here haven't really been tested. Third party doctrine might not apply in this circumstance and the court is slowly evolving with the times.
e.g. https://en.wikipedia.org/wiki/Carpenter_v._United_States
In Carpenter v. United States (2018), the Supreme Court ruled warrants are needed for gathering cell phone tracking information, remarking that cell phones are almost a “feature of human anatomy”, “when the Government tracks the location of a cell phone it achieves near perfect surveillance, as if it had attached an ankle monitor to the phone’s user”.
...[cell-site location information] provides officers with “an all-encompassing record of the holder’s whereabouts” and “provides an intimate window into a person’s life, revealing not only [an individual’s] particular movements, but through them [their] familial, political, professional, religious, and sexual associations.”
In Carpenter v. United States (2018), the Supreme Court ruled warrants are needed for gathering cell phone tracking information, remarking that cell phones are almost a “feature of human anatomy”, “when the Government tracks the location of a cell phone it achieves near perfect surveillance, as if it had attached an ankle monitor to the phone’s user”.
...[cell-site location information] provides officers with “an all-encompassing record of the holder’s whereabouts” and “provides an intimate window into a person’s life, revealing not only [an individual’s] particular movements, but through them [their] familial, political, professional, religious, and sexual associations.”
> And capitulating preemptively is bad strategy usually.
Why do you think they are doing it then?
Why do you think they are doing it then?
> It was illegal to export "strong encryption" for many years, remember? I've seen multiple reports that European lawmakers are planning to require some kind of scanning for CSAM. If this goes into effect, technology isn't going to block those laws for you. Your Purism phone will either be forced to comply or be illegal.
The point is that with a Purism phone or custom ROM on my Android phone, I could disable these "legally required" features, because the law is fucking dumb, and my rights matter more.
The law can ban E2EE, cryptocurrencies, and privacy, but so long as we have some degree of technical freedom we can and will give it the middle finger.
Apple does not offer this freedom. Here we see the walled garden of iOS getting worse and more freedom-restricting by the year. When the governments of the world demand that Apple become an arm of the dystopia, Apple will comply, and its users will have no choice but to go along with it.
Apple, knowing that it is a private company completely and utterly incapable of resisting serious government demands (ie GCBD in China) should never have developed this capability to begin with.
If Apple is going to open this Pandora's box, they ought to open up their devices too.
The point is that with a Purism phone or custom ROM on my Android phone, I could disable these "legally required" features, because the law is fucking dumb, and my rights matter more.
The law can ban E2EE, cryptocurrencies, and privacy, but so long as we have some degree of technical freedom we can and will give it the middle finger.
Apple does not offer this freedom. Here we see the walled garden of iOS getting worse and more freedom-restricting by the year. When the governments of the world demand that Apple become an arm of the dystopia, Apple will comply, and its users will have no choice but to go along with it.
Apple, knowing that it is a private company completely and utterly incapable of resisting serious government demands (ie GCBD in China) should never have developed this capability to begin with.
If Apple is going to open this Pandora's box, they ought to open up their devices too.
> ... because the law is fucking dumb, and my rights matter more.
If they aren't _everybody's_ rights then they aren't really your rights either. They are at best a privilege, and at worst something you have just been able to get away with (so far).
> The law can ban E2EE, cryptocurrencies, and privacy, but so long as we have some degree of technical freedom we can and will give it the middle finger.
Sure, in that scenario techies can secretly give it the middle finger right up until the authoritarian government they idly watched grow notices them.
If someone is seriously concerned about that happening, they could always consider trying to divert the government away from such disaster by participating.
> When the governments of the world demand that Apple become an arm of the dystopia, Apple will comply, and its users will have no choice but to go along with it.
Government demands of this sort are normally referred to as legal and regulatory compliance. Corporations, which are a legal concept allowed by the government, generally have to conform to continue to exist.
> Apple, knowing that it is a private company completely and utterly incapable of resisting serious government demands (ie GCBD in China) should never have developed this capability to begin with.
IMHO, having some portion of a pre-existing capability doesn't matter when you aren't legally allowed to challenge the request or answer "no".
If they aren't _everybody's_ rights then they aren't really your rights either. They are at best a privilege, and at worst something you have just been able to get away with (so far).
> The law can ban E2EE, cryptocurrencies, and privacy, but so long as we have some degree of technical freedom we can and will give it the middle finger.
Sure, in that scenario techies can secretly give it the middle finger right up until the authoritarian government they idly watched grow notices them.
If someone is seriously concerned about that happening, they could always consider trying to divert the government away from such disaster by participating.
> When the governments of the world demand that Apple become an arm of the dystopia, Apple will comply, and its users will have no choice but to go along with it.
Government demands of this sort are normally referred to as legal and regulatory compliance. Corporations, which are a legal concept allowed by the government, generally have to conform to continue to exist.
> Apple, knowing that it is a private company completely and utterly incapable of resisting serious government demands (ie GCBD in China) should never have developed this capability to begin with.
IMHO, having some portion of a pre-existing capability doesn't matter when you aren't legally allowed to challenge the request or answer "no".
> The point is that with a Purism phone or custom ROM on my Android phone, I could disable these "legally required" features, because the law is fucking dumb, and my rights matter more.
But if we assume a government is determined to do this, can't they find other ways to do it? If you were using Google Photos with your Purism phone, it doesn't matter what you do on your device. And you can say "well, I wouldn't use that," but maybe your ISP is convinced (or required) to do packet inspection. And then you can say, "But I'm using encryption," and the government mandates that they have a back door into all encrypted traffic that goes through their borders.
And I would submit that if we really assume a government is going to extreme lengths, then they'll make it as hard as possible to use an open phone in the first place. They'll make custom ROMs illegal. They'll go after people hosting it. They'll mandate that the phones comply with some kind of decryption standard to connect to cellular data networks. If we assume an authoritarian government bound and determined to spy on you, the assumption that we can be saved by just applying enough open source just seems pretty shaky to me.
So, I certainly don't think that a purely technological solution is enough, in the long run. This is a policy issue. I think hackers and engineers really, really want to believe that math trumps policy, but it doesn't. By all means, let's fight for strong encryption -- but let's also fight for government policy that supports it, rather than assuming encryption and open source is a guarantee we can circumvent bad policy.
But if we assume a government is determined to do this, can't they find other ways to do it? If you were using Google Photos with your Purism phone, it doesn't matter what you do on your device. And you can say "well, I wouldn't use that," but maybe your ISP is convinced (or required) to do packet inspection. And then you can say, "But I'm using encryption," and the government mandates that they have a back door into all encrypted traffic that goes through their borders.
And I would submit that if we really assume a government is going to extreme lengths, then they'll make it as hard as possible to use an open phone in the first place. They'll make custom ROMs illegal. They'll go after people hosting it. They'll mandate that the phones comply with some kind of decryption standard to connect to cellular data networks. If we assume an authoritarian government bound and determined to spy on you, the assumption that we can be saved by just applying enough open source just seems pretty shaky to me.
So, I certainly don't think that a purely technological solution is enough, in the long run. This is a policy issue. I think hackers and engineers really, really want to believe that math trumps policy, but it doesn't. By all means, let's fight for strong encryption -- but let's also fight for government policy that supports it, rather than assuming encryption and open source is a guarantee we can circumvent bad policy.
> And then you can say, "But I'm using encryption," and the government mandates that they have a back door into all encrypted traffic that goes through their borders.
And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this.
I'm personally really interested in router level malware + 0days on devices as distribution vectors rather than the typical c&c setup.
> They'll go after people hosting it.
Not too hard to imagine one being able to distribute such things across millions of ephemeral devices that are networked and incentivized to host it, all across the world, regardless of illegality in any particular jurisdiction. Technology enables this, without such, it wont be possible.
> I think hackers and engineers really, really want to believe that math trumps policy, but it doesn't
I don't think that at all, I think it comes down to incentives. I was listening to a talk the other day where someone mentioned that for the longest time (since at least wwII), governments pretty much had a monopoly on cryptographers and now there are lots of places/systems that are willing to pay more to apply cutting edge research.
> but let's also fight for government policy that supports it, rather than assuming encryption and open source is a guarantee we can circumvent bad policy.
Much more cheaper for an individual, with more immediate feedback mechanisms doing one vs another. One can also scale a lot faster than another esp since one is very much divorced from implementation.
And then one can compromise and infect millions of such backdoored devices and start feeding (much cheaper than the government enforcement implementation) spoofed data into these systems at scale on these backdoored devices that act like "swatting as a service" and completely nullify any meaning they could get from doing this.
I'm personally really interested in router level malware + 0days on devices as distribution vectors rather than the typical c&c setup.
> They'll go after people hosting it.
Not too hard to imagine one being able to distribute such things across millions of ephemeral devices that are networked and incentivized to host it, all across the world, regardless of illegality in any particular jurisdiction. Technology enables this, without such, it wont be possible.
> I think hackers and engineers really, really want to believe that math trumps policy, but it doesn't
I don't think that at all, I think it comes down to incentives. I was listening to a talk the other day where someone mentioned that for the longest time (since at least wwII), governments pretty much had a monopoly on cryptographers and now there are lots of places/systems that are willing to pay more to apply cutting edge research.
> but let's also fight for government policy that supports it, rather than assuming encryption and open source is a guarantee we can circumvent bad policy.
Much more cheaper for an individual, with more immediate feedback mechanisms doing one vs another. One can also scale a lot faster than another esp since one is very much divorced from implementation.
> Apple, knowing that it is a private company completely and utterly incapable of resisting serious government demands (ie GCBD in China) should never have developed this capability to begin with.
You are basically arguing that an iphone be incablable of doing any function at all.
Dont want the government demanding the sent/recieved data - no data sending and recieving funcitons.
Dont want the government demanding phone call intercepts - no phone call functionality.
Dont want the government demanding the contents of the screen - no screen.
The pandoras box was opened the day someone inseted a radio chip and microphone into a device. It has been open for a very long time, this is not the moment it suddenly opened.
You are basically arguing that an iphone be incablable of doing any function at all.
Dont want the government demanding the sent/recieved data - no data sending and recieving funcitons.
Dont want the government demanding phone call intercepts - no phone call functionality.
Dont want the government demanding the contents of the screen - no screen.
The pandoras box was opened the day someone inseted a radio chip and microphone into a device. It has been open for a very long time, this is not the moment it suddenly opened.
When the governments of the world demand that Apple become an arm of the dystopia, Apple will comply, and its users will have no choice but to go along with it.
I would argue that Apple is creating systems so they can't become an arm of the dystopia.
For example, even if a government somehow forced Apple to include non-CSAM hashes to the database, the system only uses hashes from multiple child protection agencies in different jurisdictions where the CSAM is the same.
So Apple only uses the hashes that are the same between org A, B and C and ignores the rest.
This, along with the audibility Apple recently announced and the other features makes it so there's literally nothing counties can do to force Apple to comply with some dystopian nightmare…
Of course, with potentially more open operating systems, it would be trivial by comparison for state actors to create a popular/custom ROM for Android that's backdoored.
I would argue that Apple is creating systems so they can't become an arm of the dystopia.
For example, even if a government somehow forced Apple to include non-CSAM hashes to the database, the system only uses hashes from multiple child protection agencies in different jurisdictions where the CSAM is the same.
So Apple only uses the hashes that are the same between org A, B and C and ignores the rest.
This, along with the audibility Apple recently announced and the other features makes it so there's literally nothing counties can do to force Apple to comply with some dystopian nightmare…
Of course, with potentially more open operating systems, it would be trivial by comparison for state actors to create a popular/custom ROM for Android that's backdoored.
>I would argue that Apple is creating systems so they can't become an arm of the dystopia.
For the life of me I can't see how catching child molesters is part of a dystopia.
For the life of me I can't see how catching child molesters is part of a dystopia.
One way to catch child molesters would be to have everyone wear a body cam at all times. Would that be dystopian enough for you?
> Will this stop those bad actors if they're determined? No, of course not, but there are so many ways they can do it already.
This is not a reason to let your guard down on security. Keeping up with securing things against bad actors is a constant battle. Tim Cook put it best [1] and I want to hear how this is not exactly what he described 5 years ago.
[1] https://youtu.be/rQebmygKq7A?t=57
This is not a reason to let your guard down on security. Keeping up with securing things against bad actors is a constant battle. Tim Cook put it best [1] and I want to hear how this is not exactly what he described 5 years ago.
[1] https://youtu.be/rQebmygKq7A?t=57
This is the refreshing analysis that I hope to see when I come to HN. Thank you for being reasonable.
> Our system involves both an on-device component where the voucher is created, but nothing is learned, and a server-side component, which is where that voucher is sent along with data coming to Apple service and processed across the account to learn if there are collections of illegal CSAM. That means that it is a service feature.
Neuenschwander seems to, maybe deliberately, be conflating : "Apple's servers have to be in the loop" and "the code can only look at photos on iCloud".
You are right, the problem is a "slippery slope," but Apple just built roller skates and there are governments trying to push us down it. Apple is in a far better position to resist those efforts if they say " we don't have this code, we will not build it, and there's no way for it to be safe for our users."
I'd say that's a little different than the slippery slope. Something more like (in)defense in depth.
Neuenschwander seems to, maybe deliberately, be conflating : "Apple's servers have to be in the loop" and "the code can only look at photos on iCloud".
You are right, the problem is a "slippery slope," but Apple just built roller skates and there are governments trying to push us down it. Apple is in a far better position to resist those efforts if they say " we don't have this code, we will not build it, and there's no way for it to be safe for our users."
I'd say that's a little different than the slippery slope. Something more like (in)defense in depth.
I have no doubt that Apple doesn't really want to implement this, and have tried to find a "non intrusive" way of doing it.
The sad truth is that once lawmakers decide they want access to all data all the time, there's nothing Apple, Google or Microsoft can do to stop that, except going out of business.
That being said, the big players in cloud storage have scanned your files for a decade. Google, Microsoft, Amazon, they all scan files being uploaded. Apple may also be doing this for iCloud Drive.
The only way to circumvent this is to use E2E encryption, either by using a service that has E2E encryption built in, or by "rolling your own", i.e. using Cryptomator. Again, this is just one law away from being outlawed.
The thing that bothers me most is that hashes are somewhat trivial to spoof[1], so if someone was to get a hold of the list of hashes, they could start sending spoof messages on a large scale, causing a false positive.
I remember when Echelon was making the rounds on the internet a couple of decades ago, and lots of people started adding X headers to their emails to cause false positives.
[1] https://towardsdatascience.com/black-box-attacks-on-perceptu...
The sad truth is that once lawmakers decide they want access to all data all the time, there's nothing Apple, Google or Microsoft can do to stop that, except going out of business.
That being said, the big players in cloud storage have scanned your files for a decade. Google, Microsoft, Amazon, they all scan files being uploaded. Apple may also be doing this for iCloud Drive.
The only way to circumvent this is to use E2E encryption, either by using a service that has E2E encryption built in, or by "rolling your own", i.e. using Cryptomator. Again, this is just one law away from being outlawed.
The thing that bothers me most is that hashes are somewhat trivial to spoof[1], so if someone was to get a hold of the list of hashes, they could start sending spoof messages on a large scale, causing a false positive.
I remember when Echelon was making the rounds on the internet a couple of decades ago, and lots of people started adding X headers to their emails to cause false positives.
[1] https://towardsdatascience.com/black-box-attacks-on-perceptu...
> The entire point of having a liberal democracy is that we are the government, and we can pull it back from authoritarianism.
The counterpoint: we (as in, the Western-aligned countries) don't have a true liberal democracy and likely never had. Not with the amount of open and veiled influence that religion (and for what it's worth, money) has in our societies - ranging from openly Christian centrist/center-right parties in Europe to entire communities in the US dominated by religious sects of all denominations.
And all of these tend to run on a "think about the children" mindset, especially regarding anything LGBT.
The result? It is very hard if not outright impossible to prevent or roll back authoritarian measures that were sold as "protect the children", since dominant religious-affiliated people and institutions (not just churches, but also thinktanks, parties and media) will put anyone in their crosshairs. Just look at how almost all blog pieces and many comments on the CSAM scanner debacle have an "I don't like pedophilia" disclaimer...
The counterpoint: we (as in, the Western-aligned countries) don't have a true liberal democracy and likely never had. Not with the amount of open and veiled influence that religion (and for what it's worth, money) has in our societies - ranging from openly Christian centrist/center-right parties in Europe to entire communities in the US dominated by religious sects of all denominations.
And all of these tend to run on a "think about the children" mindset, especially regarding anything LGBT.
The result? It is very hard if not outright impossible to prevent or roll back authoritarian measures that were sold as "protect the children", since dominant religious-affiliated people and institutions (not just churches, but also thinktanks, parties and media) will put anyone in their crosshairs. Just look at how almost all blog pieces and many comments on the CSAM scanner debacle have an "I don't like pedophilia" disclaimer...
I agree that this is a policy issue. The EU passed a new law just a last month regarding this [0].
I thought this was the implementation for the EU. If it was - it was fast?
[0] https://news.ycombinator.com/item?id=27753727
I thought this was the implementation for the EU. If it was - it was fast?
[0] https://news.ycombinator.com/item?id=27753727
I still see a lot of people thinking the EU law demands the scanning of images for CSAM, but really it permits the scanning of images again. Apparently no one noticed that EU privacy laws actually prohibited the scanning of user images until last year, when companies like Facebook stopped scanning images to avoid fines.
So Apple's move can hardly have been for the EU, but had the European Parliament not passed it, Apple would have had to disable it in the EU.
So Apple's move can hardly have been for the EU, but had the European Parliament not passed it, Apple would have had to disable it in the EU.
>Apple is treating CSAM as an engineering problem.
No they're treating it as a political and legal problem (with the UK and the EU being the furthest along on passing legislation). Their implementation is the compromise that preserves end-to-end encryption, given those political winds.
No they're treating it as a political and legal problem (with the UK and the EU being the furthest along on passing legislation). Their implementation is the compromise that preserves end-to-end encryption, given those political winds.
Agree. I think a lot of HN is seeing it as strictly an engineering problem while completely ignoring the blowing political winds.
Absolutely.
We've seen this time after time where the 3 letter agencies give companies an ultimatum: either comply or get shut down.
The worse part is that nobody can do anything about it. Under the cloak of secrecy and threats a faceless government is shaping major policy, breaking laws etc...
My only hope is that the biggest companies can speak up since they have a bit of a leverage. They can rally people behind them if the requests are borderline unethical, like spying on everybody.
If Apple can't deal with this, NOBODY else can.
We've seen this time after time where the 3 letter agencies give companies an ultimatum: either comply or get shut down.
The worse part is that nobody can do anything about it. Under the cloak of secrecy and threats a faceless government is shaping major policy, breaking laws etc...
My only hope is that the biggest companies can speak up since they have a bit of a leverage. They can rally people behind them if the requests are borderline unethical, like spying on everybody.
If Apple can't deal with this, NOBODY else can.
"either comply or get shut down'
I think if a three letter agency 'shuts down' Apple, the political blowback will be nuclear. The agency will get put on a leash if they pull some shit like tthat
I think if a three letter agency 'shuts down' Apple, the political blowback will be nuclear. The agency will get put on a leash if they pull some shit like tthat
That's why i said big companies have some leverage. People will notice.
If they don't speak up then nobody else can.
If they don't speak up then nobody else can.
>The entire point of having a liberal democracy is that we are the government, and we can pull it back from authoritarianism.
We technically can. We also can technically elect people that understand technology. But practically its 100x more likely that I, a person that works with technology for a living, magically finds a way to provide my family with a decent lifestyle doing somethign that doesn't use computers at all. And I view the chances of that happening to be almost non-existent.
We technically can. We also can technically elect people that understand technology. But practically its 100x more likely that I, a person that works with technology for a living, magically finds a way to provide my family with a decent lifestyle doing somethign that doesn't use computers at all. And I view the chances of that happening to be almost non-existent.
What's the contrarion question?
Hmm. I should perhaps have said "contrarian view," although I'm not sure it's actually even super contrarian in retrospect. Maybe more "maybe we're not asking the right questions."
[deleted]
“never doing anything with computers again, which admittedly will probably be effective.”
Increasingly I believe this is the right and probably only answer. How do we collectively accomplish this, that’s the real problem. It has to be financially and economically solved. Political, social and asymmetric solutions won’t work.
Increasingly I believe this is the right and probably only answer. How do we collectively accomplish this, that’s the real problem. It has to be financially and economically solved. Political, social and asymmetric solutions won’t work.
In this case I still think there is an engineering problem too: A bad actor only needs to get a bunch of CSAM pictures in your iCloud library to get you into big trouble.
This is not very hard to do. For example, WhatsApp has the "Save pictures to camera roll" option, on by default.
This is not very hard to do. For example, WhatsApp has the "Save pictures to camera roll" option, on by default.
Speaking of mobile OS. I am a bit of a newbie myself in this area. I am an Android user but I want to decouple from Google as much as possible. Is there an mobile OS out there that offers a similar experience to, say, Android in terms of functionalities, apps, etc without the drawback of privacy concerns?
The problem is that many Android apps require Google services to function properly. You can try two Android derivatives: CalyxOS[1] that implements a privacy-conscious subset of Google services allowing many Android apps to work properly, and GrapheneOS[2] that excludes Google services altogether at the cost of lower app compatibility[3]. Both require using Google Pixel hardware.
[1] https://calyxos.org/ [2] https://grapheneos.org/ [3] "GrapheneOS vs CalyxOS ULTIMATE COMPARISON (Battery & Speed Ft. Stock Android & iPhone)", https://www.youtube.com/watch?v=7iS4leau088
[1] https://calyxos.org/ [2] https://grapheneos.org/ [3] "GrapheneOS vs CalyxOS ULTIMATE COMPARISON (Battery & Speed Ft. Stock Android & iPhone)", https://www.youtube.com/watch?v=7iS4leau088
> GrapheneOS[2] that excludes Google services altogether at the cost of lower app compatibility[3].
It now has https://grapheneos.org/usage#sandboxed-play-services providing broader app compatibility.
That video is quite misleading and it's not the best source for accurate information about GrapheneOS.
It now has https://grapheneos.org/usage#sandboxed-play-services providing broader app compatibility.
That video is quite misleading and it's not the best source for accurate information about GrapheneOS.
Thank you for pointing this out, I will start linking to the updated material.
thanks!
Your best bets are GrapheneOS or CalyxOS. In both cases be prepared to sacrifice a lot in terms of convenience (more with GrapheneOS).
thanks!
> … the less likely it seems that it would make it radically easier for those bad actors to do so
Depends on implementation. I can easily see a possibility that the check is gated by a server side logic that can be changed at any moment without anybody knowing.
Depends on implementation. I can easily see a possibility that the check is gated by a server side logic that can be changed at any moment without anybody knowing.
> The more we learn about the way Apple actually implemented this technology, the less likely it seems that it would make it radically easier for those bad actors to do so.
Then why isn’t Apple pushing this angle?
Then why isn’t Apple pushing this angle?
> It's a policy problem. It's a governance problem. In the long run, we solve this, at least in liberal democracies, by voting people into office who understand technology, understand the value of personal encryption
Yes, it is ultimately policy problem. But the way we get people in office who understand technology is to get technological capabilities in the hands of people before they get into office, as well as the hands of those who will vote for them. Just like "bad facts make bad law", bad engineering makes bad law.
Twenty years ago we forcefully scoffed at the Clipper Chip proposal and the export ban on crypto, because they were so at odds with the actual reality of the digital environment. These days, most people's communications are mediated by large corporations operating on plaintext, and are thus are straightforward to monitor and censor. And especially when companies lead the charge, governments expect to have the same ability.
If I could hole up and rely on Free software to preserve my rights indefinitely, I wouldn't particularly care what the Surveillance Valley crowd was doing with their MITM scheme. But I can't, because Surveillance Valley is teaching governments that communications can be controlled while also fanning the flames and creating glaring examples of why they need to be controlled (cf social media "engagement" dumpster fire). And once governments expect that technology can be generally controlled, they will rule any software that does not do their bidding as some exceptional circumvention device rather than a natural capability that has always existed. This entire "trust us" cloud culture has been one big vaccination for governments versus the liberating power of technology that we were excited for two decades ago. This end result has been foreseeable since the rise of webapps, but it's hard to get software developers to understand something when their salary relies upon not understanding it.
Apart from my ][gs (and later my secondhand NeXT), I've never been a huge Apple fan. But I had hoped that by taking this recent privacy tack, they would put workable digital rights into the hands of the masses. Design their system to be solidly secure against everyone but Apple, control the app store to prevent trojans, but then stay out of users' business as software developers should. But brazenly modifying their OS, which should be working for the interests of the user, to do scanning against the interests of the user is a disappointing repudiation of the entire concept of digital rights. And so once again we're back to Free software or bust. At least the Free mobile ecosystem seems to be progressing.
Yes, it is ultimately policy problem. But the way we get people in office who understand technology is to get technological capabilities in the hands of people before they get into office, as well as the hands of those who will vote for them. Just like "bad facts make bad law", bad engineering makes bad law.
Twenty years ago we forcefully scoffed at the Clipper Chip proposal and the export ban on crypto, because they were so at odds with the actual reality of the digital environment. These days, most people's communications are mediated by large corporations operating on plaintext, and are thus are straightforward to monitor and censor. And especially when companies lead the charge, governments expect to have the same ability.
If I could hole up and rely on Free software to preserve my rights indefinitely, I wouldn't particularly care what the Surveillance Valley crowd was doing with their MITM scheme. But I can't, because Surveillance Valley is teaching governments that communications can be controlled while also fanning the flames and creating glaring examples of why they need to be controlled (cf social media "engagement" dumpster fire). And once governments expect that technology can be generally controlled, they will rule any software that does not do their bidding as some exceptional circumvention device rather than a natural capability that has always existed. This entire "trust us" cloud culture has been one big vaccination for governments versus the liberating power of technology that we were excited for two decades ago. This end result has been foreseeable since the rise of webapps, but it's hard to get software developers to understand something when their salary relies upon not understanding it.
Apart from my ][gs (and later my secondhand NeXT), I've never been a huge Apple fan. But I had hoped that by taking this recent privacy tack, they would put workable digital rights into the hands of the masses. Design their system to be solidly secure against everyone but Apple, control the app store to prevent trojans, but then stay out of users' business as software developers should. But brazenly modifying their OS, which should be working for the interests of the user, to do scanning against the interests of the user is a disappointing repudiation of the entire concept of digital rights. And so once again we're back to Free software or bust. At least the Free mobile ecosystem seems to be progressing.
[deleted]
> For instance, the "it only scans photos uploaded to iCloud" element isn't just an arbitrary limitation that can be flipped with one line of code, as some folks seem to think
It might be a happy incident if their architecture limits this feature to CSAM now, but their ToS are clearly much more general-purpose than that, strategically allowing Apple to pre-screen for any potentially illegal content. If ToS remain phrased this way, surely the implementation will catch up.
It might be a happy incident if their architecture limits this feature to CSAM now, but their ToS are clearly much more general-purpose than that, strategically allowing Apple to pre-screen for any potentially illegal content. If ToS remain phrased this way, surely the implementation will catch up.
> For instance, the "it only scans photos uploaded to iCloud" element isn't just an arbitrary limitation that can be flipped with one line of code, as some folks seem to think; as Erik Neuenschwander, head of Privacy Engineering at Apple, explained in an interview on TechCrunch[1]:
> > Our system involves both an on-device component where the voucher is created, but nothing is learned, and a server-side component, which is where that voucher is sent along with data coming to Apple service and processed across the account to learn if there are collections of illegal CSAM. That means that it is a service feature.
I don't see how that proves what Neuenschwander says it proves. Sending the voucher to the server is currently gated on whether or not the associated photo is set to be uploaded to iCloud. There's no reason why that restriction couldn't be removed.
The hard part about building this feature was doing the scanning and detection, and building out the mechanism by which Apple gets notified if anything falls afoul of the scanner. Changing what triggers (or does not trigger) the scanning to happen, or results being uploaded to Apple, is trivial.
The only thing that I think is meaningfully interesting here is that it's possible that getting hold of a phone and looking at the vouchers might not tell you anything; only after they vouchers are sent to Apple and algorithm magic happens will you learn anything. But I don't think that really matters in practice; if you can get the vouchers off a phone, you can almost certainly get the photos associated with them as well.
I read through the article you linked, and it feels pretty hand-wavy to me. Honestly I don't think I'd believe what Apple is saying here unless they release a detailed whitepaper on how the system works, one that can be vetted by experts in the field. (And then we still have to trust that what they've detailed in the paper is what they've actually implemented.)
The bottom line is that Apple has developed and deployed a method for locally scanning phones for contraband. Even if they've designed things so that the result is obfuscated unless there are many matches and/or photos are actually uploaded, that seems to be a self-imposed limitation that could be removed without too much trouble. The capability is there; not using it is merely a matter of policy.
> I know that's easy to dismiss as Pollyannaism, but "we need to protect ourselves from our own government" has a pretty dismal track record historically. The entire point of having a liberal democracy is that we are the government, and we can pull it back from authoritarianism.
Absolutely agree, but I fear we have been losing this battle for many decades now, and I'm a bit pessimistic for our future. It seems most people are very willing to let their leaders scare them into believing that they must give up liberty in exchange for safety and security.
> > Our system involves both an on-device component where the voucher is created, but nothing is learned, and a server-side component, which is where that voucher is sent along with data coming to Apple service and processed across the account to learn if there are collections of illegal CSAM. That means that it is a service feature.
I don't see how that proves what Neuenschwander says it proves. Sending the voucher to the server is currently gated on whether or not the associated photo is set to be uploaded to iCloud. There's no reason why that restriction couldn't be removed.
The hard part about building this feature was doing the scanning and detection, and building out the mechanism by which Apple gets notified if anything falls afoul of the scanner. Changing what triggers (or does not trigger) the scanning to happen, or results being uploaded to Apple, is trivial.
The only thing that I think is meaningfully interesting here is that it's possible that getting hold of a phone and looking at the vouchers might not tell you anything; only after they vouchers are sent to Apple and algorithm magic happens will you learn anything. But I don't think that really matters in practice; if you can get the vouchers off a phone, you can almost certainly get the photos associated with them as well.
I read through the article you linked, and it feels pretty hand-wavy to me. Honestly I don't think I'd believe what Apple is saying here unless they release a detailed whitepaper on how the system works, one that can be vetted by experts in the field. (And then we still have to trust that what they've detailed in the paper is what they've actually implemented.)
The bottom line is that Apple has developed and deployed a method for locally scanning phones for contraband. Even if they've designed things so that the result is obfuscated unless there are many matches and/or photos are actually uploaded, that seems to be a self-imposed limitation that could be removed without too much trouble. The capability is there; not using it is merely a matter of policy.
> I know that's easy to dismiss as Pollyannaism, but "we need to protect ourselves from our own government" has a pretty dismal track record historically. The entire point of having a liberal democracy is that we are the government, and we can pull it back from authoritarianism.
Absolutely agree, but I fear we have been losing this battle for many decades now, and I'm a bit pessimistic for our future. It seems most people are very willing to let their leaders scare them into believing that they must give up liberty in exchange for safety and security.
>It seems most people are very willing to let their leaders scare them into believing that they must give up liberty in exchange for safety and security.
What Snowden exposed is that these changes are happening in secret, with no democratic support or oversight.
Laws are being circumvented so people aren't being given the choice to support or not.
What Snowden exposed is that these changes are happening in secret, with no democratic support or oversight.
Laws are being circumvented so people aren't being given the choice to support or not.
> There's no reason why that restriction couldn't be removed.
The problem is there's no reason Apple couldn't do anything. All the ML face data gathered on device would be way more valuable than these hash vouchers.
The problem is there's no reason Apple couldn't do anything. All the ML face data gathered on device would be way more valuable than these hash vouchers.
What apple is doing is fucked up. Full stop. Mental gymnastics are required to go beyond this premise.
The issue is, it is all software settings. The iMessage filter is only active for acconts held by minors, that can be changed by Apple. Might not even be a client side setting. Also the code which can scan uploaded images can be used anywhere, scan every picture which is being stored on the device. It is all software and everything which is needed is one update.
Could you not potentially infer a lot about a person if they habe X photo content saved - via gathering that data elsewhere?
E.g. a simple one: anyone that may have saved a "winnie the pooh" meme image?
And it's not like Apple's going to publish a list of images that the system searched; though it'd be easy to differentiate between child abuse content vs. other types of images - but would it ever get out of Apple if someone went rogue or was "experimenting" to say "gather stats" of how many people have X image saved?
E.g. a simple one: anyone that may have saved a "winnie the pooh" meme image?
And it's not like Apple's going to publish a list of images that the system searched; though it'd be easy to differentiate between child abuse content vs. other types of images - but would it ever get out of Apple if someone went rogue or was "experimenting" to say "gather stats" of how many people have X image saved?
> For instance, the "it only scans photos uploaded to iCloud" element isn't just an arbitrary limitation that can be flipped with one line of code, as some folks seem to think; as Erik Neuenschwander, head of Privacy Engineering at Apple, explained in an interview on TechCrunch[1]:
>> Our system involves both an on-device component where the voucher is created, but nothing is learned, and a server-side component, which is where that voucher is sent along with data coming to Apple service and processed across the account to learn if there are collections of illegal CSAM. That means that it is a service feature.
The first paragraph does not follow from the detail in the second at all. Setting aside how abstract the language is, what about adding more complexity to the system is preventing Apple from scanning other content?
This is all a misdirect: they're saying "look at how complex this system is!" and pretending that, particularly when they built and control the entire system including it's existence, that any of that makes changing how it operates "difficult".
>> Our system involves both an on-device component where the voucher is created, but nothing is learned, and a server-side component, which is where that voucher is sent along with data coming to Apple service and processed across the account to learn if there are collections of illegal CSAM. That means that it is a service feature.
The first paragraph does not follow from the detail in the second at all. Setting aside how abstract the language is, what about adding more complexity to the system is preventing Apple from scanning other content?
This is all a misdirect: they're saying "look at how complex this system is!" and pretending that, particularly when they built and control the entire system including it's existence, that any of that makes changing how it operates "difficult".
The system isn't entirely on-device, but relies on uploading data to Apple's servers. Hence, why I said that it's not an arbitrary limitation that it only scans photos uploaded to iCloud. The system literally has to upload enough images with triggering "safety vouchers" to Apple to pass the reporting threshold, and critical parts of that calculation are happening on the server side.
I think what you're arguing is that Apple could still change what's being scanned for, and, well, yes: but that doesn't really affect my original point, which is that this is a policy/legal issue. If you assume governments are bad actors, then yes, they could pressure Apple to change this technology to scan for other things -- but if this technology didn't exist, they could just as easily pressure Apple to do it all on the servers. I think a lot of the anger comes from they shouldn't be able to do any part of this work on my device, and emotionally, I get that -- but technologically, it's hard for me not to shake the impression that "what amount happens on device vs. what amount happens on server" is a form of bikeshedding.
I think what you're arguing is that Apple could still change what's being scanned for, and, well, yes: but that doesn't really affect my original point, which is that this is a policy/legal issue. If you assume governments are bad actors, then yes, they could pressure Apple to change this technology to scan for other things -- but if this technology didn't exist, they could just as easily pressure Apple to do it all on the servers. I think a lot of the anger comes from they shouldn't be able to do any part of this work on my device, and emotionally, I get that -- but technologically, it's hard for me not to shake the impression that "what amount happens on device vs. what amount happens on server" is a form of bikeshedding.
>"what amount happens on device vs. what amount happens on server" is a form of bikeshedding
There's a massive difference between a search carried out on a device someone "owns" and carries with them at all times, and a server owned by someone else.
To claim otherwise is absurd. Hence all this backlash.
There's a massive difference between a search carried out on a device someone "owns" and carries with them at all times, and a server owned by someone else.
To claim otherwise is absurd. Hence all this backlash.
I think this is the first time they have mentioned that you will be able to compare the hash of the database on your device with a hash published in their KB article. They also detailed that the database is only the intersection of hash lists from two child safety organizations under separate governmental jurisdictions.
My immediate thought is that this could still be poisoned by Five Eyes participants, and that it does not preclude state actors forcing Apple to replicate this functionality for other purposes (which would leave the integrity of the CSAM database alone, thus not triggering the tripwire).
My immediate thought is that this could still be poisoned by Five Eyes participants, and that it does not preclude state actors forcing Apple to replicate this functionality for other purposes (which would leave the integrity of the CSAM database alone, thus not triggering the tripwire).
the opportunity being to add general functions in photo viewing apps that add a little entropy to every image (for this specific purpose), to rotate hashes, rendering the dual databases useless
monetization I guess being to hope for subscribers on github, as this could likely just be a nested dependency that many apps import. a convenient app for this specific purpose might not last long in app stores.
monetization I guess being to hope for subscribers on github, as this could likely just be a nested dependency that many apps import. a convenient app for this specific purpose might not last long in app stores.
> this could still be poisoned by Five Eyes participants, and that it does not preclude state actors forcing Apple to replicate this functionality for other purposes
The thing is, if this is your threat model you're already screwed. Apple has said they comply with laws in jurisdictions where they operate. The state can pass whatever surveillance laws they want, and I do believe Apple has shown they'll fight them to an extent, but at the end of the day they're not going to shut down the company to protect you. This all seems orthogonal to the CSAM scanning.
Additionally, as laid out in the report, the human review process means even if somehow there is a match that isn't CSAM, they don't report it until it has been verified.
The thing is, if this is your threat model you're already screwed. Apple has said they comply with laws in jurisdictions where they operate. The state can pass whatever surveillance laws they want, and I do believe Apple has shown they'll fight them to an extent, but at the end of the day they're not going to shut down the company to protect you. This all seems orthogonal to the CSAM scanning.
Additionally, as laid out in the report, the human review process means even if somehow there is a match that isn't CSAM, they don't report it until it has been verified.
I think is this a good document and it also brings to the table the threshold images count (around 30) and the alternate neural hash they will keep private (to guard against adverserial images trying to create a false positive on-device).
FWIW, I actually did an amateur threat model analysis in a comment in separate HN thread. I always thought this was called for because the initial document set was just the mathematics, not the people/process/implementation/policy risks and threat model that was the source of widespread concerns.
FWIW, I actually did an amateur threat model analysis in a comment in separate HN thread. I always thought this was called for because the initial document set was just the mathematics, not the people/process/implementation/policy risks and threat model that was the source of widespread concerns.
It’s pretty lonely over here in technical discussion land. Have we considered Reuters’s intern’s take on this?
Please don't post unsubstantive comments yourself. If other threads/comments aren't good, options include posting something better, or not posting. Degrading things even further is not a good choice (edit: particularly when the thread is new—threads are extremely sensitive to initial conditions, and a flamebait/unsubstantive comment early on can have a strongly degrading effect).
https://news.ycombinator.com/newsguidelines.html
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
https://news.ycombinator.com/newsguidelines.html
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Sincere question: why doesn’t this policy apply to the innumerable number of hot takes which are variations on the exact same, often misinformed, reaction? I’m thinking of comments like, “Apple will send me to prison for taking bath tub pics of my infant!”
I have seen many on here.
I have seen many on here.
Two answers. First, it does apply. If you see a post that ought to have been moderated but hasn't been, the likeliest explanation is that we didn't see it. We don't come close to reading everything that gets posted here—there is far too much. You can help by flagging it or emailing us at [email protected].
Second, there are degrees of these things. It's definitely bad for comments to repeat the same shallow things over and over, but the GP was particularly information-free and snarky, and mucking up this thread while claiming to be speaking in favor of technical discussion is particularly counterproductive.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
Second, there are degrees of these things. It's definitely bad for comments to repeat the same shallow things over and over, but the GP was particularly information-free and snarky, and mucking up this thread while claiming to be speaking in favor of technical discussion is particularly counterproductive.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...
I think the flagging mechanism is rife for abuse. There are clearly more people interested in low information reactions than substantive discussion. If all you act on is reports, you will be biased towards whatever they care about. That is not a good way to moderate for nuance. I’m sure you may have considered this, but I don’t think it’s coming through in the discussion of this important topic which has been mostly hysterical.
Reports are not all we act on. We read the threads too. No one can read all of them though.
Discussion on this topic has certainly been mixed, but "mostly hysterical" sounds like an exaggeration to me. People's reactions to these things are conditioned by the cognitive bias that causes us to weight the things we dislike much more strongly than the things we agree with (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...).
Discussion on this topic has certainly been mixed, but "mostly hysterical" sounds like an exaggeration to me. People's reactions to these things are conditioned by the cognitive bias that causes us to weight the things we dislike much more strongly than the things we agree with (https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...).
Mostly hysterical may technically be an exaggeration, but mostly misinformed, and often dis-informative is not.
This isn’t just about people weighing things they dislike more strongly.
It’s also about groupthink, confirmation bias, and a lack of curiosity.
HN doesn’t have an immune system against straight up misinformation.
This isn’t just about people weighing things they dislike more strongly.
It’s also about groupthink, confirmation bias, and a lack of curiosity.
HN doesn’t have an immune system against straight up misinformation.
> HN doesn’t have an immune system against straight up misinformation.
It certainly doesn't! Misinformation and disinformation are terms du jour, but as far as I can tell they are indistinguishable from old-fashioned people-being-wrong-on-the-internet. If you expect an internet forum to be immune from that...well, that's too much to expect. As far as I can tell (and moderating HN for years has really hammered this home), nearly everyone is wrong about nearly everything.
Not only that, but no one (or perhaps we can say for decorum's sake, nearly no one) really cares about the truth. We care about what we like and we want it to win against what we dislike; all the rest is rationalization. Moderating HN has really hammered that one into me as well. Such is human nature, and trying to moderate against it would be futile, not to mention a fast track to burnout. I've tried, and have the scars.
And at the same time I completely sympathize with the frustration of watching discourse on a major topic being dominated by shallow, indignant repetition that isn't engaging with the specifics of a situation.
From a moderation point of view, when a tsunami of a story washes over HN for a week or more, there's not that much we can do about it. We can prevent it from completely dominating the site; we can ask a few people not to break the site guidelines when we see that happening; that's about it. Pretending we can do much more than that would be like Canute commanding the waves.
(Btw, maybe I'm mistaking you for someone else but I have the feeling that your comments have gone, over the years, from frequently breaking the site guidelines to doing a pretty good job of respecting them. If that's true, that's amazing and I appreciate it a ton.)
Edit: I could keep adding new paragraphs to this post for almost forever, but here's another factor that moderating HN has hammered into me. It's possible for people to be wrong on both the facts and the arguments and yet for there to be some deeper justice or truth in what they're saying. Oftentimes, conflicts play out like this: someone responds by correcting facts or pointing out flaws in arguments, but they only succeed in provoking an intensified fury and end up drawing it to themselves. That's because they're answering on the level of facts-and-arguments as a way of dismissing, rather than acknowledging, that deeper level of truth or justice that people have strong feelings about. Mostly all this does is increase the amplitude of the conflict. This is very much a co-creation between the conflicting parties—i.e. between the ones who are right (or feel they are) and the other ones who are right (or feel they are). This is the real answer to complaints about groupthink, in my opinion.
It certainly doesn't! Misinformation and disinformation are terms du jour, but as far as I can tell they are indistinguishable from old-fashioned people-being-wrong-on-the-internet. If you expect an internet forum to be immune from that...well, that's too much to expect. As far as I can tell (and moderating HN for years has really hammered this home), nearly everyone is wrong about nearly everything.
Not only that, but no one (or perhaps we can say for decorum's sake, nearly no one) really cares about the truth. We care about what we like and we want it to win against what we dislike; all the rest is rationalization. Moderating HN has really hammered that one into me as well. Such is human nature, and trying to moderate against it would be futile, not to mention a fast track to burnout. I've tried, and have the scars.
And at the same time I completely sympathize with the frustration of watching discourse on a major topic being dominated by shallow, indignant repetition that isn't engaging with the specifics of a situation.
From a moderation point of view, when a tsunami of a story washes over HN for a week or more, there's not that much we can do about it. We can prevent it from completely dominating the site; we can ask a few people not to break the site guidelines when we see that happening; that's about it. Pretending we can do much more than that would be like Canute commanding the waves.
(Btw, maybe I'm mistaking you for someone else but I have the feeling that your comments have gone, over the years, from frequently breaking the site guidelines to doing a pretty good job of respecting them. If that's true, that's amazing and I appreciate it a ton.)
Edit: I could keep adding new paragraphs to this post for almost forever, but here's another factor that moderating HN has hammered into me. It's possible for people to be wrong on both the facts and the arguments and yet for there to be some deeper justice or truth in what they're saying. Oftentimes, conflicts play out like this: someone responds by correcting facts or pointing out flaws in arguments, but they only succeed in provoking an intensified fury and end up drawing it to themselves. That's because they're answering on the level of facts-and-arguments as a way of dismissing, rather than acknowledging, that deeper level of truth or justice that people have strong feelings about. Mostly all this does is increase the amplitude of the conflict. This is very much a co-creation between the conflicting parties—i.e. between the ones who are right (or feel they are) and the other ones who are right (or feel they are). This is the real answer to complaints about groupthink, in my opinion.
Not only that, but no one (or perhaps we can say for decorum's sake, nearly no one) really cares about the truth. > We care about what we like and we want it to win against what we dislike; all the rest is rationalization.
This is a bleak, valueless and ascientific view of the world. It took a while for me to process it. It is consistent with the postmodern post-truth zeitgeist of tribalism and power being the only thing that matters.
> Moderating HN has really hammered that one into me as well. Such is human nature, and trying to moderate against it would be futile, not to mention a fast track to burnout. I've tried, and have the scars.
I can very clearly see why moderating HN would cause you to adopt this view. It’s interesting because it really doesn’t match the sense of who you are as you come across in your comments.
My own view is that we are not in such a bleak world. Meaning structures are shaped by media and we are only a decade or so into the era where media has become social. This is such a monumental and unprecedented change that it’s bound to appear that there is no up or down for a while.
I think it’s transient, and doesn’t reflect a universal truth about human nature.
For one obvious contradiction - can you think of anyone who actually prefers the idea of a valueless ascientific world? If so, who, and why?
This is a bleak, valueless and ascientific view of the world. It took a while for me to process it. It is consistent with the postmodern post-truth zeitgeist of tribalism and power being the only thing that matters.
> Moderating HN has really hammered that one into me as well. Such is human nature, and trying to moderate against it would be futile, not to mention a fast track to burnout. I've tried, and have the scars.
I can very clearly see why moderating HN would cause you to adopt this view. It’s interesting because it really doesn’t match the sense of who you are as you come across in your comments.
My own view is that we are not in such a bleak world. Meaning structures are shaped by media and we are only a decade or so into the era where media has become social. This is such a monumental and unprecedented change that it’s bound to appear that there is no up or down for a while.
I think it’s transient, and doesn’t reflect a universal truth about human nature.
For one obvious contradiction - can you think of anyone who actually prefers the idea of a valueless ascientific world? If so, who, and why?
> but as far as I can tell they are indistinguishable from old-fashioned people-being-wrong-on-the-internet. If you expect an internet forum to be immune from that...well, that's too much to expect.
Also, this is a little too glib. I’m not talking about people being wrong or misinformed. I am talking about people actively spreading misinformation. These are distinguishable although I accept that they may be impractical to moderate, hence my claim about no immune system.
Also, this is a little too glib. I’m not talking about people being wrong or misinformed. I am talking about people actively spreading misinformation. These are distinguishable although I accept that they may be impractical to moderate, hence my claim about no immune system.
Responding to your edit: I think Zepto understands that and tried to direct the conversation that way. For example:
https://news.ycombinator.com/item?id=28120649
They even spent time engaging with someone who said CSAM shouldn’t matter by arguing that the fact that lots of the general population cares about it makes it a concern worth thinking about.
Even I did, in my own way: https://news.ycombinator.com/item?id=28165116
Perhaps our responses here are in frustration - that we weren’t able to engage with the matters at hand with people of similar skill and interests. I suppose one way to read your post is that HN isn’t suited for that level of discussion on breaking news with emotional resonance. Maybe another space.
They even spent time engaging with someone who said CSAM shouldn’t matter by arguing that the fact that lots of the general population cares about it makes it a concern worth thinking about.
Even I did, in my own way: https://news.ycombinator.com/item?id=28165116
Perhaps our responses here are in frustration - that we weren’t able to engage with the matters at hand with people of similar skill and interests. I suppose one way to read your post is that HN isn’t suited for that level of discussion on breaking news with emotional resonance. Maybe another space.
I may be the person you are thinking of. I don’t think I ever ‘frequently’ broke the site guidelines, but I definitely don’t do it much these days, and it’s always a mistake when I do.
With regard to the conflict dynamics - I agree, although the amplification isn’t necessarily bad.
Also I think with regard to the ‘deeper truth’, often it’s not so much ‘deeper’ as simpler.
In this case for example, people strongly dislike being distrusted - it might well be a genetic disposition since it is a crucial part of our social psychology, and Apple is making a mistake by ignoring that. This isn’t about people not trusting Apple. It’s about people not feeling trusted by Apple.
Perhaps that counts as deeper, but there isn’t much to say about it - it’s not a loud sounding insight, and so all of the other arguments get built up as a way of amplifying that signal.
With regard to the conflict dynamics - I agree, although the amplification isn’t necessarily bad.
Also I think with regard to the ‘deeper truth’, often it’s not so much ‘deeper’ as simpler.
In this case for example, people strongly dislike being distrusted - it might well be a genetic disposition since it is a crucial part of our social psychology, and Apple is making a mistake by ignoring that. This isn’t about people not trusting Apple. It’s about people not feeling trusted by Apple.
Perhaps that counts as deeper, but there isn’t much to say about it - it’s not a loud sounding insight, and so all of the other arguments get built up as a way of amplifying that signal.
I think the fact that no technical document has lasted on the front page but various rumors have is a strong point for my view. An article about internal dissent at Apple has hung on for most of the day, yet almost no comments on it engage with any of the key concepts in the article: what does it mean for an 800 post slack thread to exist? Why does it matter that the security employees don’t seem against the idea on the thread?
Similarly, I doubt anyone would be able to reliably match the comments section of any of the dozen articles about this announcement that have reached the front page in the last 8 days with the actual article content.
These are all bad signs about the healthiness of discussion.
Similarly, I doubt anyone would be able to reliably match the comments section of any of the dozen articles about this announcement that have reached the front page in the last 8 days with the actual article content.
These are all bad signs about the healthiness of discussion.
https://news.ycombinator.com/item?id=28173134 is #1 on the front page right now.
Matthew Green and Alex Stamos both wrote things about this that were on HN's front page for a long time. I'm pretty sure there have been other technical threads as well.
Matthew Green and Alex Stamos both wrote things about this that were on HN's front page for a long time. I'm pretty sure there have been other technical threads as well.
You're doing a bang up job man. It's very kind of you to continue responding to these folks when they've taken your attention rather off topic from your original comment above. Other people's bad behavior doesn't excuse your own, which also happens to be the Chinese version of the golden rule: 己所不欲，勿施於人 (literally: what you don't want, don't apply to others)
[deleted]
What’s not discussed in this Paper is how it won’t be used by third-party actors to falsely accuse someone. Does nobody remember the iCloud hack? What about instead of downloading Jennifer Lawrence’s photos, a hacker uploaded children to get someone falsely accused?
Google, Microsoft and a bunch of other services already scan for CSAM materials, yet this hasn’t been an issue so far. In fact I can’t really find a single case when someone was framed like this, despite how easy it is to fill someone’s account with CSAM and call the cops/wait.
I don’t like the privacy and property rights issues of this but the whole someone will use this to frame someone is quite BS.
I don’t like the privacy and property rights issues of this but the whole someone will use this to frame someone is quite BS.
1) how would you know someone's being framed if it's not detected as such? in this case it just looks like the target had CSAM on their machine/account and they are found guilty, despite their objections, maybe because no-one really looked closely at the evidence, or it was planted well enough.
2) if it is detected as a plant. Would you know it? Would it fly above our radar? There is evidence this happens [1], some cases are quite high profile [2].
3) This technique is already in use. Russia is allegedly using it to discredit opponents [3]. Do you think you would hear from it if it was done by secret services of any country to discredit or get rid of foreign targets or even some at home?
4) it's the ultimate weapon against foes and opponents. It's a dangerous one but I have to wonder how many estranged spouses get revenge planting evidence [4], or how many journalists or troublemakers have been the recipient of some carefully crafted planting. I mean, nearly anyone reading HN could do it sufficiently well to get someone in serious trouble, it's not really a technical issue so it's not hard to imagine it's being done.
[1]: https://www.independent.co.uk/news/uk/crime/handyman-planted...
[2]: https://www.miaminewtimes.com/news/shaq-hit-with-lawsuit-for...
[3]: https://www.nytimes.com/2016/12/09/world/europe/vladimir-put...
[4]: https://www.crimeonline.com/2019/07/04/wife-plants-child-por...
2) if it is detected as a plant. Would you know it? Would it fly above our radar? There is evidence this happens [1], some cases are quite high profile [2].
3) This technique is already in use. Russia is allegedly using it to discredit opponents [3]. Do you think you would hear from it if it was done by secret services of any country to discredit or get rid of foreign targets or even some at home?
4) it's the ultimate weapon against foes and opponents. It's a dangerous one but I have to wonder how many estranged spouses get revenge planting evidence [4], or how many journalists or troublemakers have been the recipient of some carefully crafted planting. I mean, nearly anyone reading HN could do it sufficiently well to get someone in serious trouble, it's not really a technical issue so it's not hard to imagine it's being done.
[1]: https://www.independent.co.uk/news/uk/crime/handyman-planted...
[2]: https://www.miaminewtimes.com/news/shaq-hit-with-lawsuit-for...
[3]: https://www.nytimes.com/2016/12/09/world/europe/vladimir-put...
[4]: https://www.crimeonline.com/2019/07/04/wife-plants-child-por...
Here is one case which was allegedly caused by malware. Several people were arrested and subsequently acquitted.
I would say that the opt-out preference on iPhones that by default uploads all photos to iCloud combined with a remote execution payload like Pegasus would render such an attack plausible.
https://en.wikipedia.org/wiki/United_States_v._Solon?wprov=s...
I would say that the opt-out preference on iPhones that by default uploads all photos to iCloud combined with a remote execution payload like Pegasus would render such an attack plausible.
https://en.wikipedia.org/wiki/United_States_v._Solon?wprov=s...
How would someone prove that they were framed by someone planting CSAM? Without evidence, how likely is it that they would be believed?
But the services you mentioned don’t do it on the device. The blind trust in Apple that everything will be ok is quite BS, and very naïve .
As I recall someone's neighbor once tried to frame them by downloading CSAM on their network and then calling the police or something like that.
> Does nobody remember the iCloud hack?
The iCloud hack was a few hundred cases of spear phishing.
How does your concern differ from physical access to the phone by an unauthorized user?
I think the law is the problem. Simple possession of that type of content is criminalized, as opposed to requiring the traditional elements of a crime to prosecute it.
The iCloud hack was a few hundred cases of spear phishing.
How does your concern differ from physical access to the phone by an unauthorized user?
I think the law is the problem. Simple possession of that type of content is criminalized, as opposed to requiring the traditional elements of a crime to prosecute it.
How is that not already the case for Google, Facebook, Dropbox, Microsoft, and I'm sure countless others? Or are you just suggesting that now Apple gets added to this existing long list, and that's particularly notable?
Their use of the phrase "This claim is subject to code inspection by security researchers like all other iOS device-side security claims" stood out to me.
Could someone tell me how that inspection works? Are there researchers who are given the source code?
(I posted this on another thread [0] earlier, but it's more relevant here) [0]: https://news.ycombinator.com/item?id=28175619
Could someone tell me how that inspection works? Are there researchers who are given the source code?
(I posted this on another thread [0] earlier, but it's more relevant here) [0]: https://news.ycombinator.com/item?id=28175619
Apple actually gives special devices to Security Researchers that allow them further access into the device than a normal consumer device: https://developer.apple.com/programs/security-research-devic...
In this way, third party security researchers can verify their claims. It actually works out pretty well for them since third party security researchers often find pretty severe vulnerabilities through this program.
In this way, third party security researchers can verify their claims. It actually works out pretty well for them since third party security researchers often find pretty severe vulnerabilities through this program.
That program, at least when it was introduced, required participants not to report security vulnerabilities publicly until Apple allowed them to do so, with no limits on how long that can be (see https://news.ycombinator.com/item?id=23920454 for a discussion from that time).
That makes this program particularly useless for the purpose of auditing whether Apple is adhering to its promises.
That makes this program particularly useless for the purpose of auditing whether Apple is adhering to its promises.
For security issues where the participants basically make their living indirectly by getting credit for security vulnerabilities, this carrot-and-stick potentially motivates them to stay quiet.
Meanwhile, researchers have gotten wise to notary techniques (like publishing document hashes to twitter) which would let them severely and publicly shame Apple should they sit on something that ultimately turns out to be a zero day, with much delight from/participation by the media.
For privacy/societal issues where Apple is a deliberate bad actor, they would presumably either directly be willing to break the terms of the agreement to go public, or would release information indirectly and rely on herd privacy with other researchers.
Meanwhile, researchers have gotten wise to notary techniques (like publishing document hashes to twitter) which would let them severely and publicly shame Apple should they sit on something that ultimately turns out to be a zero day, with much delight from/participation by the media.
For privacy/societal issues where Apple is a deliberate bad actor, they would presumably either directly be willing to break the terms of the agreement to go public, or would release information indirectly and rely on herd privacy with other researchers.
[deleted]
It’s not my field, but my understanding is that security researchers often disassemble and/or debug binaries without having access to the source code.
For example, as soon as an OS update is released, people will take it apart.
Having source might be nice but it’s not necessary.
For example, as soon as an OS update is released, people will take it apart.
Having source might be nice but it’s not necessary.
[deleted]
Could someone tell me how that inspection works? Are there researchers who are given the source code?
[0]: https://developer.apple.com/programs/security-research-devic...
[1]: https://www.theverge.com/2021/8/11/22620014/apple-corellium-...
> Apple will publish a Knowledge Base article containing a root hash of the encrypted CSAM hash database included with each version of every Apple operating system that supports the feature. Additionally, users will be able to inspect the root hash of the en- crypted database present on their device, and compare it to the expected root hash in the Knowledge Base article.
This is just security theater, they already sign the operating system images where the database reside. And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc.
> This feature runs exclusively as part of the cloud storage pipeline for images being up- loaded to iCloud Photos and cannot act on any other image content on the device
Until a 1-line code change happens that hooks it into UIImage.
This is just security theater, they already sign the operating system images where the database reside. And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc.
> This feature runs exclusively as part of the cloud storage pipeline for images being up- loaded to iCloud Photos and cannot act on any other image content on the device
Until a 1-line code change happens that hooks it into UIImage.
> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc.
Although this is true, the same argument already applies to "your phone might be scanning all your photos and stealthily uploading them" -- Apple having announced this program doesn't seem to have changed the odds of that.
At some point you have to trust your OS vendor.
Although this is true, the same argument already applies to "your phone might be scanning all your photos and stealthily uploading them" -- Apple having announced this program doesn't seem to have changed the odds of that.
At some point you have to trust your OS vendor.
Which is why I am confused by a lot of this backlash. Apple already controls the hardware, software, and services. I don't see why it really matters where in that chain the scanning is done when they control the entire system. If Apple can't be trusted with this control today, why did people trust them with this control a week ago?
That’s where I’m at. They could have just started doing this without even saying anything at all.
But in that case they would eventually be caught red-handed and won't get to do the "for the children" spiel and get it swept under the rug like it's about to be.
The goal is not for it to be swept under the rug. The goal is for it to deflect concerns over the coming Privacy Relay service.
The government cares far more about other things than CSAM, like terrorism, human and drug trafficking, organized crime, and fraud. Unless the CSAM detection system is going to start detecting those other things and report them to authorities, as well, it won't deflect any concerns over encryption or VPNs.
Their private relay service appears orthogonal to CSAM… it won’t make criminals and child abusers easier or harder to catch, and it doesn’t affect how people use their iCloud Photos storage.
These people are commonly prosecuted using evidence that includes server logs that showing their static IP address.
Read evidence from past trials it is obvious. See also successful and failed attempts to subpoena this info from VPN services.
Only people with iCloud will be using the relay.
It is true on the surface the photos is disconnected from the use. However, Apple only needs a solid answer that handles the bad optics of what you can do with the Tor-like anonymity of iCloud Privacy Relay.
However, if you look more closely, the CSAM service and its implementation are crafted exactly around the introduction of the relay.
Read evidence from past trials it is obvious. See also successful and failed attempts to subpoena this info from VPN services.
Only people with iCloud will be using the relay.
It is true on the surface the photos is disconnected from the use. However, Apple only needs a solid answer that handles the bad optics of what you can do with the Tor-like anonymity of iCloud Privacy Relay.
However, if you look more closely, the CSAM service and its implementation are crafted exactly around the introduction of the relay.
Yeah. I will say, though, I am happy that people are having the uncomfortable realization that they have very little control over what their iPhone does.
Now we just need everyone to have that same realization about almost all the software we use on almost all the devices we own. As a practical matter 99.99% of us operate on trust.
Remember that emission cheating scandal? Where we supposedly had a system in place to detect bad actors and yet it was detected by a rare case of some curious student exploring how things work or some such.
Which is why open(-ish[0]) things are good, because people can get curious and see how they work.
[0] I understand the emissions cheating was not supossed to be open, but a relatively open system allowed said student to take a peek and see what was going on.
[0] I understand the emissions cheating was not supossed to be open, but a relatively open system allowed said student to take a peek and see what was going on.
Agreed. I think this just shined a spotlight for a lot of people who didn’t really think about how much they had to trust Apple.
Because people (HN especially) overestimate how easy it is to develop and deploy "stealth" software that is never detected in a broad way.
The best covert exfiltration is when you can hit individual devices in a crowd, so people already have no reason to be suspicious. But you're still leaving tracks - connections, packet sizes etc. if you actually want to do anything and you only need to get caught once for the game to be up.
This on the other hand is essentially the perfect channel for its type of surveillance...because it is a covert surveillance channel! Everyone is being told to expect it to exist, that it's normal, and that it will receive frequent updates. No longer is their a danger a security researcher will discover it, its "meant" to be there.
The best covert exfiltration is when you can hit individual devices in a crowd, so people already have no reason to be suspicious. But you're still leaving tracks - connections, packet sizes etc. if you actually want to do anything and you only need to get caught once for the game to be up.
This on the other hand is essentially the perfect channel for its type of surveillance...because it is a covert surveillance channel! Everyone is being told to expect it to exist, that it's normal, and that it will receive frequent updates. No longer is their a danger a security researcher will discover it, its "meant" to be there.
They risk a whistleblower if they don't announce this news while implementing that in a country with a free press.
It's better to be forthright, or you risk your valuation on the whims of a single employee.
It's better to be forthright, or you risk your valuation on the whims of a single employee.
[deleted]
This is the conclusion that I personally arrived at. When I confronted some friends of mine with this, they gave me some good points to the contrary.
Letting Apple scan for government-related material on your device is a slippery slope to government surveillance and a hard line should be drawn here. Today, Apple may only be scanning your device for CSAM material for iCloud. Tomorrow, Apple may implement similar scanning elsewhere, gradually expand the scope of scanning to other types of content and across the entire device, implement similar government-enforcing processes on the device, and so on. It's not a good direction for Apple to be taking, regardless of how it works in this particular case. A user's device is their own personal device and anything that inches toward government surveillance on that device, should be stopped.
Another point made was that government surveillance never happens overnight. It is always very gradual. People don't mean to let government surveillance happen and yet it does because little things like this evolve. It's better to stop potential government surveillance in its tracks right now.
Letting Apple scan for government-related material on your device is a slippery slope to government surveillance and a hard line should be drawn here. Today, Apple may only be scanning your device for CSAM material for iCloud. Tomorrow, Apple may implement similar scanning elsewhere, gradually expand the scope of scanning to other types of content and across the entire device, implement similar government-enforcing processes on the device, and so on. It's not a good direction for Apple to be taking, regardless of how it works in this particular case. A user's device is their own personal device and anything that inches toward government surveillance on that device, should be stopped.
Another point made was that government surveillance never happens overnight. It is always very gradual. People don't mean to let government surveillance happen and yet it does because little things like this evolve. It's better to stop potential government surveillance in its tracks right now.
What happens if someone tries to coerce Apple into writing backdoor code? Engineers at Apple could resist, resign, slow roll the design and engineering process. They could leak it and it would get killed.
Things would have to get very very bad for that kind of pressure to work.
On the other hand, once Apple has written a backdoor enthusiastically themselves, it's a lot easier to force someone to change how it can be used. The changes are small and compliance can be immediately verified and refusal punished. To take it to its logical extreme: you cannot really fire or execute people who delay something (especially if you lake the expertise to tell how long it should take). But you can fire or execute people who refuse to flip a switch.
This technology deeply erodes Apple and its engineers' ability to resist future pressure. And the important bit here is there adversary isn't all powerful. It can coerce you to do things in secret, but its power isn't unlimited. See what happened with yahoo.[0]
https://www.reuters.com/article/us-yahoo-nsa-exclusive/exclu...
On the other hand, once Apple has written a backdoor enthusiastically themselves, it's a lot easier to force someone to change how it can be used. The changes are small and compliance can be immediately verified and refusal punished. To take it to its logical extreme: you cannot really fire or execute people who delay something (especially if you lake the expertise to tell how long it should take). But you can fire or execute people who refuse to flip a switch.
This technology deeply erodes Apple and its engineers' ability to resist future pressure. And the important bit here is there adversary isn't all powerful. It can coerce you to do things in secret, but its power isn't unlimited. See what happened with yahoo.[0]
https://www.reuters.com/article/us-yahoo-nsa-exclusive/exclu...
If you're uploading to the cloud, you have to trust a lot more than just your OS vendor (well, in the default case, your OS vendor often == your cloud vendor, but the access is a lot greater once the data is on the cloud).
And if your phone has the capability to upload to the cloud, then you have to trust your OS vendor to respect your wish if you disable it, etc.
It's curious that this is the particular breaking point on the slope for people.
The "on device" aspect just makes it more immediate feeling, I guess?
And if your phone has the capability to upload to the cloud, then you have to trust your OS vendor to respect your wish if you disable it, etc.
It's curious that this is the particular breaking point on the slope for people.
The "on device" aspect just makes it more immediate feeling, I guess?
Yeah, it's weird. Speaking purely personally, whether the scanning happens immediately-before-upload on my phone or immediately-after-upload in the cloud doesn't really make a difference to me. But this is clearly not a universal opinion.
The most-optimistic take on this I can see is that this program could be the prelude to needing to trust less people. If Apple can turn on e2e encryption for photos, using this program as the PR shield from law enforcement to be able to do it, that'd leave us having to only trust the OS vendor.
The most-optimistic take on this I can see is that this program could be the prelude to needing to trust less people. If Apple can turn on e2e encryption for photos, using this program as the PR shield from law enforcement to be able to do it, that'd leave us having to only trust the OS vendor.
> Speaking purely personally, whether the scanning happens immediately-before-upload on my phone or immediately-after-upload in the cloud doesn't really make a difference to me.
What I find interesting is that so many people find it worse to do it on device, because of the risk that they do it to photos you don't intend to upload. This is clearly where Apple got caught off-guard, because to them, on-device = private.
It seems like the issue is really the mixing of on-device and off. People seem to be fine with on-device data that stays on-device, and relatively fine with the idea that Apple gets your content if you upload it to them. But when they analyze the data on-device, and then upload the results to the cloud, that really gets people.
What I find interesting is that so many people find it worse to do it on device, because of the risk that they do it to photos you don't intend to upload. This is clearly where Apple got caught off-guard, because to them, on-device = private.
It seems like the issue is really the mixing of on-device and off. People seem to be fine with on-device data that stays on-device, and relatively fine with the idea that Apple gets your content if you upload it to them. But when they analyze the data on-device, and then upload the results to the cloud, that really gets people.
Is this really surprising to you? I'm not trying to be rude, but this is an enormous distinction.
In today's world, smartphones are basically an appendage of your body. They should not work to potentially incriminate its owner.
They should not work to potentially incriminate its owner.
But that ship has long sailed, right?
Every packet that leaves a device potentially incriminates its owner. Every access point and router is a potential capture point.
But that ship has long sailed, right?
Every packet that leaves a device potentially incriminates its owner. Every access point and router is a potential capture point.
When I use a web service, I expect my data to be collected by the service, especially if it is free of charge.
A device I own should not be allowed to collect and scan my data without my permission.
A device I own should not be allowed to collect and scan my data without my permission.
A device I own should not be allowed to collect and scan my data without my permission.
It's not scanning; it's creating a cryptographic safety voucher for each photo you upload to iCloud Photos. And unless you reach a threshold of 30 CSAM images, Apple knows nothing about any of your photos.
It's not scanning; it's creating a cryptographic safety voucher for each photo you upload to iCloud Photos. And unless you reach a threshold of 30 CSAM images, Apple knows nothing about any of your photos.
From the point of view of how image processing works, what is happening can indeed be called “scanning”.
This seems like a necessary discussion to have in preparation for widespread, default end to end encryption.
Them adding encrypted hashes to photos you don’t intend to upload is pointless and not much of a threat given the photo themselves are they. They don’t do it, but it doesn’t feel like a huge risk.
Yes, you had to trust Apple, but the huge difference with this new thing is that hiding behind CSAM gives them far more (legally obligated, in fact --- because showing you the images those hashes came from would be illegal) plausible deniability and difficulty of verifying their claims.
In other words, extracting the code and analysing it to determine that it does do what you expect is, although not easy, still legal. But the source, the CSAM itself, is illegal to possess, so you can't do that verification much less publish the results. It is this effective legal moat around those questioning the ultimate targets of this system which people are worried about.
In other words, extracting the code and analysing it to determine that it does do what you expect is, although not easy, still legal. But the source, the CSAM itself, is illegal to possess, so you can't do that verification much less publish the results. It is this effective legal moat around those questioning the ultimate targets of this system which people are worried about.
Surely they could do their image matching against all photos in iCloud without telling you in advance, and then you'd be in exactly the same boat? Google was doing this for email as early as 2014, for instance, with the same concerns about its extensibility raised by the ACLU: https://www.theguardian.com/technology/2014/aug/04/google-ch...
So in a world where Apple pushes you to set up icloud photos by default, and can do whatever they want there, and other platforms have been doing this sort of of thing for years, it's a bit startling that "on device before you upload" vs "on uploaded content" triggers far more discontent?
Maybe it's that Apple announced it at all, vs doing it relatively silently like the others? Apple has always had access to every photo on your device, after all.
So in a world where Apple pushes you to set up icloud photos by default, and can do whatever they want there, and other platforms have been doing this sort of of thing for years, it's a bit startling that "on device before you upload" vs "on uploaded content" triggers far more discontent?
Maybe it's that Apple announced it at all, vs doing it relatively silently like the others? Apple has always had access to every photo on your device, after all.
It isn't startling people trust they can opt out of iCloud photos.
If you trust that you can opt out of iCloud Photos to avoid server-side scanning, trusting that this on-device scanning only happens as part of the iCloud Photos upload process (with the only way it submits the reports being as metadata attached to the photo-upload, as far as I can tell) seems equivalent.
There's certainly a slippery-slope argument, where some future update might change that scanning behavior. But the system-as-currently-presented seems similarly trustable.
There's certainly a slippery-slope argument, where some future update might change that scanning behavior. But the system-as-currently-presented seems similarly trustable.
I trust Apple doesn't upload everyone's photos despite opting out because it would be hard to hide.
I bet it'd take a while. The initial sync for someone with a large library is big, but just turning on upload for new pictures is only a few megabytes a day. Depending on how many pictures you take, of course. And if you're caught, an anodyne "a bug in iCloud Photo sync was causing increased data usage" statement and note in the next iOS patch notes would have you covered.
And that's assuming they weren't actively hiding anything by e.g. splitting them up into chunks that could be slipped into legitimate traffic with Apple's servers.
And that's assuming they weren't actively hiding anything by e.g. splitting them up into chunks that could be slipped into legitimate traffic with Apple's servers.
No, the threat model differs entirely. Local scanning introduces a whole host of single points of failure, including the 'independent auditor' & involuntary scans, that risk the privacy & security of all local files on a device. Cloud scanning largely precludes these potential vulnerabilities.
Your phone threat model should already include "the OS author has full access to do whatever they want to whatever data is on my phone, and can change what they do any time they push out an update."
I don't think anyone's necessarily being too upset or paranoid about THIS, but maybe everyone should also be a little less trusting of every closed OS - macOS, Windows, Android as provided by Google - that has root access too.
I don't think anyone's necessarily being too upset or paranoid about THIS, but maybe everyone should also be a little less trusting of every closed OS - macOS, Windows, Android as provided by Google - that has root access too.
Sure, but that doesn't change the fact that the vulnerabilities with local scanning remain a significant superset of cloud scanning's.
Apple has built iOS off user trust & goodwill, unlike most other OSes.
Apple has built iOS off user trust & goodwill, unlike most other OSes.
Cloud Scanning vulnerability: no transparency over data use. On the phone, you can always confirm the contents of what’s added to the safety voucher’s associated data. On the cloud, anything about your photos is fair game.
Where does that fit in your set intersection?
Where does that fit in your set intersection?
> On the phone, you can always confirm the contents of what’s added to the safety voucher’s associated data.
...except you can't? Not sure where these assumptions come from.
...except you can't? Not sure where these assumptions come from.
It’s code running your device is the point, so while “you” doesn’t include everyone, it does include people who will verify this to a greater extent than if done on cloud.
It differs, but iOS already scans images locally and we really don't know what they do with the meta data, and what "hidden" categories there are.
Yes, exactly why Apple breaching user trust matters.
And how is telling you in great detail about what they’re planning to do months before they do it and giving you a way to opt out in advance a breach of trust? What more did you expect from them?
> What more did you expect from them?
Well they could not do it.
Well they could not do it.
You might prefer that, but it doesn’t violate your privacy for them to prefer a different strategy.
why even ask the question " What more did you expect from them?" if you didn't care about the answer?
I gave a pretty obvious and clear answer to that, and apparently you didn't care about the question in the first place, and have now misdirected to something else.
I am also not sure what possible definition of "privacy" that you could be using, that would not include things such as on device photo scanning, for the purpose of reporting people to the police.
Like, lets say it wasn't Apple doing this. Lets say it was the government. As in, the government required every computer that you own, to be monitored for certain photos, at which point the info would be sent to them, and they would arrest you.
Without a warrant.
Surely, you'd agree that this violates people's privacy? The only difference in this case, is that the government now gets to side step 4th amendment protections, by having a company do it instead.
I gave a pretty obvious and clear answer to that, and apparently you didn't care about the question in the first place, and have now misdirected to something else.
I am also not sure what possible definition of "privacy" that you could be using, that would not include things such as on device photo scanning, for the purpose of reporting people to the police.
Like, lets say it wasn't Apple doing this. Lets say it was the government. As in, the government required every computer that you own, to be monitored for certain photos, at which point the info would be sent to them, and they would arrest you.
Without a warrant.
Surely, you'd agree that this violates people's privacy? The only difference in this case, is that the government now gets to side step 4th amendment protections, by having a company do it instead.
My question was directed at someone who claimed their privacy was violated, and I asked them to explain how they would’ve liked their service provider to handle a difference in opinion about what to build in the future. I don’t think your comment clarifies that.
> how they would’ve liked their service provider to handle a difference in opinion about what to build in the future
And the answer is that they shouldn't implement things that violate people's privacy, such as things that would be illegal for the government to do without a warrant.
That is the answer. If it is something that the government would need a warrant for, then they shouldn't do it, and doing it would violate people's privacy.
And the answer is that they shouldn't implement things that violate people's privacy, such as things that would be illegal for the government to do without a warrant.
That is the answer. If it is something that the government would need a warrant for, then they shouldn't do it, and doing it would violate people's privacy.
You forgot, 'after it leaked'
What’s the difference between hybrid cloud/local scanning “due to a bug” checking all your files and uploading too many safety vouchers and cloud scanning “due to a bug” uploading all your files and checking them there?
...because cloud uploads require explicit user consent, practically speaking? Apple's system requires none.
Wouldn't both of those scenarios imply that the "bug" is bypassing any normal user consent? They're only practically different in that the "upload them all for cloud-scanning" one would take longer and use more bandwidth, but I suspect very few people would notice.
[deleted]
I think the difference lies in the visibility of each system in typical use. Apple's local scanning remains invisible to the user, in contrast to cloud uploading.
shuckles(1)
What about trust-but-verify ?
If the OS was open source and supported reproducible builds, you would not have to trust them, you could verify what it actually does & make sure the signed binaries they ship you actually correspond to the source code.
Once kinda wonders what they want to hide if they talks so much about user privacy yet don't provide any means for users to verify their claims.
If the OS was open source and supported reproducible builds, you would not have to trust them, you could verify what it actually does & make sure the signed binaries they ship you actually correspond to the source code.
Once kinda wonders what they want to hide if they talks so much about user privacy yet don't provide any means for users to verify their claims.
Yes, they can technically already do so, but that is not the question. The question is what can they legally do and justify with high confidence in the event of a legal challenge.
Changes to binding contractual terms that allow broad readings and provide legal justification for future overreach are dangerous. If they really are serious that they are going to use these new features in a highly limited way then they can put their money where their mouth is and add legally binding contractual terms that limit what they can do with serious consequences if they are found to be in breach. Non-binding marketing PR assurances that they will not abuse their contractually justified powers are no substitute for the iron fist of legal penalty clause.
Changes to binding contractual terms that allow broad readings and provide legal justification for future overreach are dangerous. If they really are serious that they are going to use these new features in a highly limited way then they can put their money where their mouth is and add legally binding contractual terms that limit what they can do with serious consequences if they are found to be in breach. Non-binding marketing PR assurances that they will not abuse their contractually justified powers are no substitute for the iron fist of legal penalty clause.
Yeah that's true, although to do some sort of mass scanning stealthily they would need a system exactly like what they built with this, if they tried to upload everything for scanning the data use would be enormous and give it away.
I guess it comes down to that I don't trust an OS vendor that ships an A.I. based snitch program that they promise will be dormant.
I guess it comes down to that I don't trust an OS vendor that ships an A.I. based snitch program that they promise will be dormant.
Speaking cynically, I think that them having announced this program like they did makes it less likely that they have any sort of nefarious plans for it. There's a lot of attention being paid to it now, and it's on everyone's radar going forwards. If they actually wanted to be sneaky, we wouldn't have known about this for ages.
They'd have to be transparent about it as someone would easily figure it out.You have no way of verifying the contents of that hash database. once the infrastructure is in place (i.e. on your phone) it's a lot easier to expand on it. People have short memories and are easily desensitized, after a year or two of this, everyone will forget and we'll be in uproar about it expanding to include this or that...
You're making the mistake of anthropomorphizing a corporation. Past a certain size, corporations start behaving less like people and more like computers, or maybe profit-maximizing sociopaths. The intent doesn't matter, because 5 or 10 years down the line, it'll likely be a totally different set of people making the decision. If you want to predict a corporation's behavior, you need to look at the constants (or at least, slower-changing things), like incentives, legal/technical limitations, and internal culture/structure of decision-making (e.g. How much agency do individual humans have?).
I feel that I was stating the incentives, though.
This being an area people are paying attention to makes it less likely they'll do unpopular things involving it, from a pure "we like good PR and profits" standpoint. They might sneak these things in elsewhere, but this specific on-device-scanning program has been shown to be a risk even at its current anodyne level.
This being an area people are paying attention to makes it less likely they'll do unpopular things involving it, from a pure "we like good PR and profits" standpoint. They might sneak these things in elsewhere, but this specific on-device-scanning program has been shown to be a risk even at its current anodyne level.
No they wouldn’t need a system like this. They already escrow all your iCloud Backups, and doing the scanning server side allows you to avoid any scrutiny through code or network monitoring.
> At some point you have to trust your OS vendor.
Yes, and we were trusting Apple. And now this trust is going away.
Yes, and we were trusting Apple. And now this trust is going away.
> now this trust is going away
Is it really? There are some very loud voices making their discontent felt. But what does the Venn diagram look like between 'people who are loudly condemning Apple for this' and 'people who were vehemently anti-Apple to begin with'?
My trust was shaken a bit, but the more I hear about the technology they've implemented, the more comfortable I am with it. And frankly, I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech.
Is it really? There are some very loud voices making their discontent felt. But what does the Venn diagram look like between 'people who are loudly condemning Apple for this' and 'people who were vehemently anti-Apple to begin with'?
My trust was shaken a bit, but the more I hear about the technology they've implemented, the more comfortable I am with it. And frankly, I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech.
> I'm far more worried about gov't policy than I am about the technical details. We can't fix policy with tech.
Yeah. I don't really understand the tech utopia feeling that Apple could simply turn on e2ee and ignore any future legislation to ban e2ee. The policy winds are clearly blowing towards limiting encryption in some fashion. Maybe this whole event will get people to pay more attention to policy...maybe.
Yeah. I don't really understand the tech utopia feeling that Apple could simply turn on e2ee and ignore any future legislation to ban e2ee. The policy winds are clearly blowing towards limiting encryption in some fashion. Maybe this whole event will get people to pay more attention to policy...maybe.
> Until a 1-line code change happens that hooks it into UIImage.
I really don't understand this view. You are using proprietary software, you are always an N-line change away from someone doing something you don't like. This situation doesn't change this.
If you only use open source software and advocate for others to do the same, I would understand it more.
I really don't understand this view. You are using proprietary software, you are always an N-line change away from someone doing something you don't like. This situation doesn't change this.
If you only use open source software and advocate for others to do the same, I would understand it more.
Did you verify all the binaries that you run are from compiled source code that you audited? Your BIOS?
What about your CPU and GPU firmware?
There is always a chain of trust that you end up depending on. OSS is not a panacea here.
There is always a chain of trust that you end up depending on. OSS is not a panacea here.
It's not a panacea but the most implausible the mechanism, the less likely it's going to be used on anyone but the most high value targets.
(And besides, it's far more likely that this nefarious Government agency will just conceal a camera in your room to capture your fingers entering your passwords.)
(And besides, it's far more likely that this nefarious Government agency will just conceal a camera in your room to capture your fingers entering your passwords.)
> I really don't understand this view. You are using proprietary software, you are always an N-line change away from someone doing something you don't like. This situation doesn't change this.
And I don't understand why it has to be black and white, I think the N is very important in this formula and if it is low that is a cause for concern. Like an enemy building a missile silo on an island just off your coast but promising it's just for defense.
All arguments I see is along the lines of "Apple can technically do anything they want anyways so this doesn't matter". But maybe you're right and moving to FOSS is the only solution long-term, that's what I'm doing if Apple goes through with this.
And I don't understand why it has to be black and white, I think the N is very important in this formula and if it is low that is a cause for concern. Like an enemy building a missile silo on an island just off your coast but promising it's just for defense.
All arguments I see is along the lines of "Apple can technically do anything they want anyways so this doesn't matter". But maybe you're right and moving to FOSS is the only solution long-term, that's what I'm doing if Apple goes through with this.
The size of N doesn't really matter. I'm sure Apple ships large PRs in every release, as any software company does.
Maybe not if you assume Apple is evil but for the case of Apple being good intentioned but having its hand forced, they will have a much harder time resisting a 1 line change than a mandate to spend years to develop a surveillance system
I’d leave this one out to the lawyers. I’m not one but I don’t think that the court will evaluate the number of lines of code required for help.
Apple shipped iCloud Private Relay which is a “1-line code change that hooks into CFNetwork” away from MITMing all your network connections, by this standard.
For me the standard is that I don't want any 1-line code change between me and near-perfect Orwellian surveillance.
Since your one-liners seem to be immensely dense with functional changes, I can’t understand how you trust any software.
> And there is no way to audit that the database is what they claim it is, doesn't contain multiple databases that can be activated under certain conditions, etc.
They describe a process for third parties to audit that the database was produced correctly.
They describe a process for third parties to audit that the database was produced correctly.
Do we have any idea how the NCMEC database is curated? Are there cartoons from Hustler depicting underage girls in distress? Green text stories stating they are true about illegal sexual acts? CGI images of pre-pubescent looking mythical creatures? Manga/Anime images which are sold on the Apple Store? Legitimate artistic images from books currently sold? Images of Winnie the Pooh the government has declared pornographic? From the amount of material the Feds claim is being generated every year I would have to guess all of this is included. The multi-government clause is completely pointless with the five-eyes cooperation.
The story here is that there is a black box of pictures. Apple will then use their own black box of undeclared rules to pass things along to the feds which they have not shared what would be considered offending in any way shape or form other than "we will know it when we see it". Part of the issue here is that Apple is taking the role of a moral authority. Traditionally Apple has been incredibly anti-pornography and I suspect that anything that managed to get into the database will be something Apple will just pass along.
The story here is that there is a black box of pictures. Apple will then use their own black box of undeclared rules to pass things along to the feds which they have not shared what would be considered offending in any way shape or form other than "we will know it when we see it". Part of the issue here is that Apple is taking the role of a moral authority. Traditionally Apple has been incredibly anti-pornography and I suspect that anything that managed to get into the database will be something Apple will just pass along.
Apple is manually reviewing every case to ensure it’s CSAM. You do have to trust them on that.
But if your problem is with NCMEC, you’ve got a problem with Facebook and Google who are already doing this too. And you can’t go to jail for possessing adult pornography. So even if you assume adult porn images are in the database, and Apple’s reviewers decide to forward them to NCMEC, you would still not be able to be prosecuted, at least in the US. Ditto for pictures of Winnie the Pooh. But for the rest of what you describe, simulated child pornography is already legally dicey as far as I know, so you can’t really blame Apple or NCMEC for that.
But if your problem is with NCMEC, you’ve got a problem with Facebook and Google who are already doing this too. And you can’t go to jail for possessing adult pornography. So even if you assume adult porn images are in the database, and Apple’s reviewers decide to forward them to NCMEC, you would still not be able to be prosecuted, at least in the US. Ditto for pictures of Winnie the Pooh. But for the rest of what you describe, simulated child pornography is already legally dicey as far as I know, so you can’t really blame Apple or NCMEC for that.
Facebook I completely approve of. You are trafficking data at that point if you are posting it. I just recall the days of Usenet and Napster when I would just download at random and sometimes the evil would mislabel things to cause trauma. I do not download things at random any more but when I was that age it would have been far more appropriate to notify my parents then it would be to notify the government.
In any case it is likely the government would try to negotiate a plea to get you into some predator database to help fill the law enforcement coffers even if they have no lawful case to take it to court once they have your name in their hands.
In any case it is likely the government would try to negotiate a plea to get you into some predator database to help fill the law enforcement coffers even if they have no lawful case to take it to court once they have your name in their hands.
> Ditto for pictures of Winnie the Pooh.
References to Winnie the Pooh in these discussions are about China, where images of Winnie are deemed to be coded political messages and are censored.
The concern is that Apple are building a system that is ostensibly about CSAM, and that some countries such as China will then leverage their power to force Apple to include whatever political imagery in the database as well. Giving the government there the ability to home in on who is passing around those kinds of images in quantity.
If that seems a long way indeed from CSAM, consider something more likely to fit under that heading by local government standards. There's a country today, you may have heard of, one the USA is busy evacuating its personnel from to leave the population to an awful fate, where "female teenagers in a secret school not wearing a burqa" may be deemed by the new authorities to be sexually titillating, inappropriate and illegal, and if they find out who is sharing those images, punishments are much worse than mere prison. Sadly there are a plethora of countries that are very controlling of females of all ages.
References to Winnie the Pooh in these discussions are about China, where images of Winnie are deemed to be coded political messages and are censored.
The concern is that Apple are building a system that is ostensibly about CSAM, and that some countries such as China will then leverage their power to force Apple to include whatever political imagery in the database as well. Giving the government there the ability to home in on who is passing around those kinds of images in quantity.
If that seems a long way indeed from CSAM, consider something more likely to fit under that heading by local government standards. There's a country today, you may have heard of, one the USA is busy evacuating its personnel from to leave the population to an awful fate, where "female teenagers in a secret school not wearing a burqa" may be deemed by the new authorities to be sexually titillating, inappropriate and illegal, and if they find out who is sharing those images, punishments are much worse than mere prison. Sadly there are a plethora of countries that are very controlling of females of all ages.
Drawings are prosecutable in many countries including Canada, the UK, and Australia. Also, iCloud sync is enabled by default when you set up your device, whereas the Facebook app at least is sandboxed and you have to choose to upload your photos.
> You do have to trust them on that.
If this system didn't exist, nobody would have to trust Apple.
> you would still not be able to be prosecuted
But I wouldn't want to deal with a frivolous lawsuit, or have a record in the social media of being brought CSA charges.
If this system didn't exist, nobody would have to trust Apple.
> you would still not be able to be prosecuted
But I wouldn't want to deal with a frivolous lawsuit, or have a record in the social media of being brought CSA charges.
I keep changing my mind on this. On the one hand I already operate on the assumption that uploading data to a cloud service renders that data non-private, or at least in great risk of becoming non-private in the future. This simply makes my operating assumption explicit. Also this particular implementation and its stated goals aren’t egregious.
But then there’s the slippery slope we’ve all been discussing — and the gradual precedents set for it, this being a nail in the coffin. But I sympathize with Apple for making transparent what I assume happens behind closed doors anyway. My optimistic side thinks maybe this is a sign that extensions of this project will also be made explicit. Maybe this is the only way for them to take this fight if they’re being pressured. In any case now is the time to speak out. If they extend it to texts and other documents I’m out - even as it stands I’m thinking of alternatives. But the fact we’re all discussing it now is at least a point of hope.
But then there’s the slippery slope we’ve all been discussing — and the gradual precedents set for it, this being a nail in the coffin. But I sympathize with Apple for making transparent what I assume happens behind closed doors anyway. My optimistic side thinks maybe this is a sign that extensions of this project will also be made explicit. Maybe this is the only way for them to take this fight if they’re being pressured. In any case now is the time to speak out. If they extend it to texts and other documents I’m out - even as it stands I’m thinking of alternatives. But the fact we’re all discussing it now is at least a point of hope.
> But I sympathize with Apple for making transparent what I assume happens behind closed doors anyway.
Using your devices' CPU and battery seems more egregious than doing it on their servers. If they want to help law enforcement, then they should pay for it. Of course they want to help law enforcement by forcing other people to pay the costs.
Imagine if Ford came out with a cannabis sensor in their cars that automatically called the cops on you if it detected cannabis inside the car. The sensor is allegedly only active when you're renting a Ford vehicle, not if you purchase it outright. Not a perfect analogy, but how comfortable would you find this situation?
Using your devices' CPU and battery seems more egregious than doing it on their servers. If they want to help law enforcement, then they should pay for it. Of course they want to help law enforcement by forcing other people to pay the costs.
Imagine if Ford came out with a cannabis sensor in their cars that automatically called the cops on you if it detected cannabis inside the car. The sensor is allegedly only active when you're renting a Ford vehicle, not if you purchase it outright. Not a perfect analogy, but how comfortable would you find this situation?
Let's do a more realistic example. Ford introduces a "driver safety" mechanism where the car requires a clean breathalyzer reading in order to start. If it fails it pops up a message that reminds you that drunk driving is illegal but doesn't actually stop you from starting the engine. It then sends Ford the results in an encrypted payload along with 1/30th of the decryption key.
After 30 reports someone at Ford opens the payloads, looks at the results and, decides whether to contact the police based on how damning the evidence is.
Because all the tests are performed locally you're never giving up any of your privacy unless you actually are driving drunk or are the 1/billion and get 30 false positives. I feel like this is a much stronger than the "if you have nothing to hide" meme because local testing like this lets you can have everything to hide while still revealing specific bad behavior.
Like if the TSA had scanners that somehow could only reveal banned items and nothing else I’m not sure I would even consider it a privacy violation.
After 30 reports someone at Ford opens the payloads, looks at the results and, decides whether to contact the police based on how damning the evidence is.
Because all the tests are performed locally you're never giving up any of your privacy unless you actually are driving drunk or are the 1/billion and get 30 false positives. I feel like this is a much stronger than the "if you have nothing to hide" meme because local testing like this lets you can have everything to hide while still revealing specific bad behavior.
Like if the TSA had scanners that somehow could only reveal banned items and nothing else I’m not sure I would even consider it a privacy violation.
That's not a realistic analogy because driving is a privilege you must earn, not a human right.
> Apple generates the on-device perceptual CSAM hash database through an intersection of hashes provided by at least two child safety organizations operating in separate sovereign jurisdictions – that is, not under the control of the same government. Any perceptual hashes appearing in only one participating child safety organization’s database, or only in databases from multiple agencies in a single sovereign jurisdiction, are discarded by this process, and not included in the encrypted CSAM database that Apple includes in the operating system.
Well, that's quite clever, isn't it.
It's just the sort of thing that someone who had "never heard of" the Five Eyes, or extradition treaties or illegal rendition operations involving multiple nations, might come up with.
Genius!
Well, that's quite clever, isn't it.
It's just the sort of thing that someone who had "never heard of" the Five Eyes, or extradition treaties or illegal rendition operations involving multiple nations, might come up with.
Genius!
well crap I just wrote a response lecturing you on the Five Eyes collusion before reading the last two lines of your post. :P
There exists a kind of sufficiently advanced stupidity that it can only be constructed by really smart people. People who are smart enough to rationalize doing something which is transparently wrong.
There exists a kind of sufficiently advanced stupidity that it can only be constructed by really smart people. People who are smart enough to rationalize doing something which is transparently wrong.
Curious, does any technological system used widely, standup to the threat levels you mentioned?
Probably not.
Does any technological system used widely justify itself by saying "it would take two national jurisdictions cooperating to break this?"
Does any technological system used widely justify itself by saying "it would take two national jurisdictions cooperating to break this?"
Was not defending this measure, my question was out of curiosity
China: here’s our database and here is one for Hong Kong.
They’re totally separate we promise.
They’re totally separate we promise.
Separate but identical, indeed.
If your threat model includes pervasive spying by multiple nation states, and being grabbed in the night by black helicopters, it seems unlikely you'll be overly concerned about them precisely inserting at least 30 of your photos into multiple CSAM databases and also co-ercing Apple's manual review to get you reported to NCMEC.
I don't think people are worried about multiple nation states framing them with CSAM photos - they're worried about multiple nation states in an intelligence collaboration poisoning both sets of hash lists with non-CSAM material, so that there is an intersection that makes it onto the device.
There is still that Apple human reviewer once the threshold has passed. What I would love to ask Apple is - what happens if/when their reviewers start noticing political material, religious material, etc. is being flagged on a consistent basis, thereby insinuating that the hash list has been poisoned. What's their play at that point?
There is still that Apple human reviewer once the threshold has passed. What I would love to ask Apple is - what happens if/when their reviewers start noticing political material, religious material, etc. is being flagged on a consistent basis, thereby insinuating that the hash list has been poisoned. What's their play at that point?
The document states that incorrectly flagged items are forwarded to engineering for analysis. Given their target false-positive rate (1 in 3 trillion, was it?) it seems likely that engineering would very carefully analyze a rush of false positives.
I think you have this backwards.
Given that nations already "grabbing in the night with black helicopters" (semantically, at least), and do so with impunity, it doesn't seem much of a stretch to imagine they'd potentially set someone up using this much milder sort of approach.
Given that nations already "grabbing in the night with black helicopters" (semantically, at least), and do so with impunity, it doesn't seem much of a stretch to imagine they'd potentially set someone up using this much milder sort of approach.
Is there a good argument for doing the scanning on the phone and not on the iCloud servers?
Well, theoretically this could be a step towards e2e encrypting the photos so that Apple can't see them on their servers.
This entire "secure tickets that unlock a derivative of your photos so we can review them" system would, in this interpretation, be there as a way to neuter law enforcement objections[1] to no longer being able to serve warrants for the contents of your photo library.
Now, will this actually happen? Hard to say.
[1] There was a story last year claiming that Apple wanted to turn on e2e, but the FBI got them to drop it. https://www.popularmechanics.com/technology/security/a306318...
This entire "secure tickets that unlock a derivative of your photos so we can review them" system would, in this interpretation, be there as a way to neuter law enforcement objections[1] to no longer being able to serve warrants for the contents of your photo library.
Now, will this actually happen? Hard to say.
[1] There was a story last year claiming that Apple wanted to turn on e2e, but the FBI got them to drop it. https://www.popularmechanics.com/technology/security/a306318...
No. Contrary to Craig's claims, Google & Microsoft et al. do similar hash matching on the cloud, instead of 'looking at images'.
But people consider that bad! With this system, Apple can never look at your photos in the cloud. They are a pipe for the bits, but they don't examine the contents, except on your device.
I'm not saying I necessarily agree that this is better, all-told (mixed feelings), but I think it's worth acknowledging why a rational actor might consider this superior to Google doing whatever the fuck they want with anything you upload.
I'm not saying I necessarily agree that this is better, all-told (mixed feelings), but I think it's worth acknowledging why a rational actor might consider this superior to Google doing whatever the fuck they want with anything you upload.
Photos are currently encrypted e2e already so Apple does not have the access server side to create the hash unlike their competition who chews their customers data for machine learning. https://support.apple.com/en-us/HT202303
You've misread that page -- the table just refers to content being stored in an encrypted form, even if Apple still possesses a key. There's a list below it of fully end-to-end encrypted content, which doesn't include Photos.
True enough. Apple does have the key. Well then instead the answer is that they are having you pay the CPU and energy cost instead of wasting twice the energy decrypting and hashing your photos server side.
You were sort of both right.
Apple has the keys in escrow across several HSMs separate from the cloud hosting environment. Nothing within iCloud can actually see the content, but there are processes to get access to a particular account for various reasons (such as government order). There is a separation of permissions as well as hardware-generated audit logs. Reportedly the HSM systems were specifically manipulated to not be extensible via software updates.
So it is E2E with a separate key escrow system, which Apple (wisely) does not attempt to call E2E.
Apple couldn't implement the PhotoDNA-based scanning other providers have done because Apple would need access to the escrowed keys for each user photo uploaded.
Apple has the keys in escrow across several HSMs separate from the cloud hosting environment. Nothing within iCloud can actually see the content, but there are processes to get access to a particular account for various reasons (such as government order). There is a separation of permissions as well as hardware-generated audit logs. Reportedly the HSM systems were specifically manipulated to not be extensible via software updates.
So it is E2E with a separate key escrow system, which Apple (wisely) does not attempt to call E2E.
Apple couldn't implement the PhotoDNA-based scanning other providers have done because Apple would need access to the escrowed keys for each user photo uploaded.
What is the process to make sure the next management will honor these promises?
Apple management changing would be a massive threat regardless of these premises. Apple has gone to bat for encryption and privacy to a greater degree than Google, for example, and that's because of their current management. You should not in any way assume that the next group will do the same!
Heck, what is the process for current management to honor them?
[deleted]
This seems like a pretty helpful document, and really shows how carefully the CSAM system has been designed.
Two things that stuck out to me:
1) The CSAM hash database is encrypted on-device
2) The device doesn't know the results of hash comparisons
These suggest to me that the hash comparison is going to happen inside the Secure Enclave - give it the encrypted database and the NeuralHash of an image, it gives you back a Security Voucher to upload.
I don't quite get how the hash database can stay secret though - the decryption key has to come from somewhere. AFAIK the hash database has never been made publicly available before, which probably makes it quite a high value target for security researchers and curious parties.
Two things that stuck out to me:
1) The CSAM hash database is encrypted on-device
2) The device doesn't know the results of hash comparisons
These suggest to me that the hash comparison is going to happen inside the Secure Enclave - give it the encrypted database and the NeuralHash of an image, it gives you back a Security Voucher to upload.
I don't quite get how the hash database can stay secret though - the decryption key has to come from somewhere. AFAIK the hash database has never been made publicly available before, which probably makes it quite a high value target for security researchers and curious parties.
The hash database is blinded on the server before being embedded into iOS. You’d need to access Apple’s private key or break cryptography to get access to it from the client. For the less cryptographically savvy, the “Technical Summary” is a good document on their Child Safety page.
> These suggest to me that the hash comparison is going to happen inside the Secure Enclave - give it the encrypted database and the NeuralHash of an image, it gives you back a Security Voucher to upload.
My understanding is that the hash comparison, or at any rate the counting of matches, is done in the cloud then. The way the crypto is set up is that the server has no access to the security envelope and its contents (and, after E2EE in the future, to any images) unless the number of matches exceeds the threshold.
My understanding is that the hash comparison, or at any rate the counting of matches, is done in the cloud then. The way the crypto is set up is that the server has no access to the security envelope and its contents (and, after E2EE in the future, to any images) unless the number of matches exceeds the threshold.
No secure enclave is needed, the system uses strong cryptography to protect Apple and their database providers from accountability.
I gave a semi-technical description of how private set intersection works here:
https://news.ycombinator.com/item?id=28124716
Assuming they implement it correctly and there are no apple-internal leaks we won't be discovering the content of the database.
I gave a semi-technical description of how private set intersection works here:
https://news.ycombinator.com/item?id=28124716
Assuming they implement it correctly and there are no apple-internal leaks we won't be discovering the content of the database.
That's fine, but keep it on the server I don't want this crap on my phone. I don't want to be guilty until proven innocent.
If apple can be coerced by governments to scan photos on the serverside then what is the operational purpose of a frontend scan, given what apple is publicly saying they are trying to do? This is the most confusing aspect for me.
Apple will turn it on eventually for non iCloud destined photos (and documents), you don't build a system like this not to use it.
We know Apple concedes to China's demands, and with another Snowden situation, the US would not hesitate to add classified documents to the scan list and identify individuals.
This system will catch no abusers, because they're not Airdropping photos to each other.
And if they're smart enough to use a VPN, they're not storing that on a phone thats logged into the cloud.
And if they're not using a VPN, they can be caught at download time.
We know Apple concedes to China's demands, and with another Snowden situation, the US would not hesitate to add classified documents to the scan list and identify individuals.
This system will catch no abusers, because they're not Airdropping photos to each other.
And if they're smart enough to use a VPN, they're not storing that on a phone thats logged into the cloud.
And if they're not using a VPN, they can be caught at download time.
I strongly considered switching away from Apple products last weekend; but this document has convinced me otherwise.
The threats people identify have minimal risk. If a total stranger offers you a bottle of water, you may worry about it being spiked, but him having offered the bottle doesn't make it more, or less, likely that he'll stab you after you accept it. They're separate events, no "slippery slope". It's very convincing that any alteration that would "scan" your whole phone would be caught eventually, and even if it's not, the announcement of this feature has no bearing on it. If Apple has evil intent, or is being coerced by NSLs, they would (be forced to) implement the dangerous mechanism whether this Child Safety feature existed or not. The concept of backdoors is hardly foreign to governments, and Apple didn't let any cats out of any bags. This document shows that Apple did go to great lengths to preserve privacy; including the necessity of hashes being verified in two juristictions, the fact that neither the phone nor Apple know if any images have been matched below the threshold; the lack of remote updates of this mechanism; the use of vouchers instead of sending the full image; the use of synthetic vouchers; and on and on.
Furthermore, the risks of the risks of this mechanism are lower than the existing PhotoDNA used by practically every competing service. Those have no thresholds; the human review process is obscure; there is no claim that other pictures won't be looked at.
The controversy, the fact that it uses an on-device component. But PhotoDNA-solutions fail many of Apple's design criterias, which require an on-device solution:
- database update transparency
- matching software correctness
- matching software transparency
- database and software universality
- data access restrictions.
What about the concerns that the hash databases could be altered to contain political images? PhotoDNA could be, too; but would be undetectable unlike with Apple's solutions. Worse: with serverside solutions, the server admin could use an altered hash DB only for specific target users, causing harm without arousing suspicion. Apple's design prevents this since the DB is verifiably universal to all users.
A rational look at every argument I've seen against Apple's solution indicates that it is strictly superior to current solutions, and less of a threat to users. Cryptographically generating vouchers and comparing hashes is categorically not "scanning people's phones" or a "backdoor." I think the more people understand its technical underpinnings, the less they'll see similarities with sinister dystopian cyberpunk aesthetics.
The threats people identify have minimal risk. If a total stranger offers you a bottle of water, you may worry about it being spiked, but him having offered the bottle doesn't make it more, or less, likely that he'll stab you after you accept it. They're separate events, no "slippery slope". It's very convincing that any alteration that would "scan" your whole phone would be caught eventually, and even if it's not, the announcement of this feature has no bearing on it. If Apple has evil intent, or is being coerced by NSLs, they would (be forced to) implement the dangerous mechanism whether this Child Safety feature existed or not. The concept of backdoors is hardly foreign to governments, and Apple didn't let any cats out of any bags. This document shows that Apple did go to great lengths to preserve privacy; including the necessity of hashes being verified in two juristictions, the fact that neither the phone nor Apple know if any images have been matched below the threshold; the lack of remote updates of this mechanism; the use of vouchers instead of sending the full image; the use of synthetic vouchers; and on and on.
Furthermore, the risks of the risks of this mechanism are lower than the existing PhotoDNA used by practically every competing service. Those have no thresholds; the human review process is obscure; there is no claim that other pictures won't be looked at.
The controversy, the fact that it uses an on-device component. But PhotoDNA-solutions fail many of Apple's design criterias, which require an on-device solution:
- database update transparency
- matching software correctness
- matching software transparency
- database and software universality
- data access restrictions.
What about the concerns that the hash databases could be altered to contain political images? PhotoDNA could be, too; but would be undetectable unlike with Apple's solutions. Worse: with serverside solutions, the server admin could use an altered hash DB only for specific target users, causing harm without arousing suspicion. Apple's design prevents this since the DB is verifiably universal to all users.
A rational look at every argument I've seen against Apple's solution indicates that it is strictly superior to current solutions, and less of a threat to users. Cryptographically generating vouchers and comparing hashes is categorically not "scanning people's phones" or a "backdoor." I think the more people understand its technical underpinnings, the less they'll see similarities with sinister dystopian cyberpunk aesthetics.
I understand the technologies they're proposing deploying at a decent level (I couldn't implement the crypto with my current skills, but what they're doing in the PSI paper makes a reasonable amount of sense).
The problem is that this "hard" technological core (the crypto) is subject to an awful lot of "soft" policy issues around the edge - and there's nothing but "Well, we won't do that!" in there.
Plus, the whole Threat Model document feels like a 3AM brainstorming session thrown together. "Oh, uh... we'll just use the intersection of multiple governments hashes, and, besides, they can always audit the code!" Seriously, search the threat model document for the phrase "subject to code inspection by security researchers" - it's in there 5x. How, exactly, does one go about getting said code?
Remember, national security letters with gag orders attached exist.
Also, remember, when China and Apple came to a head over iCloud server access, Apple backed down and gave China what they wanted.
Even if this, alone isn't enough to convince you to move off Apple, are you comfortable with the trends now clearly visible?
The problem is that this "hard" technological core (the crypto) is subject to an awful lot of "soft" policy issues around the edge - and there's nothing but "Well, we won't do that!" in there.
Plus, the whole Threat Model document feels like a 3AM brainstorming session thrown together. "Oh, uh... we'll just use the intersection of multiple governments hashes, and, besides, they can always audit the code!" Seriously, search the threat model document for the phrase "subject to code inspection by security researchers" - it's in there 5x. How, exactly, does one go about getting said code?
Remember, national security letters with gag orders attached exist.
Also, remember, when China and Apple came to a head over iCloud server access, Apple backed down and gave China what they wanted.
Even if this, alone isn't enough to convince you to move off Apple, are you comfortable with the trends now clearly visible?
> Even if this, alone isn't enough to convince you to move off Apple, are you comfortable with the trends now clearly visible?
Still much better than all but the most esoteric inconvenient alternatives.
Still much better than all but the most esoteric inconvenient alternatives.
Then people need to start asking themselves if privacy is really a value they really hold or is just an empty, bandwagon idealism.
Because sacrificing privacy for convenience is why we got to this point.
Because sacrificing privacy for convenience is why we got to this point.
And if those are all that's left that meet your criteria for a non-abusive platform, then... well, that's what you've got to work with. Maybe try to improve those non-abusive platforms.
I'm rapidly heading there. I'm pretty sure I won't run Win11 given the hardware requirements (I prefer keeping older hardware running when it still fits my needs) and the requirement for an online Microsoft account for Win11 Home (NO, and it's pretty well stupid that I have to literally disconnect the network cable to make an offline account on Win10 now, and then disable the damned nag screens).
If Apple is going full in on this whole "Your device is going to work against you" thing they're trying for, well... I'm not OK with that either. That leaves Linux and the BSDs. Unfortunately, Intel isn't really OK in my book either with the fact that they can't reason about their chips anymore (long rant, but L1TF and Plundervolt allowing pillage of the SGX guarantees tells me Intel can't reason about their chips)... well. Hrm. AMD or ARM it is, and probably not with a very good phone either.
At this point, I'm going down that road quite quickly, far sooner than I'd hoped, because I do want to live out what I talk about with regards to computers, and if the whole world goes a direction I'm not OK with, well, OK. I'll find alternatives. I accept that unless things change, I'm probably no more than 5-10 years away from simply abandoning the internet entirely outside work and very basic communications. It'll suck, but if that's what I need to do to live with what I claim I want to live by, that's what I'll do.
"I think this is a terrible idea and I wish Apple wouldn't do it, but I don't care enough about it to stop using Apple products" is a perfectly reasonable stance, but it does mean that Apple now knows they can do more of this sort of thing and get away with it. Good luck with the long term results of allowing this.
I'm rapidly heading there. I'm pretty sure I won't run Win11 given the hardware requirements (I prefer keeping older hardware running when it still fits my needs) and the requirement for an online Microsoft account for Win11 Home (NO, and it's pretty well stupid that I have to literally disconnect the network cable to make an offline account on Win10 now, and then disable the damned nag screens).
If Apple is going full in on this whole "Your device is going to work against you" thing they're trying for, well... I'm not OK with that either. That leaves Linux and the BSDs. Unfortunately, Intel isn't really OK in my book either with the fact that they can't reason about their chips anymore (long rant, but L1TF and Plundervolt allowing pillage of the SGX guarantees tells me Intel can't reason about their chips)... well. Hrm. AMD or ARM it is, and probably not with a very good phone either.
At this point, I'm going down that road quite quickly, far sooner than I'd hoped, because I do want to live out what I talk about with regards to computers, and if the whole world goes a direction I'm not OK with, well, OK. I'll find alternatives. I accept that unless things change, I'm probably no more than 5-10 years away from simply abandoning the internet entirely outside work and very basic communications. It'll suck, but if that's what I need to do to live with what I claim I want to live by, that's what I'll do.
"I think this is a terrible idea and I wish Apple wouldn't do it, but I don't care enough about it to stop using Apple products" is a perfectly reasonable stance, but it does mean that Apple now knows they can do more of this sort of thing and get away with it. Good luck with the long term results of allowing this.
> And if those are all that's left that meet your criteria for a non-abusive platform, then... well, that's what you've got to work with. Maybe try to improve those non-abusive platforms.
There's always the potential to work on the underlying laws. Not every problem can be solved by tech alone.
There's always the potential to work on the underlying laws. Not every problem can be solved by tech alone.
[deleted]
> If Apple has evil intent, or is being coerced by NSLs, they would (be forced to) implement the dangerous mechanism whether this Child Safety feature existed or not.
Apple used to fight implementing dangerous mechanisms. And succeeded.[1]
[1] https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute
Apple used to fight implementing dangerous mechanisms. And succeeded.[1]
[1] https://en.wikipedia.org/wiki/FBI–Apple_encryption_dispute
Their “you can’t compel us to build something” argument was for building a change to the passcode retry logic, which is presumably as simple as a constant change. Certainly building a back door in this system is at least as difficult, so the argument still stands.
In that specific case at least the FBI is asking Apple to produce new firmware that will bypass existing protection on an existing device - basically asking Apple to root a locked down phone which would likely require them breaking their own encryption or finding vulnerabilities in their own firmware. This is not exactly technically trivial since anything of this sort that’s already known would be patched out.
In this case it seems only a matter of policy that Apple submits CSAM hashes to devices for scanning. No new software needs to be written, no new vulnerability found. There’s no difference between a hash database of CSAM, FBI terrorists photos, or Winnie the Pooh memes. All that’s left is Apple’s words that their policy is capable of standing up to the legal and political pressures of countries it operates in. That seems like a far lower bar.
In this case it seems only a matter of policy that Apple submits CSAM hashes to devices for scanning. No new software needs to be written, no new vulnerability found. There’s no difference between a hash database of CSAM, FBI terrorists photos, or Winnie the Pooh memes. All that’s left is Apple’s words that their policy is capable of standing up to the legal and political pressures of countries it operates in. That seems like a far lower bar.
No, they were asked to sign a software update which would bypass the application processor enforced passcode retry limit on an iPhone 5C which had no secure element.
> The second protection [against mis-inclusion of non CSAM hashes] is human review: there is no automated reporting in Apple’s system. All positive matches must be visually confirmed by Apple as containing CSAM before Apple will disable the account and file a report with the child safety organization.
I don't understand this at all. As I understand it, part of the problem is that — in the US — Apple isn't legally allowed to review, transmit or do anything else with suspected CSAM images, so they can't have a manual review process (or even check that their neural hashing is working as expected on the real dataset).
Does anyone else have any idea of what this is trying to describe?
If Apple really are somehow reviewing flagged photos to confirm that they're CSAM and not maliciously flagged files before sending any reports, then that does make the system substantially more resilient to Five Eyes abuse (not that I wish that job on anyone).
Edit: There's more context later in the document
> First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.
This is more confusing: visually comparison using a second perceptual hash doesn't actually to provide any protection against mis-inclusion of non-CSAM images: it just double-checks that the image really was a match in the database (ie. protects against hash-collision errors), but it does't check that the database contained actual CSAM.
Apple explicitly says that this process protects against mis-inclusion though, which doesn't make sense to me yet.
I don't understand this at all. As I understand it, part of the problem is that — in the US — Apple isn't legally allowed to review, transmit or do anything else with suspected CSAM images, so they can't have a manual review process (or even check that their neural hashing is working as expected on the real dataset).
Does anyone else have any idea of what this is trying to describe?
If Apple really are somehow reviewing flagged photos to confirm that they're CSAM and not maliciously flagged files before sending any reports, then that does make the system substantially more resilient to Five Eyes abuse (not that I wish that job on anyone).
Edit: There's more context later in the document
> First, as an additional safeguard, the visual derivatives themselves are matched to the known CSAM database by a second, independent perceptual hash. This independent hash is chosen to reject the unlikely possibility that the match threshold was exceeded due to non-CSAM images that were adversarially perturbed to cause false NeuralHash matches against the on-device encrypted CSAM database. If the CSAM finding is confirmed by this independent hash, the visual derivatives are provided to Apple human reviewers for final confirmation.
This is more confusing: visually comparison using a second perceptual hash doesn't actually to provide any protection against mis-inclusion of non-CSAM images: it just double-checks that the image really was a match in the database (ie. protects against hash-collision errors), but it does't check that the database contained actual CSAM.
Apple explicitly says that this process protects against mis-inclusion though, which doesn't make sense to me yet.
Your phone transmits the images with their security envelope (which was computed on device and contains neural hash and "visual derivative") to the iCloud server. During that process, Apple does not know whether there's any CSAM in it, so they can transmit legally.
Then the server determines whether the number of matches exceeds the threshold. Only if that is the case (by crypto magic) can the security envelope of the flagged images only (by crypto magic) be unlocked, and the "visual derivative" be reviewed.
(Note that if (at a later stage) E2EE is enabled for the photos, the images themselves would never be accessible by Apple or LE, whether flagged or not, if I understand the design correctly).
Then the server determines whether the number of matches exceeds the threshold. Only if that is the case (by crypto magic) can the security envelope of the flagged images only (by crypto magic) be unlocked, and the "visual derivative" be reviewed.
(Note that if (at a later stage) E2EE is enabled for the photos, the images themselves would never be accessible by Apple or LE, whether flagged or not, if I understand the design correctly).
Your understanding seems correct. After a positive evaluation from Apple, the CyberTipline report is filed to NCMEC, which operates as a clearinghouse and notifies law enforcement.
Law enforcement then gets a court order, which will cause Apple to release requested/available information about that account.
If photos later are outside the key escrow system, Apple would not be able to release the photos encryption keys, and would only be able to share 'public' photo share information which the user has opted to share without encryption to make it web-accessible.
Presumably in this case, the visual derivatives would still be used as evidence, but there would be 5th amendment arguments around forcing access to a broader set of photos.
Law enforcement then gets a court order, which will cause Apple to release requested/available information about that account.
If photos later are outside the key escrow system, Apple would not be able to release the photos encryption keys, and would only be able to share 'public' photo share information which the user has opted to share without encryption to make it web-accessible.
Presumably in this case, the visual derivatives would still be used as evidence, but there would be 5th amendment arguments around forcing access to a broader set of photos.
Your understanding is incorrect. Apple can, and is in fact required to, verify that they have actual CSAM before forwarding it to the Cyber Tip line. At that point, they must delete the information within 60 days.
Interesting. In that case, do you know why they talk about reviews only seeing "visual derivatives" (from the second perceptual hash)?
Either these 'derivatives' basically contain the original image (so reviewers can verify that it's actual CSAM) and there's no point in using derivatives at all, or they're more abstract (eg. 8x8 pixellated images) in which case the reviewer can't see the actual content (but could confirm a database match).
Edit: I was able to find the answer, they suggest that the 'visual derivative' is something like a 'low-resolution version' of the original image, so the content should still be clearly visible.
Either these 'derivatives' basically contain the original image (so reviewers can verify that it's actual CSAM) and there's no point in using derivatives at all, or they're more abstract (eg. 8x8 pixellated images) in which case the reviewer can't see the actual content (but could confirm a database match).
Edit: I was able to find the answer, they suggest that the 'visual derivative' is something like a 'low-resolution version' of the original image, so the content should still be clearly visible.
Apple claims that “data minimization” is one of their privacy pillars, and this was probably an attempt at that. You could imagine an inverted colors or lower res image gets the point across without subjecting you or Apple to all the high fidelity images you have in your library.
> Apple can, and is in fact required to, verify that they have actual CSAM
It's not Apple's job to decide if something is CSAM or not. They're required to report it if they suspect it is, even if they can't confirm it.
It's not Apple's job to decide if something is CSAM or not. They're required to report it if they suspect it is, even if they can't confirm it.
It’s simple, the reviewers will be cops (so viewing CP is legal).
They will pass on/hit report at rates that make a FISA judge blush.
They will pass on/hit report at rates that make a FISA judge blush.
I have long been interested in what a professional-grade threat model from a large FAANG/SV organization is. Is this a representative model?
Microsoft came up with DREAD and STRIDE and they suggest there threat models are more elaborate.
Would love to see more representative examples!
Microsoft came up with DREAD and STRIDE and they suggest there threat models are more elaborate.
Would love to see more representative examples!
This feels more like a marketing document to me than a workaday threat model. It's fairly handwavey, and the goal seems to be to convince rather than to do a hardnosed analysis.
Not that it's not useful—I found it convincing—but I doubt this is what a real threat model looks like. Not that I actually know. I'd be interested in seeing a real one too.
Not that it's not useful—I found it convincing—but I doubt this is what a real threat model looks like. Not that I actually know. I'd be interested in seeing a real one too.
I have no doubt the features work exactly the way Apple said they do. Seriously.
I very much doubt that they will refuse to scan for whatever China asks. I very much doubt they'll risk the PRC shutting down Foxconn factories.
I very much doubt Apple will ultimately be able to resist scanning for whatever the US Government asks in a national security letter attached to a gag order. They will take them to a secret court session which we'll never hear about, the court will rule in the government's favor, and Apple will be forced to comply and not allowed to tell anybody.
This happened to Cloudflare and they were gagged while fighting it for 4 years. They didn't win in court, but the government eventually dropped the case as it was no longer needed. And this started under Obama, not Trump.
Now, will Apple bother to fight this out in court? I think they probably will. But we won't know about that until they either definitively win or the government gives up. If the government wins, on the other hand, we'll NEVER hear about it.
I very much doubt that they will refuse to scan for whatever China asks. I very much doubt they'll risk the PRC shutting down Foxconn factories.
I very much doubt Apple will ultimately be able to resist scanning for whatever the US Government asks in a national security letter attached to a gag order. They will take them to a secret court session which we'll never hear about, the court will rule in the government's favor, and Apple will be forced to comply and not allowed to tell anybody.
This happened to Cloudflare and they were gagged while fighting it for 4 years. They didn't win in court, but the government eventually dropped the case as it was no longer needed. And this started under Obama, not Trump.
Now, will Apple bother to fight this out in court? I think they probably will. But we won't know about that until they either definitively win or the government gives up. If the government wins, on the other hand, we'll NEVER hear about it.
> Now, will Apple bother to fight this out in court? I think they probably will.
Why would they? Each year they give up customer data on over 150,000 users without a fight when the US government simply asks them for it[1].
[1] https://www.apple.com/legal/transparency/us.html
Why would they? Each year they give up customer data on over 150,000 users without a fight when the US government simply asks them for it[1].
[1] https://www.apple.com/legal/transparency/us.html
“Human review and reporting” - fact checkers?
Or are we to assume that their AI is 100% effective.
This will not end well..
Or are we to assume that their AI is 100% effective.
This will not end well..
What strikes me about this paper is that there are no names of the people who wrote it on the title page. Why were they afraid to put their name(s) on this?
Seeing HN over the last week, I can think of a few reasons…
Yeah seeing as its just a white paper there really isn't much need, and the possibility of retribution is immense on this one. I know a lot of pissed off people, let alone some of the nuts who this is angering and who could dox or do worse to the authors.
Sounds like it still comes down to trust. Quoting from the paper:
"Apple will refuse all requests to add non-CSAM images to the perceptual CSAM hash database; third party auditors can confirm this through the process outlined before. Apple will also refuse all requests to instruct human reviewers to file reports for anything other than CSAM materials for accounts that exceed the match threshold."
"Apple will refuse all requests to add non-CSAM images to the perceptual CSAM hash database; third party auditors can confirm this through the process outlined before. Apple will also refuse all requests to instruct human reviewers to file reports for anything other than CSAM materials for accounts that exceed the match threshold."
It's already a lie, considering NCMEC's database already has non-CSAM images. It won't ever be true, since the hundreds and possibly thousands of entities with direct and indirect access to the database can upload SunnyMeadow.jpg labeled as CSAM content, and it's blindly accepted.
If Apple is receiving a hash labeled as "CSAM", how can they possibly make this guarantee? They do not know it's CSAM. It's unverifiable and comes down to "trust us".
Remember, NCMEC is an unaccountable organisation inextricably linked with the FBI and the U.S. government. It is not subject to FOIA requests. Its database has never been audited, and it is known to be a mess already.
If Apple is receiving a hash labeled as "CSAM", how can they possibly make this guarantee? They do not know it's CSAM. It's unverifiable and comes down to "trust us".
Remember, NCMEC is an unaccountable organisation inextricably linked with the FBI and the U.S. government. It is not subject to FOIA requests. Its database has never been audited, and it is known to be a mess already.
> "hundreds and possibly thousands of entities with direct and indirect access to the database can upload SunnyMeadow.jpg labeled as CSAM content, and it's blindly accepted."
According to whom?
> "it is known to be a mess already."
According to whom? You've posted a number of claims about the CSAM database administration process on HN and the one instance where you've cited a source, it's been a junk source.
According to whom?
> "it is known to be a mess already."
According to whom? You've posted a number of claims about the CSAM database administration process on HN and the one instance where you've cited a source, it's been a junk source.
To me, the fact that they are making such a strong statement is a big deal. If they had plans or thought that they would be forced to add other content they would not be so direct in this statement. I hope I am not wrong. As you said: trust.
I don’t like the idea of stuff running on my device, consuming my battery and data, when the only point is to see if I am doing something wrong?
An analogy I can come up with is: the government hires people to visit your house every day, and while they’re there they need your resources (say, food, water, and electricity). In other words, they use up some of the stuff you would otherwise be able to use only for yourself — constantly — and the only reason they’re there is to see if you’re doing anything wrong. Why would you put up with this?
An analogy I can come up with is: the government hires people to visit your house every day, and while they’re there they need your resources (say, food, water, and electricity). In other words, they use up some of the stuff you would otherwise be able to use only for yourself — constantly — and the only reason they’re there is to see if you’re doing anything wrong. Why would you put up with this?
A lot of things work this way. The government is quite literally taxing some of your income and using it to check if you're doing anything wrong.
When a cop is checking your car for parking violations, they're using some of your taxpayer dollars to see if you're doing something wrong. You don't benefit at all from your car getting checked. But you probably benefit from everyone else's cars being checked, which is why we have this system.
Or similarly, you don't benefit from airport security searching your luggage. It's a waste of your time and invasion of your privacy if you don't have anything forbidden and even worse for you if you do have something forbidden. But you help pay for airport security anyway because it might benefit you when they search other people's luggage.
You don't benefit at all from having your photos scanned. But you (or someone else) might benefit from everyone else's photos being scanned (if it helps catch child abusers or rescue abused children). It's just a question of whether you think the benefit is worth the cost.
When a cop is checking your car for parking violations, they're using some of your taxpayer dollars to see if you're doing something wrong. You don't benefit at all from your car getting checked. But you probably benefit from everyone else's cars being checked, which is why we have this system.
Or similarly, you don't benefit from airport security searching your luggage. It's a waste of your time and invasion of your privacy if you don't have anything forbidden and even worse for you if you do have something forbidden. But you help pay for airport security anyway because it might benefit you when they search other people's luggage.
You don't benefit at all from having your photos scanned. But you (or someone else) might benefit from everyone else's photos being scanned (if it helps catch child abusers or rescue abused children). It's just a question of whether you think the benefit is worth the cost.
So 3rd amendment defense? These are digital soldiers being quartered in our digital house.
Probably a clarification of the 4th in order; there are multiple levels of government and private institutions at play to disguise that the government is pressuring private institutions to do search and seizure on their behalf.
A better fitting analogy would be you ask FedEx to pick up a package at your house for delivery, and they have a new rule that they won't pick up packages unless they can look inside to make sure that they are packed correctly and do not contain anything that FedEx does not allow.
When they open and look into the package while in your house for the pickup, they are using your light to see and your heating/cooling is keeping the driver comfortable while they inspect the package.
When they open and look into the package while in your house for the pickup, they are using your light to see and your heating/cooling is keeping the driver comfortable while they inspect the package.
I'd say the analogy is closer to, they'll x-ray your package to make sure there aren't any prohibited/dangerous items.
The point of it is to make sure iCloud Photos remains a viable service in light of real and perceived regulatory threats, and possibly leave the door open to end to end encryption in the future.
This is a false dichotomy that is being pushed constantly online. E2EE is possible without this tech and in fact is only useful without tech like this to act as a MITM.
There is no regulatory threat within the US that could require this happen, if this was demanded by the government it would be a blatant violation of the 4th amendment. Apple should have stood their ground if this was in response to perceived government pressure.
There are two gov. issues. One is if the FBI comes knocking and the other is new laws. The government could absolutely write a law banning e2ee so that when the FBI does knock with a warrant they can get access.
In the past, Apple has done what they can to stand their ground against the first, but they (like any other company) will have to comply with any laws passed.
Whether the 'if a warrant is obtained the gov. should be granted access' law is a violation of the 4th amendment remains to be seen.
In the past, Apple has done what they can to stand their ground against the first, but they (like any other company) will have to comply with any laws passed.
Whether the 'if a warrant is obtained the gov. should be granted access' law is a violation of the 4th amendment remains to be seen.
When the FBI does knock with a warrant they already get access to data on iCloud. iCloud is not meaningfully encrypted to prevent this.
The FBI is unable to get a warrant to search data on everyones phones, regardless of what laws are passed. They might be able to get a warrant to search all the data on apple's server (I would consider this unlikely, but I don't know of precedent in either direction), but that data is fundamentally not on everyones phones. This isn't a novel legal question, you cannot search everyones devices without probable cause that "everyone" committed a crime.
The FBI is unable to get a warrant to search data on everyones phones, regardless of what laws are passed. They might be able to get a warrant to search all the data on apple's server (I would consider this unlikely, but I don't know of precedent in either direction), but that data is fundamentally not on everyones phones. This isn't a novel legal question, you cannot search everyones devices without probable cause that "everyone" committed a crime.
IMHO, the relevant regulations are pretty carefully worded to make 4th amendment defenses harder. In this case, Apple has some liability for criminal activity on their platform whether it is was E2E encrypted or not.
So they open a huge back door instead of waiting for a threat and fighting that?
[deleted]
Having read the document, I am even more confident in Apple's approach than before. I commend Apple for its transparency.
A curious outcome of the scrutiny that the system design is receiving is that it will make it less effective at the task of catching actual collections of CSAM.
Page eleven promises something I haven’t seen before: that Apple will publish the match threshold for each version. Meaning, even if after all this, there are abusers still using iCloud photos to manage their illegal collection of CSAM, they can check back to know what number of images is safe to keep.
A twisted version of the Streisand effect.
Page eleven promises something I haven’t seen before: that Apple will publish the match threshold for each version. Meaning, even if after all this, there are abusers still using iCloud photos to manage their illegal collection of CSAM, they can check back to know what number of images is safe to keep.
A twisted version of the Streisand effect.
If you knew about the existence of this program and were determined to keep a collection of CSAM on your phone I don't understand why someone would choose to dance around specific thresholds rather than just disable iCloud sync.
Setting arguments aside, if ever there is a CSAM scanner I’d let loose in my account is Apples.
[deleted]
Burn it all down.
If you can't understand the theory and reimplement it then you shouldn't even look at it.
The internet is full of pathological wizards and witches. Come here for obscure books but otherwise stay far away.
If you can't understand the theory and reimplement it then you shouldn't even look at it.
The internet is full of pathological wizards and witches. Come here for obscure books but otherwise stay far away.
> The system is designed so that a user need not trust Apple, any other single entity, or even any set of possibly-colluding entities from the same sovereign jurisdiction (that is, under the control of the same government)
Who's to say authoritarian governments don't team up? We just saw this happen over the last decade with the rise of nationalism everywhere.
We already know governments can put significant pressure on private companies. Don't capitulate, Apple. Blow the whistle. You're getting screwed and you have our support if you just stand up for what you said 5 years ago.
Who's to say authoritarian governments don't team up? We just saw this happen over the last decade with the rise of nationalism everywhere.
We already know governments can put significant pressure on private companies. Don't capitulate, Apple. Blow the whistle. You're getting screwed and you have our support if you just stand up for what you said 5 years ago.
A key point that needs to be mentioned: we strongly dislike being distrusted.
It might well be a genetic heritage. Being trusted in a tribe is crucial to survival, and so is likely wired deep into our social psychology.
Apple is making a mistake by ignoring that. This isn’t about people not trusting Apple. It’s about people not feeling trusted by Apple.
Because of this, it doesn’t matter how trustworthy the system is or what they do to make it less abusable. It will still represent distrust of the end user, and people will still feel that in their bones.
People argue about the App Store and not being trusted to install their own Apps etc. That isn’t the same. We all know we are fallible and a lot of people like the protection of the store, and having someone to ‘look after’ them.
This is different and deeper than that. Nobody want to be a suspect for something they know they aren’t doing. It feels dirty.
It might well be a genetic heritage. Being trusted in a tribe is crucial to survival, and so is likely wired deep into our social psychology.
Apple is making a mistake by ignoring that. This isn’t about people not trusting Apple. It’s about people not feeling trusted by Apple.
Because of this, it doesn’t matter how trustworthy the system is or what they do to make it less abusable. It will still represent distrust of the end user, and people will still feel that in their bones.
People argue about the App Store and not being trusted to install their own Apps etc. That isn’t the same. We all know we are fallible and a lot of people like the protection of the store, and having someone to ‘look after’ them.
This is different and deeper than that. Nobody want to be a suspect for something they know they aren’t doing. It feels dirty.
The part Federighi's "interview" where he can't understand how people perceive this as a back door [1] seems incredibly out of touch. The back door is what everyone is talking about. Someone at Apple should at least be able to put themselves in their critics' shoes for a moment. I guess we need to wait to hear Tim Cook explain how this is not what he described 5 years ago [2].
[1] https://youtu.be/OQUO1DSwYN0?t=425
[2] https://youtu.be/rQebmygKq7A?t=57
[1] https://youtu.be/OQUO1DSwYN0?t=425
[2] https://youtu.be/rQebmygKq7A?t=57
It’s not a back door in any sense of the word. That’s why he is surprised people see it as one.
It really only does what they say it does, and it really is hard to abuse.
But that doesn’t matter. The point is that even so, it makes everyone into a suspect, and that feels wrong.
It really only does what they say it does, and it really is hard to abuse.
But that doesn’t matter. The point is that even so, it makes everyone into a suspect, and that feels wrong.
> It’s not a back door in any sense of the word.
It is. It's a simple matter for two foreign governments to decide they don't want their people to criticize the head of state with memes, and then insert such images into the database Apple uses for scanning.
Apple's previous position on privacy was to make such snooping impossible because they don't have access to the data. Now they are handing over access.
What I and thousands of others online are describing ought to be understandable by anyone at Apple. The fact that an Apple exec can sit there in a prepared interview and look perplexed about how people could see this as a back door is something I don't understand at all. This "closing his ears" attitude may indicate he is full of it.
It is. It's a simple matter for two foreign governments to decide they don't want their people to criticize the head of state with memes, and then insert such images into the database Apple uses for scanning.
Apple's previous position on privacy was to make such snooping impossible because they don't have access to the data. Now they are handing over access.
What I and thousands of others online are describing ought to be understandable by anyone at Apple. The fact that an Apple exec can sit there in a prepared interview and look perplexed about how people could see this as a back door is something I don't understand at all. This "closing his ears" attitude may indicate he is full of it.
Edit: I reconsidered my previous reply.
That really doesn’t sound like anything I’d describe as a “back door”. A back door implies general purpose access. A system which required the collision of multiple governments and Apple and their child abuse agencies simply is not that.
One of the casualties of this debate is that people are using terms that make things sound worse than they are. If you can’t get at my filesystem, you don’t have a back door. I understand the motive for stating the case as harshly as possible, but I think it’s misguided.
Having said this, I would find it interesting to hear what Federighi would say about this potential abuse case.
That really doesn’t sound like anything I’d describe as a “back door”. A back door implies general purpose access. A system which required the collision of multiple governments and Apple and their child abuse agencies simply is not that.
One of the casualties of this debate is that people are using terms that make things sound worse than they are. If you can’t get at my filesystem, you don’t have a back door. I understand the motive for stating the case as harshly as possible, but I think it’s misguided.
Having said this, I would find it interesting to hear what Federighi would say about this potential abuse case.
> A system which required the collision of multiple governments and Apple and their child abuse agencies simply is not that.
Agree to disagree.
> I would find it interesting to hear what Federighi would say about this potential abuse case.
Personally I would not. That's a political consideration and not something I want to hear a technologist weigh in on while defending their technology. Apple's previous stance, with which I agree, was to not give humans any chance to abuse people's personal data,
https://youtu.be/rQebmygKq7A
Agree to disagree.
> I would find it interesting to hear what Federighi would say about this potential abuse case.
Personally I would not. That's a political consideration and not something I want to hear a technologist weigh in on while defending their technology. Apple's previous stance, with which I agree, was to not give humans any chance to abuse people's personal data,
https://youtu.be/rQebmygKq7A
> Personally I would not. That's a political consideration and not something I want to hear a technologist weigh in on while defending their technology.
It’s not. The abuse case flows from their architecture. Perhaps it isn’t as ‘easy’ as getting multiple countries to collude with Apple. If the architecture can be abused the way you think it can, that is a technical problem as well as a political one.
It’s not. The abuse case flows from their architecture. Perhaps it isn’t as ‘easy’ as getting multiple countries to collude with Apple. If the architecture can be abused the way you think it can, that is a technical problem as well as a political one.
You can't solve the human-bias problem with technology. That's the whole reason Apple didn't want to build in a back door in the first place.
You may not be able to solve the bias problem altogether, but you can definitely change the threat model and who you have to trust.
Apple’s model has always involved trusting them. This model involves trusting other people in narrow ways. The architecture determines what those ways are.
Apple’s model has always involved trusting them. This model involves trusting other people in narrow ways. The architecture determines what those ways are.
Trusting Apple to not scan my device in the past was easy because as an engineer I know I would speak up if I saw that kind of thing secretly happening, and I know security researchers would speak up if they detected it.
Now Apple will scan the device and we must trust that 3rd parties will not abuse the technology by checking for other kinds of imagery such as memes critical of heads of state.
The proposed change is so much worse than the previous state of things.
Now Apple will scan the device and we must trust that 3rd parties will not abuse the technology by checking for other kinds of imagery such as memes critical of heads of state.
The proposed change is so much worse than the previous state of things.
> checking for other kinds of imagery such as memes critical of heads of state.
Do you live in a country where the head of state wants to check for such memes?
Do you live in a country where the head of state wants to check for such memes?
Probably. You underestimate humans if you don't think any of us will try to squash things that make us look bad.
How does this attack even work? So some government poisons the database with political dissident memes and suddenly Apple starts getting a bunch of new reports which when reviewed are obviously not CSAM.
If the government can force Apple to also turn over these reports then they could have just made Apple add their political meme database directly and it's already game over.
If the government can force Apple to also turn over these reports then they could have just made Apple add their political meme database directly and it's already game over.
More like, the government says Apple can't operate there unless they include what they say is illegal.
Apple is run by humans who are subject to influence and bias. Who knows what policy changes will come in Apple's future. Apple's previous stance was to not hand over data because they don't have access to it. This change completely reverses that.
Apple is run by humans who are subject to influence and bias. Who knows what policy changes will come in Apple's future. Apple's previous stance was to not hand over data because they don't have access to it. This change completely reverses that.
[deleted]
I would say that it is a back door, because it would work even if iCloud Photos were E2E encrypted. It may not be a generic one, but one that is specific to a purpose Apple decided is rightful. And there is no guarantee that Apple (or authorities) won't decide that there are other rightful purposes.
> I would say that it is a back door, because it would work even if iCloud Photos were E2E encrypted.
Backdoor is defined by the Oxford Dictionary as "a feature or defect of a computer system that allows surreptitious unauthorized access to data."
The system in question requires you to upload the data to iCloud Photos for the tickets to be meaningful and actionable. Both your phone and iCloud services have EULA which call out and allow for such scanning to take place, and Apple has publicly described how the system works as far as its capabilities and limitations. In the sense that people see this as a change in policy (IIRC the actual license agreement language changed over a year ago) , Apple has also described how to no longer use the iCloud Photos service.
One less standard usage is not about unauthorized access but specifically to private surveillance (e.g. "Clipper chip") - but I would argue that the Clipper chip was a case where the surveillance features were specifically not being talked about, hence it still counting as "unauthorized access".
But with a definition that covers broad surveillance instead of unauthorized access, it would still be difficult to classify this as a back door. Such surveillance arguments would only pertain to the person's phone and not to information the user chose to release to external services like iCloud Photos.
To your original point, it would still work with iCloud Photos did not have the key escrow, albeit with less data being capable of being able to be turned over to law enforcement. However iCloud Photos being an external system would still mean this is an intentional and desired feature (presumably) by the actual system owners (Apple).
Backdoor is defined by the Oxford Dictionary as "a feature or defect of a computer system that allows surreptitious unauthorized access to data."
The system in question requires you to upload the data to iCloud Photos for the tickets to be meaningful and actionable. Both your phone and iCloud services have EULA which call out and allow for such scanning to take place, and Apple has publicly described how the system works as far as its capabilities and limitations. In the sense that people see this as a change in policy (IIRC the actual license agreement language changed over a year ago) , Apple has also described how to no longer use the iCloud Photos service.
One less standard usage is not about unauthorized access but specifically to private surveillance (e.g. "Clipper chip") - but I would argue that the Clipper chip was a case where the surveillance features were specifically not being talked about, hence it still counting as "unauthorized access".
But with a definition that covers broad surveillance instead of unauthorized access, it would still be difficult to classify this as a back door. Such surveillance arguments would only pertain to the person's phone and not to information the user chose to release to external services like iCloud Photos.
To your original point, it would still work with iCloud Photos did not have the key escrow, albeit with less data being capable of being able to be turned over to law enforcement. However iCloud Photos being an external system would still mean this is an intentional and desired feature (presumably) by the actual system owners (Apple).
I see. My interpretation doesn't hold up given your definitions of back door.
I bet the authorities would be happy with a surveillance mechanism disclosed in the EULA, though. Even if such a system is not technically a back door, I am opposed to it and would prefer Apple to oppose it.
Edit: I just noticed that you had already clarified your argument in other replies. I am sorry to make you repeat it.
I bet the authorities would be happy with a surveillance mechanism disclosed in the EULA, though. Even if such a system is not technically a back door, I am opposed to it and would prefer Apple to oppose it.
Edit: I just noticed that you had already clarified your argument in other replies. I am sorry to make you repeat it.
It has proven very difficult to oppose laws meant to deter child abuse and exploitation.
Note that while the EFF's mission statement is about defending civil liberties, they posted two detailed articles about Apple's system without talking about the questionable parts of the underlying CSAM laws. There was nothing about how the laws negatively impact civil liberties and what the EFF might champion there.
The problem is that the laws themselves are somewhat uniquely abusable and overreaching, but they are meant to help reduce a really grotesque problem - and reducing aspects like detection and reporting is not going to be as effective against the underlying societal issue.
Apple has basically been fighting this for the 10 years since the introduction of iCloud Photo, saying they didn't have a way to balance the needs to detect CSAM material without impacting the privacy of the rest of users. PhotoDNA was already deployed at Microsoft and being deployed by third parties like Facebook when iCloud Photo launched.
Now it appears that Apple was working a significant portion on that time toward trying to build a system that _did_ attempt to accomplish a balance between social/regulatory responsibility and privacy.
But such a system has to prop technical systems and legal policies against one another to make up the shortcomings of each, which make it a very complex and nuanced system.
Note that while the EFF's mission statement is about defending civil liberties, they posted two detailed articles about Apple's system without talking about the questionable parts of the underlying CSAM laws. There was nothing about how the laws negatively impact civil liberties and what the EFF might champion there.
The problem is that the laws themselves are somewhat uniquely abusable and overreaching, but they are meant to help reduce a really grotesque problem - and reducing aspects like detection and reporting is not going to be as effective against the underlying societal issue.
Apple has basically been fighting this for the 10 years since the introduction of iCloud Photo, saying they didn't have a way to balance the needs to detect CSAM material without impacting the privacy of the rest of users. PhotoDNA was already deployed at Microsoft and being deployed by third parties like Facebook when iCloud Photo launched.
Now it appears that Apple was working a significant portion on that time toward trying to build a system that _did_ attempt to accomplish a balance between social/regulatory responsibility and privacy.
But such a system has to prop technical systems and legal policies against one another to make up the shortcomings of each, which make it a very complex and nuanced system.
But the catch is: all the incumbents already treated your data as if you were guilty until proven innocent. Apple’s transparency about that change may have led people to internalize that, but it’s been the de facto terms of most cloud relationships.
What I personally don’t understand is why Apple didn’t come out with a different message: we’ve made your iPhone so secure that we’ll let it vouch for your behalf when it sends us data to store. We don’t want to see the data, and we won’t see any of it unless we find that lots of the photos you send us are fishy. Android could never contemplate this because they can’t trust the OS to send vouchers for the same photos it uploads, so instead they snoop on everything you send them.
It seems like a much more win-win framing that emphasizes their strengths.
What I personally don’t understand is why Apple didn’t come out with a different message: we’ve made your iPhone so secure that we’ll let it vouch for your behalf when it sends us data to store. We don’t want to see the data, and we won’t see any of it unless we find that lots of the photos you send us are fishy. Android could never contemplate this because they can’t trust the OS to send vouchers for the same photos it uploads, so instead they snoop on everything you send them.
It seems like a much more win-win framing that emphasizes their strengths.
> What I personally don’t understand is why Apple didn’t come out with a different message: we’ve made your iPhone so secure that we’ll let it vouch for your behalf when it sends us data to store. We don’t want to see the data, and we won’t see any of it unless we find that lots of the photos you send us are fishy.
That is PR speak that would have landed worse in tech forums. I respect them more for not doing this.
The core issue is this performs scans, without your approval, on your local device. Viruses already do that and it's something techies have always feared governments might impose. That private companies are apparently being strong-armed into doing it is concerning because it means the government is trying to circumvent the process of public discussion that is typically facilitated by proposing legislation.
That is PR speak that would have landed worse in tech forums. I respect them more for not doing this.
The core issue is this performs scans, without your approval, on your local device. Viruses already do that and it's something techies have always feared governments might impose. That private companies are apparently being strong-armed into doing it is concerning because it means the government is trying to circumvent the process of public discussion that is typically facilitated by proposing legislation.
> without your approval
This isn’t true. You can always turn off iCloud Photo Library and just store the photos locally or use a different cloud provider.
This isn’t true. You can always turn off iCloud Photo Library and just store the photos locally or use a different cloud provider.
I hate this argument. Pressing yes to the T&C once when you setup an Apple account doesn’t exactly constitute my approval imo (even if it does legally).
There’s no disable button or even clear indication that it’s going on.
There’s no disable button or even clear indication that it’s going on.
> There’s no disable button or even clear indication that it’s going on.
iCloud Photos is an on-off switch.
In terms of clearly indicating everything that is going on within the service, that is just not possible for most non-tech users. It appears to have been pretty difficult even for those familiar with things like cryptography and E2E systems to understand the nuances and protections in place.
Instead, the expectation should be that using _any_ company's cloud hosting or cloud synchronization features is fundamentally sharing your data with them. It should then be any vendor's responsibility give give customers guarantees on which to evaluate trust.
iCloud Photos is an on-off switch.
In terms of clearly indicating everything that is going on within the service, that is just not possible for most non-tech users. It appears to have been pretty difficult even for those familiar with things like cryptography and E2E systems to understand the nuances and protections in place.
Instead, the expectation should be that using _any_ company's cloud hosting or cloud synchronization features is fundamentally sharing your data with them. It should then be any vendor's responsibility give give customers guarantees on which to evaluate trust.
No, the comment above meant there's no explicitly marked disable button. People who don't follow tech news won't hear about this.
Let's not pretend anyone is really "opting in" on this feature.
Anyone who doesn’t like it can switch off iCloud Photo Library.
I can imagine many people doing that in response to this news.
I can imagine many people doing that in response to this news.
The fact that you can opt out for now does not set me at ease.
Nor me. I will not opt out because I think there is no threat to me and I like iCloud photos. That doesn’t mean I like the presence of this mechanism.
> That doesn’t mean I like the presence of this mechanism.
I don't think I ever said you did.
I don't think I ever said you did.
No - but you did suggest there was no opting in. I’m pointing out that just because I’m not entirely happy with the choice doesn’t mean it isn’t a choice.
> doesn’t mean it isn’t a choice.
People who haven't heard what Apple is doing are not presented with a choice. Not everyone follows tech news so closely.
People who haven't heard what Apple is doing are not presented with a choice. Not everyone follows tech news so closely.
I agree this is vastly better than what anyone else is doing, and you know I understand the technology.
However, I don’t think any framing would have improved things. I think it was always going to feel wrong.
I would prefer they don’t do this because it feels bad to be a suspect even in this abstract and in-practice harmless way.
Having said that, having heard from people who have investigated how bad pedophile activity actually is, I can imagine being easily persuaded back in the other direction.
I think thins is about the logic of evolutionary psychology, not the logic of cryptography.
My guess is that between now and the October iPhone release we are going to see more media about the extent of the problem they are trying to solve.
That is how Apple wins this.
However, I don’t think any framing would have improved things. I think it was always going to feel wrong.
I would prefer they don’t do this because it feels bad to be a suspect even in this abstract and in-practice harmless way.
Having said that, having heard from people who have investigated how bad pedophile activity actually is, I can imagine being easily persuaded back in the other direction.
I think thins is about the logic of evolutionary psychology, not the logic of cryptography.
My guess is that between now and the October iPhone release we are going to see more media about the extent of the problem they are trying to solve.
That is how Apple wins this.
> Having said that, having heard from people who have investigated how bad pedophile activity actually is, I can imagine being easily persuaded back in the other direction.
There are terrible things out there that we should seek to solve. They should not be solved by creating 1984 in the literal sense, and certainly not by the company that became famous for an advertisement based on that book [1].
Apple, take your own advice and Think Different [2].
[1] https://www.youtube.com/watch?v=VtvjbmoDx-I
[2] https://www.youtube.com/watch?v=5sMBhDv4sik
There are terrible things out there that we should seek to solve. They should not be solved by creating 1984 in the literal sense, and certainly not by the company that became famous for an advertisement based on that book [1].
Apple, take your own advice and Think Different [2].
[1] https://www.youtube.com/watch?v=VtvjbmoDx-I
[2] https://www.youtube.com/watch?v=5sMBhDv4sik
> creating 1984 in the literal sense
I take it you haven’t read 1984.
When Craig Federighi straps a cage full of starving rats to my face, I’ll concede this point.
I take it you haven’t read 1984.
When Craig Federighi straps a cage full of starving rats to my face, I’ll concede this point.
China has tiger chairs. Should we move closer to their big brother systems?
No but this has nothing to do with that.
In case you missed it, I think this is probably a bad move.
I just don’t think these arguments about back doors and creeping totalitarianism are either accurate or likely to persuade everyday users when they weigh them up against the grotesque nature of child exploitation.
In case you missed it, I think this is probably a bad move.
I just don’t think these arguments about back doors and creeping totalitarianism are either accurate or likely to persuade everyday users when they weigh them up against the grotesque nature of child exploitation.
> I just don’t think these arguments about back doors and creeping totalitarianism are either accurate or likely to persuade everyday users when they weigh them up against the grotesque nature of child exploitation.
Agree to disagree. This opens the door to something much worse in my opinion.
Agree to disagree. This opens the door to something much worse in my opinion.
I don’t think I agree. If you think this boils down to instinct, do you think a story about coming together to save the next generation will work well on people cynical enough to see TLAs around every corner? At the very least, I feel like Apple should probably make a concession to the conspiracy minded so that they can bleach any offensive bits from their device and use them as offline-first DEFCON map viewing devices, or some such.
> do you think a story about coming together to save the next generation will work well on people cynical enough to see TLAs around every corner?
Absolutely not. I don’t think they need to persuade people who are convinced of their iniquity.
What they need is an environment in which those people look like they are arguing over how many dictatorships need to collude to detect anti-government photos and how this constitutes literal 1984, while Apple is announcing a way to deter horrific crimes against American children that they have seen on TV.
Absolutely not. I don’t think they need to persuade people who are convinced of their iniquity.
What they need is an environment in which those people look like they are arguing over how many dictatorships need to collude to detect anti-government photos and how this constitutes literal 1984, while Apple is announcing a way to deter horrific crimes against American children that they have seen on TV.
That's not at all clear.
A lot of people like strong border controls for instance, even if it means when they return to their country from abroad they have to go through more checks or present more documents to get in. Or consider large gated communities where you have to be checked by a guard to get in.
Many peoples seem fine with being distrusted as long as the distrust is part of a mechanism to weed out those who they feel really are not trustworthy and it is not too annoying for them to prove that they are not one of those people when they encounter a trust check.
A lot of people like strong border controls for instance, even if it means when they return to their country from abroad they have to go through more checks or present more documents to get in. Or consider large gated communities where you have to be checked by a guard to get in.
Many peoples seem fine with being distrusted as long as the distrust is part of a mechanism to weed out those who they feel really are not trustworthy and it is not too annoying for them to prove that they are not one of those people when they encounter a trust check.
Your examples are not a good analogy. The distrust is transient and then you are cleared. This is a stare of permanently being a suspect.
However, it may certainly be the case that in the end people in general do accept this as a price worth paying to fight pedophiles.
However, it may certainly be the case that in the end people in general do accept this as a price worth paying to fight pedophiles.
Here's what I don't get: who is importing CSAM into their camera roll in the first place? I for one have never felt the urge to import regular, legal porn into my camera roll. Who the hell is going to be doing that with stuff they know will land them in prison? Who the hell co-mingles their deepest darkest dirtiest secret amongst pictures of their family and last night’s dinner?
I can believe that some people might be so technically illiterate that they'd upload CSAM to a private Facebook group thinking it was secure. But as far as I'm aware it's not possible to password protect photo albums using the built-in Photos app.
I can believe that some people might be so technically illiterate that they'd upload CSAM to a private Facebook group thinking it was secure. But as far as I'm aware it's not possible to password protect photo albums using the built-in Photos app.
Actually it occurs to me that perhaps they're noticing a lot of CSAM is being produced with smartphone cameras and are hoping to snag the phone which produced the originals. If they can get new content into their database before the photographer deletes them, they might find the phone which took the originals—and then find the child victim.
Beyond implausible, but then most law enforcement tends to rely on someone eventually doing something stupid.
Beyond implausible, but then most law enforcement tends to rely on someone eventually doing something stupid.
It is only supposed to detect CSAM already known to NCMEC, not identify new images.
That's my point. If the original photographer doesn't delete them by the time they're known to NCMEC and the hash database is updated, they would match. As I said, I recognise this is beyond implausible.
NCMEC's database does not only contain CSAM. It has never been audited. It's full of false positives. They are immune from FOIA requests. They are accountable to nobody. They work so closely with the FBI that there are FBI agents working on the database directly.
NCMEC is essentially an unaccountable, unauditable department of the FBI that also happens to be incompetent (the amount of non-CSAM in the database is large).
NCMEC is essentially an unaccountable, unauditable department of the FBI that also happens to be incompetent (the amount of non-CSAM in the database is large).
You don't know what's in it, but you do know it's full of false positives? I wonder, do you know how many of those false positives are flagged as A1?
I know for a fact that it is full of false positives, there's also public sources making the same claim. [1]
[1] https://www.hackerfactor.com/blog/index.php?/archives/929-On...
[1] https://www.hackerfactor.com/blog/index.php?/archives/929-On...
I clicked on your link hoping for serious analysis. I would have settled for interesting analysis. I was disappointed. It's just a guy who found one MD5 hash collision. The paragraph was written in a way that makes it unclear whether source of this specific hash was in fact NCMEC or if it was "other law enforcement sources". So which was it? Did this person follow up with the source to confirm whether his hash match was a false positive or a hash collision?
So in short, the source for your claims has evidence of between zero and one false positives.
Unimpressive would be an understatement.
So in short, the source for your claims has evidence of between zero and one false positives.
Unimpressive would be an understatement.
It was not a hash collision. It was a false positive. The way the hashing solution works doesn't really allow for a false positive except in extraordinarily rare circumstances.
It matched a man holding a monkey full clothed as a CSAM image, direct from NCMEC's database.
He encountered a 20% false positive rate while running a moderately popular image service, with an admittedly low sample size. It's still evidence, and given NCMEC is immune from oversight, FOIA and accountability, it's concerning.
Also, the fact I know there are false positives does not stem from that post. I know it independently, but since you asked for a source stronger than "just trust a random internet guy", I gave you one.
He's not the only person making the claim though, others throughout these threads with industry knowledge have confirmed what I already knew. If you're asking me to reveal how I know, I'm afraid you'll be disappointed. I'd rather be accused of lying than elaborate.
It matched a man holding a monkey full clothed as a CSAM image, direct from NCMEC's database.
He encountered a 20% false positive rate while running a moderately popular image service, with an admittedly low sample size. It's still evidence, and given NCMEC is immune from oversight, FOIA and accountability, it's concerning.
Also, the fact I know there are false positives does not stem from that post. I know it independently, but since you asked for a source stronger than "just trust a random internet guy", I gave you one.
He's not the only person making the claim though, others throughout these threads with industry knowledge have confirmed what I already knew. If you're asking me to reveal how I know, I'm afraid you'll be disappointed. I'd rather be accused of lying than elaborate.
> It was not a hash collision. It was a false positive.
Again, this was an MD5. It's literally impossible to assert that it was a false positive with absolute certainty. A hash collision is not outside the realm of possibility, especially with MD5. Apparently no attempt was made to chase this up. And we still don't know whether it was the NCMEC database or "other law enforcement sources".
You continue to claim that it was "direct from NCMEC's database" but again, that isn't asserted by your source.
> He encountered a 20% false positive rate
He encountered one potential false positive. Converting one data point into a percentage is exactly why earlier I described this nonsense as being disingenuous. The fact that you would cite this source and then defend their statistical clown show is, in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up.
Again, this was an MD5. It's literally impossible to assert that it was a false positive with absolute certainty. A hash collision is not outside the realm of possibility, especially with MD5. Apparently no attempt was made to chase this up. And we still don't know whether it was the NCMEC database or "other law enforcement sources".
You continue to claim that it was "direct from NCMEC's database" but again, that isn't asserted by your source.
> He encountered a 20% false positive rate
He encountered one potential false positive. Converting one data point into a percentage is exactly why earlier I described this nonsense as being disingenuous. The fact that you would cite this source and then defend their statistical clown show is, in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up.
What you're missing is the statistical probability of two MD5 hashes colliding, which is astronomically unlikely.
Your argument is essentially that a collision is more likely than a false positive, which would imply a false positive rate of 0.00% in NCMEC's database based on its size.
It's clear that nothing will convince you if you believe that humans managing a database will never, ever make a mistake after over 300,000,000 entries. Because if they make one single mistake, then it supports my argument -- a false positive becomes substantially more likely statistically than a hash collision.
You're also providing a pretty large red herring with your suggestion that he could've simply asked NCMEC if it was a false positive or a hash collision. NCMEC would never provide that information, because the database is highly secret.
Given those statistics, I think that source is more than valid.
>in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up
I am happy to let you believe that I'm lying.
One industry insider whose employer works with NCMEC cited a false positive rate of 1 in 1,000. [1] The product he works in is used in conjunction with NCMEC's database. Elsewhere, in press releases, the company cites a failure rate of 1% (presumably both false positives and false negatives) [2]
[1] https://news.ycombinator.com/item?id=21446562
[2] https://www.prnewswire.com/news-releases/thorns-automated-to...
Your argument is essentially that a collision is more likely than a false positive, which would imply a false positive rate of 0.00% in NCMEC's database based on its size.
It's clear that nothing will convince you if you believe that humans managing a database will never, ever make a mistake after over 300,000,000 entries. Because if they make one single mistake, then it supports my argument -- a false positive becomes substantially more likely statistically than a hash collision.
You're also providing a pretty large red herring with your suggestion that he could've simply asked NCMEC if it was a false positive or a hash collision. NCMEC would never provide that information, because the database is highly secret.
Given those statistics, I think that source is more than valid.
>in my opinion, strong positive evidence that your other citation-free assertions are all entirely made up
I am happy to let you believe that I'm lying.
One industry insider whose employer works with NCMEC cited a false positive rate of 1 in 1,000. [1] The product he works in is used in conjunction with NCMEC's database. Elsewhere, in press releases, the company cites a failure rate of 1% (presumably both false positives and false negatives) [2]
[1] https://news.ycombinator.com/item?id=21446562
[2] https://www.prnewswire.com/news-releases/thorns-automated-to...
Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data. I am merely pushing back at your evidence-free claim that "it's full of false positives."
Once again, you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". You're reading far more into your sources than they are saying.
And now you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are clearly very confused.
Once again, you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources". You're reading far more into your sources than they are saying.
And now you are conflating claims of false positives in the origin database with rates of false positives in a perceptual hashing algorithm. You are clearly very confused.
>Nowhere have I claimed that the NCMEC databases are entirely devoid of miscategorised data
If NCMEC's databases have a false positive rate of 1 in 1,000, do you realise that means a false positive rate is substantially more likely than a hash collision?
>I am merely pushing back at your evidence-free claim that "it's full of false positives."
1 in 1,000 is "full of false positives" by my own standards, to which I've posted evidence from an employee whose company works directly with NCMEC.
>you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources".
NCMEC and law enforcement are one and the same. FBI employees work directly at NCMEC. [1] Law enforcement have direct access to the database, including for categorisation purposes. To suggest law enforcement's dataset is tainted yet NCMEC's is not doesn't make any sense to me.
>You are clearly very confused
Address the 1 in 1,000 claim. Thorn is an NCMEC partner and a Thorn employee has claimed a false positive rate of 1 in 1,000. In press releases, Thorn is even less sure at 1% failure rates. Thorn uses perceptual hashing.
I can't see how you can simultaneously claim the database isn't "full of false positives" while acknowledging a failure rate as abysmal as 1 in 1,000.
I also didn't conflate anything, both a hash collision and a perceptual hash collision are less likely than a false positive, by an extraordinary margin. Apple claims their algorithm has a collision 1 in 1,000,000,000,000 times. Compare that to 1 in 1,000.
The database is full of false positives, and now presumably you'll deny both industry claims and NCMEC partner claims.
[1] https://www.fbi.gov/investigate/violent-crime/cac
If NCMEC's databases have a false positive rate of 1 in 1,000, do you realise that means a false positive rate is substantially more likely than a hash collision?
>I am merely pushing back at your evidence-free claim that "it's full of false positives."
1 in 1,000 is "full of false positives" by my own standards, to which I've posted evidence from an employee whose company works directly with NCMEC.
>you continue to assume that the MD5 collision cited was from a NCMEC corpus and not "other law enforcement sources".
NCMEC and law enforcement are one and the same. FBI employees work directly at NCMEC. [1] Law enforcement have direct access to the database, including for categorisation purposes. To suggest law enforcement's dataset is tainted yet NCMEC's is not doesn't make any sense to me.
>You are clearly very confused
Address the 1 in 1,000 claim. Thorn is an NCMEC partner and a Thorn employee has claimed a false positive rate of 1 in 1,000. In press releases, Thorn is even less sure at 1% failure rates. Thorn uses perceptual hashing.
I can't see how you can simultaneously claim the database isn't "full of false positives" while acknowledging a failure rate as abysmal as 1 in 1,000.
I also didn't conflate anything, both a hash collision and a perceptual hash collision are less likely than a false positive, by an extraordinary margin. Apple claims their algorithm has a collision 1 in 1,000,000,000,000 times. Compare that to 1 in 1,000.
The database is full of false positives, and now presumably you'll deny both industry claims and NCMEC partner claims.
[1] https://www.fbi.gov/investigate/violent-crime/cac
Apparently 70 million images have been reported by other providers (if I'm reading [1] correctly, which I'm not sure I am, since the paywall hides it before I can read closely).
Assuming that's correct, the answer is "a lot of people." I find that eminently plausible.
[1] https://www.nytimes.com/2020/02/07/us/online-child-sexual-ab...
Assuming that's correct, the answer is "a lot of people." I find that eminently plausible.
[1] https://www.nytimes.com/2020/02/07/us/online-child-sexual-ab...
It looks like well over 90% of that is Facebook and the bulk of the remainder is Google and Yahoo who crawl the web and run large email services. None of those are analogous to importing CSAM into your smartphone's photo library.
Why don't they scan files as they're downloaded, the way we scan for malware and viruses on PCs? This way, we can warn people _before_ a law is broken!
This is Apple's trying to shift the focus onto technicalities. Remotely monitored CCTV cams in your house is never a good idea, no matter what the treat model is.
If they continue down this road, they will not only have effectively created the smartphone category, but also coincidentally destroyed it.
If our devices are designed to spy on us, we're frankly not even going to use them anymore. I wonder if they forgot that using this thing is optional? You'll see a resurgence of single purpose dumb devices.
If our devices are designed to spy on us, we're frankly not even going to use them anymore. I wonder if they forgot that using this thing is optional? You'll see a resurgence of single purpose dumb devices.
I just found the perfect video clip to represent Apple's response to launching this feature: https://youtu.be/-7XchZICFQU?t=17
My phone is an extension of my brain, it is my most trusted companion. It holds my passwords, my mail, my messages, my photos, my plans and notes, it holds the keys to my bank accounts and my investments. I sleep with it by my bed and I carry it around every day. It is my partner in crime.
Now apple are telling me that my trusted companion is scanning my photos as they are uploaded to iCloud, looking for evidence that apple can pass to the authorities? They made my phone a snitch?
This isn't about security or privacy, I don't care about encryption or hashes here, this is about trust. If I can't trust my phone, I can't use it.
Now apple are telling me that my trusted companion is scanning my photos as they are uploaded to iCloud, looking for evidence that apple can pass to the authorities? They made my phone a snitch?
This isn't about security or privacy, I don't care about encryption or hashes here, this is about trust. If I can't trust my phone, I can't use it.
Technically they could have been scanning the photos already to power some AI algorithms or whatever else.
I think this is a nudge for folks to wake up and see the reality of what it means to use the cloud. We are leasing storage space from Apple in this case.
Technically, it’s no different than a landlord checking up on their tenants to make sure “everything is okay.”
And technically, if you do not like iCloud, don’t use it and roll your own custom cloud storage! After all, it’s redundancy and access the cloud provides. And Apple provides APIs to build them.
Hell, with the new Files app on iOS i just use a Samba share with my NAS server (just a custom built desktop with RAID 5 and Ubuntu.)
I think this is a nudge for folks to wake up and see the reality of what it means to use the cloud. We are leasing storage space from Apple in this case.
Technically, it’s no different than a landlord checking up on their tenants to make sure “everything is okay.”
And technically, if you do not like iCloud, don’t use it and roll your own custom cloud storage! After all, it’s redundancy and access the cloud provides. And Apple provides APIs to build them.
Hell, with the new Files app on iOS i just use a Samba share with my NAS server (just a custom built desktop with RAID 5 and Ubuntu.)
> Technically, it’s no different than a landlord checking up on their tenants to make sure “everything is okay.”
By this logic, you don't own your phone but rent it from Apple?
By this logic, you don't own your phone but rent it from Apple?
Technically I own my iPhone but the software is licensed to me (not a lawyer but that’s my understanding).
We probably also sign something that allows Apple to do what they will with our images—albeit we retain the copyrights to them.
Just to reiterate: the landlord metaphor is referring to software which we lease from Apple—not the device itself.
This is yet another case where I want to emphasize the importance of OPEN hardware and software designs. If we truly want ownership, we have to take ownership into our hands (which means a lot of hard work). Most commercial software is licensed so no we don’t own anything in whole that runs commercial software.
We probably also sign something that allows Apple to do what they will with our images—albeit we retain the copyrights to them.
Just to reiterate: the landlord metaphor is referring to software which we lease from Apple—not the device itself.
This is yet another case where I want to emphasize the importance of OPEN hardware and software designs. If we truly want ownership, we have to take ownership into our hands (which means a lot of hard work). Most commercial software is licensed so no we don’t own anything in whole that runs commercial software.
They do scan the photos on my device, to provide useful functionality to me. All good.
If they scan photos in iCloud, that's up to apple, I can use it or not. No problem.
With this new setup, my device can betray me.I think that is different and crosses a line.
No they really couldn't, because someone investigating, monitoring and reverse engineering the device traffic might have noticed, it would be leaked by a whistleblower, there are plenty of ways this could ruin Apple.
Not so now, they are in the clear either way because Apple themselfes cannot look into the hashes databank. So the backdoor is there, the responsibility of the crime of spying is forwarded to different actors, which even cannot be monitored by Apple.
This is truly a devilish device they thought up. Make misuse possible, exonerate any responsibility to outside actors, act naive as if hands are clean.
Not so now, they are in the clear either way because Apple themselfes cannot look into the hashes databank. So the backdoor is there, the responsibility of the crime of spying is forwarded to different actors, which even cannot be monitored by Apple.
This is truly a devilish device they thought up. Make misuse possible, exonerate any responsibility to outside actors, act naive as if hands are clean.
This is the thing Apple doesn't realize: our phone's OS vendor has one job: not to betray us.
They have betrayed us.
I have purchased my last iPhone.
They have betrayed us.
I have purchased my last iPhone.
good luck finding a replacement, you'll end up using a custom rom if you are serious about tracking and privacy, and that custom rom will be... fully dependent on donations.
I'd pay for it as a subscription service TBH. It would be good for security minded corps and nerds.
of all the items in the list, this is what the govt is interested in :
> It is my partner in crime.
> It is my partner in crime.
A threat model without a discussion on threats. Nice.
Here's the thing I keep circling around: assume that bad actors, government or otherwise, want to target political dissidents using internet-enabled smartphones. The more we learn about the way Apple actually implemented this technology, the less likely it seems that it would make it radically easier for those bad actors to do so. For instance, the "it only scans photos uploaded to iCloud" element isn't just an arbitrary limitation that can be flipped with one line of code, as some folks seem to think; as Erik Neuenschwander, head of Privacy Engineering at Apple, explained in an interview on TechCrunch[1]:
> Our system involves both an on-device component where the voucher is created, but nothing is learned, and a server-side component, which is where that voucher is sent along with data coming to Apple service and processed across the account to learn if there are collections of illegal CSAM. That means that it is a service feature.
Will this stop those bad actors if they're determined? No, of course not, but there are so many ways they can do it already. They'll get cloud storage providers to give them access. If they can't, they'll get network providers to give them access. And if those bad actors are, as many people fear, government actors, then they have tools far more potent than code: they have laws. It was illegal to export "strong encryption" for many years, remember? I've seen multiple reports that European lawmakers are planning to require some kind of scanning for CSAM. If this goes into effect, technology isn't going to block those laws for you. Your Purism phone will either be forced to comply or be illegal.
I wrote in a previous comment on this that one of Silicon Valley's original sins is that we tend to treat all problems as if they're engineering problems. Apple is treating CSAM as an engineering problem. Most of the discussion on HN about how horrible and wrong Apple is here still treats it as an engineering problem, though: well, you can get around this by just turning off iCloud Photos or never using Apple software or throwing your iPhone in the nearest lake and switching to Android, but only the right kind of Android, or maybe just never doing anything with computers again, which admittedly will probably be effective.
Yet at the end of the day, this isn't an engineering problem. It's a policy problem. It's a governance problem. In the long run, we solve this, at least in liberal democracies, by voting people into office who understand technology, understand the value of personal encryption, and last but certainly not least, understand the value of, well, liberal democracy. I know that's easy to dismiss as Pollyannaism, but "we need to protect ourselves from our own government" has a pretty dismal track record historically. The entire point of having a liberal democracy is that we are the government, and we can pull it back from authoritarianism.
The one thing that Apple is absolutely right about is that expanding what those hashes check for is a policy decision. Maybe where those hashes get checked isn't really what we need to be arguing about.
[1]: https://techcrunch.com/2021/08/10/interview-apples-head-of-p...