Shopify employees accessed customer databases without authorization
38 comments
The only icing in the cake is that at least Shopify has been both transparent and quick - it's only taken a couple months and they've managed to get bottom of the case. Couple months might seem long but from what I've seen it takes about a year of lag time from the start of the breach to when the company finds out/acknowledges.
In any case I'm wondering - how did Shopify discover this intrusion? Do they check logs regularly? Did they receive a tip off?
In any case I'm wondering - how did Shopify discover this intrusion? Do they check logs regularly? Did they receive a tip off?
This is a common requirement for security-conscious organizations, especially those with HIPAA or PCI requirements. For shopify, this likely was originally created as a customer requirement, so that clients could monitor their staff. The typical setup is to generate internal user logs and feed it into a SIEM of some type, potentially with custom rules to do some checking. Alternatively, this may very well have been caught by a type of DLP (data loss protection) or network monitoring product
It seems like employees are becoming the weakest link in cloud security. If Google will be breached one day, most probably it will happen not because of a technical vulnerability, but due to employee sabotage.
I'm pretty sure that at exactly this moment somewhere someone criminal is already analyzing organization structures, employee profiles, internal security policies and tools of the cloud giants.
I'm pretty sure that at exactly this moment somewhere someone criminal is already analyzing organization structures, employee profiles, internal security policies and tools of the cloud giants.
That already happened a decade ago: https://www.wired.com/2010/09/google-spy/
Speaking as an ex-Facebook engineer it would be incredibly easy to get nefarious people employed there in an engineering role. Once inside, they have access to -all- user data; there is no actual access control (there are some basic access checks built into the Facebook application to keep you from accessing "private" data fields by accident, but all you have to do is edit that code and remove the access check and recompile Facebook on your laptop and you can access anything in the production database -- including peoples private Messenger chats).
Facebook warn you when you are hired not to actually do this, because they have auditing systems to watch for it and you will be fired (supposedly) but for people employed by some other agency specifically for the purpose of getting high-value private data out of Facebook, being fired by Facebook for doing so is part of the expected outcome and no big deal.
A well funded agency could easily keep getting people hired at Facebook to get whatever data they want, as often as they want. Facebook is constantly trying to keep their hiring pipelines full and despite the image Facebook likes to portray, it isn't "only the best talent" that gets a job there. There are some very smart, capable people at Facebook, but there are a ton of very mediocre engineers that lucked out in the hiring process, as well. It's really just a numbers game to get in.
I'm sure much the same is true at Google/Twitter/etc.
Facebook warn you when you are hired not to actually do this, because they have auditing systems to watch for it and you will be fired (supposedly) but for people employed by some other agency specifically for the purpose of getting high-value private data out of Facebook, being fired by Facebook for doing so is part of the expected outcome and no big deal.
A well funded agency could easily keep getting people hired at Facebook to get whatever data they want, as often as they want. Facebook is constantly trying to keep their hiring pipelines full and despite the image Facebook likes to portray, it isn't "only the best talent" that gets a job there. There are some very smart, capable people at Facebook, but there are a ton of very mediocre engineers that lucked out in the hiring process, as well. It's really just a numbers game to get in.
I'm sure much the same is true at Google/Twitter/etc.
Then it wouldn't be unreasonable to assume that such nefarious people have already infiltrated Facebook and other tech giants. And they are not amateurs. Given the extraordinary value of access to the information possessed by the big tech corporations, we can expect a level of sophistication that would be on par with Cold War-era espionage.
Didn't receive an email, but are they just now referring to the incident that took place September 23 2020?
https://www.cbc.ca/news/business/shopify-data-breach-1.57351...
https://www.cbc.ca/news/business/shopify-data-breach-1.57351...
> not the result of some sort of technical vulnerability
So what then? Did they deceive or coerce somebody?
So what then? Did they deceive or coerce somebody?
I reckon it was more about internal employees abusing their permissions and access to customer data and being sloppy when attempting to extract it.
That wouldn't be "without authorization" then.
Yes it would. If you fail to put up a fence around your yard, that doesn’t mean I’m authorized to walk in and steal your garden gnomes. Sure, it would be ideal if authorization were enforced with technical barriers, but just because a computer system thinks someone is authorized to access data doesn’t mean they have the legal authority to do so.
Let’s say I own a company like Shopify. I have an agreement with my customers: I won’t use consumers’ personal details for anything other than processing orders. Sure, I have the technical authority to poke around in the database if I so desire. I could technically take all that data and sell it to the highest bidder. That doesn’t mean I’m legally or procedurally authorized to do so.
Employees have procedures they are expected to follow. Many employees have significant permissions to access data. It’s unreasonable to bar all employees from accessing sensitive data at a technical level; people need to be able to fix problems when things go wrong. If a group of developers conspire to push malicious or faulty changes to production, including the developers who are supposed to be reviewing code and preventing such things from happening, that doesn’t change the fact that they are not authorized to exfiltrate data.
You can go online and buy a set of keys used by first responders. For $25, you can get into just about any commercial building. Are you authorized to actually use those keys on someone else’s property? Not without a contract, but that doesn’t mean there’s a technical barrier in place.
Let’s talk about Shopify’s datacenters! They probably colocate or use a cloud service or whatever. Ultimately, data is stored in datacenters. Someone like Deviant Ollam will have no trouble waltzing right in the front door. He might already have the keys. Or maybe he colocates at the same datacenter, getting him a good chunk of the way there. Is he legally authorized to access customer data, despite having the tools at his disposal to access it with minimal difficulty and likely no digital hackery? No, he is not.
Let’s say I own a company like Shopify. I have an agreement with my customers: I won’t use consumers’ personal details for anything other than processing orders. Sure, I have the technical authority to poke around in the database if I so desire. I could technically take all that data and sell it to the highest bidder. That doesn’t mean I’m legally or procedurally authorized to do so.
Employees have procedures they are expected to follow. Many employees have significant permissions to access data. It’s unreasonable to bar all employees from accessing sensitive data at a technical level; people need to be able to fix problems when things go wrong. If a group of developers conspire to push malicious or faulty changes to production, including the developers who are supposed to be reviewing code and preventing such things from happening, that doesn’t change the fact that they are not authorized to exfiltrate data.
You can go online and buy a set of keys used by first responders. For $25, you can get into just about any commercial building. Are you authorized to actually use those keys on someone else’s property? Not without a contract, but that doesn’t mean there’s a technical barrier in place.
Let’s talk about Shopify’s datacenters! They probably colocate or use a cloud service or whatever. Ultimately, data is stored in datacenters. Someone like Deviant Ollam will have no trouble waltzing right in the front door. He might already have the keys. Or maybe he colocates at the same datacenter, getting him a good chunk of the way there. Is he legally authorized to access customer data, despite having the tools at his disposal to access it with minimal difficulty and likely no digital hackery? No, he is not.
Thanks for the thoughtful comment. You make several excellent points which I really appreciate!
> It’s unreasonable to bar all employees from accessing sensitive data at a technical level
This is the heart of the confusion. Sensitive data must be locked down (e.g., encrypted) and access tightly controlled so only employees with a legitimate purpose have read access. Since this is a "technical" solution to the problem, I would label the original data breach a "technical" vulnerability.
On the hand, the "developers conspire to push malicious or faulty changes to production" scenario is not a technical vulnerability; it falls into the category of deceit/fraud à la social engineering. Of course there are technical means you could try to foil exfiltration, but generally this sort of attack is prevented by non-technical means e.g. code review.
> It’s unreasonable to bar all employees from accessing sensitive data at a technical level
This is the heart of the confusion. Sensitive data must be locked down (e.g., encrypted) and access tightly controlled so only employees with a legitimate purpose have read access. Since this is a "technical" solution to the problem, I would label the original data breach a "technical" vulnerability.
On the hand, the "developers conspire to push malicious or faulty changes to production" scenario is not a technical vulnerability; it falls into the category of deceit/fraud à la social engineering. Of course there are technical means you could try to foil exfiltration, but generally this sort of attack is prevented by non-technical means e.g. code review.
Two Tell HNs in two days about Shopify with some weak cases. I smell some stock shorting strategy here.
We have recourse against platform employees who snoop user data for personal reasons, and even share it with their friends or political organizations? Literally thought that was a perk of their jobs.
Someone should tell reddit/google/facebook/amazon as that will blow things up pretty badly.
Wait until they are subject to normal privacy regulations that require the companies to list the names of people who have accessed their user data.
Someone should tell reddit/google/facebook/amazon as that will blow things up pretty badly.
Wait until they are subject to normal privacy regulations that require the companies to list the names of people who have accessed their user data.
Something like this should probably legislated https://cloud.google.com/access-transparency
But it doesn't stop the government from bulk copying "business data" via powers granted by the Patriot Act.
But it doesn't stop the government from bulk copying "business data" via powers granted by the Patriot Act.
From what I've heard from ex-FB employees, they are told repeatedly during onboarding that snooping for personal reasons is a first-offensive firable offense, to the point that engineers often won't try to look up information even when debugging prod issues.
How do you protect against this sort of thing as a SaaS developer?
Prevent direct database access or at least allow it only from jump servers which don’t allow file transfers.
For troubleshooting purposes create debugging tools. Log and check their usage. When things mature, you can even require multiple admins to work together for certain actions.
Minimize human access to production envs. Automate deployments. When access is needed, use jump servers and block file transfers (or force them to go through channel that is audited).
Do review logs and alerts on regular basis. Put effort to minimize false alerts and excessive logging. Quite when reviewing logs you just notice things that “don’t look right”.
Nothing is 100% secure, but also people with bad intensions don’t always have unlimited skills/energy/time.
For troubleshooting purposes create debugging tools. Log and check their usage. When things mature, you can even require multiple admins to work together for certain actions.
Minimize human access to production envs. Automate deployments. When access is needed, use jump servers and block file transfers (or force them to go through channel that is audited).
Do review logs and alerts on regular basis. Put effort to minimize false alerts and excessive logging. Quite when reviewing logs you just notice things that “don’t look right”.
Nothing is 100% secure, but also people with bad intensions don’t always have unlimited skills/energy/time.
A step in the right direction is to use encrypted backend data / databases. This is still fraught with problems, but it provides another layer of protection and can demonstrate the difference between "The data was just sitting there" and "We had to manually exfiltrate the encryption keys to read the data".
It's not perfect, but it adds another layer to prove malicious intent.
It's not perfect, but it adds another layer to prove malicious intent.
In the vast majority of situations where I have seen unauthorized use of data in this fashion it was customer support people. All the encryption in the world doesn't help if you build tools that allow relatively low paid and under-trained employees to access the data at will.
Effective things you can do to reduce risk:
Effective things you can do to reduce risk:
- Deploy honeypot accounts that email multiple people if they are accessed
- Create a two-man system that requires a second person to approve "risky" things
- Require a 2FA token for your internal tools
- Email the user when they log in from a new IP address
- Have support staff use two different browsers - one to access the ticket system and another to access internal toolsI've thought about this, and researched it too, and see two problems with encrypting user data in the DB:
1. You would have to use an external search engine to index user data, and that would need encryption too.
2. If the user forgets their password then their data is inaccessible.
The encryption key doesn’t have to be per-user.
If your data is stored in a 3rd party database its common to just use one secret key (which only your app has access to) for all the data to prevent the 3rd party from reading it.
If your data is stored in a 3rd party database its common to just use one secret key (which only your app has access to) for all the data to prevent the 3rd party from reading it.
Employees should have access to customers' data on a need to know basis. Most employees do not need access so should not have access.
Then, there should be an audit trail of all accesses and this should be known to employees. First that dissuades employees from acting improperly, second that allows the company to verify that they indeed do not act improperly and to track down culprits if something happens.
Then, there should be an audit trail of all accesses and this should be known to employees. First that dissuades employees from acting improperly, second that allows the company to verify that they indeed do not act improperly and to track down culprits if something happens.
I sometimes go out of my way to hide my identity from sites/services I sign up with. Easiest way to get doxxed is for someone to ask one of their polticially-aligned buddies working at a site to pull up your info.
I mean... its only news because it got out. If you seriously believe companies arent accessing your data its borderline delusional.
Any system admin accessing to a server is able to take a copy of any data stored (or even transiently present on its network interface) on it.
If he is a spy/robber, if he is corruptible or threatened... a third party will obtain this copy. For the main culprit this doesn't induce any risk (where is the evidence?). This is absolutely not as with your bank, for example, which cannot really steal money without you taking notice.
How serious people are willing to store confidential data on any rented or hosted server is completely beyond me. Then some of their competitors' proposals are "just a little bit" better than theirs', or seem to have a pretty good grasp on some R&D or customer database.
Many here work on some cloud thing, most are honest and some will be upset by my comment. This is not about you but about rotten fruits in the basket.
If he is a spy/robber, if he is corruptible or threatened... a third party will obtain this copy. For the main culprit this doesn't induce any risk (where is the evidence?). This is absolutely not as with your bank, for example, which cannot really steal money without you taking notice.
How serious people are willing to store confidential data on any rented or hosted server is completely beyond me. Then some of their competitors' proposals are "just a little bit" better than theirs', or seem to have a pretty good grasp on some R&D or customer database.
Many here work on some cloud thing, most are honest and some will be upset by my comment. This is not about you but about rotten fruits in the basket.
Um what? The problem was the lack of internal authorization to do so. Do you not see how that's a huge liability? It's basically an "inside job". If one employee can do it, then anybody with similar credentials can.
The "oh shit" scenario is when the stolen data is used against to commit crimes against customers, e.g., identity theft, stalking, you name it.
The "oh shit" scenario is when the stolen data is used against to commit crimes against customers, e.g., identity theft, stalking, you name it.
Again, youre assuming internal controls exist implicitly. They dont. Theyre a risk exercise not a requirement.
They have very loose controls at shopify - no ISMS, no standard key controls - bluntly, it’s a miracle they haven’t had much worse happen yet. They’re not even ISO27001 compliant or certified.
This may have been true in the past, however per https://www.shopify.com/security they are SOC2-certified (SOC2 is significantly more common in North America), they are certainly PCI Level 1, and have GDPR/CCPA compliance requirements. You can also see their 2019 Transparency Report: https://www.shopify.com/security/transparency-report/report-.... It is still possible that their SOC2 and PCI reports could have a number of exceptions, but I would be surprised at this point in their maturity cycle.
Most small SaaS companies, the sort I have worked for, have literally zero controls for this sort of thing. I would expect more of a 100+ billion dollar company like Shopify, but frankly I am not surprised.
The "employee access to customer data isn't protected" sits as unsolved an opportunity canvas/brief in almost every SaaS company. You can get to a fair amount of controls with little to no code and only with process changes (aka SoC and ISO certifications), which is also what SaaS security teams spend quite a bit of time on. There are a fair amount of problems to be solved here.
Agree. And as much as policies are in place, it is not unusual to see csv exports downloaded to local laptops for analysis as part of work.
[deleted]
How was the event discovered?
Shopify, the company whose software runs the Fangamer store (and more than a million others online), has informed us that an internal security event it has been investigating since late last year included Fangamer customer data. Information regarding customer financial accounts and payment cards was not affected, but we are writing to make you aware of the situation.
According to Shopify, certain members of its support team used their Shopify credentials to obtain archived customer data from several hundred stores without authorization. The team members accessed data associated with order fulfillment — names, addresses, email addresses, cart contents, and phone numbers — but did not access or acquire any financial-account or payment-card information.
We are extremely frustrated and sorry to be sending you this email; Fangamer's internal development team takes data security extremely seriously. Data not in Fangamer's Shopify store — including Kickstarter backer information, account information and passwords, and email addresses used to sign up for our newsletter — was not accessed, and the store continues to operate as normal. Fangamer Japan, which operates as a separate store, was also not affected.
Shopify has terminated the employees who did this and eliminated the vulnerabilities that made it possible. Shopify has also reported that it will be providing any other relevant information to us as its investigation continues, and we'll pass along any new material details. If you have any questions, though, please contact us at [email protected].
Thank you, Fangamer