Hacker confirms access through infostealer infection [withdrawn](hudsonrock.com)
hudsonrock.com
Hacker confirms access through infostealer infection [withdrawn]
https://www.hudsonrock.com/blog/snowflake-massive-breach-access-through-infostealer-infection
234 comments
@dang is a no-op, consider contacting at the email in the footer.
But on the other hand, that they yanked the blog post is interesting and shouldn't be glossed over by just linking to the archive. What changed?
Edit: It looks like Snowflake's official response denies essentially all of the claims in TFA. Maybe Hudson Rock got taken in by a dishonest source and pulled the article when they realized their mistake?
https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
But on the other hand, that they yanked the blog post is interesting and shouldn't be glossed over by just linking to the archive. What changed?
Edit: It looks like Snowflake's official response denies essentially all of the claims in TFA. Maybe Hudson Rock got taken in by a dishonest source and pulled the article when they realized their mistake?
https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
I don’t think the source was dishonest. Also, Snowflake hasn’t really denied anything. That press release still reads the same, and still acknowledges that customers were impacted.
Also, both infosec groups and journalists were given sample data. On top of that, Live Nation explicitly said in their SEC filing that the breach happened through a third-party cloud provider, which is what Snowflake is.
I think that if Ticketmaster starts to send out breach emails to their customers next week, that will put any suspicions about this being fake to rest.
It’s difficult to imagine ShinyHunters getting duped as well, since they are the initial strongest link to this whole thing.
vx-underground says that Mandiant and Crowdstrike worked with Snowflake and tl;dr of it is that Snowflake wasn’t breached. That is also strange. They weren’t breached but some customers were affected? :jackie_chan_holding_his_head_pic:
Also, both infosec groups and journalists were given sample data. On top of that, Live Nation explicitly said in their SEC filing that the breach happened through a third-party cloud provider, which is what Snowflake is.
I think that if Ticketmaster starts to send out breach emails to their customers next week, that will put any suspicions about this being fake to rest.
It’s difficult to imagine ShinyHunters getting duped as well, since they are the initial strongest link to this whole thing.
vx-underground says that Mandiant and Crowdstrike worked with Snowflake and tl;dr of it is that Snowflake wasn’t breached. That is also strange. They weren’t breached but some customers were affected? :jackie_chan_holding_his_head_pic:
extremely likely scenario. Hudson tried to write a splashy hit piece and was met with reality.
Yes, that must be what happened. Maybe it was also traced back to the person based on previous posts or something like that, and they got cold fee.
Quietly pulling the piece without a retraction is pretty shady as well
Well, they doxed the alleged employee that they claimed was the source of the breach and likely got conned into what could have been a short-selling scam.
Life comes at you fast…
Life comes at you fast…
Yeah that's bad. I've put "[withdrawn]" in the title and taken out the name of the company for now, since it seems unfair to leave it in there. If the claims turn out to have been accurate, we can put it back.
Yeah, the piece itself had a tone that made me suspicious of the company, but this silent retraction makes me certain that they're untrustworthy.
Hi, Felipe at Snowflake here.
Here is the latest from Snowflake on this issue: https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
We'll keep updating that URL with any further news.
Here is the latest from Snowflake on this issue: https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
We'll keep updating that URL with any further news.
So, just to be clear because I found your link filled with a bit too much corporate speak:
1. The linked Hudson Rock post is explicitly claiming that the breaches at Ticketmaster and Santander Bank were caused by a Snowflake employee whose credentials were compromised.
2. This bullet point, "We did find evidence that similar to impacted customer accounts, the threat actor obtained personal credentials to and accessed a demo account owned by a former Snowflake employee. It did not contain sensitive data." (emphasis mine) says pretty clearly to me, then, that Snowflake believes the Hudson Rock account to be false.
So is that a correct understanding then?
1. The linked Hudson Rock post is explicitly claiming that the breaches at Ticketmaster and Santander Bank were caused by a Snowflake employee whose credentials were compromised.
2. This bullet point, "We did find evidence that similar to impacted customer accounts, the threat actor obtained personal credentials to and accessed a demo account owned by a former Snowflake employee. It did not contain sensitive data." (emphasis mine) says pretty clearly to me, then, that Snowflake believes the Hudson Rock account to be false.
So is that a correct understanding then?
The OP Hudson Rock writes something that I understand is saying: This was more than a breach of one customer's credentials, they got some employee creds and they weren't protected by 2 factor so they got into other customer accounts using that engineer's creds.
The snowflake writeup reads to me as if a customer's account creds got compromised - and it implied to me that was the end of it, no central or other account access on thoes creds. Nothing about this use of some employee account info that didn't have 2 factor auth on it.
1. I'm sure snowflake wants all access creds of any kind for their internal employees to use 2fa.
2. It used to be at least as a customer you could create a name/password without 2fa to log in to your own info there if you wanted to, like say as a customer you create a db or table and want to access it.
The snowflake writeup reads to me as if a customer's account creds got compromised - and it implied to me that was the end of it, no central or other account access on thoes creds. Nothing about this use of some employee account info that didn't have 2 factor auth on it.
1. I'm sure snowflake wants all access creds of any kind for their internal employees to use 2fa.
2. It used to be at least as a customer you could create a name/password without 2fa to log in to your own info there if you wanted to, like say as a customer you create a db or table and want to access it.
For 1. Those accounts would be set up by customers, so if they didn’t require 2FA, it didn’t happen.
Will there be any direct comment regarding the article here?
The linked post now contains unambiguous denials of the majority of claims in TFA, and TFA has been taken down.
the ad for protection services?
Yes, I know... I've seen that, also the posts on Reddit/HN and so on, but I am just curious if there is some truth to it.
snowflake has released a handful of statements and articles. giving attention to the ad would only be promoting it. unlikely to be allowed.
The salesforce.com servers are temporarily unable to respond to your request. We apologize for the inconvenience.
Thank you for your patience, and please try again in a few moments.
Sill down.
“…Visit http://trust.salesforce.com for current system status and availability.”
[deleted]
The screenshots of the chat logs are really something. This firm claims to be in communication with the actual criminal, and the actual criminal says that using their firm would have helped prevent the breach.
I have updated my sense of the firm's trustworthiness accordingly.
I have updated my sense of the firm's trustworthiness accordingly.
This is just pure speculation, but it kind of looks like the hacker was being ignored by Snowflake, so they somehow got in touch with Hudson Rock and offered them this promotional opportunity (to break the news, more than the throwaway line in the article) with the goal of retaliating against Snowflake for failing to pay the ransom. And Hudson Rock agreed to play along and hype up the story, presenting it as a bigger breach than it really was. One wonders whether Hudson Rock was the first they went to, or just the first to take them up on the offer.
It's also possible that the firm is being trolled by the "threat actor."
Are you trying to say that the threat actor is just going up to firms they're trying to extort and telling them lies? Criminals just going around lying to people? Don't they know that's against the law?
You joke, but these threat actors live and die by their reputation. Either they’re being honest, or this is a one-off or exit.
I mean, most people aren't criminals... what are the odds of someone being a DOUBLE criminal!?
That particular exchange is bizarre and cartoonish. I don’t know what to make of it.
“should have bought protection from Hudson Rock could have saved them this one”
“yes i agree it wouldve helped for sure”
“should have bought protection from Hudson Rock could have saved them this one”
“yes i agree it wouldve helped for sure”
It's a common euphemism in ransomware and protection rackets in general. One of my favourites is the message the akira group leaves in infected machines that goes something like:
Btw I looked at what they provide as evidence of data removal (2) and it's literally just the stdout of `rm -vrf data` lol. I mean, I get that it's impossible to provide evidence of absence, plus the victims have no leverage anyway, but I dig the theatrics.
Congratulations, you have passed a surprise information
security audit and become a victim of ransomware.
[...]
We offer:
1) full decryption assistance
2) evidence of data removal
3) security report on vulnerabilities we found
4) guarantees not to publish or sell your data
5) guarantees not to attack you in the future
Btw I looked at what they provide as evidence of data removal (2) and it's literally just the stdout of `rm -vrf data` lol. I mean, I get that it's impossible to provide evidence of absence, plus the victims have no leverage anyway, but I dig the theatrics.
The screenshots make this feel entirely fabricated or entirely marketing motivated.
If it's somehow real, omitting the hudsonrock message would be good sense.
I have mentally black listed this company.
If it's somehow real, omitting the hudsonrock message would be good sense.
I have mentally black listed this company.
Sounds like implied extortion to me.
they're also totally wrong about what they had access to . . .
in that you trust them less?
Absolutely the case for me. I don't give Snowflake much here, but Hudson Rock sells this exact type of "protection" and so far including BBC, no other independent verification?
This from the GP's link does it: “should have bought protection from Hudson Rock could have saved them this one”
This from the GP's link does it: “should have bought protection from Hudson Rock could have saved them this one”
We should thread carefully on this one.
It might be that they genuinely geeked out.
Hudson reputation would forever be scarred (badly) if they tried to manipulate the narrative.
Going down this hole also means we discredit the perpetrator, even if he did specifically reach out to Hudson.
Just wanted to say this so we don’t immediately jump to conclusions.
It might be that they genuinely geeked out.
Hudson reputation would forever be scarred (badly) if they tried to manipulate the narrative.
Going down this hole also means we discredit the perpetrator, even if he did specifically reach out to Hudson.
Just wanted to say this so we don’t immediately jump to conclusions.
It's the first time I heard from Hudson and they didn't start out great reputation wise for me
At least based on the wording of the perpetrator, Snowflake really did have the system designed in a way where a single administrator account gives you carte blanche to everything.
> On may 31st, Snowflake released a statement in which they claim that they are investigating an industry-wide identity-based attacks that have impacted “some” of their customers.
https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
This gives credibility to both Ticketmaster and Santander stories.
Wow! Could be one of the biggest dumps of all time if the threat actors did everything correctly?
> On may 31st, Snowflake released a statement in which they claim that they are investigating an industry-wide identity-based attacks that have impacted “some” of their customers.
https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
This gives credibility to both Ticketmaster and Santander stories.
Wow! Could be one of the biggest dumps of all time if the threat actors did everything correctly?
If the threat actor has played it right, there is a high possibility that this will be the largest data breach in history.
There's no evidence of that at all. The screenshot shows a few Snowflake professional services demo accounts only. These are accounts used by the sales engineer to demo features to customers.
It's possible the attacker was able to deduce some information about certain customers, but they would not then be able to connect to those accounts to extract data as those accounts should not be accessible from the public Internet at all, and should require corporate authentication.
It's possible the attacker was able to deduce some information about certain customers, but they would not then be able to connect to those accounts to extract data as those accounts should not be accessible from the public Internet at all, and should require corporate authentication.
[deleted]
So the account was without 2FA protection?
In my experience with Snowflake support about a year ago, an administrator of the customer's account had to explicitly grant access to Snowflake in order for the Snowflake team to see or do anything -- and if I recall correctly the access had an expiry.
That’s if they were following procedure. But on these internal systems, there are often hacks around the procedure that folks with the right mindset can easily find.
...All with just one successful malware-as-a-service attack against the right employee. (Lumma was the malware-as-a-service used.)
Interestingly, there's another adjacent story on the frontpage about Pegasus being used against NGOs in Eastern Europe. The principle of least authority is important, but also device security is important!
https://news.ycombinator.com/item?id=40535912
Interestingly, there's another adjacent story on the frontpage about Pegasus being used against NGOs in Eastern Europe. The principle of least authority is important, but also device security is important!
https://news.ycombinator.com/item?id=40535912
That's what the article implies, but I think it's overblown. They provide enough information (unfortunately) to identify the employee whose credentials were stolen, and she's a Sales Engineer. The data seems to have come from her own Snowflake account, which was used to build demos for customers or prospective customers. It's quite possible that those customers granted her access to some of their actual data, which was used in those demos, but it's a far cry from unfettered access to the customer's Snowflake database itself. It's also quite possible that the hacker exfiltrated fake-but-realistic data used for demo purposes and doesn't know the difference.
> They provide enough information (unfortunately) to identify the employee whose credentials were stolen, and she's a Sales Engineer.
I'm not previously familiar with Hudson Rock, nor how "standard" disclosures around this work, but identifying the breached employee felt like an extremely shitty move to me. If a single infected laptop of a sales engineer (i.e. not even an admin with extensive access rights) resulted in a breach this large, the root cause problem is not the sales engineer - and I'd note that Hudson Rock says as much in their article.
I'm not previously familiar with Hudson Rock, nor how "standard" disclosures around this work, but identifying the breached employee felt like an extremely shitty move to me. If a single infected laptop of a sales engineer (i.e. not even an admin with extensive access rights) resulted in a breach this large, the root cause problem is not the sales engineer - and I'd note that Hudson Rock says as much in their article.
But don't you see, we fired the problem so no more worries!
Oh, what's that. How did we change our hiring process to avoid hiring a problem again?
Sorry, my phones buzzing and I need to go.
--
Although obviously yes the problem isn't with hiring, it's with the system where a what should be fairly untrusted device shouldn't be able to exfiltrate a ton of data without setting a flag off somewhere.
Oh, what's that. How did we change our hiring process to avoid hiring a problem again?
Sorry, my phones buzzing and I need to go.
--
Although obviously yes the problem isn't with hiring, it's with the system where a what should be fairly untrusted device shouldn't be able to exfiltrate a ton of data without setting a flag off somewhere.
The problem isn't the employee or the hiring process. It's the security infrastructure! One compromised account, supposedly from sales, shouldn't bring down the whole company.
Exactly, how is an SE privileged enough to cause a problem? Or for the activities to go unnoticed?
Like I would be very humiliated to have a system under my care that had this problem.
Like I would be very humiliated to have a system under my care that had this problem.
By customer giving their account permission to access customer's dataset.
The prospective customer copies their data to Snowflake so Snowflake can demonstrate their awesomeness with the customer data.
This is the problem of Hudson Rock making conjecture or trying to be authoritative when tbey don't know the process.
One SE is working on many accounts. Snowflake SEs don't build within the client environment typically. They set up a demo account like you or I do with the $400 in credits. SEs are constantly starting these. Why? They expire after the fact. The SE builds in the created demo account and shows the client. After 30 days Snowflake locks the account (no credit card) and subsequently drops the demo instance and data.
For an SE to do the work the customer can do one or more of the following: The customer's SF instance shares data to that demo instance created by the SE AND/OR the customer has given access to that Snowflake SE through SSO.
Either way, this is more of orgs not being restrictive in their security posture. There is nothing novel about this exploit other than they found an SE who was working very hard and clients who had not properly scoped the security permissions of an employee/contractoe/guest.
One SE is working on many accounts. Snowflake SEs don't build within the client environment typically. They set up a demo account like you or I do with the $400 in credits. SEs are constantly starting these. Why? They expire after the fact. The SE builds in the created demo account and shows the client. After 30 days Snowflake locks the account (no credit card) and subsequently drops the demo instance and data.
For an SE to do the work the customer can do one or more of the following: The customer's SF instance shares data to that demo instance created by the SE AND/OR the customer has given access to that Snowflake SE through SSO.
Either way, this is more of orgs not being restrictive in their security posture. There is nothing novel about this exploit other than they found an SE who was working very hard and clients who had not properly scoped the security permissions of an employee/contractoe/guest.
yeah, the perpetrator is wrong: nothing in production was accessed, no customer data either. totally being mischaracterized in the hudson post.
I don't work for Snowflake but I spend a lot of time working with them and their SE organisation.
When working and building demos with clients, SEs create demonstration environments on the same $400 Snowflake demo accounts anyone can. To build demos the client would grant access to that SE. The SE would take some of the data to the demo environment and then work on it. This is further confirmed by the name of the environment Hudson Rock just published.
As far as I can tell, this is a process issue of clients not expiring an ID of someone who they were sharing data with and a threat actor swiping credentials. There is nothing novel about this as there is no exploit.
Also congrats Hudson Rock you just outed a person who was taken due to having malware on their computer. This is no different then if you gave a contractor credentials and they had those swiped. Dicks.
When working and building demos with clients, SEs create demonstration environments on the same $400 Snowflake demo accounts anyone can. To build demos the client would grant access to that SE. The SE would take some of the data to the demo environment and then work on it. This is further confirmed by the name of the environment Hudson Rock just published.
As far as I can tell, this is a process issue of clients not expiring an ID of someone who they were sharing data with and a threat actor swiping credentials. There is nothing novel about this as there is no exploit.
Also congrats Hudson Rock you just outed a person who was taken due to having malware on their computer. This is no different then if you gave a contractor credentials and they had those swiped. Dicks.
Just because there isn’t a “novel exploit” doesn’t mean this isn’t a big deal.
Snowflake is susceptible to their SE’s having credentials stolen. These credentials can bypass MFA. And per the article, they have no expiry. That’s strikes one, two, and three.
Snowflake’s security practices lead to a situation where their customers are either required, or at minimum encouraged, to share access to broad datasets with Snowflake employees. That’s strike four.
Yes, there is also issue here that the customers are responsible themselves for not granting too broad access, and that’s on them. But it’s also on Snowflake for not having a better system that doesn’t require this access, or at minimum not having better oversight and control over this transitive access.
Once these accounts are granted access to a customer’s data, they aren’t “demo accounts” anymore. They’re real accounts, with real, very valuable data, and they should be treated as such.
Edit to add: it is worth noting that Snowflake claims the demo account did not have access to customer data and wasn’t the source of the leak, which is in contradiction with what the attackers claim.
Snowflake is susceptible to their SE’s having credentials stolen. These credentials can bypass MFA. And per the article, they have no expiry. That’s strikes one, two, and three.
Snowflake’s security practices lead to a situation where their customers are either required, or at minimum encouraged, to share access to broad datasets with Snowflake employees. That’s strike four.
Yes, there is also issue here that the customers are responsible themselves for not granting too broad access, and that’s on them. But it’s also on Snowflake for not having a better system that doesn’t require this access, or at minimum not having better oversight and control over this transitive access.
Once these accounts are granted access to a customer’s data, they aren’t “demo accounts” anymore. They’re real accounts, with real, very valuable data, and they should be treated as such.
Edit to add: it is worth noting that Snowflake claims the demo account did not have access to customer data and wasn’t the source of the leak, which is in contradiction with what the attackers claim.
Indeed, the less novel the exploit the more embarassing it is.
Data stolen because of some crazy multi-exploit zero day chain. Well that is understandable, i don't blame the company.
Data stolen because no 2FA support? In 2024 that is just embarassing.
Data stolen because of some crazy multi-exploit zero day chain. Well that is understandable, i don't blame the company.
Data stolen because no 2FA support? In 2024 that is just embarassing.
Yes, that's also what I think must have happened—missing 2FA. But it seems it's also not mandatory within Snowflake accounts also, from what I understand from the general message.
I've never used Snowflake and assumed that because you push all your data into it, it probably has 2FA enabled by default. Is it optional?
I've never used Snowflake and assumed that because you push all your data into it, it probably has 2FA enabled by default. Is it optional?
I don't quite understand how Snowflake works.
My understanding was that you had to grant storage access (e.g. S3) and compute access (e.g. EC2) from your account to Snowflake, which would then use said resources to perform queries that you issue from their hosted web UI.
In that case it would mean stealing the Snowflake demo account of a SE should not expose your data unless you forgot to revoke their access to your underlying resources.
Can someone explain if that is how it works?
My understanding was that you had to grant storage access (e.g. S3) and compute access (e.g. EC2) from your account to Snowflake, which would then use said resources to perform queries that you issue from their hosted web UI.
In that case it would mean stealing the Snowflake demo account of a SE should not expose your data unless you forgot to revoke their access to your underlying resources.
Can someone explain if that is how it works?
No, Snowflake runs it's own storage and compute (on either AWS, GCP, or Azure depending on what you pick).
You can definitely bring your own storage (e.g. store your data in your own s3 buckets and integrate it with snowflake) using storage integrations and external tables.
See https://docs.snowflake.com/en/user-guide/data-load-s3-config...
Personally believe this is the right approach as the data resides in a location fully under the company's control. You could ditch snowflake and the data still resides in your s3 buckets for reuse with a another platform (just remove the iam permissions for snowflake).
See https://docs.snowflake.com/en/user-guide/data-load-s3-config...
Personally believe this is the right approach as the data resides in a location fully under the company's control. You could ditch snowflake and the data still resides in your s3 buckets for reuse with a another platform (just remove the iam permissions for snowflake).
Yes, federated queries (external tables) are supported but that is a lot slower than ingesting the data into Snowflake's storage and querying it. Since Snowflake's pricing model is based on computation time, querying external tables are usually more costly because of worse performance.
And the network ingress/egress costs are higher for cross account/region transfers.
So the customer data is actually stored on Snowflakes AWS accounts?
What difference does it make what underlying storage / provider it uses then?
Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?
What difference does it make what underlying storage / provider it uses then?
Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?
> So the customer data is actually stored on Snowflakes AWS accounts?
Yes.
> Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?
Yes. It's covered comprehensively in their docs, along with the caveats.
> What difference does it make what underlying storage / provider it uses then?
"Snowflake does not charge data ingress fees. However, a cloud storage provider might charge a data egress fee for transferring data from the provider to your Snowflake account."
unsaid: "...and you have to pay for that".
Note that when they say 'your Snowflake account' they mean our cloud account which we own, and which we run our workloads in, which we refer to as 'your' snowflake account.
Tangibly speaking, what means is that if you want to check up your billing; you go through snowflake; you can't login to a cloud console and see the actual charges the cloud vendor is charging.
> What difference does it make what underlying storage / provider it uses then?
They pass the specific underlying cloud vendor costs on to you (with, I guess, some markup, though you have no way of know what that is :)
Yes.
> Also does that mean every data query to snowflake goes out/in to/from internet at egress/Ingress costs?
Yes. It's covered comprehensively in their docs, along with the caveats.
> What difference does it make what underlying storage / provider it uses then?
"Snowflake does not charge data ingress fees. However, a cloud storage provider might charge a data egress fee for transferring data from the provider to your Snowflake account."
unsaid: "...and you have to pay for that".
Note that when they say 'your Snowflake account' they mean our cloud account which we own, and which we run our workloads in, which we refer to as 'your' snowflake account.
Tangibly speaking, what means is that if you want to check up your billing; you go through snowflake; you can't login to a cloud console and see the actual charges the cloud vendor is charging.
> What difference does it make what underlying storage / provider it uses then?
They pass the specific underlying cloud vendor costs on to you (with, I guess, some markup, though you have no way of know what that is :)
Ironically this model most resembles Teradata, which used to sell their own proprietary hardware/software combination at exorbitant rates.
Snowflake compute instance types cost about $.30-.40 an hour on EC2, so it's quite a markup.
As far as security I do believe they allow the customers to set their own storage keys, so there may be some isolation from a global breach.
Snowflake compute instance types cost about $.30-.40 an hour on EC2, so it's quite a markup.
As far as security I do believe they allow the customers to set their own storage keys, so there may be some isolation from a global breach.
Snowflake usually unloads data to an internal stage bucket in the same region as your snowflake account. If you use an s3 gateway endpoint getting that data is free of egress charges.
At the snowflake size you get custom price lists from cloud operators.
But I think there was also support for peering with client VPCs (or equivalents) which is why they support AWS, Azure, and GCP - you choose the location that is most fitting for linking with your cloud/physical workloads.
But I think there was also support for peering with client VPCs (or equivalents) which is why they support AWS, Azure, and GCP - you choose the location that is most fitting for linking with your cloud/physical workloads.
This was the case in ~2019 but are you sure this is still true? I think you can “bring your own account” with Snowflake, but I’m honestly not certain because their docs aren’t exactly clear about it…
I always wondered why snowflake doesn't just install a control plane on customers own cloud resources a la databricks. Seems like they'd be able to mitigate a lot of liability that way.
All storage/compute/networking etc. is handled snowflake side.
For various reasons, you're not getting to touch the actual DB bits.
You can, IIRC, use snowflake-hosted connectors to access external data though.
And there's a "data marketplace" of sorts where clients can publish/consume datasets.
For various reasons, you're not getting to touch the actual DB bits.
You can, IIRC, use snowflake-hosted connectors to access external data though.
And there's a "data marketplace" of sorts where clients can publish/consume datasets.
Databricks -does- work that way, iirc.
[deleted]
Having spent a lot of time in the Snowflake ecosystem, a few things to understand.
Snowflake is both a platform and a database. Customers can implement any level of security within the system. For example, when Snowflake introduced OAuth functionality, they put a lot of pressure for us to implement it in our tool. We're small enough and they are a significant partner so we priortised and got it into our platform so that a customer CAN implement it if they want too.
The keyword here is CAN. As a platform with database functionality, Snowflake allows customers to implement any level of security for access. So each instance of Snowflake https://XX123456.us-east-1.snowflakecomputing.com/ security is independent. So let's answer some questions
* "Was this a production breach", No, this was not a production breach. This is not when hackers went into the Azure and were able to elevate credentials and see all accounts.
* "But doesn't Snowflake use MFA", yes and on INTERNAL systems and the ability to connect to the PROD environment of SF it would absolutely have that. However, every customer has the ability to configure a user's credentials for access in their instance https://XX123456.us-east-1.snowflakecomputing.com/. Likely, the SE was granted access to the client's instance through the use of a user name and password.
* "But why not use MFA then Snowflake", this comes back to Okta. You have to create a trust relationship between providers. That's easy when you are only talking about your own employees. They all sit in your same directory. However, to set up a separate trust relationship for another domain, well now we have to get the security team of the client involved. All this so that someone can take a little bit of data and move it into another environment in order to do a demonstration. Not likely.
* "Well that just sounds stupid, my application has SSO enabled." True, but as also a database - Snowflake has to account for entire ecosystem that don't even support SSO-OAuth. For example, two major products that do not support SSO connections to Snowflake: Azure Machine Learning and Zapier. Both only support user name / password connections to Snowflake. So if you were an SE trying to show the value of Snowflake and its integration with AzureML; the ONLY way you could do that would be through user name / password functionality.
The issue in all of this is we are so used to thinking of SaaS as just an application. Databases have to support all sorts of application weirdness. Here is my gut thought on how this went down.
* The SE was working with multiple clients. These clients in an effort to save time and not wanting to have their security team involved and try and create a trust relationship with Snowflake went ahead and created a user name and password for the SE into the client environment vs setting up SSO. Again, this is ONLY controlled by the client.
* The SE at some point had active access to multiple instances of Snowflake to their various clients; with various levels of access depending upon the skill and fastidiousness of the client when they created the account for the Snowflake SE into their account. Again, how the SE is connecting to Snowflake is the same way any other user and application are doing so. I know this for a fact as SEs have to create their own throwaway demo instances https://XX123456.us-east-1.snowflakecomputing.com/. SEs don't have access into the backend of Snowflake.
* At some point this SE had malware installed on to their computer. At this point, this person was cooked and any customer who had given them access to the environment.
As for Hudson Rocks, they can go f themselves for doxing the SE. It was quite trivial to find who this person was and based on the Snowflake announcement they no longer have a job. That person was having an awful week already, now there name is out there. So again, F Hudson for doxing. It would have been easy to blind them out of the exchange (they did the hacker), but did not do so for the victim.
Snowflake is both a platform and a database. Customers can implement any level of security within the system. For example, when Snowflake introduced OAuth functionality, they put a lot of pressure for us to implement it in our tool. We're small enough and they are a significant partner so we priortised and got it into our platform so that a customer CAN implement it if they want too.
The keyword here is CAN. As a platform with database functionality, Snowflake allows customers to implement any level of security for access. So each instance of Snowflake https://XX123456.us-east-1.snowflakecomputing.com/ security is independent. So let's answer some questions
* "Was this a production breach", No, this was not a production breach. This is not when hackers went into the Azure and were able to elevate credentials and see all accounts.
* "But doesn't Snowflake use MFA", yes and on INTERNAL systems and the ability to connect to the PROD environment of SF it would absolutely have that. However, every customer has the ability to configure a user's credentials for access in their instance https://XX123456.us-east-1.snowflakecomputing.com/. Likely, the SE was granted access to the client's instance through the use of a user name and password.
* "But why not use MFA then Snowflake", this comes back to Okta. You have to create a trust relationship between providers. That's easy when you are only talking about your own employees. They all sit in your same directory. However, to set up a separate trust relationship for another domain, well now we have to get the security team of the client involved. All this so that someone can take a little bit of data and move it into another environment in order to do a demonstration. Not likely.
* "Well that just sounds stupid, my application has SSO enabled." True, but as also a database - Snowflake has to account for entire ecosystem that don't even support SSO-OAuth. For example, two major products that do not support SSO connections to Snowflake: Azure Machine Learning and Zapier. Both only support user name / password connections to Snowflake. So if you were an SE trying to show the value of Snowflake and its integration with AzureML; the ONLY way you could do that would be through user name / password functionality.
The issue in all of this is we are so used to thinking of SaaS as just an application. Databases have to support all sorts of application weirdness. Here is my gut thought on how this went down.
* The SE was working with multiple clients. These clients in an effort to save time and not wanting to have their security team involved and try and create a trust relationship with Snowflake went ahead and created a user name and password for the SE into the client environment vs setting up SSO. Again, this is ONLY controlled by the client.
* The SE at some point had active access to multiple instances of Snowflake to their various clients; with various levels of access depending upon the skill and fastidiousness of the client when they created the account for the Snowflake SE into their account. Again, how the SE is connecting to Snowflake is the same way any other user and application are doing so. I know this for a fact as SEs have to create their own throwaway demo instances https://XX123456.us-east-1.snowflakecomputing.com/. SEs don't have access into the backend of Snowflake.
* At some point this SE had malware installed on to their computer. At this point, this person was cooked and any customer who had given them access to the environment.
As for Hudson Rocks, they can go f themselves for doxing the SE. It was quite trivial to find who this person was and based on the Snowflake announcement they no longer have a job. That person was having an awful week already, now there name is out there. So again, F Hudson for doxing. It would have been easy to blind them out of the exchange (they did the hacker), but did not do so for the victim.
Ironically they pulled the article. Did they also dox the wrong person?
(I condemn doxing in any way)
Calling the SE a "victim" is debatable. If you work environment is infected with malware, you screwed up.
"Typically, Lumma has been distributed disguised as cracked or fake popular software like VLC or ChatGPT. Recently though, threat actors have also delivered the malware through emails containing payloads in the form of attachments or links impersonating well-known companies."
I would not be suprised if this was a recruiter-attack-vector.
Calling the SE a "victim" is debatable. If you work environment is infected with malware, you screwed up.
"Typically, Lumma has been distributed disguised as cracked or fake popular software like VLC or ChatGPT. Recently though, threat actors have also delivered the malware through emails containing payloads in the form of attachments or links impersonating well-known companies."
I would not be suprised if this was a recruiter-attack-vector.
This is the description of one of Hudson Rock's main products, "Bayonet".
"Imagine getting access to a lead-generation platform featuring hundreds of thousands of compromised companies around the world with active vulnerabilities that you can convert into customers."
I've seen and dealt with a couple of these types of companies. It's a pretty sleazy tactic, and it's low skill/effort from a technical point of view as well.
"Imagine getting access to a lead-generation platform featuring hundreds of thousands of compromised companies around the world with active vulnerabilities that you can convert into customers."
I've seen and dealt with a couple of these types of companies. It's a pretty sleazy tactic, and it's low skill/effort from a technical point of view as well.
Doing some more digging, this is where the data is sourced
"Hudson Rock acquires and purchases compromised data directly from top-tier threat actors operating in closed circle hacking groups. What sets our data apart is its quality in providing high accessibility to hacker groups looking for potential targets, and the speed in which we make it available to clients compared to other threat intelligence companies. Our operational knowhow, and our boots-on-the-ground approach to cybercrime originates from the IDF's 8200 Cybercrime division, and its efforts to thwart nation-state adversaries and professional threat actors."
https://cavalier.hudsonrock.com/docs
This would imply that they are financially engaging with and supporting cyber criminals.
"Hudson Rock acquires and purchases compromised data directly from top-tier threat actors operating in closed circle hacking groups. What sets our data apart is its quality in providing high accessibility to hacker groups looking for potential targets, and the speed in which we make it available to clients compared to other threat intelligence companies. Our operational knowhow, and our boots-on-the-ground approach to cybercrime originates from the IDF's 8200 Cybercrime division, and its efforts to thwart nation-state adversaries and professional threat actors."
https://cavalier.hudsonrock.com/docs
This would imply that they are financially engaging with and supporting cyber criminals.
About as ethical as those other 8200 alums, NSO. The ethics of the IDF on full display.
NSO wasn't founded by 8200 alumni, however companies such as the following were:
* Checkpoint
* Palo Alto Networks
* Waze
* Wiz
* Cybereason
Does your theory hold up? no, but why not generalize
* Checkpoint
* Palo Alto Networks
* Waze
* Wiz
* Cybereason
Does your theory hold up? no, but why not generalize
They've supposedly founded >1K companies, so easy to cherry-pick for any desired conclusion, but point taken.
Just a side effect of any apartheid regime, it is a deeply morally corrupted nation to the very core.
Generalizing over the whole nation? Hmm... Where have we seen this?..
In south africa?
Care to explain how a conflict between two national groups that are indigenous to the same land is similar to a racial system of segregation including beaches and restrooms, based on race theory with different racial classifications?
Or are you using a word that describes something different just because it evokes negative emotions?
Or are you using a word that describes something different just because it evokes negative emotions?
I see you are ok with what you are... Well, no harm trying - I highly recommend you reading this: https://mo-husseini.medium.com/50-completely-true-things-1ce...
That screenshot where the attacker agrees that buying their services would help makes me have a very strong suspicion this attack is either faked, sponsored by or in some way Hudson Rock is in cahoots with the attackers. That screenshot is a pure 100% advertisement manufactured by Hudson Rock one way or another.
As they usually say.. I'd check whoever is running Hudson Rock's hard drive.
As they usually say.. I'd check whoever is running Hudson Rock's hard drive.
The original HN link redirects to this company's home page now, so the blog post was likely an ad campaign to get traffic.
This is the point where the company's domain should get banned.
[deleted]
I just had a quick read through a couple more posts on HR, and a lot of them end with something along the lines of "heh, should have bought protection from us", reeks like a racket.
The whole blog post reeks of extreme self-congratulation and youre right, a total scum move to expose the victim. Altogether very weak performance from Hudson Rock.
[deleted]
It seems somehow like an Indiehacker page, probably taken too lightheartedly. Additionally, the related pages have no real contact details.
Snowflake’s customers are the victim, not Snowflake.
By victim I referred to the Snowflake employee that had their credentials taken
logrot(1)
Agreed, mentioning the login name of the compromised account seems really unprofessional and unnecessary.
This was a really effective anti-ad for Hudson Rock.
The entry seems written by someone lacking maturity.
The candor in the screencap'd chat conversation is novel, and will probably drive clicks.
But in its unedited form serves as dirty laundry, and including the language from the threat actor is both unnecessary and inappropriate.
I can't tell if the threat actor agreeing HR would have potentially helped avert this problem is a good endorsement or not.
On one hand testimony from a threat of a product's effectiveness would be good, but on the other, this is a little up close and personal of an endorsement from someone actively ransoming so many companies and putting so much data at risk.
The candor in the screencap'd chat conversation is novel, and will probably drive clicks.
But in its unedited form serves as dirty laundry, and including the language from the threat actor is both unnecessary and inappropriate.
I can't tell if the threat actor agreeing HR would have potentially helped avert this problem is a good endorsement or not.
On one hand testimony from a threat of a product's effectiveness would be good, but on the other, this is a little up close and personal of an endorsement from someone actively ransoming so many companies and putting so much data at risk.
You saw someone on the internet use a bad word and you felt offended?
> But in its unedited form serves as dirty laundry, and including the language from the threat actor is both unnecessary and inappropriate.
A threat actor has intent to hold a company ransom for $20 million and your first reaction is to feign offense that the word 'retarded' appeared in a chat log? These are not charm school graduates.
A threat actor has intent to hold a company ransom for $20 million and your first reaction is to feign offense that the word 'retarded' appeared in a chat log? These are not charm school graduates.
> The entry seems written by someone lacking maturity.
Agree with that.
Agree with that.
Generally, I like the page and the openness of the API behind it. It is much more common for people to talk about haveibeenpwned as a source for leaked credentials, but the site claims to have over 20 million computer entries from log stealers ... and every computer has XX password.. But yes probably this was written in a hurry to catch the wave?
From the official Snowflake response[0]:
>We believe this is the result of ongoing industry-wide, identity-based attacks with the intent to obtain customer data. Research indicates that these types of attacks are performed with our customers’ user credentials that were exposed through unrelated cyber threat activity. To date, we do not believe this activity is caused by any vulnerability, misconfiguration, or malicious activity within the Snowflake product.
[0]https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
>We believe this is the result of ongoing industry-wide, identity-based attacks with the intent to obtain customer data. Research indicates that these types of attacks are performed with our customers’ user credentials that were exposed through unrelated cyber threat activity. To date, we do not believe this activity is caused by any vulnerability, misconfiguration, or malicious activity within the Snowflake product.
[0]https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
They seem to have unpublished the article, but it’s still available on the Wayback machine: https://web.archive.org/web/20240531140540/https://www.hudso...
Not a good look. If they got something wrong, a retraction is appropriate after making these kinds of claims and accusations. Simply removing the post without explanation is irresponsible and unprofessional.
I'm sure the Snowflake legal department is ready to open a can of whoop ass on them if that's the case.
They’ve definitely gotten the repeatedly-named Snowflake employee in a lot of hot water.
May have been a cease desist from Snowflake and they don't want to take the heat alone?
This article is claiming that the Ticketmaster breach from a few days ago was actually a much broader hack affecting 400+ companies, all through a Snowflake employee's stolen credentials. This seems like a pretty big story that's only being reported on hudsonrock.com now.
I haven't heard of Hudson Rock before, does anyone know if they are a reputable source?
I haven't heard of Hudson Rock before, does anyone know if they are a reputable source?
BBC News report of a substantial hack of Santander bank; linked to Snowflake.
https://www.bbc.co.uk/news/articles/c6ppv06e3n8o
BBC just linked back to Hudson Rock's allegation FWIW, they don't have any independent confirmation
Great, so these companies do not give a flying fuck about their customer data in making sure the data stored at cloud storage companies are end to end encrypted.
To think these random cloud storage companies can access your bank information is utterly shocking.
To think these random cloud storage companies can access your bank information is utterly shocking.
> To think these random cloud storage companies can access your bank information is utterly shocking.
Honestly this sort of thing shouldn't be shocking at all.
Honestly this sort of thing shouldn't be shocking at all.
It’s been a while since I’ve been a Snowflake customer, but I do recall that Snowflake has a mode where the customer owns their own encryption key for their data. Snowflake employees (even admins with the highest access) have no access to the customer’s data unless the customer grants explicit access. It’d take a pretty serious breach on their compute notes to exfiltrate data.
https://docs.snowflake.com/en/user-guide/security-encryption...
https://docs.snowflake.com/en/user-guide/security-encryption...
Not surprised at all. Doesn't even depend on cloud vendors - I'm thinking back to the 2023 MOVEit vulnerability which resulted in the release of a ton of customer info from banks' own internal infrastructure.
Can a tool like Snowflake work if it doesn’t have access to the unencrypted data?
No. E2E encryption doesn't really apply here.
Snowflake employees need time to sell off all their shares. This news will hurt the SNOW stock price big.
> I haven't heard of Hudson Rock before, does anyone know if they are a reputable source?
I first learned of Hudson Rock after their "CEO" started spamming every security-related subreddit with low-effort blogspam over a period of months alleging numerous breaches. They've had several accounts banned, both by Reddit moderators and administrators.
Personally, I would no consider them a reputable or reliable source.
I first learned of Hudson Rock after their "CEO" started spamming every security-related subreddit with low-effort blogspam over a period of months alleging numerous breaches. They've had several accounts banned, both by Reddit moderators and administrators.
Personally, I would no consider them a reputable or reliable source.
[deleted]
Snowflake says it's not their fault, it's customer's fault apparently:
https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
If https://www.hudsonrock.com/blog/snowflake-massive-breach-acc... has not completely fabricated the story, Snowflake are tiny bit less than truthful.
> Research indicates that these types of attacks are performed with our customers’ user credentials that were exposed through unrelated cyber threat activity. To date, we do not believe this activity is caused by any vulnerability, misconfiguration, or malicious activity within the Snowflake product. Throughout the course of our ongoing investigation, we have promptly informed the limited number of customers who we believe may have been impacted.
They are way too wishy-washy:
* "Research indicates": Their own research or just general security research out there.
* "...do not believe this activity is caused by any vulnerability, misconfiguration, or malicious activity within the Snowflake product": I think they know exactly how it happened. They enumerate all possible scenarios and methods it didn't happen: misconfiguration, vulnerability, malicious activity within the product. But skillfully skip the explanation for how it actually happened.
They were contacted by the bad actor if hudsonrock is to be believed, so they probably had a good idea how it happened.
https://community.snowflake.com/s/question/0D5VI00000Emyl00A...
If https://www.hudsonrock.com/blog/snowflake-massive-breach-acc... has not completely fabricated the story, Snowflake are tiny bit less than truthful.
> Research indicates that these types of attacks are performed with our customers’ user credentials that were exposed through unrelated cyber threat activity. To date, we do not believe this activity is caused by any vulnerability, misconfiguration, or malicious activity within the Snowflake product. Throughout the course of our ongoing investigation, we have promptly informed the limited number of customers who we believe may have been impacted.
They are way too wishy-washy:
* "Research indicates": Their own research or just general security research out there.
* "...do not believe this activity is caused by any vulnerability, misconfiguration, or malicious activity within the Snowflake product": I think they know exactly how it happened. They enumerate all possible scenarios and methods it didn't happen: misconfiguration, vulnerability, malicious activity within the product. But skillfully skip the explanation for how it actually happened.
They were contacted by the bad actor if hudsonrock is to be believed, so they probably had a good idea how it happened.
Or they had no idea and assumed that customers had a breach, so blamed them without doing a proper in-depth analysis.
The hacker alleges that they weren't even expiring refresh tokens, that is pretty huge if true, it's just a massive, glaring issue.
The hacker alleges that they weren't even expiring refresh tokens, that is pretty huge if true, it's just a massive, glaring issue.
I think it's unlikely:
* If Hudson Rock is to be believed, with the chat screenshot, Snowflake was notified and asked for a ransom. It would seem odd that all their 400 customers were all hacked randomly at the same time, by only one hacker group just based on those customers' own bad credentials
* They enumerate all the possible ways they were not hacked, and seems to skips the exact one way they were hacked. It's like saying: "It wasn't A, C, D, E, or F" as the causes. Hmm, they skipped B it seems, I wonder why...
* If Hudson Rock is to be believed, with the chat screenshot, Snowflake was notified and asked for a ransom. It would seem odd that all their 400 customers were all hacked randomly at the same time, by only one hacker group just based on those customers' own bad credentials
* They enumerate all the possible ways they were not hacked, and seems to skips the exact one way they were hacked. It's like saying: "It wasn't A, C, D, E, or F" as the causes. Hmm, they skipped B it seems, I wonder why...
I think the following is also totally possible:
1. The attacker quoted in the Hudson Rock article did breach the sales engineer's account as described.
2. The data in those accounts, however, was just demo data (Snowflake unambiguously says the compromised employee account did not have sensitive data) and it seems possible the attacker is overstating the impact of their specific breach.
3. I've seen no evidence elsewhere that "400 customers" of Snowflake were breached. So it at least seems plausible that just Ticketmaster and Santander had their accounts breached because their own employee creds were stolen and that gave access to their Snowflake data.
I definitely agree the Snowflake announcement had too much corporate speak but from my plain reading of it they are explicitly denying that their employee's stolen creds resulted in a breach of real PII.
1. The attacker quoted in the Hudson Rock article did breach the sales engineer's account as described.
2. The data in those accounts, however, was just demo data (Snowflake unambiguously says the compromised employee account did not have sensitive data) and it seems possible the attacker is overstating the impact of their specific breach.
3. I've seen no evidence elsewhere that "400 customers" of Snowflake were breached. So it at least seems plausible that just Ticketmaster and Santander had their accounts breached because their own employee creds were stolen and that gave access to their Snowflake data.
I definitely agree the Snowflake announcement had too much corporate speak but from my plain reading of it they are explicitly denying that their employee's stolen creds resulted in a breach of real PII.
Don’t they hint at how it happened in what you quoted?
> performed with our customers’ user credentials that were exposed through unrelated cyber threat activity
Unless they are just making this up, they seem to think that credentials were obtained elsewhere. They also say at the end that they told customers to review their account settings. So they are directing the blame away from themselves.
You’re right that this seems to conflict with Hudson Rock. Unfortunately our sources are the company that allegedly was hacked and a company that shamelessly doxed an employee in the course of using this event to promote their product. I think we’ll need to wait for more details.
> performed with our customers’ user credentials that were exposed through unrelated cyber threat activity
Unless they are just making this up, they seem to think that credentials were obtained elsewhere. They also say at the end that they told customers to review their account settings. So they are directing the blame away from themselves.
You’re right that this seems to conflict with Hudson Rock. Unfortunately our sources are the company that allegedly was hacked and a company that shamelessly doxed an employee in the course of using this event to promote their product. I think we’ll need to wait for more details.
> You’re right that this seems to conflict with Hudson Rock. Unfortunately our sources are the company that allegedly was hacked and a company that shamelessly doxed an employee in the course of using this event to promote their product. I think we’ll need to wait for more details.
Fair point. Doxing the employee is shitty, no doubt. Makes Hudson Rock look bad as well. But at the same time, I was giving them some credibility as it would seems if they completely fabricated the screenshots, they might as well file for bankruptcy, given the market they seem to operate in.
Fair point. Doxing the employee is shitty, no doubt. Makes Hudson Rock look bad as well. But at the same time, I was giving them some credibility as it would seems if they completely fabricated the screenshots, they might as well file for bankruptcy, given the market they seem to operate in.
The article doesn't seem very consistent with the headline of "hundreds of breached customers"
1. The password for lift/okta is only allowing access to a servicenow portal and not customer accounts, so the refresh token issue seems restricted to the servicenow portal and unrelated to any actual customer data being exposed from customer Snowflake accounts
2. The screenshot with 10 corporate accounts compromised shows 4 different Snowflake account credentials (one of which appears to be a personal demo account) so that might explain up to 3 customers being compromised but there's no details showing other customers being compromised.
Assuming all of the SE's credentials were compromised for all of the customers they were working with, we can probably say the total customers compromised would be in the low double digits (each customer account would have had to provision access to the SE individually)
Big leap to say that literally the entirety of Snowflake's customer base is compromised from a "refresh token issue" (in the internal Okta portal) that isn't even linked to any customer Snowflake account
1. The password for lift/okta is only allowing access to a servicenow portal and not customer accounts, so the refresh token issue seems restricted to the servicenow portal and unrelated to any actual customer data being exposed from customer Snowflake accounts
2. The screenshot with 10 corporate accounts compromised shows 4 different Snowflake account credentials (one of which appears to be a personal demo account) so that might explain up to 3 customers being compromised but there's no details showing other customers being compromised.
Assuming all of the SE's credentials were compromised for all of the customers they were working with, we can probably say the total customers compromised would be in the low double digits (each customer account would have had to provision access to the SE individually)
Big leap to say that literally the entirety of Snowflake's customer base is compromised from a "refresh token issue" (in the internal Okta portal) that isn't even linked to any customer Snowflake account
Very possible there were creds etc accessible in Servicenow that could have been used to move laterally from there. Conjecture, obviously.
Without knowing exactly how the compromised account is set up, and what access is granted, it may be difficult to say. At "security focused large telecom" I am aware of, you would be surprised what level of tech has access to what (though of course all access is logged).
> Okta
Just how many pwns involve this keyword? Between Okta itself being pwned and it allowing stupid shit like ignoring expiry, I am strongly inclined to believe that rolling your own login system might be the strongest security posture these days. Last I heard, Okta doesn't use any form of FIDO internally: how completely and utterly worthless.
Just how many pwns involve this keyword? Between Okta itself being pwned and it allowing stupid shit like ignoring expiry, I am strongly inclined to believe that rolling your own login system might be the strongest security posture these days. Last I heard, Okta doesn't use any form of FIDO internally: how completely and utterly worthless.
Okta is a single point of failure but its still probably way better then what the average developer will throw together under deadline.
Why in the world would obtaining a Snowflake employee’s credentials allow you to then obtain Snowflake’s customers’ data? Doesn’t this imply that people working at Snowflake can see all of the data that I put in it?
Admittedly I don’t have much experience with Snowflake, but as a baseline I expect better from a “cloud storage giant”.
Admittedly I don’t have much experience with Snowflake, but as a baseline I expect better from a “cloud storage giant”.
There’s something missing here.
From what the Hudson Rock article shows, they were able to use an SE’s creds to access their demo account. This is not a customer account and shouldn’t (but of course could) contain sensitive info. It’s not clear to me how this snowballed into a larger breach.
Perhaps customers had granted this SE access to their accounts and the data within. Or perhaps there’s a deeper hack. But this isn’t clear to me from what I’ve read.
From what the Hudson Rock article shows, they were able to use an SE’s creds to access their demo account. This is not a customer account and shouldn’t (but of course could) contain sensitive info. It’s not clear to me how this snowballed into a larger breach.
Perhaps customers had granted this SE access to their accounts and the data within. Or perhaps there’s a deeper hack. But this isn’t clear to me from what I’ve read.
I was just going to post the same thing. The files that they show in the screenshots are things like PROGRESSIVE_BID_CHANGE_202405271129.csv. Looks suspiciously like the Snowflake Sales Engineer's data for their job role closing a deal with Progressive, not Progressive's own data. And there's no reason to think that a SE would have broad access to customer data. There may be some overlap, but I doubt it contains sensitive customer data owned by Progressive.
You’re thinking “bid” is in reference to Snowflake bidding for progressive as a client?
I’d say thats not likely, I work in fintech and the first thing this filename indicates to me is a CSV feed of market data for bid prices (https://en.m.wikipedia.org/wiki/Bid_price)
This is a common type of dataset a firm would dump into a datalake to use as reference data lookups against other more sensitive data (for pricing trades, etc.)
I’d say thats not likely, I work in fintech and the first thing this filename indicates to me is a CSV feed of market data for bid prices (https://en.m.wikipedia.org/wiki/Bid_price)
This is a common type of dataset a firm would dump into a datalake to use as reference data lookups against other more sensitive data (for pricing trades, etc.)
Something similar happened at a previous employer. Contractor was hired to do a big data PoC, and they managed to cajole access to a prod data dump for a more impactful demo.
They then managed to load all this PII data into an ElasticSearch instance that was open to the internet and was discovered by threat actors.
I wouldn’t be surprised to find that something similar happened here, where an unscrubbed prod dataset was shared for a better demo.
They then managed to load all this PII data into an ElasticSearch instance that was open to the internet and was discovered by threat actors.
I wouldn’t be surprised to find that something similar happened here, where an unscrubbed prod dataset was shared for a better demo.
There may have been administrative access that was not properly secured.
No, that sounds about right. This is a new, agile, cloud-first company that grew very quickly and has faced significant turnover. You don't get such growth by doing everything right.
Looking at linked-in, the unlucky employee could be someone in a sales role, with only 7 months of tenure. Every company has a few sysadmins with a scary amount of reach, but that's not what happened here.
Edit: A ServiceNow access request flow with poor internal controls would explain it.
Looking at linked-in, the unlucky employee could be someone in a sales role, with only 7 months of tenure. Every company has a few sysadmins with a scary amount of reach, but that's not what happened here.
Edit: A ServiceNow access request flow with poor internal controls would explain it.
>> This is a new, agile, cloud-first company that grew very quickly and has faced significant turnover.
This is not really true of Snowflake, which is not some 2-person YOLO startup, and it's also pretty irrelevant as the weakest link is often a single employee regardless of the size or industry of the company. In my experience the support and security is way better than average - example: as a client of both Snowflake and Sisense, Snowflake reached out to me about the Sisense breach before Sisense did.
This is not really true of Snowflake, which is not some 2-person YOLO startup, and it's also pretty irrelevant as the weakest link is often a single employee regardless of the size or industry of the company. In my experience the support and security is way better than average - example: as a client of both Snowflake and Sisense, Snowflake reached out to me about the Sisense breach before Sisense did.
Its support and security posture could very well be better than average.
Looking a other breaches (Qlik Attunity, Microsoft AAD, ...) indicates that being better then average is not enough if you're a sufficiently attractive target.
It's not a new tiny company. It's about 12 years old with 7000 employees. They know they are dead if they are not hot on security, so at the moment I would take this story with a big pinch of salt. Quite possible certain customer configurations have been attacked, but that is a different thing.
(new) sales person with an uber account that has access to carte blanche customer data. This is not only a disaster, if true, but also violates probably every certification under the sun, if they had any at all. Reminder Snowflake is a couple of sales persons from Oracle and a techie.
I'm not sure it does, perhaps it violates the spirit but not the letter.
You need a way to give your employees access to customer data; for support cases. So you build a "request access" form in your ITSM. Now you can tick off every box related to certification: There is a process. Only authorized persons have access. Every aspect of it can be audited.
Later, perhaps sales people (the 1000's of new joiners) start using it as well for lead generation. It's a lot easier to sell if you know how your product is used by other companies in the same industry.
Much later, someone's account is compromised, makes the same requests and it gets waved through. Why wouldn't it ? It is a valid request made by a current employee of the company. What other criteria would apply ? This is not a bank.
You need a way to give your employees access to customer data; for support cases. So you build a "request access" form in your ITSM. Now you can tick off every box related to certification: There is a process. Only authorized persons have access. Every aspect of it can be audited.
Later, perhaps sales people (the 1000's of new joiners) start using it as well for lead generation. It's a lot easier to sell if you know how your product is used by other companies in the same industry.
Much later, someone's account is compromised, makes the same requests and it gets waved through. Why wouldn't it ? It is a valid request made by a current employee of the company. What other criteria would apply ? This is not a bank.
Aside all things stated that are wrong from security perspective - how about limit the qty and rate any such support account has access to? Breaching an account shouldn't give you access to dump everything out the gate. Even if that is the case, where are other measures alerting there's a stream of egress going on? This sounds like systemic issue which most certs are all about.
> What other criteria would apply?
Many companies have processes that require 2 or more humans in the loop for sensitive prod data.
Many companies have processes that require 2 or more humans in the loop for sensitive prod data.
You're lucky if it's only 2 and the approval process takes less than months.
[deleted]
Was their intent to dox the employee while discussing this beach? They show the employee’s username, which is easily Googleable.
But hey, they blot out the name of the alleged attacker in chat, because of privacy...
And IP
the username is probably the only proof they have that they were talking to an actual member of the hacker group.
Snowflake internal staff do not have access to read customer data, unless a customer grants it. Customers can use their own KMS to generate table keys.
Snowflake has a lot of security features. But still, customers may well misconfigure their own Snowflake accounts and therefore be vulnerable.
A well configured Snowflake account:
- does not allow any access from the public Internet. Network policies set by the customer should restrict access to corporate networks only. - does not allow authentication unless with MFA or via corporate IDP / SAML - has dynamic masking / tokenisation
Snowflake seems to have most of the Fortune 500 as customers. If Snowflake itself was somehow penetrated and all controls circumvented, it would certainly be huge and you'd be reading about a lot more than Santander and Ticketmaster.
At this point it seems more like the "AWS Hack" that affected CapOne back in the day (that was CapOne's fault, not AWS!).
Snowflake has a lot of security features. But still, customers may well misconfigure their own Snowflake accounts and therefore be vulnerable.
A well configured Snowflake account:
- does not allow any access from the public Internet. Network policies set by the customer should restrict access to corporate networks only. - does not allow authentication unless with MFA or via corporate IDP / SAML - has dynamic masking / tokenisation
Snowflake seems to have most of the Fortune 500 as customers. If Snowflake itself was somehow penetrated and all controls circumvented, it would certainly be huge and you'd be reading about a lot more than Santander and Ticketmaster.
At this point it seems more like the "AWS Hack" that affected CapOne back in the day (that was CapOne's fault, not AWS!).
"Snowflake internal staff do not have access to read customer data"
Do engineers in Snowflake have access to production systems?
Do engineers in Snowflake have access to production systems?
They don't have customer key access and can't assume customer identity but ultimately yes, via a multi-eye approval process there is access to the prod infra - but this is extremely tightly secured, and not something a phishing attack on a single sales engineer could ever achieve.
Many enterprise customers additionally use standard third party crypto libraries to tokenise and/or encrypt sensitive fields before storage in any warehouse/database such as Snowflake or Redshift.
This is a similar principle to using client-side encryption for S3. The infra provider (AWS in that case) can never read the data.
Many enterprise customers additionally use standard third party crypto libraries to tokenise and/or encrypt sensitive fields before storage in any warehouse/database such as Snowflake or Redshift.
This is a similar principle to using client-side encryption for S3. The infra provider (AWS in that case) can never read the data.
> Snowflake internal staff do not have access to read customer data
By default, no. But it is standard operating procedure for sales engineers to request and be given access to customer data so they can build demos.
By default, no. But it is standard operating procedure for sales engineers to request and be given access to customer data so they can build demos.
It is not standard operating procedure, and demos wouldn't be done with production data. In fact, most enterprises would have contracts in place with Snowflake that explictly state that Snowflake staff can not be granted access to their Snowflake accounts (this is actually the default in the Snowflake enterprise professional services contract now).
You can understand it: Snowflake lawyers are naturally reluctant to have their staff be granted access to any customer's accounts.
It is however quite common to have Snowflake PS guys have limited access to customer dev environments.
However, such access, even in dev, should always be network restricted.
You can understand it: Snowflake lawyers are naturally reluctant to have their staff be granted access to any customer's accounts.
It is however quite common to have Snowflake PS guys have limited access to customer dev environments.
However, such access, even in dev, should always be network restricted.
Wow that's what i call a anti-ad for Hudson-Rock, should have bought services from a public relation firm...yes i agree, it would have helped for sure.
Are theirs HQ located in Hollywood?
Are theirs HQ located in Hollywood?
Hudson Rock exposing the name of the employee whose credentials were stolen shows me that Hudson Rock is not a serious company. I hope Snowflake is supporting her.
Snowflake’s blog about this incident mentions “former employee”. If she was actually fired as a result of this, I would hope that the entire security organization also went with her.
"To understand how the hack was carried out, the threat actor explains that they were able to sign into a Snowflake employee’s ServiceNow account using stolen credentials, thus bypassing OKTA which is located on lift.snowflake.com."
I'm unfamiliar with ServiceNow and their website seems to be just marketing bable.
Can someone explain the role of ServiceNow in this scenario and why it important?
I'm unfamiliar with ServiceNow and their website seems to be just marketing bable.
Can someone explain the role of ServiceNow in this scenario and why it important?
ServiceNow is a system used for many ticketing, CMDB, etc. tasks, including automation.
That said, the quoted sentence boils down "the cookie we stole authorized us to service now instance bypassing SSO prompt".
That said, the quoted sentence boils down "the cookie we stole authorized us to service now instance bypassing SSO prompt".
I don't know anything about Snowflake or enterprise sales, but I'm wondering how that sales process works. A sales representative sets up a demo account. Then what, they load it with real customer data, with the customer's cooperation?
It seems like if a potential customer screws this up then they are wrong, but also, sales might be wrong to encourage them to share their data insecurely, and a sales process that encourages insecure sharing of enterprise data by potential customers might be bad company policy.
Perhaps doing it right would be tricky when it requires educating the customer. And it might get in the way of sales.
It seems like if a potential customer screws this up then they are wrong, but also, sales might be wrong to encourage them to share their data insecurely, and a sales process that encourages insecure sharing of enterprise data by potential customers might be bad company policy.
Perhaps doing it right would be tricky when it requires educating the customer. And it might get in the way of sales.
>A sales representative sets up a demo account. Then what, they load it with real customer data, with the customer's cooperation?
Better than Fujitsu who actually went into UK Subpostmasters accounts and 'adjusted' their accounts live. No links - there are too many, but I would direct you to the various testimonies of the CEO of the PO, paula vennells (no capital letters for her) over the last....15 years.
Better than Fujitsu who actually went into UK Subpostmasters accounts and 'adjusted' their accounts live. No links - there are too many, but I would direct you to the various testimonies of the CEO of the PO, paula vennells (no capital letters for her) over the last....15 years.
The practice of how Snowflake team uses customer data and builds/helps them and their access is not expired is still on Snowflake; such access should be temporary in nature, tied to an active ticket upon its closure to auto expire. The fact that Snowflake has managed to attain (and/or keep?) their FedRamp High certification despite this, is a separate story for some investigative journalists (not just for the label but also to look into what public sector data might be at risk or already compromised...)
This is my first ever HN post, and I actually got distracted/took timeout from studing for a Snowflake cert (what are the odds; first ever story I saw about Snowflake on HN).
So, I was reading the comments and then in my mind I said to myself, this sounds like something connected with a Pegasus type company. The very next line in the comment I was reading was: "our boots-on-the-ground approach to cybercrime originates from the IDF's 8200 Cybercrime division"!
As others have noted, doxing the SE seems unnecessary...unless, that was part of the threat/proposal to Snowflake. You can imagine companies that had worked with that SE being concerned/need reassuring they're not affected.
I wouldn't at all be surprised if someone had bet against Snowflake stock before this story broke, if the story was hyped up enough or it was bad enough to spook the market.
Snowflake "strongly recommends" using 2FA for the admin account role, but users are free to decide whether to use it or not. Snowflake's website states: "MFA is enabled on a per-user basis; however, at this time, users are not automatically enrolled in MFA. To use MFA, users must enrol themselves." - I assume a future update will change that so they can enable it by default. MFA related questions always appear in lower-level Snowflake certs.
I'd assume (as a consultant) It would be against company policy to use a username/password for client work. Sometimes that's one of the first bad practices we see working with new clients. Perhaps that's why the SE is an ex-employee.
So, I was reading the comments and then in my mind I said to myself, this sounds like something connected with a Pegasus type company. The very next line in the comment I was reading was: "our boots-on-the-ground approach to cybercrime originates from the IDF's 8200 Cybercrime division"!
As others have noted, doxing the SE seems unnecessary...unless, that was part of the threat/proposal to Snowflake. You can imagine companies that had worked with that SE being concerned/need reassuring they're not affected.
I wouldn't at all be surprised if someone had bet against Snowflake stock before this story broke, if the story was hyped up enough or it was bad enough to spook the market.
Snowflake "strongly recommends" using 2FA for the admin account role, but users are free to decide whether to use it or not. Snowflake's website states: "MFA is enabled on a per-user basis; however, at this time, users are not automatically enrolled in MFA. To use MFA, users must enrol themselves." - I assume a future update will change that so they can enable it by default. MFA related questions always appear in lower-level Snowflake certs.
I'd assume (as a consultant) It would be against company policy to use a username/password for client work. Sometimes that's one of the first bad practices we see working with new clients. Perhaps that's why the SE is an ex-employee.
>should have bought protection from Hudson Rock
Damn that is so not classy. Who approved that?
Damn that is so not classy. Who approved that?
From their marketing page:
> Snowflake’s single platform eliminates data silos
I guess so, especially now.
> Snowflake’s single platform eliminates data silos
I guess so, especially now.
> The data from these companies was put up for sale on the Russian-speaking cybercrime forum
Just russia being russia, as usual.
Just russia being russia, as usual.
The screenshot shows the post in English. The website domain is Indian. The seller contact xmpp.cn in China. But no, we'll keep blaming Russia for everything.
I agree. Every single security thing that happens is put on Russia.
Typical propaganda.
If we were pre-Ukraine, it’d be China getting all the blame (like it used to be).
I’m not pro-Russia, I’m just anti-bullshit.
Typical propaganda.
If we were pre-Ukraine, it’d be China getting all the blame (like it used to be).
I’m not pro-Russia, I’m just anti-bullshit.
There’s a reason a lot of malware won’t install on a user’s computer if it detects they’re using a Russian keyboard.
>I agree. Every single security thing that happens is put on Russia.
This is transparently false. Both Russia and China have copped blame for various recent attacks. And the fact of the matter is that Russian hackers are extremely active at the moment, and were even before 2022.
See for example: https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
>If we were pre-Ukraine, it’d be China getting all the blame (like it used to be).
Russia was behind the Solarwinds hack, which is the most widely covered one in the past decade. Ironically the Chinese were also independently exploiting Solarwinds to break into government agencies, but that didn't get much attention.
This is transparently false. Both Russia and China have copped blame for various recent attacks. And the fact of the matter is that Russian hackers are extremely active at the moment, and were even before 2022.
See for example: https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
>If we were pre-Ukraine, it’d be China getting all the blame (like it used to be).
Russia was behind the Solarwinds hack, which is the most widely covered one in the past decade. Ironically the Chinese were also independently exploiting Solarwinds to break into government agencies, but that didn't get much attention.
When they don’t know who did something, they say Russia.
I suppose it makes sense to say that I live in the UK. Depending on where you are in the world, you may be hearing blame on Russia passed a lot less.
But in the UK, it is constant. It is tiresome.
Regarding how active Russian hackers are, I bet they are not any more active than GCHQ, NSA etc. they are all at it. God, even Israel will sell their Pegasus to any dictator who will pay them the money.
I suppose it makes sense to say that I live in the UK. Depending on where you are in the world, you may be hearing blame on Russia passed a lot less.
But in the UK, it is constant. It is tiresome.
Regarding how active Russian hackers are, I bet they are not any more active than GCHQ, NSA etc. they are all at it. God, even Israel will sell their Pegasus to any dictator who will pay them the money.
Have we already forgotten about all the cybercrime originating from Ukraine?
People in ex-USSR also speak/write Russian, so while I have no doubt Russians frequent the forum, lots of others from their sphere of influence also visit that forum. Don't assume everyone on HN is a 5-eyes resident.
Your daily evidence that modern Russia is essentially just an organized crime ring with oil reserves and nukes.
Yes, any country that is not your ally or vassal is an "organized crime ring". It's safer to be with oil and nukes, than without them you know.
I don't think naming things that apply to USA also, is quite the win you think it is.
That's just "government"
In response to this incident, I just put together a Python CLI to help you monitor and take action against suspicious sessions in your account.
https://github.com/Titan-Systems/titan-security-tools
https://github.com/Titan-Systems/titan-security-tools
Where are the blocked ip’s from? Also rapeflake? Hesitant to google that
Snowflake published these rules earlier today - https://community.snowflake.com/s/article/Communication-ID-0...
Ahh thanks!
[deleted]
It's unclear if it's customer metadata or real customer data data?
Santander claims:
"Following an investigation, we have now confirmed that certain information relating to customers of Santander Chile, Spain and Uruguay, as well as all current and some former Santander employees of the group had been accessed. Customer data in all other Santander markets and businesses are not affected."
https://www.santander.com/en/stories/statement
"Following an investigation, we have now confirmed that certain information relating to customers of Santander Chile, Spain and Uruguay, as well as all current and some former Santander employees of the group had been accessed. Customer data in all other Santander markets and businesses are not affected."
https://www.santander.com/en/stories/statement
But this does not mention Snowflake so it's on 100% clear it's the same root cause. And if it is the root cause being discussed elsewhere here namely that a customer engineer with partial data for demos... could that really have had access to this level of data?
> were able to sign into a Snowflake employee’s ServiceNow account using stolen credentials, thus bypassing OKTA
It's probably not technically relevant to the breach, but it's at least interesting that the CEO of Snowflake is the former CEO of ServiceNow: https://en.wikipedia.org/wiki/Frank_Slootman
It's probably not technically relevant to the breach, but it's at least interesting that the CEO of Snowflake is the former CEO of ServiceNow: https://en.wikipedia.org/wiki/Frank_Slootman
[deleted]
I'm more than disappointed in Hudson Rock. They lead with confirmation of a breach, but detail no more than allegations and swagger. The decision to not redact the login is not just a moral failure, but it shows that they do not understand the subject matter that they purport to be experts in. Shameful.
This. Thank you.
[deleted]
Protecting customer data from compromised insiders can be pretty hard. They often need the access to do their jobs. Still, in this case it's was far too easy - just one session cookie and a password shouldn't itself by sufficient to compromise all your customers.
It sounds like they found a way to bypass MFA on snowflake (because snowflake didn’t expire session cookies), and stole an employees credential, obtained via a “Lumma-type Infostealer” which I guess is just a key logger in browser extensions and fake versions of software…
something doesn't add up, because I don't see how this extrapolates from stealing privileged Snowflake employee credentials. How does that become a keylogger on a client's computer?
Yeah it is a bit muddled honestly. I had to read it a couple times and I still don’t completely get what happened:
1. Employee installs a key logger
2. Snowflake does not expire session cookies
3. Malware steals their session cookie and password, so can bypass employee MFA/okta
4. ???
5. Somehow this one employee has admin access to 4000 snowflake instances
1. Employee installs a key logger
2. Snowflake does not expire session cookies
3. Malware steals their session cookie and password, so can bypass employee MFA/okta
4. ???
5. Somehow this one employee has admin access to 4000 snowflake instances
Step 4 is right in the article:
"they were able to sign into a Snowflake employee’s ServiceNow account using stolen credentials, thus bypassing OKTA which is located on lift.snowflake.com.
Following the infiltration, the threat actor claims that they were able to generate session tokens, which enabled them to exfiltrate massive amounts of data from the company"
"they were able to sign into a Snowflake employee’s ServiceNow account using stolen credentials, thus bypassing OKTA which is located on lift.snowflake.com.
Following the infiltration, the threat actor claims that they were able to generate session tokens, which enabled them to exfiltrate massive amounts of data from the company"
Yes, but how should ServiceNow create session tokens if it is not part of the SSO system? I don't know enough about ServiceNow, but I think every large company has some products that are not part of their-SSO system. So that makes sense, but I am not sure about the next step.
I think they mean regenerating servicenow's own tokens/cookies, without hitting okta. so SN's session would still be valid.
I'm not surprised. A few months ago I was reverse engineering their login flow (I was writing a driver for Elixir & Rust), and was getting MANY stack traces and weird bugs happening with their authentication flow, especially around OAuth.
I'll bet there's a few of those reputedly 400 companies keeping an eye on the news of this and another eye on their stick price.
If data gets out, especially financial data being reported to investors, that spells curtains for a company generally.
If data gets out, especially financial data being reported to investors, that spells curtains for a company generally.
The article now seems to have been deleted and redirects back to their homepage.
The query that Snowflake provides to identify privileged users is wrong.
I wrote a better version here: https://gist.github.com/titan-teej/924dcb42604a98b90d6419262...
TL;DR they don't check for users who can transitively gain admin access.
I wrote a better version here: https://gist.github.com/titan-teej/924dcb42604a98b90d6419262...
TL;DR they don't check for users who can transitively gain admin access.
It's surprising that SnowFlake didn't pay the $20M ransom. Seems like a no-brainer compared to the reputational damage this would cause.
I honestly don't understand why someone would pay a ransom. They just hacked you. Do you think they'd actually follow through with keeping quiet?
I've read Genestealer and thought, what?
> To put it bluntly, a single credential resulted in the exfiltration of potentially hundreds of companies that stored their data using Snowflake, with the threat actor himself suggesting 400 companies are impacted
You only have to "fail" once, as they say.-
You only have to "fail" once, as they say.-
Template for writing GDPR subject access requests [0] for whoever it may concern. It is a bit too harsh, but it may be useful for one to know what info may have been compromised.
https://www.linkedin.com/pulse/nightmare-letter-subject-acce...
https://www.linkedin.com/pulse/nightmare-letter-subject-acce...
[deleted]
Imagine having your enterprise data warehouse stolen.
[deleted]
How did Snowflake not have 2FA?
https://web.archive.org/web/20240531140540/https://www.hudso...