Okta: “We made a mistake” delaying the Lapsus$ hack disclosure(bleepingcomputer.com)
bleepingcomputer.com
Okta: “We made a mistake” delaying the Lapsus$ hack disclosure
https://www.bleepingcomputer.com/news/security/okta-we-made-a-mistake-delaying-the-lapsus-hack-disclosure/
66 comments
I wonder if their radius agent still includes all the dev and test time libraries. It use to, or may still include jars related to Maven and JUnit. Not confidence inspiring.
BTW, nvidia is going open source driver or Lapsus did release the code of the hardware and software?
That's not how copyright works. It would be illegal to distribute anything proprietary that Lapsus$ released, and it would never make it's way into any mainstream distributions let alone code hosting providers.
Lapsus threatened to release all of their Nvidia data if Nvidia didn't open source their drivers. Gp is asking if either outcome happened (though I don't think they did).
I'm not sure what this means: https://www.collabora.com/news-and-blog/blog/2022/03/23/how-...
> We're not actually writing one here but it'll make the examples easier if we pretend we are. Just for the sake of example, I'm going to pick on NVIDIA because... Why not? Such a driver is clearly missing and really should happen soon. (Hint! Hint!) We're going to call this hypothetical new Vulkan driver NVK. It's short and obvious.
> We're not actually writing one here but it'll make the examples easier if we pretend we are. Just for the sake of example, I'm going to pick on NVIDIA because... Why not? Such a driver is clearly missing and really should happen soon. (Hint! Hint!) We're going to call this hypothetical new Vulkan driver NVK. It's short and obvious.
I’d be much more understanding if
1. They didn’t try to downplay the leak
2. Didn’t double down on the leak not being a big deal later
3. Didn’t specialize in authentication and security
1. They didn’t try to downplay the leak
2. Didn’t double down on the leak not being a big deal later
3. Didn’t specialize in authentication and security
Exactly. Okta has given absolutely zero indication that they will notify us next time they have a security event, either.
We pay them exorbitant costs so that they don't make these mistakes. How dare they outsource my security when I'm paying them a premium not to.
I am a very unhappy customer who is very interested in Keycloak.
I am a very unhappy customer who is very interested in Keycloak.
I'm not sure Keycloak is a viable alternative for most businesses. Security software as a whole tends to be _extremely_ difficult to run securely and at scale.
Honestly, most of these companies would be better off using Google, Azure or AWS' SSO-as-a-Service product (if that's what you're hoping to get out of Keycloak).
That's not to say that I don't appreciate that there's an open-source alternative out there, however.
Honestly, most of these companies would be better off using Google, Azure or AWS' SSO-as-a-Service product (if that's what you're hoping to get out of Keycloak).
That's not to say that I don't appreciate that there's an open-source alternative out there, however.
I have a feeling Cloudflare is going to be a new entrant into this space in the next 6-12 months.
I’ve got a similar feeling and I’m witnessing it through their Zero Trust product. All the rails for SSO/SAML are coming together.
Interesting enough is it looks like it will be provider agnostic.
You could use the “raw” saml endpoint provided by the service, a Google Identity endpoint, Okta provided saml endpoint, shibboleth on-prep protected by Tunnels, jumpcloud etc.
There’s even an saml/SSO preview of what data will be sent to the application upon authZ by the Identity Provider. There’s configuration rules already in place (AuthN) that can be applied to Organizational Units based upon the user’s metadata.
It’s a pretty clear bet at this point that Cloudflare will be making an entrance. Considering they used Okta internally performing a rapid investigation of the breach, (1) is the right thing to do as a service provider/rails to the internet (2) is strong product marketing for their future product (3) can be used to gain internal support for replacing Okta with their own product
Interesting enough is it looks like it will be provider agnostic.
You could use the “raw” saml endpoint provided by the service, a Google Identity endpoint, Okta provided saml endpoint, shibboleth on-prep protected by Tunnels, jumpcloud etc.
There’s even an saml/SSO preview of what data will be sent to the application upon authZ by the Identity Provider. There’s configuration rules already in place (AuthN) that can be applied to Organizational Units based upon the user’s metadata.
It’s a pretty clear bet at this point that Cloudflare will be making an entrance. Considering they used Okta internally performing a rapid investigation of the breach, (1) is the right thing to do as a service provider/rails to the internet (2) is strong product marketing for their future product (3) can be used to gain internal support for replacing Okta with their own product
One can only hope.
For those too lazy to click through, this comment is the CTO of cloudflare asking what features people would like in response to a comment: "Matthew Prince has a lot of tweets today about how he might have to begrudgingly enter the IAM space given how disappointed he is, how serious are you guys about this? "
> Honestly, most of these companies would be better off using Google, Azure or AWS' SSO-as-a-Service product (if that's what you're hoping to get out of Keycloak).
The thing is, your Keycloak instance is not going to matter to any hacker, particularly if it's inside a VPN and not reachable from the Internet - and while we're at it, fuck zero-trust because it is essentially the same level of stupidity as using Okta, you're once again putting all your eggs into the basket of whatever provider you choose.
Your SSO-as-a-service provider however? They're the juiciest target out there that is. Everyone from secret services over enemy nation states to your average cyber-criminal is looking to get access there. And as we've seen, all it takes is a couple teenagers and a couple thousand dollars.
Good network design costs a lot of money to set up, particularly to limit the scope of an attack (e.g. because the VPN software had a vulnerability), but it's orders of magnitude better in the long run than to outsource core IT to some incompetent fools with subcontractors.
The thing is, your Keycloak instance is not going to matter to any hacker, particularly if it's inside a VPN and not reachable from the Internet - and while we're at it, fuck zero-trust because it is essentially the same level of stupidity as using Okta, you're once again putting all your eggs into the basket of whatever provider you choose.
Your SSO-as-a-service provider however? They're the juiciest target out there that is. Everyone from secret services over enemy nation states to your average cyber-criminal is looking to get access there. And as we've seen, all it takes is a couple teenagers and a couple thousand dollars.
Good network design costs a lot of money to set up, particularly to limit the scope of an attack (e.g. because the VPN software had a vulnerability), but it's orders of magnitude better in the long run than to outsource core IT to some incompetent fools with subcontractors.
> The thing is, your Keycloak instance is not going to matter to any hacker, particularly if it's inside a VPN and not reachable from the Internet.
This doesn't make it particularly usable as SSO...
>Good network design costs a lot of money to set up, particularly to limit the scope of an attack (e.g. because the VPN software had a vulnerability), but it's orders of magnitude better in the long run than to outsource core IT to some incompetent fools with subcontractors.
This is exactly my point. Most businesses not not have the resources to maintain this level of infrastructure.
Additionally, I'm personally of the opinion that walled gardens with VPN entry points are a particularly good choice for modern businesses these days. Even the White House OMB is pushing the beyondcorp model in their recent recommendations for ZT.
This doesn't make it particularly usable as SSO...
>Good network design costs a lot of money to set up, particularly to limit the scope of an attack (e.g. because the VPN software had a vulnerability), but it's orders of magnitude better in the long run than to outsource core IT to some incompetent fools with subcontractors.
This is exactly my point. Most businesses not not have the resources to maintain this level of infrastructure.
Additionally, I'm personally of the opinion that walled gardens with VPN entry points are a particularly good choice for modern businesses these days. Even the White House OMB is pushing the beyondcorp model in their recent recommendations for ZT.
> This doesn't make it particularly usable as SSO...
Why? Your core IT should not be visible from outside a VPN anyway, and if you're in a VPN you can use your Keycloak or whatever SAML system as you wish.
> Most businesses not not have the resources to maintain this level of infrastructure.
And right here is the problem: too many businesses see IT simply as a cost center instead of as what it is: a vital part of the business. You can't even run a grocery store without computers any more, and even a grocery store is a juicy target for criminals given that credit card data is processed there (not to mention employment records that can be used for identity impersonation).
People simply go and attach whatever bullshit devices from HVAC controllers to crappy 10$ IoT surveillance cameras fresh off of Alibaba on their core network and in some cases even "convenience wifi for customers", and then they wonder why either hackers or the feds come knocking. Jesus.
Why? Your core IT should not be visible from outside a VPN anyway, and if you're in a VPN you can use your Keycloak or whatever SAML system as you wish.
> Most businesses not not have the resources to maintain this level of infrastructure.
And right here is the problem: too many businesses see IT simply as a cost center instead of as what it is: a vital part of the business. You can't even run a grocery store without computers any more, and even a grocery store is a juicy target for criminals given that credit card data is processed there (not to mention employment records that can be used for identity impersonation).
People simply go and attach whatever bullshit devices from HVAC controllers to crappy 10$ IoT surveillance cameras fresh off of Alibaba on their core network and in some cases even "convenience wifi for customers", and then they wonder why either hackers or the feds come knocking. Jesus.
There is Amazon KeyCloak, might meets someones needs https://www.amazonaws.cn/en/solutions/keycloak-on-aws/
> We pay them exorbitant costs so that they don't make these mistakes.
The problem with this approach is that, repeated by hundreds of companies at scale, paints such a massive target on the vendor, that a breach becomes fundamentally inevitable.
I expect this sort of thing happens to all big auth providers, from AWS to Google; it's just dealt with more efficiently and discreetly.
The problem with this approach is that, repeated by hundreds of companies at scale, paints such a massive target on the vendor, that a breach becomes fundamentally inevitable.
I expect this sort of thing happens to all big auth providers, from AWS to Google; it's just dealt with more efficiently and discreetly.
[deleted]
That "mistake" will cost them under GDPR and from lost customer trust.
But if they are still a customer, they won't give a damn about their losing trust. As long as the company doesn't lose the $$ coming from them being a customer then that's all that matters
Indeed.
As I said in another comment thread about this event, I’ve been watching my leadership team’s response to this as we have deep Okta integration, therefore so do several of our customers using our platform.
Said response has been pretty interesting and amounted to: “it’s working isn’t it?” when shown this event and the disclosures. Both executive and functional Security leadership teams practically shrugged.
Very glad I’m not the one making that call.
As I said in another comment thread about this event, I’ve been watching my leadership team’s response to this as we have deep Okta integration, therefore so do several of our customers using our platform.
Said response has been pretty interesting and amounted to: “it’s working isn’t it?” when shown this event and the disclosures. Both executive and functional Security leadership teams practically shrugged.
Very glad I’m not the one making that call.
Market forces at the most basic.
> "We made a mistake"
You could even say, they had a lapse of judgment. Sorry for the pun.
You could even say, they had a lapse of judgment. Sorry for the pun.
Is it really fair to call it a “delayed disclosure” if the people who disclosed it were the sixteen year old chavs who popped your billion dollar security business? Are we supposed to believe that Okta would have disclosed this in due time anyway, regardless of whether the teenagers who broke into their system posted screenshots of it on Twitter?
They blatantly lied. A third-party auth provider, where trust is absolutely crucial, and they lied about being breached. Repeatedly.
And even now, they are not coming out and being honest like "we lied because we were scared about liability, but the person behethis decision has been fired". It's not good enough.
Sorry, but they have some serious work to do if they want to regain that trust. I for one, will not be using their services again.
And even now, they are not coming out and being honest like "we lied because we were scared about liability, but the person behethis decision has been fired". It's not good enough.
Sorry, but they have some serious work to do if they want to regain that trust. I for one, will not be using their services again.
Once again, the attempted cover up is almost always worst than the original issue. Okta could have come out and leaned into the process issues they found, how they are improving, etc... and probably walked away with increased trust. Instead, they have strung out a bad situation and look worse every day.
Looks like it is worse than that they lied. If they genuinely believed that it’s just a password reset attempt and relied on their subcontractor to investigate the incident, their security is shit.
I would posit that it is more likely than not that ALL companies lie about security and intrusions. This one will pass on into the night and is unlikely to affect the company, because security is not something most people want to think about. Passwords will not be changed, protocols will not be updated, and it will happen again.
I think this take is a little bit naive given that a lot of the customers using Okta are themselves regulated by things like healthcare records legislation, financial services regulations, GDPR, and similar. It isn't an option to not use effective security or to claim that it's somebody else's responsibility when you were holding data that has been exfiltrated illegitimately
It's been frustrating dealing with their constant double speak and contradictions in their blog posts. The worst example was them saying that it's not a breach and then showing how they've been breached by the very definition of the term.
The only rational reason I can come up with for this duplicitous talk is for the shareholders; with this double speak they can reduce just how much their shares fall. I'm not on a high horse, I'm positive that most of our orgs was do the same thing.
And I'm not surprised that they're just like all other large companies that choose money/PR over accuracy. What is maddening is that they made it so hard to zero in on required information to help us assess risk; they're at the center of a lot of security workflows, and accurate information is more critical from them than it is from others.
I fear for Auth0 and what it ~~may~~ will become under Okta's 'culture'.
The only rational reason I can come up with for this duplicitous talk is for the shareholders; with this double speak they can reduce just how much their shares fall. I'm not on a high horse, I'm positive that most of our orgs was do the same thing.
And I'm not surprised that they're just like all other large companies that choose money/PR over accuracy. What is maddening is that they made it so hard to zero in on required information to help us assess risk; they're at the center of a lot of security workflows, and accurate information is more critical from them than it is from others.
I fear for Auth0 and what it ~~may~~ will become under Okta's 'culture'.
"And I'm not surprised that they're just like all other large companies that choose money/PR over accuracy."
Whenever you hear "Public Relations" just replace it with "Propaganda" since that is what is always was.
https://blog.apaonline.org/2020/07/06/recently-published-boo...
"How Propaganda Became Public Relations: Foucault and the Corporate Government of the Public is a philosophical investigation into the transformative effect propaganda has on us as individuals, on us as publics, and on society more broadly. I argue that propaganda does far worse than just lie to us: modern propaganda aims to transform us into the kind of subjects who carry out a particular line of conduct freely and as a key part of our identity. "
Every company engages in propaganda. The fact that you believe Okta from the start was because of their propaganda.
Whenever you hear "Public Relations" just replace it with "Propaganda" since that is what is always was.
https://blog.apaonline.org/2020/07/06/recently-published-boo...
"How Propaganda Became Public Relations: Foucault and the Corporate Government of the Public is a philosophical investigation into the transformative effect propaganda has on us as individuals, on us as publics, and on society more broadly. I argue that propaganda does far worse than just lie to us: modern propaganda aims to transform us into the kind of subjects who carry out a particular line of conduct freely and as a key part of our identity. "
Every company engages in propaganda. The fact that you believe Okta from the start was because of their propaganda.
Also worried about Auth0. I was about to use it for a project when the acquisition happened and I’ve held off specifically because all of the negatives I’ve heard about Okta.
It's hard to justify Auth0 when you can use cognito or AFDS often for free as an addon to things you already need. Or something like firebase (or supabase or appwrite etc). I would trust any of those solutions more than Okta, which is pretty old school.
This has moved up my schedule for porting from Auth0 to Cognito. But, honestly, I'm still putting it off. From the outside, Cognito looks like one of those AWS systems that's just going to be a nightmare to learn and use.
Hiya. You should check out FusionAuth (disclosure: I'm an employee). We have a comparable offering to Auth0.
From talking to other users, Cognito is okay as long as you don't need any real customization. It's also great if you need to get AWS credentials on login. But there are plenty of folks who start with Cognito, run into its limitations, and move elsewhere (even if they stay with AWS for other service). Here's a story about someone who moved from Cognito to FusionAuth: https://fusionauth.io/blog/2020/11/18/reconinfosec-fusionaut...
From talking to other users, Cognito is okay as long as you don't need any real customization. It's also great if you need to get AWS credentials on login. But there are plenty of folks who start with Cognito, run into its limitations, and move elsewhere (even if they stay with AWS for other service). Here's a story about someone who moved from Cognito to FusionAuth: https://fusionauth.io/blog/2020/11/18/reconinfosec-fusionaut...
It's crazy to me that David Bradbury is still shoving Sitel under that bus instead of claiming responsibility for his failure. Is he hoping for a golden parachute here or is the organization so rotten that they cannot admit fault?
Sitel deserves to be shoved under a bus. I worked at Sitel, and it’s a complete shit show. Believe everything you hear.
If Sitel is so bad, I wonder why Bradbury approved their use? Maybe he just feels responsible but is not willing to admit it.
In my humble opinion, cost. Sitel is a “lowest bidder” churn machine. It opens big call centers in low employment areas, and just churns and burns.
Cubicle farms, unprofessional managers constrained by ops managers who play favorites, and fiefdoms.
Why? There are only two positions at Sitel: On the phone, or off, and no one who has ever gotten off the phone wants to ever go back, so their relations between Ops and Team Managers are always strained.
In a word, at Sitel crap rolls down hill and collects there until it drowns you.
Cubicle farms, unprofessional managers constrained by ops managers who play favorites, and fiefdoms.
Why? There are only two positions at Sitel: On the phone, or off, and no one who has ever gotten off the phone wants to ever go back, so their relations between Ops and Team Managers are always strained.
In a word, at Sitel crap rolls down hill and collects there until it drowns you.
Hopefully the union drive at Amazon has spillover effects across the IT sector. Worker protections are badly needed.
No more unions. Co-ops > unions. Unions (at least American ones) are adversarial in an unproductive way. E.g. fighting automated subway trains.
Why not both?
There are so many red flags here, it's not even funny anymore.
* a third party has barely restricted, deep access to all customer data
* the "SuperUser" app can apparently have logged in users idling around in a VM, waiting for someone to come along and use it without any automatic logout and re-authentication
* a single account accessing 300+ customers in a few days doesn't trigger any alerts
* they detect a compromise, and do absolutely nothing about it for months, except letting the third party order a security audit; they patiently wait for a report; they don't even audit the access logs
* only a screenshot posted online triggers an audit of access logs and a public response
* they still try to blame the third party and the security firm for their own (basically outrageous) inactivity
All of this by a company entrusted with the most critical gatekeeping functionality of systems, used by many large enterprises and expected to have top notch security.
* a third party has barely restricted, deep access to all customer data
* the "SuperUser" app can apparently have logged in users idling around in a VM, waiting for someone to come along and use it without any automatic logout and re-authentication
* a single account accessing 300+ customers in a few days doesn't trigger any alerts
* they detect a compromise, and do absolutely nothing about it for months, except letting the third party order a security audit; they patiently wait for a report; they don't even audit the access logs
* only a screenshot posted online triggers an audit of access logs and a public response
* they still try to blame the third party and the security firm for their own (basically outrageous) inactivity
All of this by a company entrusted with the most critical gatekeeping functionality of systems, used by many large enterprises and expected to have top notch security.
There are operational red flags but not the ones you mentioned. What a lot of people forget in these situations is that these are support agents and they need access to customer data to do their job. Additionally, I bet accessing 300 accounts per week is a totally normal thing for a support agent to do. Sure you could write some alarms that trigger in certain cases but it’s very hard to do without creating alarm fatigue.
The issue here is that 1) an employee was able to be phished and/or have malware installed on their device and 2) support agents should only have access to accounts where they have been assigned a ticket, and agents should have no control over which tickets they get assigned.
The issue here is that 1) an employee was able to be phished and/or have malware installed on their device and 2) support agents should only have access to accounts where they have been assigned a ticket, and agents should have no control over which tickets they get assigned.
> these are support agents and they need access to customer data to do their job.
In these cases, the support agents should not have the ability to open support tickets or modify the companies on them - and then you can give them superuser access to companies with currently open support tickets (preferably those they are assigned only).
In these cases, the support agents should not have the ability to open support tickets or modify the companies on them - and then you can give them superuser access to companies with currently open support tickets (preferably those they are assigned only).
Yup. That's point #2 in my post :)
> deep access to all customer data
"All" is not correct, they outlined what data was in the Superuser utility. It certainly has access, but not "all" data.
> the "SuperUser" app can apparently have logged in users idling around in a VM, waiting for someone to come along and use it without any automatic logout and re-authentication
Well it was an employee laptop, not a VM (so the news says). But regardless of this, the question is what is the session length of superuser? It could be 20mn, it could be 1hr, it could be 8hr. There is some balance for security and usability of the team who actually uses that tool. Even if the session length is reasonably short, this was an insider willingly giving up access to their machine. So likely the insider logged in and sent the attackers a message saying "here you go" and let them work for as long as the session would last. Heck, the insider may even repeatedly login for them.
> a single account accessing 300+ customers in a few days doesn't trigger any alerts
Almost every support interaction would likely require the access of that customer in the superuser utility. Just a dozen cases an hour for an 8 hour day, is nearly 100 accounts accessed per day. That doesn't even seem like a lot of cases per hour.
I'm not saying Okta is in the right here. But throwing around a lot of FUD doesn't help the situation.
"All" is not correct, they outlined what data was in the Superuser utility. It certainly has access, but not "all" data.
> the "SuperUser" app can apparently have logged in users idling around in a VM, waiting for someone to come along and use it without any automatic logout and re-authentication
Well it was an employee laptop, not a VM (so the news says). But regardless of this, the question is what is the session length of superuser? It could be 20mn, it could be 1hr, it could be 8hr. There is some balance for security and usability of the team who actually uses that tool. Even if the session length is reasonably short, this was an insider willingly giving up access to their machine. So likely the insider logged in and sent the attackers a message saying "here you go" and let them work for as long as the session would last. Heck, the insider may even repeatedly login for them.
> a single account accessing 300+ customers in a few days doesn't trigger any alerts
Almost every support interaction would likely require the access of that customer in the superuser utility. Just a dozen cases an hour for an 8 hour day, is nearly 100 accounts accessed per day. That doesn't even seem like a lot of cases per hour.
I'm not saying Okta is in the right here. But throwing around a lot of FUD doesn't help the situation.
Definitely some flags and horrible communication, but will any customers actually leave? Changing an IdM provider can easily run into 7-figures for large organizations.
I could definitely see some bigger companies moving away after this. Lots of them hauled ass after the log4j chaos, for example.
And lot will stay, because one tenet of corporate security is to outsource whatever possible and make it someone else's problem, shifting the liability out of the house. Check box ticked.
Not always. It is difficult to hire people with a deep knowledge of complex systems such as the Windows suite and it makes sense, security wise to protect this insured of having chaos in your configurations.
An MS environment is today extraordinary complex and it is very easy to make a mistake setting it up, maintaining and integrating other stuff with it.
An MS environment is today extraordinary complex and it is very easy to make a mistake setting it up, maintaining and integrating other stuff with it.
Most probably nobody wanted to be the messanger of the bad news, perhaps due to the fear of being made a scapegoat?
Their entire business is managing authentication and they don't use hardware tokens for all employees. Welcome to clown town.
At this company, a VP eng/CTO type person asked me how I would handle a situation where my teammate was underperforming. I am pretty sure that I failed this interview section and the whole interview loop because the company value of transparency meant that I should have said I would publicly humiliate the person instead of beginning by speaking to them privately.
Not surprised. This "VP eng/CTO" is somewhat notorious I guess. I had an interview with them and everyone kept warning me about his part of the interview loop which I thought was odd. He apparently prides himself in being the most difficult part of their interview loop. He gave me some impromptu coding exercise that he appeared to be making up as he went along. I thought it was very bizarre that the VP of Engineering/CTO was giving coding tests and improvised ones at that. I had already completed 4 rounds of coding tests at that point and he was the last stop on the loop.
At least they're true to their value of transparency in showing to candidates that they're an awful place to work
What of the rumors about AWS keys being leaked through slack channels that the support account had access to?
That was plastered all over the HN and reddit threads a few days ago, no mention of it from Okta yet. Did that turn out to be bogus?
That was plastered all over the HN and reddit threads a few days ago, no mention of it from Okta yet. Did that turn out to be bogus?
I missed that, might you or someone else have a link? I'm not seeing it.
It was claimed in the $lapsus Telegram chat by $lapsus themselves: https://twitter.com/_MG_/status/1506345542837628932/photo/1
From the screenshot:
> I don't think storing AWS keys within Slack would comply to any of these standards?
From the screenshot:
> I don't think storing AWS keys within Slack would comply to any of these standards?
Is there a reason a Okta needs to contract with a third party provider for support? I mean their entire business is security. Is this just penny-pinching or is there a legitimate reason that a publicly traded company can't provide their own support staff?
This whole thing is like an exercise in plausible deniability, corporate double-speak, blame shifting and arrogance.
This whole thing is like an exercise in plausible deniability, corporate double-speak, blame shifting and arrogance.