Okta’s Investigation of the January 2022 Compromise(okta.com)
okta.com
Okta’s Investigation of the January 2022 Compromise
https://www.okta.com/blog/2022/03/oktas-investigation-of-the-january-2022-compromise/
123 comments
> And then go on to write paragraphs of detail and a timeline that explicitly shows for a five day period an unauthorized user had full super user access to the service.
it sounds like they built the support tool (with its unfortunate name) such that it places limited trust in support contractors and the information technology that supports them. because of this they're able to identify potentially affected customers and even audit all activities that have taken place. in other words, the system worked as designed.
this is very different (and orders of magnitude less severe) than an actual compromise of the production service itself.
it sounds like they built the support tool (with its unfortunate name) such that it places limited trust in support contractors and the information technology that supports them. because of this they're able to identify potentially affected customers and even audit all activities that have taken place. in other words, the system worked as designed.
this is very different (and orders of magnitude less severe) than an actual compromise of the production service itself.
A person who should not have had access to the system gained near full admin access for five whole days. That is the textbook definition of a breach of security.
It doesn't matter if they did it by fooling or paying a low level CS rep to get access to their account vs. using their 'leet hacking skillz' to pwn the electronic defenses. A breach is a breach and the CSO of all people has to own up to that fact.
It doesn't matter if they did it by fooling or paying a low level CS rep to get access to their account vs. using their 'leet hacking skillz' to pwn the electronic defenses. A breach is a breach and the CSO of all people has to own up to that fact.
> A person who should not have had access to the system gained near full admin access for five whole days.
"This is an application built with least privilege in mind to ensure that support engineers are granted only the specific access they require to perform their roles. They are unable to create or delete users. They cannot download customer databases."
not being able to create or delete users seems a far cry from "near full admin access."
"This is an application built with least privilege in mind to ensure that support engineers are granted only the specific access they require to perform their roles. They are unable to create or delete users. They cannot download customer databases."
not being able to create or delete users seems a far cry from "near full admin access."
Why? We only know that they are not able to create new users.
But at least changing passwords of existing accounts seemed possible according to the screenshots, perhaps disabling 2FA as well.
As long as Okta does not provide a list of what the user was able to do, we don't know
But at least changing passwords of existing accounts seemed possible according to the screenshots, perhaps disabling 2FA as well.
As long as Okta does not provide a list of what the user was able to do, we don't know
At my own workplace (a SaaS service), we cannot change passwords, we can only send reset links (and the reset email goes to the end user).
I can't see any place in the screenshots where they are setting a user's password, only sending reset links (which would do nothing if the support user does not have access to system email or end users email).
Also at my workplace, 2FA is not enforced by an IdP (like Okta), but by our own application (and therefore could not be disabled at the IdP level).
I can't see any place in the screenshots where they are setting a user's password, only sending reset links (which would do nothing if the support user does not have access to system email or end users email).
Also at my workplace, 2FA is not enforced by an IdP (like Okta), but by our own application (and therefore could not be disabled at the IdP level).
How do you send a reset password email with a SSO platform that gates email?
If you have 2FA a OTP can be sent to a trusted device or app to allow a password reset without having access to email?
Is your workplace Okta? They built a custom internal tool. Until they enumerate exactly what the attacker had access to it’s hard to infer anything.
The article specifically addresses this, right? They log every interaction from support engineers and are going to report to the affected customers what actions were performed.
As others have mentioned, the slack access is potentially more concerning.
As others have mentioned, the slack access is potentially more concerning.
Based on the screenshots and Okta's description (which absolutely needs to be more clear), the attacker couldn't directly change passwords, they could only trigger password reset emails.
The linked article states that SuperUser is the name of their internal software. It's a confusing name, but at no point did the attacker have access to "near full admin access", based off the report.
The relevant question is "near-full admin access of a system that does what?"
If Okta has implemented a BeyondCorp model, the attackers had access with known limits. It doesn't matter how much access if the machine they've accessed is itself untrusted. That's the punchline of an episode of Star Trek: Lower Decks where they allowed an evil computer to take over the lighting systems on a starship; while they repaired the ship, all the computer could do was flick the lights on and off angrily.
If Okta has implemented a BeyondCorp model, the attackers had access with known limits. It doesn't matter how much access if the machine they've accessed is itself untrusted. That's the punchline of an episode of Star Trek: Lower Decks where they allowed an evil computer to take over the lighting systems on a starship; while they repaired the ship, all the computer could do was flick the lights on and off angrily.
On the topic of 'leet hacking skillz' I personally find it strange that so many Silicon Valley companies only care abut leetcode for the interview process.
You can be a security expert and no tech company will care unless you cram leetcode non-stop.
On the plus side the lack of security focus makes for a lot of opportunities for the stock market when betting against overvalued tech companies with this problem!
You can be a security expert and no tech company will care unless you cram leetcode non-stop.
On the plus side the lack of security focus makes for a lot of opportunities for the stock market when betting against overvalued tech companies with this problem!
security design is pretty much the only thing that resembles real engineering in software. other than safety critical systems, it's pretty much the most common thing that can actually go wrong and harm people.
one day there will probably be professional engineering certifications for software engineers, and those certifications will largely center around building secure software and systems. (with maybe some focus on safety critical systems as well)
it's funny when you think about how young the field is.
one day there will probably be professional engineering certifications for software engineers, and those certifications will largely center around building secure software and systems. (with maybe some focus on safety critical systems as well)
it's funny when you think about how young the field is.
The screenshots by the attacking group claimed to have access to thousands of slack channels, many of which they say included API keys. It's likely that some or many of those were to prod services.
This is the actual egregious thing in the breach. All of the support tool stuff seems like it was acting as designed and prevented the breach for sprawling.
The idea that at a security focused company they are passing around secrets via public slack channels is wild. That’s the thing they need to address.
The idea that at a security focused company they are passing around secrets via public slack channels is wild. That’s the thing they need to address.
[deleted]
> an unauthorized user had full super user access to the service
From the article:
> The majority of support engineering tasks are performed using an internally-built application called SuperUser or SU for short, which is used to perform basic management functions of Okta customer tenants. This does not provide “god-like access” to all its users. This is an application built with least privilege in mind to ensure that support engineers are granted only the specific access they require to perform their roles. They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories.
From the article:
> The majority of support engineering tasks are performed using an internally-built application called SuperUser or SU for short, which is used to perform basic management functions of Okta customer tenants. This does not provide “god-like access” to all its users. This is an application built with least privilege in mind to ensure that support engineers are granted only the specific access they require to perform their roles. They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories.
From circulating screenshots, and the omission in the listing here, it looks like they used this account to reset user passwords and MFA on a bunch of tenants. Not being able to create or delete users is meaningless.
Why is it a meaningless distinction to prevent account creation but allow password reset?
Most commercial services allow unauthenticated password reset just by hitting a web form. It would send a password reset link to the correct place and the person would either ignore it, follow through and have a new password or report it. It doesn’t impact the service access at all.
Meanwhile creating an account would allow new access to the system. I’m having a hard time figuring out how that’s not a world of difference?
Most commercial services allow unauthenticated password reset just by hitting a web form. It would send a password reset link to the correct place and the person would either ignore it, follow through and have a new password or report it. It doesn’t impact the service access at all.
Meanwhile creating an account would allow new access to the system. I’m having a hard time figuring out how that’s not a world of difference?
Apparently the "password reset" is more of a "password set" feature.
Where was that outlined? All of the screen dumps I saw showed what looked like a reset.
Why is password reset considered a big issue? If I know your email / login I can go reset your password for a bunch of stuff now without any breach.
Fair call. That's an important clarification.
Kinda unrelated, but how did job title inflation get this bad?
Support _Engineer_ that is using a prebuilt application to do stuff like reset MFA. In what world does this have anything to do with engineering?
Support _Engineer_ that is using a prebuilt application to do stuff like reset MFA. In what world does this have anything to do with engineering?
[deleted]
Nope. You are just not understanding it.
To simplify: The hackers had minimal access to stuff and couldn’t do much.
To simplify: The hackers had minimal access to stuff and couldn’t do much.
They could see the names and contacts of employees at Okta’s customers and reset their credentials at a bare minimum per the leaked screenshot. That doesn’t seem that minimal?
Based on the screenshots, they could trigger password reset emails. They couldn't directly set the new password.
Not defending Okta's response here; I think it has been quite terrible.
Not defending Okta's response here; I think it has been quite terrible.
The CSO is implying that the super user was able to get into the system but did not have access to sensitive data, or at least did not exfiltrate that data. It's possible if their defenses are layered enough. The security model is defense in depth, rather than the "moat" concept of hard on the outside, soft on the inside. Micro-breaches are expected; it's about detecting and mitigating them. I'm not saying the CSO is telling the truth, but that's how you reconcile someone having access to the system without accessing sensitive data.
> Support engineers use a number of customer support tools to get their job done including Okta’s instances of Jira, Slack, Splunk, RingCentral, and support tickets through Salesforce.
I like how it just glosses over access to all the other tools which often contain a treasure trove of data. Just Slack can give an attacker worst case credentials pasted into channels and best case loads of information for more targeted social engineering attacks. LAPSUS$ even stated they had access to over 8K channels.
I like how it just glosses over access to all the other tools which often contain a treasure trove of data. Just Slack can give an attacker worst case credentials pasted into channels and best case loads of information for more targeted social engineering attacks. LAPSUS$ even stated they had access to over 8K channels.
Reading the other trending article on LAPSUS$ (https://news.ycombinator.com/item?id=30774406), access to Slack suits their MO perfectly in terms of mining it for data for more sophisticated escalation of access via social engineering.
I view this hack as a public service - we're seeing how awfully these big security companies actually handle security in the worst case. Let that be a lesson to everyone who is in favour of mass centralisation. There needs to be a security solution where Okta essentially provides the skeleton of the infrastructure but not the entire solution, such that an Okta compromise does not compromise everyone who uses them.
It's wild that apparently near full admin access to the service was handed out to subcontractors of subcontractors. There was so little oversight and so many levels of indirection that it took months to figure out the full impact of the breach and incident. This does not seem like a healthy or responsible security policy for a company whose primary business is... authentication security.
No, definitely not full admin access. Just a support tool called “SuperUser”.
It was a customer service rep who could send password reset emails.
Systems listed were that app and SaaS offerings. The laptop was vendor owned and managed.
Given my experience, it probably had 0 access to anything “inside” Okta. There was no mention of vendor VPN or Virtual Desktop access. Or access to any internal system beyond what was a horribly named call center support tool.
They really wouldn’t have passed the necessary audits for their enterprise and government customer base if they allowed 3rd party devices “real” access.
Beyond a “hacker” getting access, there is very little trust in (likely) contract employees of a contract company.
The fact that the tool is named Super User is killing them more than anything here.
It was a customer service rep who could send password reset emails.
Systems listed were that app and SaaS offerings. The laptop was vendor owned and managed.
Given my experience, it probably had 0 access to anything “inside” Okta. There was no mention of vendor VPN or Virtual Desktop access. Or access to any internal system beyond what was a horribly named call center support tool.
They really wouldn’t have passed the necessary audits for their enterprise and government customer base if they allowed 3rd party devices “real” access.
Beyond a “hacker” getting access, there is very little trust in (likely) contract employees of a contract company.
The fact that the tool is named Super User is killing them more than anything here.
But it was cheap!
And don't forget that they meet all the governments BS standards and auditing checks. So lots of government contractors use them.
One company doing a bad job does not mean it's impossible or even uncommon to do a good job. Also, if you wanted to hedge against Okta... feel free. You can U2F 2FA your services behind Okta or in front of it. We use GSuite SSO, but everywhere we can set 2FA outside of it we do so.
> does not mean it's impossible or even uncommon to do a good job.
The only way one can reach this conclusion is by ignoring all the breaches / CVEs that happened in large companies during the past few years.
Nowadays I just assume every company is crap at security unless proven otherwise.
The only way one can reach this conclusion is by ignoring all the breaches / CVEs that happened in large companies during the past few years.
Nowadays I just assume every company is crap at security unless proven otherwise.
I'm pretty up on my breaches :)
Yes, most companies are bad at security. Some aren't though. The problem right now is figuring out which ones are - there's very little signal.
Yes, most companies are bad at security. Some aren't though. The problem right now is figuring out which ones are - there's very little signal.
While true, Okta isn't some minor player that we can just wave away like this. I bet lots of other similar big companies will have similar issues - this is about much more than just their technical merit.
> One company doing a bad job does not mean it's impossible or even uncommon to do a good job.
We all know it is very uncommon to do a great job. Everyone has been breached sooner or later. Any anyone who has worked in engineering or security in tech companies knows how often security concerns are underprioritized far behind more visible but less important work.
Hedging is a good answer. Relying on a single point of failure that, if it ever fails open, will expose everything at once? Not a smart idea.
We all know it is very uncommon to do a great job. Everyone has been breached sooner or later. Any anyone who has worked in engineering or security in tech companies knows how often security concerns are underprioritized far behind more visible but less important work.
Hedging is a good answer. Relying on a single point of failure that, if it ever fails open, will expose everything at once? Not a smart idea.
Sounds like the main point of disagreement is the definition of a single word, "breached". Otherwise, everyone is in agreement about what happened, right? I agree with your definition of the word for whatever it's worth.
At this point, it would be helpful for Okta to stop using the term "breached", because apparently they aren't using the word in the same way everyone else is, and it's a point of contention.
At this point, it would be helpful for Okta to stop using the term "breached", because apparently they aren't using the word in the same way everyone else is, and it's a point of contention.
I actually think the definitional conflict might be on the term "Okta service" -
> the Okta service has not been breached
If you consider the Okta service as the infrastructure acutally providing auth services to customers - then it wasn't breached. Only Okta support services were breached and that did not (going with Okta's line here) actually allow for somebody to access the actual Okta auth services.
> the Okta service has not been breached
If you consider the Okta service as the infrastructure acutally providing auth services to customers - then it wasn't breached. Only Okta support services were breached and that did not (going with Okta's line here) actually allow for somebody to access the actual Okta auth services.
Everyone else is using the plain English definition because it’s accurate.
Okta is intentionally using their own definition to down-play what’s happened.
A breach is simply ‘to overcome defences’ (ie to get access to something you shouldn’t). In this case their defence against someone else accessing the super user application was the support employee and their credentials, but this defence was clearly overcome by the hackers.
I agree that they need to stop using the word.
Okta is intentionally using their own definition to down-play what’s happened.
A breach is simply ‘to overcome defences’ (ie to get access to something you shouldn’t). In this case their defence against someone else accessing the super user application was the support employee and their credentials, but this defence was clearly overcome by the hackers.
I agree that they need to stop using the word.
Agreed. It's obvious that lawyers are deeply involved already at this point.
The most likely reason they dont wanna use the common sense term "breached" is because it implies legally a breach of contract, which means liability and getting sued and having to show up in front of congress.
Very sad to see their response be so intransparent and flawed. I wish technical people would be more involved in writing these responses, not lawyers.
The most likely reason they dont wanna use the common sense term "breached" is because it implies legally a breach of contract, which means liability and getting sued and having to show up in front of congress.
Very sad to see their response be so intransparent and flawed. I wish technical people would be more involved in writing these responses, not lawyers.
The breach is technical in nature. The disclosure and response is a legal matter.
They dont wanna use the word "breached" because lawyers write these things - and legally breach has a different meaning than security wise. Very sad that they already prepare on the legal front rather then being open and transparent.
Was coming around to make this point that in the cybersecurity world the word "breach" can have a very particular meaning with legal implications. As a cybersecurity professional, we are trained not to refer to anything as a breach while communicating findings of investigations or alerting organizations to activities that we see unless that has already been legally established.
There is plenty concerning about their response to this situation, and this phrasing can be confusing, but from my POV in the industry this choice of words is understandable.
There is plenty concerning about their response to this situation, and this phrasing can be confusing, but from my POV in the industry this choice of words is understandable.
I think you have misunderstood at least parts of it.
"superuser" is a tool which can be used for various tasks. It is not the same thing as "full super user access". As I understand it, the tool superuser doesn't allow you to do whatever you want, like accessing any piece of data in the databases. It is used by their support engineers.
Not trying to say there was no impact, but "full super user access" traditionally implies more than being able to run one specific tool with the name superuser.
I think people who have written about this and looked at the screen shots has been sloppy and spread misinformation.
"superuser" is a tool which can be used for various tasks. It is not the same thing as "full super user access". As I understand it, the tool superuser doesn't allow you to do whatever you want, like accessing any piece of data in the databases. It is used by their support engineers.
Not trying to say there was no impact, but "full super user access" traditionally implies more than being able to run one specific tool with the name superuser.
I think people who have written about this and looked at the screen shots has been sloppy and spread misinformation.
I’m finding it hard to figure out what the role of a CSO is after this incident. They seem to be the least knowledgeable about the product they’re creating as to offer the highest degree of plausible deniability possible.
I believe part of the job description for CxO jobs is "willing to bend reality as per needs of the company"
[deleted]
At least he didn't blame the intern...
What a weird and potentially misleading statement.
* They stress that the compromised account wasn't able to "create/delete users or download customer databases", but not what it could do. Could it change passwords of accounts and add 2fa methods, allowing them to take over rarely/never accessed users? Disable 2fa? Change account permissions? List user accounts and metadata to build a user account DB for further attacks? The application is named "SuperUser" ...
* It took public posting of a screenshot to trigger an audit of access logs, two months after the compromise was detected!
* "Only" 2.5% of customers were accessed. That's supposed to be a good thing? Those were certainly the most valuable targets.
* Concludes everything is just fine and no corrective actions need to be taken, but affected customers might want to do their own analysis... Huh?
Sounds a lot like damage control.
* They stress that the compromised account wasn't able to "create/delete users or download customer databases", but not what it could do. Could it change passwords of accounts and add 2fa methods, allowing them to take over rarely/never accessed users? Disable 2fa? Change account permissions? List user accounts and metadata to build a user account DB for further attacks? The application is named "SuperUser" ...
* It took public posting of a screenshot to trigger an audit of access logs, two months after the compromise was detected!
* "Only" 2.5% of customers were accessed. That's supposed to be a good thing? Those were certainly the most valuable targets.
* Concludes everything is just fine and no corrective actions need to be taken, but affected customers might want to do their own analysis... Huh?
Sounds a lot like damage control.
> Sounds a lot like damage control.
Yup, the first line made my hackles stand up; surely there's no security incidents raised for a user-invoked operation like adding MFA to a subcontractor's account? Audit logging is fine, but that's a user-initiated operation that already requires (if they have their shit in order) username, password, existing MFA if applicable, and an e-mail confirmation. To the user. It's all by the user.
Does Github raise the alarms if I change something in my MFA settings?
Yup, the first line made my hackles stand up; surely there's no security incidents raised for a user-invoked operation like adding MFA to a subcontractor's account? Audit logging is fine, but that's a user-initiated operation that already requires (if they have their shit in order) username, password, existing MFA if applicable, and an e-mail confirmation. To the user. It's all by the user.
Does Github raise the alarms if I change something in my MFA settings?
The MFA tokens you use on Github are provided by you. The MFA tokens used by Okta employees/contractors may be provided by Okta, therefore Okta may know the token that was attempted to be added was not provided by them, and that is why it may have been blocked and a security alert generated. GitHub can't do that unless they are the only ones providing the tokens to their users.
It looks like it's got flagged because the key was added from a new location
The entire message has a tone of being entirely true but not representative of the entire truth. And that just leaves us all hanging with more questions because it doesn't tell us what we really need to know.
2.5% of all customers were accessed by all Sintel employees for the period in question. How many customers did the particular affected Sintel employee access?
They assessed all the actions that took place by the Sintel employees. Were any of them sensitive?
The tool doesn't allow them to create or delete users. Does it allow them to modify users so that an attacker can take over an account?
The attacker had access to "Jira, Slack, Splunk, RingCentral, and support tickets through Salesforce". Doesn't sound bad on its own but have those tools been evaluated to ensure they don't have sensitive information? Do we know what the Sintel employee in question accessed from each of those tools for the time period?
The employee's machine was logged into using an RDP session. Was the employee's machine assessed to ensure they weren't storing other sensitive information? While not allowed, a lot of support engineers will quickly drop in "temporary" sensitive stuff into random notepads and what not. Was this assessed? Also, why does a support engineer have a machine that has the ability to enable an inward bound RDP session at all?? How was that not the first trigger since this is a known attack vector for support agents.
Given the possibly small blast radius of this issue, Okta had all the opportunity to turn this into a communications win for them and just build a multiple of trust really quickly. All these half communications have utterly botched it and that's just really disappointing. We'll still continue to use them because the switching costs just don't justify moving off Okta. But Okta has really taken a hit in their "trust bank".
2.5% of all customers were accessed by all Sintel employees for the period in question. How many customers did the particular affected Sintel employee access?
They assessed all the actions that took place by the Sintel employees. Were any of them sensitive?
The tool doesn't allow them to create or delete users. Does it allow them to modify users so that an attacker can take over an account?
The attacker had access to "Jira, Slack, Splunk, RingCentral, and support tickets through Salesforce". Doesn't sound bad on its own but have those tools been evaluated to ensure they don't have sensitive information? Do we know what the Sintel employee in question accessed from each of those tools for the time period?
The employee's machine was logged into using an RDP session. Was the employee's machine assessed to ensure they weren't storing other sensitive information? While not allowed, a lot of support engineers will quickly drop in "temporary" sensitive stuff into random notepads and what not. Was this assessed? Also, why does a support engineer have a machine that has the ability to enable an inward bound RDP session at all?? How was that not the first trigger since this is a known attack vector for support agents.
Given the possibly small blast radius of this issue, Okta had all the opportunity to turn this into a communications win for them and just build a multiple of trust really quickly. All these half communications have utterly botched it and that's just really disappointing. We'll still continue to use them because the switching costs just don't justify moving off Okta. But Okta has really taken a hit in their "trust bank".
> 2.5% of all customers were accessed by all Sintel employees for the period in question. How many customers did the particular affected Sintel employee access?
Most likely, they don't write it because they may not know themselves. It is not certain that it gets logged perhaps?
Most likely, they don't write it because they may not know themselves. It is not certain that it gets logged perhaps?
It's interesting how you need to read what they don't write to actually figure out what they're saying:
They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories.
So they can likely arbitrarily access/impersonate or at least password reset existing users, and probably reconfigure the accounts in creative ways.
For transparency, these customers will receive a report that shows the actions performed on their Okta tenant by Sitel during that period of time. We think this is the best way to let customers assess the situation for themselves.
So they have no idea which of the actions were or weren't legitimate and make it their customers' problem.
They are unable to create or delete users. They cannot download customer databases. They cannot access our source code repositories.
So they can likely arbitrarily access/impersonate or at least password reset existing users, and probably reconfigure the accounts in creative ways.
For transparency, these customers will receive a report that shows the actions performed on their Okta tenant by Sitel during that period of time. We think this is the best way to let customers assess the situation for themselves.
So they have no idea which of the actions were or weren't legitimate and make it their customers' problem.
At least they are finally notifying.
> This is an application built with least privilege in mind
Uh huh, makes sense
> Named SuperUser
Uhh...
It lists all the operations that it can't do, but not what it can do. Can they download a private SAML certificate? Can they impersonate a user? Can they configure SSO and MFA settings? Can they download audit logs?
Uh huh, makes sense
> Named SuperUser
Uhh...
It lists all the operations that it can't do, but not what it can do. Can they download a private SAML certificate? Can they impersonate a user? Can they configure SSO and MFA settings? Can they download audit logs?
> Can they download a private SAML certificate?
Oh, that's a good one. Definitely something that the software should not allow, because I can't see a legitimate reason for this (allowing to download the certificate is fine, but not the key).
Oh, that's a good one. Definitely something that the software should not allow, because I can't see a legitimate reason for this (allowing to download the certificate is fine, but not the key).
This was my topmost question too. The report very cleanly omits any and all mentions of SAML signing certificates.
Solar Winds was the first known incident to escalate to so called "Golden SAML" attack. If the support staff had access to signing certificates, then that would open the door to a wide-scale exploitation of Okta's clients.
A shower of Golden SAMLs, if you like.
Solar Winds was the first known incident to escalate to so called "Golden SAML" attack. If the support staff had access to signing certificates, then that would open the door to a wide-scale exploitation of Okta's clients.
A shower of Golden SAMLs, if you like.
Caution to fellow readers: Put down your drink before reading the last line of this post.
Recent and related:
New Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30774193 - March 2022 (24 comments)
Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30769537 - March 2022 (220 comments)
Also:
DEV-0537 (LAPSUS$) Criminal actor targeting organizations - https://news.ycombinator.com/item?id=30774406 - March 2022 (0 comments)
Lapsus$ hackers leak 37GB of Microsoft's alleged source code - https://news.ycombinator.com/item?id=30763623 - March 2022 (117 comments)
New Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30774193 - March 2022 (24 comments)
Updated Okta Statement on Lapsus$ - https://news.ycombinator.com/item?id=30769537 - March 2022 (220 comments)
Also:
DEV-0537 (LAPSUS$) Criminal actor targeting organizations - https://news.ycombinator.com/item?id=30774406 - March 2022 (0 comments)
Lapsus$ hackers leak 37GB of Microsoft's alleged source code - https://news.ycombinator.com/item?id=30763623 - March 2022 (117 comments)
"were taken from a Sitel support engineer’s computer upon which an attacker had obtained remote access using RDP. This device was owned and managed by Sitel. The scenario here is analogous to walking away from your computer at a coffee shop"
It really is not analogous at all.
That RDP was enabled, let alone could be accessed from outside the network is worrying.
To me this would appear that, to save a buck, they outsource a lot of functions that then meant customer security was partly out of their hands, and relied upon another company having their security ducks in a row. I'm sure in their marketing materials they boast about state-of-the-art security, but that's only as good as the weakest point in the chain.
It really is not analogous at all.
That RDP was enabled, let alone could be accessed from outside the network is worrying.
To me this would appear that, to save a buck, they outsource a lot of functions that then meant customer security was partly out of their hands, and relied upon another company having their security ducks in a row. I'm sure in their marketing materials they boast about state-of-the-art security, but that's only as good as the weakest point in the chain.
Maybe they meant:
The scenario here is analogous to walking away from your computer at a coffee shop... with your computer unlocked and logged in to various things
The scenario here is analogous to walking away from your computer at a coffee shop... with your computer unlocked and logged in to various things
We can extend that - "after we've promised our customers that a computer would never be left unlocked and logged in to sensitive things in a public place, but would instead be behind multiple locked doors."
It's really not the "see, this is something you might do! It's not so bad!" out they thought it would be.
It's really not the "see, this is something you might do! It's not so bad!" out they thought it would be.
> The sharing of these screenshots is embarrassing for myself and the whole Okta team.
It speaks volumes that their embarrassment is so important that it was the second sentence of the whole investigation, while there is literally not a single word of apology to the actual customers who were compromised.
It speaks volumes that their embarrassment is so important that it was the second sentence of the whole investigation, while there is literally not a single word of apology to the actual customers who were compromised.
They are individually reaching out to the actual customers who were compromised. You are making quite an assumption without seeing the individual emails.
I'm not assuming anything, I'm commenting on the public post here.
As communicated in stern words to Okta, my company unnecessarily spend many people hours on this. IT had to investigate if we were impacted by this, and on top of that issued a password reset for the entire company.
A swift communication by Okta could have avoided this all together. It seems they care more about their shareholders than their customers.
A swift communication by Okta could have avoided this all together. It seems they care more about their shareholders than their customers.
> It seems they care more about their shareholders than their customers.
isn't this how publicly-traded companies are supposed to work? I agree on critizicing that approach and capitalism model, but I don't understand how that isn't common knowledge here.
isn't this how publicly-traded companies are supposed to work? I agree on critizicing that approach and capitalism model, but I don't understand how that isn't common knowledge here.
Of course the sole purpose of a publicly traded company is to maximise the revenue for its shareholders, however, you can take a long term or short term approach on this.
Okta appeared to have kept this under wraps to prevent a shareholders backlash (short term approach) However, as a result they achieved the opposite, as the share price is still down this morning. This may of course be a temporarily glitch, however, I can see it resulting in a temporary loss of revenue. If I would be evaluating Okta versus a different solution right now, this may well sway my decision.
Okta appeared to have kept this under wraps to prevent a shareholders backlash (short term approach) However, as a result they achieved the opposite, as the share price is still down this morning. This may of course be a temporarily glitch, however, I can see it resulting in a temporary loss of revenue. If I would be evaluating Okta versus a different solution right now, this may well sway my decision.
It doesn't have to be. If the shareholders want the company to otherwise cease operations and throw a big ice cream party for all the shareholders every Friday they can choose to do that. There's nothing that forces a public company to focus only on maximizing shareholder value, they just have to be open and honest about the goals of the company and try and meet the shareholder expectations.
I suspect there are plenty of current customers that are considering a change, not just prospects.
No, it is not how publicly traded companies are supposed to work.
Publicly traded companies can set whatever priorities they want; the law only requires certain levels of accurate reporting. If shareholders think a company is too focused on customers, or not enough on shareholders, their options are simply to complain or sell the stock, or both.
Publicly traded companies can set whatever priorities they want; the law only requires certain levels of accurate reporting. If shareholders think a company is too focused on customers, or not enough on shareholders, their options are simply to complain or sell the stock, or both.
They can vote for different board members or do shareholder initiatives as well.
It could be argued that if you don't care about your customers, you won't be in business long enough to please your shareholders.
Companies who care about their customers usually do better in the long run. You can look up "long term greedy" for an example.
> The majority of support engineering tasks are performed using an internally-built application called SuperUser or SU for short, which is used to perform basic management functions of Okta customer tenants.
Pretty ominous name. I wouldn’t hand out “super user” accounts to support engineers from contracting firms for “basic duties in handling inbound support queries”.
Pretty ominous name. I wouldn’t hand out “super user” accounts to support engineers from contracting firms for “basic duties in handling inbound support queries”.
"Although that individual attempt was unsuccessful, out of an abundance of caution, we reset the account and notified Sitel who engaged a leading forensic firm to perform an investigation."
Really don't like that they're still unwilling to actually say the name of the 'leading forensic firm'.
Really don't like that they're still unwilling to actually say the name of the 'leading forensic firm'.
All while happily throwing Sitel under the bus.
This whole communication from Okta is a total debacle. They are trying desperately to under play the event and deflect blame in the poor transparency after the event all while not once mentioning that they plan on making any remediations or changing anything in their day to day operations after this incident to prevent it from happening again.
This whole communication from Okta is a total debacle. They are trying desperately to under play the event and deflect blame in the poor transparency after the event all while not once mentioning that they plan on making any remediations or changing anything in their day to day operations after this incident to prevent it from happening again.
> "I am greatly disappointed by the long period of time that transpired between our notification...once WE received the Sitel summary report WE should have moved more swiftly to understand its implications."
This is an example of total non-ownership. He is the CSO. It should be "I" or "My" and not diffusing responsibility onto his team with "We".
In my book you should celebrate your successes as a team ("we", "our") but failures are ALWAYS on leaders ("I", "my").
This is an example of total non-ownership. He is the CSO. It should be "I" or "My" and not diffusing responsibility onto his team with "We".
In my book you should celebrate your successes as a team ("we", "our") but failures are ALWAYS on leaders ("I", "my").
I'm curious about Okta's customer facing audit logs. It seems they do offer audit logs [1] [2] but I can't find the documentation on what all events are included and if actions by support are (which they should). Google for example includes transparency events in it's audit log [3], CloudTrail also shows actions taken by AWS support.
I've talked about this before [4] but just having internal audit logs doesn't cut it nowadays, even in this case it took two months to check the audit logs, you should give your customers access to their audit logs, ideally in near realtime so they can do proactive monitoring.
[1] https://help.okta.com/en/prod/Content/Topics/Reports/Reports...
[2] https://developer.okta.com/docs/reference/api/system-log/
[3] https://developers.google.com/admin-sdk/reports/v1/appendix/...
[4] https://apptrail.com/blog/2022/03/07/internal-vs-customer-fa...
I've talked about this before [4] but just having internal audit logs doesn't cut it nowadays, even in this case it took two months to check the audit logs, you should give your customers access to their audit logs, ideally in near realtime so they can do proactive monitoring.
[1] https://help.okta.com/en/prod/Content/Topics/Reports/Reports...
[2] https://developer.okta.com/docs/reference/api/system-log/
[3] https://developers.google.com/admin-sdk/reports/v1/appendix/...
[4] https://apptrail.com/blog/2022/03/07/internal-vs-customer-fa...
Apparently support actions can be found with "user.session.impersonation" from what I read.
Oh man, it sucks that they were hacked, but at least they weren't hacked.
After seeing this incident, and particularly Okta’s response to this, I will make it a personal mission to ensure that Okta is not used anywhere that I work.
So if I'm reading this right, Okta was aware of a "compromise" of one of their sub-processors that impacted an unknown number of their customers/end users. They then waited more than 2 months before performing their own rudimentary analysis of the audit log to see what actions that sub-processor may have taken during the "compromise".
Their CSO writes, "Over the past 24 hours we have analyzed more than 125,000 log entries to ascertain what actions were performed by Sitel during the relevant period. We have determined that the maximum potential impact is 366 (approximately 2.5% of) customers whose Okta tenant was accessed by Sitel."
IANAL and these are only my opinions but it seems like:
(a) Their DPO chose not to notify (GDPR Art. 33) without having the full picture or thought waiting several months for sub-processor's report was a justifiable reason for delaying notification
(b) They failed to perform their own basic forensic activities in light of a "compromise" and only reviewed logs on March 22nd
(c) Have terrible taste in naming their support app "Super User"
In my opinion they are also down playing the importance of the data that may have been compromised. For example do the hackers now know which accounts have MFAs attached to and which don't. What the password policies are (e.g. strength, number attempts, etc.)
Their CSO writes, "Over the past 24 hours we have analyzed more than 125,000 log entries to ascertain what actions were performed by Sitel during the relevant period. We have determined that the maximum potential impact is 366 (approximately 2.5% of) customers whose Okta tenant was accessed by Sitel."
IANAL and these are only my opinions but it seems like:
(a) Their DPO chose not to notify (GDPR Art. 33) without having the full picture or thought waiting several months for sub-processor's report was a justifiable reason for delaying notification
(b) They failed to perform their own basic forensic activities in light of a "compromise" and only reviewed logs on March 22nd
(c) Have terrible taste in naming their support app "Super User"
In my opinion they are also down playing the importance of the data that may have been compromised. For example do the hackers now know which accounts have MFAs attached to and which don't. What the password policies are (e.g. strength, number attempts, etc.)
The compromised tenant looks like it was specifically _not_ one of the EMEA ones so GDPR wouldn't be relevant here.
I think the issue is that they just wouldn’t know. They didn’t know which customers were impacted. They didn’t know which users personal data might have been compromised. They most likely don’t have the ability to determine whether a user is a EU resident or not as this information would reside with their customers HR systems which all points to having to notify to avoid the legal complications.
If the tenant had 1 or more European employees in their system, then yes GDPR is likely relevant.
Again, sounds very defensive and not honest. Like rather than say the account had least priveleges and did not have "God" level access, you should state what authorities they did have, for eg: resetting pws, mfa's, etc...
david bradbury will probably fail upwards, must be nice
That is deeply ironic and also hilarious in this context.
Here’s video: https://youtu.be/fAADWfJO2qM
[deleted]
The first 71 minutes of the response timeline are exemplary. But then two gaps emerge:
1. (Bad) Why did it take an entire day to notify the contractor?
2. (Much, much worse) Why is there no mention of any investigation into all the historical actions taken by that support agent?
I remember a couple years ago we were using Okta for AWS federated access. One day a bunch of role associations disappeared. Ended up badgering support for a couple weeks since the audit logs were empty before they finally admitted it was a "bad migration"
Apparently they had some internal issue with orphaned records or something and deployed a code "fix" that ended up deleting legitimate records in a way that wasn't auditable from the customer side.
I think it took about 4-6 weeks of trying to escalate through support and account reps until we actually got an answer. It was also very surprising to see role associations just disappear without a trace (we associated Okta groups with AWS IAM roles in the Okta integration)
Apparently they had some internal issue with orphaned records or something and deployed a code "fix" that ended up deleting legitimate records in a way that wasn't auditable from the customer side.
I think it took about 4-6 weeks of trying to escalate through support and account reps until we actually got an answer. It was also very surprising to see role associations just disappear without a trace (we associated Okta groups with AWS IAM roles in the Okta integration)
[deleted]
Is there a list of services that use Okta?
What a weird and dumb response. The fact that they keep updating their response will result in a backfire.
What the best free place to subscribe to, to get notified of hacks like this? Some place that is quick at getting them added/listed and notifying people. As I often hear about it in the news first which is day+ after it’s released and not soon enough
Depends on the type of hack. Haveibeenpwned is a classic, but i don't think it covers cases like this, it's more for username/password/PII breaches.
I'd say twitter perhaps but you have to follow the right sources
David Bradbury, grow a backbone. You were breached and you did not notify your customers.
"Prior to Okta, David was the Senior Vice President and Chief Security Officer at Symantec" So, his prior experience is as CSO at a company who's principal business is selling fake security products. Ho, boy.
tl;dr - Companies cheerfully handed over their golden skeleton and city (company) keys over to a third party service provider that offers SSO and now, after they got hacked by some kiddy that writes like a teenager, we found out that they also just cook with water (or less: api keys in slack). Now the public company communication is a reputation crushing web of lies, inaccuracies and whining.
The stock hasn't crashed yet. I see opportunity.
Could have made 500% on PUTS today alone though.
[deleted]
oh no, my company just rolled out Okta too...
I don't understand how OKTA is +4.34%/1M and 10.26%/5D. This is bat-shit crazy lol. Anyway I put a 10,000USD 5x sell at 166.19, let's see how it goes from here :D :D :D
hahha. downvote at will. I just made $2k with literally twenty mouse clicks.
Congrats! If you bought PUTS at market open today (or yesterday) you could have made 500% today.
I'm excited about this and in the future will be checking stock price first thing when I see these smaller publicly listed tech companies getting hit with big hacks.
For MSFT (and companies of that size) this type of news doesn't move the stock that much but with current volatility there will be plenty of great plays in the future for smaller companies.
I'm excited about this and in the future will be checking stock price first thing when I see these smaller publicly listed tech companies getting hit with big hacks.
For MSFT (and companies of that size) this type of news doesn't move the stock that much but with current volatility there will be plenty of great plays in the future for smaller companies.
"In this post, I want to provide a timeline and my perspective on what has transpired, and where we are today with this investigation. I hope that it will illuminate why I am confident in our conclusions that the Okta service has not been breached and there are no corrective actions that need to be taken by our customers."
And then go on to write paragraphs of detail and a timeline that explicitly shows for a five day period an unauthorized user had full super user access to the service.
He explicitly says, "The report from the forensic firm highlighted that there was a five-day window of time between January 16-21, 2022 when the threat actor had access to the Sitel environment, which we validated with our own analysis."
This is some unbelievable double speak to try to claim that Okta was not breached. Any trust I had in this company is completely in the toilet, just based on this response. How is the CEO not responding to this too? The hole just keeps getting dug deeper the more they say.