How Not to Acknowledge a Data Breach(krebsonsecurity.com)
krebsonsecurity.com
How Not to Acknowledge a Data Breach
https://krebsonsecurity.com/2019/04/how-not-to-acknowledge-a-data-breach/
56 comments
Companies I have worked at that have outsourced their IT infrastructure have almost always consistently showed a lack of care, experience, training and competence when it came to looking after our systems. In many cases I really believe that it ultimately cost the company more in lost efficiencies and downtime than if they had kept the infrastructure expertise in-house.
One things I always wonder - Why do IT folks, people who are supposedly expert in computers, fall for phishing emails etc? You'd think they'd know better.
Couple of years ago I was working at the offshore office of a large internet company. They claimed that their platform touched nearly 15-30% of internet users. They had a large Security Operations Center to ensure every threat was monitored and mitigated.
Our software and systems were frequently flagged for security issues. Any suggestions to alleviate the issue fell on deaf ears.
The onshore US IT team always complained about how security changes will make life difficult for them. The biggest critic was someone who had once opened a phishing mail and got his password stolen (using memory dumps in Windows).
While the offshore team was afraid that pushing US onshore team too hard might put their jobs in danger.
And as if getting hacked wasn't enough, these guys exchanged plain text files containing everyone's salary via email. No amount of training or meeting helped.
Finally, a roundabout solution was put in place. Give US folks two laptops - one with heavy encryption for work and second, for checking mails.
But they wouldn't budge on spending salaries in plain text. They said, it was easier for them to manage this way.
Couple of years ago I was working at the offshore office of a large internet company. They claimed that their platform touched nearly 15-30% of internet users. They had a large Security Operations Center to ensure every threat was monitored and mitigated.
Our software and systems were frequently flagged for security issues. Any suggestions to alleviate the issue fell on deaf ears.
The onshore US IT team always complained about how security changes will make life difficult for them. The biggest critic was someone who had once opened a phishing mail and got his password stolen (using memory dumps in Windows).
While the offshore team was afraid that pushing US onshore team too hard might put their jobs in danger.
And as if getting hacked wasn't enough, these guys exchanged plain text files containing everyone's salary via email. No amount of training or meeting helped.
Finally, a roundabout solution was put in place. Give US folks two laptops - one with heavy encryption for work and second, for checking mails.
But they wouldn't budge on spending salaries in plain text. They said, it was easier for them to manage this way.
> Why do IT folks, people who are supposedly expert in computers, fall for phishing emails etc?
Phishing is a numbers game. Make it convincing enough, send to enough people, and wait for someone who's too busy/tired to think to fall for it.
> The onshore US IT team always complained about how security changes will make life difficult for them.
Because that's absolutely true and valid complaint. From their point of view, they're being paid for doing jobs, and then the company starts spending money to prevent them from doing their jobs. I've been on the receiving end of this in the past, where only sanity of our internal IT team prevented making all software development take 3 times longer, because the company decided to apply some completely bullshit "security practices" they found in ISO-whatever compliance handbook.
Security needs to work with people, not against people. You can't just announce security changes that utterly destroy existing workflows, without helping develop equivalent replacement workflow (with all corner cases accounted for), and while still expecting the same amount of work from people. You'll just see people push back hard, and then ignore the new changes to the extent possible while still trying to meet their deadlines.
Phishing is a numbers game. Make it convincing enough, send to enough people, and wait for someone who's too busy/tired to think to fall for it.
> The onshore US IT team always complained about how security changes will make life difficult for them.
Because that's absolutely true and valid complaint. From their point of view, they're being paid for doing jobs, and then the company starts spending money to prevent them from doing their jobs. I've been on the receiving end of this in the past, where only sanity of our internal IT team prevented making all software development take 3 times longer, because the company decided to apply some completely bullshit "security practices" they found in ISO-whatever compliance handbook.
Security needs to work with people, not against people. You can't just announce security changes that utterly destroy existing workflows, without helping develop equivalent replacement workflow (with all corner cases accounted for), and while still expecting the same amount of work from people. You'll just see people push back hard, and then ignore the new changes to the extent possible while still trying to meet their deadlines.
The dumbest security restriction I've dealt with is not giving devs local admin access to their work compeers. In a previous job I had to contact my manager every time I needed to install or update software. Sometimes he had to get approval from the IT gods before he could do anything. I usually just figured out ways to run applications without installing them.
It was a huge waste of time. From what I understand a dev installed something bad one time so they punished everyone. Then again that's usually how these things happen.
It was a huge waste of time. From what I understand a dev installed something bad one time so they punished everyone. Then again that's usually how these things happen.
At a previous employer that policy was continually causing problems and I'd been able to fly under their radar since all they did was issue hardware to myself and my minions†. Eventually IT staff came around needing the root password for my dev machine and things got awkward very quickly when I explained they didn't have internal clearance to log into it, much less view any data, and they'd have to go ask a senior vice president for direct approval in writing.
They were some of the best IT people I've worked with, but they'd just run into a situation where they were the people who needed permission and they had to create exceptions on the fly. There's always going to be a place where the process breaks down and either you give the IT people room to adjust their policies or you make them sit on their hands and pretend nothing is going on.
† Not my terminology.
They were some of the best IT people I've worked with, but they'd just run into a situation where they were the people who needed permission and they had to create exceptions on the fly. There's always going to be a place where the process breaks down and either you give the IT people room to adjust their policies or you make them sit on their hands and pretend nothing is going on.
† Not my terminology.
Oh yes. Taking away admin access was one of the dumb ISO-whatever security restrictions they tried to impose on us top-down. We could retain it iff we signed a form that put any and all responsibility and blame for any problems coming from our admin access on us personally. Fortunately, right people in the right places arranged it so the forms got "lost", and admin access was not taken away.
I'm in a similar situation right now. In order to install new software onto my computer I have to get IT to add it to a company-wide whitelist, so I don't bother.
This is a pretty good “culture fit” question to ask a prospective employer in an interview. If the answer is wrong it won’t be the only thing wrong.
> Because that's absolutely true and valid complaint.
I agree that it is a valid complaint.
But simple rules like - Use a thick/thin client or terminals instead of hopping on to a server or use complex passwords with password manager if required or the easiest one - don't open personal emails on work machine, should be easy to follow.
I agree that it is a valid complaint.
But simple rules like - Use a thick/thin client or terminals instead of hopping on to a server or use complex passwords with password manager if required or the easiest one - don't open personal emails on work machine, should be easy to follow.
Quick bullet points on the steps a company should perform when a data breech comes to light?
If you don't have internal IT folks and an incident response team already in place you're going to need to bring in some outside assistance.
- Take all compromised or infected systems offline immediately and have them carefully examined to determine what data those systems contained (customer, internal, and employee) and what was exposed to attackers.
- look for additional signs of compromise in other systems as well paying close attention to those connected with the ones that were compromised
- log and record everything extensively throughout the process
- consult with your lawyers to see what legal obligations you have to disclose the breach publicly
- notify everyone directly impacted by the breach as soon as you've identified them so they can start taking steps to protect themselves. Don't wait until you have all the details, you can update them as more information is discovered.
- Make sure you have professionals to perform the investigation and re-secure your impacted systems before they are put back online. This isn't the time to count on that one guy in the office who "knows computers real good" to fix it.
- continue to keep extensive logs and keep a close eye on everything for a time after the attack to make sure you're not compromised again.
If you can manage even that much you're doing better than most of the companies I've seen who were hacked. I've seen banks leave compromised systems connected to the network for months after being notified. I've seen some pretty large companies refuse to bring in outside help because they don't want the expense even after they get hacked over and over. Many small to midsize companies try to handle everything as quietly as possible and never tell anyone what happened.
- Take all compromised or infected systems offline immediately and have them carefully examined to determine what data those systems contained (customer, internal, and employee) and what was exposed to attackers.
- look for additional signs of compromise in other systems as well paying close attention to those connected with the ones that were compromised
- log and record everything extensively throughout the process
- consult with your lawyers to see what legal obligations you have to disclose the breach publicly
- notify everyone directly impacted by the breach as soon as you've identified them so they can start taking steps to protect themselves. Don't wait until you have all the details, you can update them as more information is discovered.
- Make sure you have professionals to perform the investigation and re-secure your impacted systems before they are put back online. This isn't the time to count on that one guy in the office who "knows computers real good" to fix it.
- continue to keep extensive logs and keep a close eye on everything for a time after the attack to make sure you're not compromised again.
If you can manage even that much you're doing better than most of the companies I've seen who were hacked. I've seen banks leave compromised systems connected to the network for months after being notified. I've seen some pretty large companies refuse to bring in outside help because they don't want the expense even after they get hacked over and over. Many small to midsize companies try to handle everything as quietly as possible and never tell anyone what happened.
It can be a complex, painful process. It will involve looking at log files for just about anything, resetting some set of passwords, engaging outside counsel, engaging a forensic/IR firm.
And it is often a race against various competing priorities. For example, if there is an attacker actively working into your system, one temptation is to totally turn that ingress off. However, if the attacker is sophisticated, then they know that they have been detected, and may well find yet another route into your systems that is harder for you to spot.
This can be proactively addressed by having an IR plan, and occasional table-top exercises too address select scenarios. Executive buy-in is critical.
And it is often a race against various competing priorities. For example, if there is an attacker actively working into your system, one temptation is to totally turn that ingress off. However, if the attacker is sophisticated, then they know that they have been detected, and may well find yet another route into your systems that is harder for you to spot.
This can be proactively addressed by having an IR plan, and occasional table-top exercises too address select scenarios. Executive buy-in is critical.
This is very interesting facts that cyber security is something required specialized knowledge and training to quickly respond with appropriate info rather than jumping around that actually creates confusion around the industry as such.
Who uses outsourcing firms like Wipro (government agencies)? What do they do? It seems to me, living in the bubble of haute tech, that the only things you ever hear about these firms are (a.) they exist and (b.) they are very large.
One of the things they do is man the call centers in which the call agents are only allowed to: a) open tickets b) follow a script intended for people who didn't do basic troubleshooting before calling tech support c) be very sorry that your business is losing money due to the service interruption d) give you vague non-answers as to when the team with the authority to actually fix things but which is not allowed to interact directly with customers will actually get around to working on your issue, and e) ask you to kindly rate them highly on the customer satisfaction survey that you will be automatically connected to after this call.
Another thing they do is supply you with an endless cascade of developers who rotate in and out of your dev team on a 3-month basis. In those three months they will a) tell your IT team that they need a newer laptop because their current one gets slow when trying to run 3 instances of Visual Studio concurrently b) wonder why hard drive failures are correlated with rough handling of the laptop c) ask IT to troubleshoot their compilation errors d) only save their code locally instead of to a shared repo e) get upset when 3 months of said locally saved code, which IT was never told about, gets lost when the laptop, previously issued to the dev they replaced, is re-imaged before redeployment as per company policy/standard-procedure f) thereafter dodge turning in laptops to IT after their rotation ends, instead giving it directly to their replacement, causing IT's stores of deployable laptops to get depleted, and thus causing IT to come hunting after the never-turned-in laptops, because the next wave of 3-month devs is coming and hardware to issue them is needed, and finally g) treat company-issued laptops so roughly that when IT eventually does them back, what was brand shiny and new 3 months prior is now chipped, cracked, scratched and covered in grease and crumbs, and has a full kitten's worth of fluff accumulation in the cooling fan.
Okay, that last one was a bit hyperbolic, but every single point there did happen, and more than once.
Yes, I have been embittered by contractors from these large Indian IT firms, both as an end user 'supported' by them and as a co-worker supporting them. My experience with them has not been a positive one.
Another thing they do is supply you with an endless cascade of developers who rotate in and out of your dev team on a 3-month basis. In those three months they will a) tell your IT team that they need a newer laptop because their current one gets slow when trying to run 3 instances of Visual Studio concurrently b) wonder why hard drive failures are correlated with rough handling of the laptop c) ask IT to troubleshoot their compilation errors d) only save their code locally instead of to a shared repo e) get upset when 3 months of said locally saved code, which IT was never told about, gets lost when the laptop, previously issued to the dev they replaced, is re-imaged before redeployment as per company policy/standard-procedure f) thereafter dodge turning in laptops to IT after their rotation ends, instead giving it directly to their replacement, causing IT's stores of deployable laptops to get depleted, and thus causing IT to come hunting after the never-turned-in laptops, because the next wave of 3-month devs is coming and hardware to issue them is needed, and finally g) treat company-issued laptops so roughly that when IT eventually does them back, what was brand shiny and new 3 months prior is now chipped, cracked, scratched and covered in grease and crumbs, and has a full kitten's worth of fluff accumulation in the cooling fan.
Okay, that last one was a bit hyperbolic, but every single point there did happen, and more than once.
Yes, I have been embittered by contractors from these large Indian IT firms, both as an end user 'supported' by them and as a co-worker supporting them. My experience with them has not been a positive one.
password resets? The one time I had to work with goverment IT they had a call center. They were trying to introduce a VPN product that required a super old version of IE / Java to connect (we literally had to run Windows XP with IE to get this thing to work). This was the "new" software that was like 10 years old when purchased.
I'll skip all the drama - but one thing you could do is call the help desk to get your password reset because of one clusterf after another. It actually worked great. Hi, my username is JoeBob. Ok JoeBob, your new password is XXXX.
That was it. This is a system with super long passwords that had to be changed ridiculously often, and an account lock feature after a few messed up entries which required a password reset and lots of temp staff who came and went among other issues (there were two layers of passwords and people constantly got them confused).
So they had a metric TON of password calls. Despite all the drama with passwords, you could get your password reset just by knowing your username and the number to call for resets. It was brilliant and did save a TON of time, but I had to laugh at the security of the system given your username was derived directly from your name and was widely available in reports etc.
I never mentioned anything though because the thought of a more complicated procedure to deal with for all the staff would have been a nightmare.
I'll skip all the drama - but one thing you could do is call the help desk to get your password reset because of one clusterf after another. It actually worked great. Hi, my username is JoeBob. Ok JoeBob, your new password is XXXX.
That was it. This is a system with super long passwords that had to be changed ridiculously often, and an account lock feature after a few messed up entries which required a password reset and lots of temp staff who came and went among other issues (there were two layers of passwords and people constantly got them confused).
So they had a metric TON of password calls. Despite all the drama with passwords, you could get your password reset just by knowing your username and the number to call for resets. It was brilliant and did save a TON of time, but I had to laugh at the security of the system given your username was derived directly from your name and was widely available in reports etc.
I never mentioned anything though because the thought of a more complicated procedure to deal with for all the staff would have been a nightmare.
That c) particularly hit home. I had a contractor insist a T_PAAMAYIM_NEKUDOTAYIM is usually caused by a RAM fault and sent people to replace hardware.
Geez. That employee sounds almost as awful as the place you work.
employees, and worked. The individual items were distributed across the various individual contractors cycling in and out; no one was guilty of all of those points, but most were guilty of some. If it was one mostly-static set of contractors who could've been grown into actually productive devs who cared about their craft, that would have been one thing, and I'd have felt proud to have contributed to said growth. But an endless parade of people with whom communication is difficult will eventually have a corrosive effect upon the soul.
I haven't worked there for years. When I was laid off during the acquisition, I didn't protest, because I was about to look for a new job anyway. My only regret was not being allowed to finish building the new image deployment system so the persons to take up my duties wouldn't have to deal with yet-another-half-finished-system-built-by-people-no-longer-working-here.
I haven't worked there for years. When I was laid off during the acquisition, I didn't protest, because I was about to look for a new job anyway. My only regret was not being allowed to finish building the new image deployment system so the persons to take up my duties wouldn't have to deal with yet-another-half-finished-system-built-by-people-no-longer-working-here.
The area I see these firms get a ton of work is in Enterprise Resource Planning (ERP) work. TONS of companies run their HR / Finance / Sales / etc departments on things like Salesforce and Oracle and whatever, and there are huge numbers of consultants whose jobs are to build custom installations over and over. I've never worked in the field, but my wife did. It's definitely reaaaally different from the way that the rest of the software industry works.
Its also a hugely lucrative ones for people who like working in Software as a job as opposed to Software Developers who like programming, enjoy the craft etc. Not to say that such people aren't good at their craft; just that they're more vocational: they understand certain technologies and know many different ways to configure them which is what businesses need and pay them good money for.
It can also be a stressful one if the tech you're familiar with goes out of style. Lots of re-training involved, years of experience is basically worthless, etc.
It can also be a stressful one if the tech you're familiar with goes out of style. Lots of re-training involved, years of experience is basically worthless, etc.
Yeah, when I first encountered the field I was quite confused. I’d interview developers who had worked for years, but only knew about really weirdly narrow areas of SAP or whatever and definitely couldn’t display the kinds of skills that we usually evaluate for. It’s so different as to feel like a completely different field; we just happen to share titles.
A company I worked at bought Firewire driver code from them (this was 15-20 years ago). The code was really, really terrible and we eventually had to replace it with our own code (which was 20x shorter, faster, and more spec compliant).
In retrospect, and as much as we in engineering hated their code, it was probably worth what they paid for it just because it got us 75% there and took pressure off the team while we created the replacement code.
In retrospect, and as much as we in engineering hated their code, it was probably worth what they paid for it just because it got us 75% there and took pressure off the team while we created the replacement code.
A few years back I met with a few engineers working for Wipro. They were designing the dashboard electronics behind the new BMW 5 or 7 series. I was shocked to find that out at the time. Found some news about it here in case someone is curious on more info: https://www.wipro.com/newsroom/press-releases/archives/wipro...
Wipro has over 1200 clients worldwide.
From their Q4 FY18 presentation: https://www.wipro.com/content/dam/nexus/en/investor/quarterl...
Supposedly they are used by 150 of the Fortune 500.
From their Q4 FY18 presentation: https://www.wipro.com/content/dam/nexus/en/investor/quarterl...
Supposedly they are used by 150 of the Fortune 500.
I hope that this will start reducing over time. They clearly lied and dissembled in that conference call.
I wonder if it's because it's multiple layers down the "company stack." Company A hires company B hires company C and C uses Wipro.
It's a fun sight when all of the subcontractor of subcontractor of subcontractor recursion chains form a beanstalk into the heavens /peanutgallery
non-"tech" companies use Wipro etc for their websites and internal IT tools. All the stuff that's too boring or custom for Microsoft and Google etc.
A lot of places also contract out their server administration or firewall administration to places like Wipro. A fraction of a shared resource is a lot cheaper than several full time employees.
I've never seen anyone happy with the arrangement, management is merely satisfied by the cost savings. Quality is far (far) lower, but price is far lower as well.
I've never seen anyone happy with the arrangement, management is merely satisfied by the cost savings. Quality is far (far) lower, but price is far lower as well.
[deleted]
People who do RFPs for projects.
Sometimes I think I'm pretty hip to the jargon but every once in a while I learn a new one. IoC is an "Indicator of compromise."
The context was indeed all wrong for "Inversion of Control" but I wasn't going to ask the Urban Dictionary what it meant at work.
Afaik I it’s a pretty common term. Altering/monitoring teams often times hit up product security teams for IoCs they can use in their alerting implementation.
It is common; everything in security is an ambiguous acronym.
"C-and-C", for "command-and-control," is particularly vague and has multiple presentations-- C+C (Music Factory? An equation?), CnC (machining for materials fabrication), C2 (tutoring? Dicarbons?), C&C (a law firm?), CC (credit card).
APT (advanced persistent threat or Aptitude package manager?) is also annoying.
If you're curious, a whole bunch of security-related acronyms are reflected in the various product names and descriptions here:
https://github.com/hslatman/awesome-threat-intelligence
"C-and-C", for "command-and-control," is particularly vague and has multiple presentations-- C+C (Music Factory? An equation?), CnC (machining for materials fabrication), C2 (tutoring? Dicarbons?), C&C (a law firm?), CC (credit card).
APT (advanced persistent threat or Aptitude package manager?) is also annoying.
If you're curious, a whole bunch of security-related acronyms are reflected in the various product names and descriptions here:
https://github.com/hslatman/awesome-threat-intelligence
C&C also used to stand for the old game series Command & Conquer.
[deleted]
It is a common term - in the infosec world... outside of that, not so much.
I wonder what percentage of HN readers are on alerting/monitoring terms!
This place worse than most, tends to eschew in-jokes and memes as marks of belonging and membership, and rely instead on unexplained jargon.
This place worse than most, tends to eschew in-jokes and memes as marks of belonging and membership, and rely instead on unexplained jargon.
> This place worse than most, tends to eschew in-jokes and memes as marks of belonging and membership, and rely instead on unexplained jargon.
The unexplained jargon isn't any worse here than anywhere else. Everything from motorcycling to woodworking to sewing has its own arcane terminology for simple concepts to ostracize the unlearned.
Yes, you too can get icy stares from a table full of middle-aged women at the sewing club for failing to be born with the knowledge that "the thing that does edges" is actually called a "serger."
The unexplained jargon isn't any worse here than anywhere else. Everything from motorcycling to woodworking to sewing has its own arcane terminology for simple concepts to ostracize the unlearned.
Yes, you too can get icy stares from a table full of middle-aged women at the sewing club for failing to be born with the knowledge that "the thing that does edges" is actually called a "serger."
Overlocker when not in the US.
I found it helped me that "indicators of compromise" is written out in full in the relevant paragraph, and then just abbreviated in the summary.
It's rough if you're skimming though (and who has time to read everything on the web?)
It's rough if you're skimming though (and who has time to read everything on the web?)
Very common in Incident Response and Network Security Monitoring.
How not to report about how not to acknowledge a data breach includes dubbing silly music over the other guy speaking to further prove your smugness. While it is always fun to say I told you so, you can still be professional about it.
If I’m not mistaken, the video was made by someone else.
Correct, the video was made by:
https://www.youtube.com/channel/UCc5jsl5zRbbGbXO0AB4aW4w
Graham Cluley
Graham Cluley
[deleted]
[deleted]
The only reason these kinds of companies continue to exist is the continuing stupidity of American Executives who will go to any lengths to not pay a decent wage for their Software operations. Fuck them, and they deserve exactly this for fucking over good quality talent.
Nobody would be surprised if a bridge built by cheapo contractors failed. Its much the same way with Software.