Breach at IT Outsourcing Giant Wipro(krebsonsecurity.com)
krebsonsecurity.com
Breach at IT Outsourcing Giant Wipro
https://krebsonsecurity.com/2019/04/experts-breach-at-it-outsourcing-giant-wipro/
70 comments
A breach that involves a technical vulnerability is one of those things that happens, gets patched and everyone moves on.
One of the things when establishing a contract with a company like Wipro is that they operate on your systems from locked down rooms where disks, thumb drives, etc. are not allowed. A secure private link is setup to your corporate environment to ensure that your customer's data is not available to the outsourced firm to do with as they please.
A breach on Wipro which allowed an attacker to gain access to 11 customer systems (i.e. maybe some Fortune 500 companies) to me means they should pretty much go bankrupt because any sane customer will stop doing business with them as soon as they can. It speaks of incompetence and complete disregard for common safeguards. How can a breach into Wipro's corporate system in any way lead an attacker to Wipro's customer's data? An obvious one could be their employees sending customer credentials via the Wipro corporate email.
One of the things when establishing a contract with a company like Wipro is that they operate on your systems from locked down rooms where disks, thumb drives, etc. are not allowed. A secure private link is setup to your corporate environment to ensure that your customer's data is not available to the outsourced firm to do with as they please.
A breach on Wipro which allowed an attacker to gain access to 11 customer systems (i.e. maybe some Fortune 500 companies) to me means they should pretty much go bankrupt because any sane customer will stop doing business with them as soon as they can. It speaks of incompetence and complete disregard for common safeguards. How can a breach into Wipro's corporate system in any way lead an attacker to Wipro's customer's data? An obvious one could be their employees sending customer credentials via the Wipro corporate email.
HN has a predominant US reader base. Its natural that they compare everything else in the world vs US. For ex: If China excels in bullet trains, a common rhetoric will be that they copied something from the US, China's human rights issues and then goes on to say about some failed bullet train project in California. I am not sure how those discussions invokes "intellectual curiosity", but still gets to the top of HN. I got sick of this. So I created a filter with some ML out of HN RSS feeds which filters posts that it thinks is not on par with "intellectual curiosity" theme. The filter has shown great improvements as I continue to give it valid inputs. This includes all those toxic bashing like in this one.
Can any of the security folks on here tell me what good secure systems really look like? If I wanted to build a company infrastructure from scratch what would "default secure"
look like? I am fairly sure I know what a good software engineering process looks like, but if I guessed a secure infrastructure I would be concerned I am
missing basics. (Hence no examples to get us started)
Necessary but not sufficient:
- Services do transport encryption, authentication, and authorization even to internal callers; presence on the LAN does not confer any privileged access.
- Identity and permissions that are user-friendly enough not to encourage workarounds, sophisticated enough to implement principle of least privilege, centralized enough to keep up to speed with personnel changes (including instant lockout on termination).
- Applications are developed with an awareness of common vulnerabilities and associated defenses (i.e. OWASP top 10). Sensitive applications are vetted by security researchers for high-end vulnerabilities. Off the shelf applications and libraries are kept up to date.
- Current ground truth and complete change history for everything about production is known and documented. Something like Puppet manifests in Git comprehensively describe everything that has been done to a VM from its baseline image, every rule in a hardware firewall, etc.
- Activities are logged; logs are correlated and analyzed for suspicious activity; alerts are promptly and competently investigated; credentials found to be compromised can be easily rotated; systems found to be vulnerable can be quickly patched or isolated.
- A quorum of insiders is required for especially sensitive operations; this is not just policy or code, but backstopped by math and hardware. At the root of trust you will find things like Shamir's Secret Sharing, HSMs, signing ceremonies, etc. and not a dude with a private key on his workstation.
I'm by no means a security expert, but as far as I'm aware, this stuff will make you a harder target than most.
- Services do transport encryption, authentication, and authorization even to internal callers; presence on the LAN does not confer any privileged access.
- Identity and permissions that are user-friendly enough not to encourage workarounds, sophisticated enough to implement principle of least privilege, centralized enough to keep up to speed with personnel changes (including instant lockout on termination).
- Applications are developed with an awareness of common vulnerabilities and associated defenses (i.e. OWASP top 10). Sensitive applications are vetted by security researchers for high-end vulnerabilities. Off the shelf applications and libraries are kept up to date.
- Current ground truth and complete change history for everything about production is known and documented. Something like Puppet manifests in Git comprehensively describe everything that has been done to a VM from its baseline image, every rule in a hardware firewall, etc.
- Activities are logged; logs are correlated and analyzed for suspicious activity; alerts are promptly and competently investigated; credentials found to be compromised can be easily rotated; systems found to be vulnerable can be quickly patched or isolated.
- A quorum of insiders is required for especially sensitive operations; this is not just policy or code, but backstopped by math and hardware. At the root of trust you will find things like Shamir's Secret Sharing, HSMs, signing ceremonies, etc. and not a dude with a private key on his workstation.
I'm by no means a security expert, but as far as I'm aware, this stuff will make you a harder target than most.
Foucs... You can't protect everything, but can ensure that handling of at least some truly important data is as paranoid as it can be.
Protecting a company's FTP server that is open to thousands of employees is not a doable task, for example
And the same is true of human knowledge, a company saying that all and everything within its walls is super secret, can't truly hold anything secret.
At one of my first jobs in Canada, an owner of the company was very clear on the point what is a commercial secret and what isn't. Whenever there was a meeting genuinely demanding it, he clearly stated at the start "this is a commercial secret covered by confidentiality agreement."
Protecting a company's FTP server that is open to thousands of employees is not a doable task, for example
And the same is true of human knowledge, a company saying that all and everything within its walls is super secret, can't truly hold anything secret.
At one of my first jobs in Canada, an owner of the company was very clear on the point what is a commercial secret and what isn't. Whenever there was a meeting genuinely demanding it, he clearly stated at the start "this is a commercial secret covered by confidentiality agreement."
Focus* typo...
Corporate mindset understanding the importance of security is more important than "implement LAPS" or "follow modern password policy guidelines instead of ones from 2002".
If you look at a lot of the big breaches, they have some pretty common patterns. Old operating systems (XP, 7, etc.), unpatched software, excessive vendor access. This isn't because they don't have money to manage these things, it's that other business priorities with immediately visible results have taken priority. "This business software sales needs was designed for Windows XP" takes prioritization over "It's unsafe to use anything older than Windows 10 on our network". If another department and IT have a conflict, the other department wins because it brings in revenue.
If you have people in the chain of authority above IT who support IT, and understand that securing your infrastructure prevents catastrophes on the same level as fires and the PR disasters, you will generally do much better than businesses who don't. People need to understand that IT/security personnel are not "annoying" them, but trying to help them avoid catastrophes they don't even understand.
If you look at a lot of the big breaches, they have some pretty common patterns. Old operating systems (XP, 7, etc.), unpatched software, excessive vendor access. This isn't because they don't have money to manage these things, it's that other business priorities with immediately visible results have taken priority. "This business software sales needs was designed for Windows XP" takes prioritization over "It's unsafe to use anything older than Windows 10 on our network". If another department and IT have a conflict, the other department wins because it brings in revenue.
If you have people in the chain of authority above IT who support IT, and understand that securing your infrastructure prevents catastrophes on the same level as fires and the PR disasters, you will generally do much better than businesses who don't. People need to understand that IT/security personnel are not "annoying" them, but trying to help them avoid catastrophes they don't even understand.
Site down ATM, Google Cache doesn't have it, but good old Wayback machine has it: https://web.archive.org/web/20190415214511/https://krebsonse...
> "The company has robust internal processes and a system of advanced security technology in place to detect phishing attempts and protect itself from such attacks."
Somehow I don't think this is a phishing attack.
Somehow I don't think this is a phishing attack.
>”Wipro’s systems were seen being used as jumping-off points for digital fishing expeditions targeting at least a dozen Wipro customer systems.“
Well, it looks like it was used as a launching point to phish their (Wipro’s) clients. They probably had s pretty good catch. I imagine one of their clients alerted them to the issues.
Well, it looks like it was used as a launching point to phish their (Wipro’s) clients. They probably had s pretty good catch. I imagine one of their clients alerted them to the issues.
Confirms phishing. https://www.reuters.com/article/us-wipro-cybercrime/indias-w...
Humans are the weakest link
Might be interesting to annotate the biggest data breaches with each victim's outsourcing partner.
https://en.wikipedia.org/wiki/List_of_data_breaches
https://en.wikipedia.org/wiki/List_of_data_breaches
I know of at least one very large customer where management of the VPN appliance and firewall controlling access of Wipro vendors was outsourced... to Wipro.
I had a problem with traffic black holing all of a sudden everything going nowhere with Wipro once.
What we found was that suddenly a some ports with traffic distended for the internet bounced and the black holing started. A Wipro rep proudly announced that they had caused the port bouncing. You see they found that someone (Wipro... but they didn't know it was their own people at this point) had cabled around the firewalls a year earlier because something wasn't working, and never hooked them back up.
So now we were cabled to the firewalls properly... the firewalls that as far as anyone could tell had nonsensical configs that all pointed to a null route.
So for at least a year, maybe more, there was no firewall. The end customer was a large financial institution.
What we found was that suddenly a some ports with traffic distended for the internet bounced and the black holing started. A Wipro rep proudly announced that they had caused the port bouncing. You see they found that someone (Wipro... but they didn't know it was their own people at this point) had cabled around the firewalls a year earlier because something wasn't working, and never hooked them back up.
So now we were cabled to the firewalls properly... the firewalls that as far as anyone could tell had nonsensical configs that all pointed to a null route.
So for at least a year, maybe more, there was no firewall. The end customer was a large financial institution.
"India’s third-largest IT outsourcing company — was dealing with a multi-month intrusion from an assumed state-sponsored attacker.
"Both sources, who spoke on condition of anonymity, said Wipro’s systems were seen being used as jumping-off points for digital fishing expeditions targeting at least a dozen Wipro customer systems."
Well, I suppose it's the step you'd expect but a state-actor engaging in a broad fishing trip still seems like a new thing. Can we expect whatever state will be installing their official botnet in whatever country next?
Well, I suppose it's the step you'd expect but a state-actor engaging in a broad fishing trip still seems like a new thing. Can we expect whatever state will be installing their official botnet in whatever country next?
Whelp, life comes at you fast: https://www.bankinfosecurity.asia/interviews/wipros-new-ciso...
Dude has 11 industry-recognized certifications in the domains of IT, information security, security framework and secure enterprise architecture.
So I guess Wipro will have everything under control.
So I guess Wipro will have everything under control.
My lord the "frictionless" buzzword BS in that made me feel like I was actually losing brain cells:
> He offers insights on:
> How privileged access management will fit into a frictionless security approach; > The role of artificial intelligence, machine learning and blockchain in enabling frictionless security and faster threat detection;
I seriously considered registering justaddblockchain.com after reading that to document all the places where people feel they look smarter by just adding "blockchain!" to random technical discussions.
> He offers insights on:
> How privileged access management will fit into a frictionless security approach; > The role of artificial intelligence, machine learning and blockchain in enabling frictionless security and faster threat detection;
I seriously considered registering justaddblockchain.com after reading that to document all the places where people feel they look smarter by just adding "blockchain!" to random technical discussions.
Holy crap... That's not what frictionless means. That's literally the definition of appropriate levels of security based on risk profile, but even high levels of security should be frictionless, i.e. they should work the same way every time and should be relatively easy to do, which is what 90% of MFA apps are. What's not frictionless is when your security team needlessly causes you to change your very secure password every 90 days because someone told them that was a good idea. It leads to sticky notes and easily cracked passwords that use formulas i.e. Spring2019! Summer2019! And so on like in so many corporations due to poorly thought out security implementation. Longer time before forced password changes coupled with either a physical key or software token that don't have to be remembered are way better.
"State-sponsored attacker": A euphemism to deflect blame for your own inadequate security practices.
The attacker is happily sponsored by the state of affairs at the company.
This surprises me... not one bit.
I have yet to find the first multibillion outsourcing company that was any good.
All the great people are immediately hired by the company they’re actually working for, so they’re always left with the mediocre and terrible.
I have yet to find the first multibillion outsourcing company that was any good.
All the great people are immediately hired by the company they’re actually working for, so they’re always left with the mediocre and terrible.
I have some friends who were in an Indian outsourcing company which shall remain nameless. One of them described how they had an "A Team" which was made up of smart, sharp individuals. When they wanted a contract, they'd send this A-team to really impress the clients. Once they had the contract in hand, this A-team would be tasked to get the next contract, and replaced with run-of-the-mill average (or even below average) talent to save on the cost. Rinse, lather repeat.
I don't think that's an Indian outsourcing issue, so much as an artifact of the dynamics of the consulting environment.
Your above average talent gets trusted with high stakes, client-visible scenarios like new business development, RFP pitches, and high-visibility fire fighting. The roles tend to be a combination of subject matter expertise, solutions architect, client management, and sales.
Once you have a contract in hand, theoretically the engagement is structured at that point. It no longer needs the independent competence of members of your A Team, but rather competent project/program management, client/engagement management, and execution resources.
That's the stage where a lot of problems get introduced, and it's not always a result of cost cutting or profitability measures. A lot of it comes down to operational competency (both for the agency and the client), as well as the structure of the contract itself.
The biggest thing I've learned in my time consulting is what to do (and not do) when structuring a contract for a consulting/outsourcing engagement. Everything else cascades from there.
Your above average talent gets trusted with high stakes, client-visible scenarios like new business development, RFP pitches, and high-visibility fire fighting. The roles tend to be a combination of subject matter expertise, solutions architect, client management, and sales.
Once you have a contract in hand, theoretically the engagement is structured at that point. It no longer needs the independent competence of members of your A Team, but rather competent project/program management, client/engagement management, and execution resources.
That's the stage where a lot of problems get introduced, and it's not always a result of cost cutting or profitability measures. A lot of it comes down to operational competency (both for the agency and the client), as well as the structure of the contract itself.
The biggest thing I've learned in my time consulting is what to do (and not do) when structuring a contract for a consulting/outsourcing engagement. Everything else cascades from there.
> That's the stage where a lot of problems get introduced
At enterprise scale it comes down to how well the consulting team can understand and worm their way into the business processes. They need an inside ally who can enable the consultants to navigate the silos, and not just some internal PHB barking orders.
Every enterprise org chart has a different flavor of dysfunction.
At enterprise scale it comes down to how well the consulting team can understand and worm their way into the business processes. They need an inside ally who can enable the consultants to navigate the silos, and not just some internal PHB barking orders.
Every enterprise org chart has a different flavor of dysfunction.
I worked at a consulting/product agency where the requirement was always to work at the same place as the client, and to have a client stakeholder embedded within directly within the team. The projects that felt the most successful were the ones where we got these requirements really right.
I think the problem stems more from the nature of technology itself.
I run a marketing consultancy. I can't really offload my work to cheaper talent because my deliverables can be easily evaluated. Good writing and good design, after all, are universally accessible.
But unless you're a programmer, you can't really make out good code from bad code.
I run a marketing consultancy. I can't really offload my work to cheaper talent because my deliverables can be easily evaluated. Good writing and good design, after all, are universally accessible.
But unless you're a programmer, you can't really make out good code from bad code.
It's not only that. The main problem is that it's virtually impossible to come up with specifications, which are precise enough that they can be implemented 1:1.
Outsourcing companies will aim for the lowest common denominator, which the contract allows and with which they can get away with.
Case in point? I can exactly tell you what the specs for the first time reporting system I worked with at MajorCorp said, without ever having seen those specs:
Must be able to enter hours
That's exactly what we could do. Enter hours. Not minutes, not ten minute intervals, not quarter - or half hours, no; hours.
When I get such a spec I hit back with questions. Send that to an outsourcer and they won't see any need to provide a yota more than what the spec says.
In other words: You entering a world of pain, which looks good on spreadsheet but really only that.
Outsourcing companies will aim for the lowest common denominator, which the contract allows and with which they can get away with.
Case in point? I can exactly tell you what the specs for the first time reporting system I worked with at MajorCorp said, without ever having seen those specs:
Must be able to enter hours
That's exactly what we could do. Enter hours. Not minutes, not ten minute intervals, not quarter - or half hours, no; hours.
When I get such a spec I hit back with questions. Send that to an outsourcer and they won't see any need to provide a yota more than what the spec says.
In other words: You entering a world of pain, which looks good on spreadsheet but really only that.
> But unless you're a programmer, you can't really make out good code from bad code.
That's not always true. We recently dropped a small consulting company (~20 people, US based) that was awarded a small (but not necessarily trivial) project and really bungled things up. The end product was delayed numerous times, noticeably sluggish and riddled with bugs. It would have been an easy win for this company and would have led to more recurring work, but for whatever reason, they decided to assign the project to a single junior programmer without any oversight.
That's not always true. We recently dropped a small consulting company (~20 people, US based) that was awarded a small (but not necessarily trivial) project and really bungled things up. The end product was delayed numerous times, noticeably sluggish and riddled with bugs. It would have been an easy win for this company and would have led to more recurring work, but for whatever reason, they decided to assign the project to a single junior programmer without any oversight.
I worked for a company who outsourced a few times and found that the cost was actually higher in some cases and as you describe there would often be immediate turnover to folks who were far worse, and at worst a liability.... and even those guys would turn over all the time. They'd try another company, same result.
Most confusing was when things were straight horrible ... there was nothing anyone can do. I don't know if the contracts say "you give us your money and just deal with whatever we give you" but that's how people would act. The outsourced company could do all sorts of absurd things that would have serious consequences had it occurred with domestic support / new procedures and etc. But when the outsource company did it and it was as plain as day "nothing we can do about it".
Most confusing was when things were straight horrible ... there was nothing anyone can do. I don't know if the contracts say "you give us your money and just deal with whatever we give you" but that's how people would act. The outsourced company could do all sorts of absurd things that would have serious consequences had it occurred with domestic support / new procedures and etc. But when the outsource company did it and it was as plain as day "nothing we can do about it".
That's part of the business plan of most consulting busineses and especially of big name consulting companies.
Principal consultants, who actually really know their shit, are called in for the contract negotiations. Those are of course not the folks, which you get to work on the project.
The actual contractors are usually rather junior, often fresh of a plane and range in quality from quite good to absolutely abyssimal. With big name consulting companies they can still be billed at a daily rate of 2K +.
In other words, and less politely put, the consulting business is pretty much a bait and switch.
Principal consultants, who actually really know their shit, are called in for the contract negotiations. Those are of course not the folks, which you get to work on the project.
The actual contractors are usually rather junior, often fresh of a plane and range in quality from quite good to absolutely abyssimal. With big name consulting companies they can still be billed at a daily rate of 2K +.
In other words, and less politely put, the consulting business is pretty much a bait and switch.
This is a pretty standard approach, not just an "Indian outsourcing company thing".
eg I used to do contract work with Fujitsu some years ago, as one of the implementation people in their projects team. We'd get things set up and working well, document the heck out of it for the ops team, and then hard it over to said ops team for daily running.
It's done this way so the more experienced / creative solution finding types ;) (eg expensive) can be engaged to solve the initial problem + implement the solution. Once that's working it just needs people to run the daily playbook and look after minor issues that crop up.
eg I used to do contract work with Fujitsu some years ago, as one of the implementation people in their projects team. We'd get things set up and working well, document the heck out of it for the ops team, and then hard it over to said ops team for daily running.
It's done this way so the more experienced / creative solution finding types ;) (eg expensive) can be engaged to solve the initial problem + implement the solution. Once that's working it just needs people to run the daily playbook and look after minor issues that crop up.
I worked with some other outsourcing companies from other part of the world, and seen exactly this pattern: The initial team was sharp and smart, to gain stakeholder confidence, they stayed for a few months, setting up demo and prototype. Later they would slowly bring in other team members and started to withdraw themselves from the project. Eventually, those A-team was gone.
Those late-joiners had absolutely no clue on what was going on, and also didn't have the competency at all to hold the helm. Result? A much mediocre rewrite of the whole platform which never fulfilled the requirements.
Those late-joiners had absolutely no clue on what was going on, and also didn't have the competency at all to hold the helm. Result? A much mediocre rewrite of the whole platform which never fulfilled the requirements.
I’ve seen the version of that where one individual does the phone interview, and a different one shows up on-site.
The consulting agency was prospecting for enterprise clients who would not pay attention close enough to such details.
The consulting agency was prospecting for enterprise clients who would not pay attention close enough to such details.
They problem is that the "A team" can impress clients by cutting corners, rushing stuff and applying fairly generic solutions that won't work long term as requirements get more specific and more data goes in. Then they've moved on to new things before having a chance to learn from their mistakes.
This is the story of literally every single B2B service company in every industry I have dealt with (media, IT, ad-tech, analytics, catering)
Basically any industry where manpower cost is the biggest cost and you have a fixed-ish price B2B contract, the only way to squeeze out a higher margin is to lower manpower cost.
Basically any industry where manpower cost is the biggest cost and you have a fixed-ish price B2B contract, the only way to squeeze out a higher margin is to lower manpower cost.
That does not prove one bit that the A team was technically performant. It just proves that bullshit sells.
It happens to every company. And – so my pet theory goes – this is inevitable.
Any company over size x is, by definition, staffed by average people. That’s the very definition of average. What, then, would lead anyone to believe that a sufficiently large company could possibly be full of geniuses? It just can’t happen.
(FWIW this is why I find the companies that do seem to break the mould – Apple, for example – so interesting.)
Any company over size x is, by definition, staffed by average people. That’s the very definition of average. What, then, would lead anyone to believe that a sufficiently large company could possibly be full of geniuses? It just can’t happen.
(FWIW this is why I find the companies that do seem to break the mould – Apple, for example – so interesting.)
Apple surely has plenty of average people too. Arguably the key to operating a large business is to figure out how to optimize the placement and tasking of your geniuses/motivated people with vision/whatever for the greatest gains whilst using the average staff effectively to keep the general workload off of said individuals.
> Any company over size x is, by definition, staffed by average people.
Average for that company, not average across profession.
> What, then, would lead anyone to believe that a sufficiently large company could possibly be full of geniuses? It just can’t happen.
It absolutely can and should (though it's another question why it rarely does). It requires expending continuous effort though - effort to bring in above-average people, to raise the level of people on-board so they stay above average, and to get rid of people that are below average.
This is actually what life itself does. The default, lowest-energy state of matter is a mixed and utterly uninteresting soup of everything. Living things continuously expend energy to maintain an organized, higher-energy state against the threat of average. They continuously pump entropy out.
Average for that company, not average across profession.
> What, then, would lead anyone to believe that a sufficiently large company could possibly be full of geniuses? It just can’t happen.
It absolutely can and should (though it's another question why it rarely does). It requires expending continuous effort though - effort to bring in above-average people, to raise the level of people on-board so they stay above average, and to get rid of people that are below average.
This is actually what life itself does. The default, lowest-energy state of matter is a mixed and utterly uninteresting soup of everything. Living things continuously expend energy to maintain an organized, higher-energy state against the threat of average. They continuously pump entropy out.
I'm not going to debate whether any "companies" are good, but there are more than a few "deals" that have worked very well for the client. Good deals require a lot factors to get right.
I always wondered why, and I came to following conclusion:
1. Orgs outsourcing projects are optimising on costs.
2. Outsourcing IT companies are optimising on margins(nearly the same thing).
If orgs outsourcing projects start optimising on quality of deliverables or security, or speed of delivery, we will see drastic change in behaviour of these IT companies.
IT companies will be better incentivised to hire smart graduates, invest in trainings etc.
If orgs outsourcing projects start optimising on quality of deliverables or security, or speed of delivery, we will see drastic change in behaviour of these IT companies.
IT companies will be better incentivised to hire smart graduates, invest in trainings etc.
Maybe, they could pay market price than? Oh, the only target is to be cheap.. Okay then
To be fair, the OP was referring to all large outsourcing companies. There are a lot of outsourcing companies that charge well over the rate for a full time hire (they don't necessarily pay their employees that, but...) I worked briefly for one such "body shop" and I found that the company billed my time at a rate of about twice what they paid me. As far as I can tell, it tends to slot in between a full time employee and an independent consultant/contractor in terms of price.
Why do companies pay this premium? For one, it may be because they have a budget for a "6 month project" (wink, wink, nudge, nudge) but not for a full time hire. Sometimes it's because they think that they'll be able to pick and choose the best developers from inside a large organisation (and sometimes it works out that way... almost never, but sometimes...) But really it's just a matter of being naive, I think.
The company I used to work for (as an independent contractor... ahem...) went through a phase where the CEO was absolutely convinced that it didn't pay to hire developers -- they were a PITA (always demanding weird things, never showing up before 10:30 am, never dressing properly for work, etc, etc). They always wanted raises and you have to track progress and negotiate with them on price every year. You have to pay benefits and if a person is on disability on maternity, you've got to do something about it. You constantly need to be reviewing CVs, interviewing and hiring. If you get someone really bad, you can't actually reasonably fire them for a long time. Etc, etc, etc. So he tried a few experiments with hiring from outsourcing companies (all of which were charging considerably more than any of us were getting paid, I will add)... and it wasn't a disaster, but it wasn't really any better. But the cherry on the cake was, as soon as the project was done, they fled and no amount of pleading could get those developers back again (as you can imagine). So the CEO realised that having a crack development team was definitely worth the effort and the company really transformed as a result.
However, a lot of companies are still stuck in the mindset where it's best to have as few full time developers as possible.
Why do companies pay this premium? For one, it may be because they have a budget for a "6 month project" (wink, wink, nudge, nudge) but not for a full time hire. Sometimes it's because they think that they'll be able to pick and choose the best developers from inside a large organisation (and sometimes it works out that way... almost never, but sometimes...) But really it's just a matter of being naive, I think.
The company I used to work for (as an independent contractor... ahem...) went through a phase where the CEO was absolutely convinced that it didn't pay to hire developers -- they were a PITA (always demanding weird things, never showing up before 10:30 am, never dressing properly for work, etc, etc). They always wanted raises and you have to track progress and negotiate with them on price every year. You have to pay benefits and if a person is on disability on maternity, you've got to do something about it. You constantly need to be reviewing CVs, interviewing and hiring. If you get someone really bad, you can't actually reasonably fire them for a long time. Etc, etc, etc. So he tried a few experiments with hiring from outsourcing companies (all of which were charging considerably more than any of us were getting paid, I will add)... and it wasn't a disaster, but it wasn't really any better. But the cherry on the cake was, as soon as the project was done, they fled and no amount of pleading could get those developers back again (as you can imagine). So the CEO realised that having a crack development team was definitely worth the effort and the company really transformed as a result.
However, a lot of companies are still stuck in the mindset where it's best to have as few full time developers as possible.
Programming is still a very complex craft and requires sensible people with reasonable skill levels to implement reliable systems. This seems to be totally ignored by most non-tech American companies who treat Software much the same way as, say, buying a phone service. There are a lot of American businesses that are suffering because of this and its sad but I'm not sure how they can be convinced that having good programmers who are compensated reasonably well is such an important thing to have.
Maybe I'm saying this from the perspective of a programmer and not a business owner, IDK. Cloud stuff seems promising: a nice way to deliver turnkey solutions instead of custom crap that needs an outsourced body shop to maintain.
Maybe I'm saying this from the perspective of a programmer and not a business owner, IDK. Cloud stuff seems promising: a nice way to deliver turnkey solutions instead of custom crap that needs an outsourced body shop to maintain.
I think a lot of it is that companies don't see software as something you need to sustain. They see it as a tool that helps them with the rest of their business. It makes sense to them to go out an buy that tool. For example, if you are a brewery and you need kegs, you go out and buy kegs. You don't hire a whole bunch of coopers and devote a fair percentage of your business building barrels. That's seen as an old fashioned, wasteful way to do business. Unless software is your actual product, why do you have a team building it? You might even buy custom built kegs, but once you have them, you don't need a team building kegs -- you just need to hire someone occasionally to replace old ones.
I think from that perspective it makes total sense. Unfortunately, software is an embodiment of your business requirements and that you business requirements will probably be changing from day to day. Once you realise that, you can see that you need a team to be constantly changing the software.
In bonsai (trees in pots) there is a saying that a bonsai is only "finished" when it is dead -- because it keeps growing and changing. You can never say, "I'm done" because tomorrow the tree is different. I think this is also true of software: it's only "done" when nobody is using it any more (or, I suppose, when it's so unimportant that nobody cares that it isn't correct).
I think until we get the singularity, we will always need "programmers" -- people who translate the business requirements into something that the computer can act on. In fact, it's mostly thinking about the requirements that is important. The actual encoding of the computer instructions is (hopefully!) not that difficult.
I think from that perspective it makes total sense. Unfortunately, software is an embodiment of your business requirements and that you business requirements will probably be changing from day to day. Once you realise that, you can see that you need a team to be constantly changing the software.
In bonsai (trees in pots) there is a saying that a bonsai is only "finished" when it is dead -- because it keeps growing and changing. You can never say, "I'm done" because tomorrow the tree is different. I think this is also true of software: it's only "done" when nobody is using it any more (or, I suppose, when it's so unimportant that nobody cares that it isn't correct).
I think until we get the singularity, we will always need "programmers" -- people who translate the business requirements into something that the computer can act on. In fact, it's mostly thinking about the requirements that is important. The actual encoding of the computer instructions is (hopefully!) not that difficult.
> In fact, it's mostly thinking about the requirements that is important. The actual encoding of the computer instructions is (hopefully!) not that difficult.
Jesus please don't draw conclusions about things you have no idea about. Would you say that designing and developing machines that replace humans bit by bit is not difficult (hopefully!).
Jesus please don't draw conclusions about things you have no idea about. Would you say that designing and developing machines that replace humans bit by bit is not difficult (hopefully!).
> Jesus please don't draw conclusions about things you have no idea about
That's a curious choice of wording. I have considerable experience encoding requirements as computer programs. According to the latest Stack Overflow survey, I seem to be in the 99th percentile in terms of coding experience ;-) I find that getting the requirements right is dramatically more difficult than encoding the instructions for the computer (even if you happen to write a big ball of hairy code).
Consider that we have 3 types of errors that can occur in software: errors in requirements, errors in translating those requirements to code, and errors in expressing that code. In other words, we can build the wrong system because we got the requirements wrong, we can build the wrong system because our code doesn't implement the requirements that we gathered, and we can have errors where the implementation has effects that we didn't consider.
There is a reasonable chance that we can build systems that given requirements will encode them into working code. This can take care of the second 2 classes of errors. However, if you look at where most of your time as a developer goes to: it is rewriting the code where the requirements were insufficient or incorrect. Until we have an AI system that can replicate human levels of thinking we won't be able to go from "I need a sales report" to code because there is no way for the system to reason about what should be in a sales report -- and especially no way to ask relevant questions about the business to write a sales report that is good for that business.
No matter how good we get at automatically translating requirements to code, you are still going to need a person to gather the requirements and think in excruciating detail about exactly what you need. You may have noticed that normal people can't do that job. They will say, "I need a sales report" and you say "What should it contain?" -- the answer will be "The sales information". Keeping track of all the mind-boggling little details and exactly how it will affect the rest of the system is still going to be a job for a human -- until we can build a system that is capable as a human (and not just any human -- a human that is able to think very hard about requirements). This will not happen in my lifetime. It might not even happen in your lifetime (which by the very nature of statistics is likely to be longer than mine ;-) ).
The "hopefully!" part was meant to be a nod to the occasional situation where people build systems that are far more complicated than the problem they are solving. Hopefully you are not building such a system ;-)
I hope that cleared up what I was trying to say. I have to admit that I am not very clear on what you were trying to say.
That's a curious choice of wording. I have considerable experience encoding requirements as computer programs. According to the latest Stack Overflow survey, I seem to be in the 99th percentile in terms of coding experience ;-) I find that getting the requirements right is dramatically more difficult than encoding the instructions for the computer (even if you happen to write a big ball of hairy code).
Consider that we have 3 types of errors that can occur in software: errors in requirements, errors in translating those requirements to code, and errors in expressing that code. In other words, we can build the wrong system because we got the requirements wrong, we can build the wrong system because our code doesn't implement the requirements that we gathered, and we can have errors where the implementation has effects that we didn't consider.
There is a reasonable chance that we can build systems that given requirements will encode them into working code. This can take care of the second 2 classes of errors. However, if you look at where most of your time as a developer goes to: it is rewriting the code where the requirements were insufficient or incorrect. Until we have an AI system that can replicate human levels of thinking we won't be able to go from "I need a sales report" to code because there is no way for the system to reason about what should be in a sales report -- and especially no way to ask relevant questions about the business to write a sales report that is good for that business.
No matter how good we get at automatically translating requirements to code, you are still going to need a person to gather the requirements and think in excruciating detail about exactly what you need. You may have noticed that normal people can't do that job. They will say, "I need a sales report" and you say "What should it contain?" -- the answer will be "The sales information". Keeping track of all the mind-boggling little details and exactly how it will affect the rest of the system is still going to be a job for a human -- until we can build a system that is capable as a human (and not just any human -- a human that is able to think very hard about requirements). This will not happen in my lifetime. It might not even happen in your lifetime (which by the very nature of statistics is likely to be longer than mine ;-) ).
The "hopefully!" part was meant to be a nod to the occasional situation where people build systems that are far more complicated than the problem they are solving. Hopefully you are not building such a system ;-)
I hope that cleared up what I was trying to say. I have to admit that I am not very clear on what you were trying to say.
Sorry I'm dense but what is the 6 month contract wink wink nudge nudge?
It's a 6 month engagement that renews in perpetuity. Mainly as a way to avoid having employer obligations.
Also known as a Permalancer - who can also massively increase their take home pay and reduce their tax.
After the Microsoft lawsuit, those perpetual renewals pretty much dried up. Any company that still does it is taking a risk.
Take a tour in Europe, France or Switzerland, I know some contractors who have been in the same client, if not team (through various reorg) for more than two decades.
Myself,5y same client over a 7years period, each time in a different team or close to.
Wonder how long it'll last. Current government in Poland recently announced they're planning a crackdown on "permalancers", with a recent interview calling people regularly billing just one client "not true entrepreneurs", and therefore not entitled to privileges of sole proprietorship.
I've been doing this (not quite on the 5 year scale) in both UK and Australia.
Now Microsoft only allows contractors for 18 months in a given 24 month period (i.e. mandatory 6 month cooling-off period).
Google, Facebook, CIA, Pentagon... all have been hacked.
> All the great people are immediately hired by the company they’re actually working for, so they’re always left with the mediocre and terrible.
By "great people" I assume you mean the best talent. I assume you also mean that corporate politics (bootlicking, backstabbing, parasitic behaviour...) has no role to play in these hirings.
> All the great people are immediately hired by the company they’re actually working for, so they’re always left with the mediocre and terrible.
By "great people" I assume you mean the best talent. I assume you also mean that corporate politics (bootlicking, backstabbing, parasitic behaviour...) has no role to play in these hirings.
In my best Wipro-excuse voice: "... see, thing is:..."
... they have a process for everything. It's just a very convoluted, silo'd, circuitous process that takes several days to get anything done, with several steps including: throw a support ticket over the wall, and let a subprocess pick it up. On and on we go, until we have a lax process that is impossible to fix, on to which, all we can do is pump more, cheap, low skilled talent. But hey, it's creating a middle class somewhere.
[deleted]
[deleted]
What doesn’t surprise me is Wipro refusing to comment. This would have never come out if they weren’t targeting from inside the house.
would we classify this as a friction-less security breach?
A few years ago I worked with Wipro as they would contact me (technical support for some products) on behalf of their customers.
The incompetence was astounding, and I worked in support for a long time, and Wipro was really astounding. Everything from security to just understanding what we were telling them was mindbogglingly bad. It wasn't a language barrier, they simply didn't have many / sometimes anyone who understood the technology on the most basic level.
Wipro would open tickets dozens at a time claiming there was some sort of technical issue, but they often couldn't explain what if anything they tried. We would find the equipment at factory defaults, last boot time was when it was in the factory.... but now it was a P1 ticket because "it didn't work and it needs to be up and running by the end of the day". Then we'd ask what how they wanted it configured and they ... wouldn't know. Then they'd escalate through sales and the executives claiming we had been "working with them for weeks and were not helping".
Then they would go silent and not respond for days or weeks only to reappear later as angry as ever that we hadn't done anything when our last questions to them might be as simple as "what isn't working?".
It was worse when they actually tried configuring things as they were masters at nonsensical configurations, looping cables back into the same equipment they came from and etc. You could look at their systems that were "working" and it was errors everywhere and you couldn't trust anything you saw.
Even internally Wipro would tell us that they "can't tell" the "other team" (another team inside Wipro working with the same customer) that they need to change their configuration. They would just repeat that they can't tell them that ... and we'd be stuck because it's obvious the "other team" is configured wrong. I'd tell them to let me be the bad guy and tell them on a call, but nope. So things would just not work.
It was a common occurrence as things got worse that we would eventually end up on a conference call with Wipro and their end customer and their customer's perception was entirely off. There was no way it was miscommunication, they were straight lying to their customer all along. Often we'd have to break the news to the customer that we haven't been working on the issue for weeks, we just heard about it today, nobody can tell us how they want the product configured on the most basic level...
The only thing worse than that situation was to look up these customer's of Wipro and see they scrapped their own IT departments in favor of outsourcing, and I'm not sure they had more than a couple people who understood what was really going on.
The incompetence was astounding, and I worked in support for a long time, and Wipro was really astounding. Everything from security to just understanding what we were telling them was mindbogglingly bad. It wasn't a language barrier, they simply didn't have many / sometimes anyone who understood the technology on the most basic level.
Wipro would open tickets dozens at a time claiming there was some sort of technical issue, but they often couldn't explain what if anything they tried. We would find the equipment at factory defaults, last boot time was when it was in the factory.... but now it was a P1 ticket because "it didn't work and it needs to be up and running by the end of the day". Then we'd ask what how they wanted it configured and they ... wouldn't know. Then they'd escalate through sales and the executives claiming we had been "working with them for weeks and were not helping".
Then they would go silent and not respond for days or weeks only to reappear later as angry as ever that we hadn't done anything when our last questions to them might be as simple as "what isn't working?".
It was worse when they actually tried configuring things as they were masters at nonsensical configurations, looping cables back into the same equipment they came from and etc. You could look at their systems that were "working" and it was errors everywhere and you couldn't trust anything you saw.
Even internally Wipro would tell us that they "can't tell" the "other team" (another team inside Wipro working with the same customer) that they need to change their configuration. They would just repeat that they can't tell them that ... and we'd be stuck because it's obvious the "other team" is configured wrong. I'd tell them to let me be the bad guy and tell them on a call, but nope. So things would just not work.
It was a common occurrence as things got worse that we would eventually end up on a conference call with Wipro and their end customer and their customer's perception was entirely off. There was no way it was miscommunication, they were straight lying to their customer all along. Often we'd have to break the news to the customer that we haven't been working on the issue for weeks, we just heard about it today, nobody can tell us how they want the product configured on the most basic level...
The only thing worse than that situation was to look up these customer's of Wipro and see they scrapped their own IT departments in favor of outsourcing, and I'm not sure they had more than a couple people who understood what was really going on.
Well I look around in an Indian IT shop and it's all people brow beating each other and chatting with each other. As if it's all an FD income to them and no one wants to seriously work.
With the increase in MSP style operations with an IT companys systems having root access across all their client's systems IT companies are going to be massive targets for bad actors. There's already been a few cases of all of a compaies clients being ransomwared.
If you've ever had the pleasure to work with people from wipro,ipsoft,atos etc to name a few this should not surprise you.
In the last 6 months there have been security breaches at Facebook [1] Google [2] Cisco [3] and look at those threads and some of these breaches are extremely amateurish and the general consensus is these things happen and the top voted responses mirror this attitude.
Yet on the same site on the threads about India, China and non US companies we see some kind of dissonance where these are reframed as showhow affecting these companies uniquely because of 'poor standards' and 'mediocre engineers' and the top voted responses reflect this.
Far from informed discussion this not only demonizes entire groups but creates and perpetuates prejudice that will no doubt impact everything from recruitment to general behavior. And this continues on discussions beyond security to things like corruption, surveillance and other issues.
[1] https://news.ycombinator.com/item?id=19565918
[2] https://news.ycombinator.com/item?id=18170174
[3] https://news.ycombinator.com/item?id=19507225