Bank of England to crack down on 'secretive' cloud computing services(itnews.com.au)
itnews.com.au
Bank of England to crack down on 'secretive' cloud computing services
https://www.itnews.com.au/news/bank-of-england-to-crack-down-on-secretive-cloud-computing-services-567310
122 comments
How is the problem of PII better solved on premises?
Think about it more abstractly from the perspective of trust and # of actors involved.
If you run 100% of your IT workload on-prem, the ability to control the flow of data can be boiled down into a physical exercise of following fiber channel cables in your own datacenter. Having a unified set of firewall rules that define your entire public interface also helps a lot.
You can actually make deterministic guarantees to your customers that not only your own systems are secure, but also that the systems of your vendors and other 3rd parties are as well. The moment you start configuring site-to-site VPNs with 3rd parties across which you intend to transact sensitive business knowledge, you are surrendering an entire mountain of security constraints.
If we are being honest with ourselves, a lot of shops that are 100% on-prem probably have worse security practices than AWS, et. al. Perhaps the biggest hazard is really the hybrid model. If some fintech went 100% into the cloud without even an HSM on-prem to worry about, then you could probably have a solid argument on the other side of the spectrum. Also, remember that multi-cloud might seem like a resiliency measure, but it also adds another target to your back.
The middle ground is where all the pain seems to be. Hybrid cloud usually means more required trust than most organizations ever wanted to enter into. I frequently find myself as the harbinger of bad news when I get into deep-dive technical calls with some of our customers. Turns out a lot of the other vendors we work with like to bend the truth in order to make a quick buck. Many perverse incentives are pulling these massive organizations into hilariously-contorted IT stances, and some of us are starting to see a consulting opportunity.
If you run 100% of your IT workload on-prem, the ability to control the flow of data can be boiled down into a physical exercise of following fiber channel cables in your own datacenter. Having a unified set of firewall rules that define your entire public interface also helps a lot.
You can actually make deterministic guarantees to your customers that not only your own systems are secure, but also that the systems of your vendors and other 3rd parties are as well. The moment you start configuring site-to-site VPNs with 3rd parties across which you intend to transact sensitive business knowledge, you are surrendering an entire mountain of security constraints.
If we are being honest with ourselves, a lot of shops that are 100% on-prem probably have worse security practices than AWS, et. al. Perhaps the biggest hazard is really the hybrid model. If some fintech went 100% into the cloud without even an HSM on-prem to worry about, then you could probably have a solid argument on the other side of the spectrum. Also, remember that multi-cloud might seem like a resiliency measure, but it also adds another target to your back.
The middle ground is where all the pain seems to be. Hybrid cloud usually means more required trust than most organizations ever wanted to enter into. I frequently find myself as the harbinger of bad news when I get into deep-dive technical calls with some of our customers. Turns out a lot of the other vendors we work with like to bend the truth in order to make a quick buck. Many perverse incentives are pulling these massive organizations into hilariously-contorted IT stances, and some of us are starting to see a consulting opportunity.
> You can actually make deterministic guarantees to your customers that not only your own systems are secure, but also that the systems of your vendors and other 3rd parties are as well.
You can make a "deterministic" guarantee, whatever that is, that your systems are secure? That's seems pretty bold and probably dangerous, no?
You can make a "deterministic" guarantee, whatever that is, that your systems are secure? That's seems pretty bold and probably dangerous, no?
> That's seems pretty bold and probably dangerous, no?
Its not dangerous in my experience. The more dangerous angle for me is this belief that it is impossible (or hopelessly difficult) to build a secure system.
The reality is that it is only possible if you are willing to take total ownership of the entire vertical. If you control every single byte that enters and exits your enterprise, you can prove that things are secure. Is it practical to do this in all cases? No. Is it feasible in theory and in certain cases? Absolutely.
If you buy into the 3rd party hosting game, you instantly lose control over the critical variables you would need to in order to create the opportunity for these sorts of guarantees to exist in the first place. You (and your customers) will be stuck wondering about side channel damage and human factors that you have no direct control over. When you own the hardware and the real estate it is parked on top of, you can start to reel these things back in really quickly with powerful policy frameworks (2-person rules for critical changes, mandatory checklists, etc). These sorts of policies seem to work really well for very tricky areas like keeping our nuclear weapons from doing inappropriate things.
Its not dangerous in my experience. The more dangerous angle for me is this belief that it is impossible (or hopelessly difficult) to build a secure system.
The reality is that it is only possible if you are willing to take total ownership of the entire vertical. If you control every single byte that enters and exits your enterprise, you can prove that things are secure. Is it practical to do this in all cases? No. Is it feasible in theory and in certain cases? Absolutely.
If you buy into the 3rd party hosting game, you instantly lose control over the critical variables you would need to in order to create the opportunity for these sorts of guarantees to exist in the first place. You (and your customers) will be stuck wondering about side channel damage and human factors that you have no direct control over. When you own the hardware and the real estate it is parked on top of, you can start to reel these things back in really quickly with powerful policy frameworks (2-person rules for critical changes, mandatory checklists, etc). These sorts of policies seem to work really well for very tricky areas like keeping our nuclear weapons from doing inappropriate things.
How do you know your own programmers aren't introducing security bugs? Or are even acting against your interests intentionally? It happens to every other software developer, why not you?
Do you build all your own hardware from raw materials? How do you know everyone in the supply chains is perfectly secure?
Attacks have succeeded against the CIA, against RSA, Google, and many others. Nuclear weapons plans have been stolen. I would not trust a vendor who claimed they could guarantee security.
Do you build all your own hardware from raw materials? How do you know everyone in the supply chains is perfectly secure?
Attacks have succeeded against the CIA, against RSA, Google, and many others. Nuclear weapons plans have been stolen. I would not trust a vendor who claimed they could guarantee security.
> If we are being honest with ourselves, a lot of shops that are 100% on-prem probably have worse security practices than AWS, et. al.
Does it matter? You have the same freedom to fuck security up setting your AWS infrastructure as you have setting your on-prem infrastructure. All the very competent AWS staff is able to do is add less risk, they can't save you from anything.
Does it matter? You have the same freedom to fuck security up setting your AWS infrastructure as you have setting your on-prem infrastructure. All the very competent AWS staff is able to do is add less risk, they can't save you from anything.
Good ol IAM basically being an incoherent Json programming language documented in gibberish.
>the ability to control the flow of data can be boiled down into a physical exercise of following fiber channel cables in your own datacenter
I suppose the infamous Equifax breach was due to a secret fiber optic cable running out of their datacenter?
I suppose the infamous Equifax breach was due to a secret fiber optic cable running out of their datacenter?
> I suppose the infamous Equifax breach was due to a secret fiber optic cable running out of their datacenter?
No, of course not. But when you're dealing with physical infrastructure you can actually touch, it's much clearer and more certain what you're dealing with.
No, of course not. But when you're dealing with physical infrastructure you can actually touch, it's much clearer and more certain what you're dealing with.
I don't think that is as true as you make it sound. I have managed on-prem and cloud infrastructure and am much more confident that my cloud servers are secure because a whole lot of stuff is done by the provider who know a lot more than me between them.
Even on a really simple on-prem scenario, you have switches to configure, vlans to setup, hardware drivers, a gazillion updates to make all the time and a tonne of employees making it all very difficult. The fact I can see it physically doesn't realy help me that much.
Even on a really simple on-prem scenario, you have switches to configure, vlans to setup, hardware drivers, a gazillion updates to make all the time and a tonne of employees making it all very difficult. The fact I can see it physically doesn't realy help me that much.
For one, at a certain scale you shouldn't be running an inhouse DC solo, you can get away with it more in the cloud but at a certain ace again you want more manpower for review/auditing and brain/man power.
Most of what you described aren't actually principle attack vectors either. The primary vectors are the same between on premise and cloud. Misconfigured VPNs, stolen VPN credentials, poor network segmentation (Cloud absolutely does not fix this for you, you still need brain power and auditing to find accidental misconfigs).
No it was due to the officially-endorsed fiber optic cables sitting in plain sight and the fact that they do business with so many other parties.
I work with some intermediate vendors in this space (they have direct access to the credit bureau data), and their security mechanisms are of concern. I am under some very strict NDA constraints, but I can say that there are serious problems and I am not surprised that breaches occur with regular frequency.
You can barely trust your own in-house developers to get these things right. How can you possibly hope to trust many other additional parties to get it right simultaneously as well?
I work with some intermediate vendors in this space (they have direct access to the credit bureau data), and their security mechanisms are of concern. I am under some very strict NDA constraints, but I can say that there are serious problems and I am not surprised that breaches occur with regular frequency.
You can barely trust your own in-house developers to get these things right. How can you possibly hope to trust many other additional parties to get it right simultaneously as well?
Just my observations, but some business entities run into legal challenges that vary greatly by industry. B2B customers have a contractual relationship with you, not your hosting provider. If your hosting provider has an "oops we leaked your data" they only breached the contract with their vendor, not themselves. The contract can specify how data is managed. Adding to this, some companies/banks legal controls are compatible with SOC2 controls of a 3rd party data processor being audited and some companies are not. Some companies can cite the 3rd parties audited certifications and some can not based on preexisting legal contractual agreements with their own customers. I am not a lawyer, but had to sit in many meetings with lawyers and businesses and this is a real issue they have to address. I have also worked for a large bank. Rules, regulations and contracts around 3rd party data processors and financial institutions can get very complicated. There are a myriad of additional complicating variables that go beyond regulations. Legal obligations also vary by relationship. If a bank has preexisting relationships with other banks, they may be obligated to get approval from the other banks to change how their data is managed. Amending contracts is non-trivial. This rabbit hole can get very deep.
I'm a banking attorney and I think you've hit the nail on the head. Vendor management is popular with regulators, particularly with respect to info sec and data privacy, and the navigating the contractual responsibilities can quickly become burdensome.
Getting the lawyers on both sides to agree to language for the ongoing certification(s) that will also satisfy the internal auditors, the outside auditors, the regulators... suddenly something that sounded simple when the contract was signed takes several meetings and a lot of back and forth.
Getting the lawyers on both sides to agree to language for the ongoing certification(s) that will also satisfy the internal auditors, the outside auditors, the regulators... suddenly something that sounded simple when the contract was signed takes several meetings and a lot of back and forth.
What's the worst that can happen in a on-premises attack, Vs the worst that could happen if AWS was hacked?
The amount of financial data that could be exploited at once is magnitudes larger in a popular cloud. I don't think it's strange that a regulator might look at that failure point with some trepidation.
The amount of financial data that could be exploited at once is magnitudes larger in a popular cloud. I don't think it's strange that a regulator might look at that failure point with some trepidation.
On the other hand, at least AWS, Azure etc. all have a vested interest in doing things well and securely. At a bank, most employees know nothing and care nothing about the HW and SW systems, they just use and abuse them.
Why do you think banks don't have vested interest?
And I don't expect my cleaner to service my car. Banks have DBAs, network and physical security experts on tap (just like cloud providers do).
And I don't expect my cleaner to service my car. Banks have DBAs, network and physical security experts on tap (just like cloud providers do).
Well they do have the vested interest but they are predictably irrational in another one - trying to make more money and cutting costs from "non-core businesses". It isn't rational but certain sorts of predictable stupidity are more common in some contexts than others.
Smaller surface area to guard for one.
The surface area of vulnerabilities like spectre and meltdown is much lower
because you can control physical access to the hardware
> Once the risks are reviewed in open and honest ways, we find that virtually all of our clients would prefer to keep our solution on-prem.
I hope this kind of risk assessment becomes more common. I'm used to people not caring until things blow up on their faces.
I hope this kind of risk assessment becomes more common. I'm used to people not caring until things blow up on their faces.
Well, I guess you are speaking about companies who don't know what they do. But this article speaks about banks, who have hundreds of servers, ability to recover anything, full-time employed ops teams, monitoring & automation in place for 20 years already. Moving to cloud provides very little advantage (definitely not financially) to such companies. They are not SaaS who might need to double their infrastructure overnight. Lots of advantages provided by cloud has been in place years before clouds because of VMWare. Another thing is existing bank apps are often monoliths (when you are happy, in Java with Spring, C# .NET, when not, in Visual Basic, Cobol, PowerBuilder, SAP) or huge interconnected services, huge databases with thousands of stored procedures, SOAP APIs in place. This is not a place where you would leverage ElasticBeanstalk or AWS Lambda.
> Moving to cloud provides very little advantage (definitely not financially) to such companies.
As someone who has worked as a software developer for big NY banks for the past 25 years, that's simply not true.
The answer is, it's complicated. JPMorganChase for instance has a $12 billion annual IT spend. They do a LOT of different things. Admittedly certain things are best left on prem for regulatory audit points (more with respect to resilience/business continuity rather than security.) But a substantial portion of it could be moved to the cloud at some cost savings.
Additionally the brittleness of the service and database infrastructure is a pathology of the on-prem environment rather than an argument for it. Cohorts of SA's and DBA's are wasting their time doing work which in a modern environment would be scripted and more flexible.
As someone who has worked as a software developer for big NY banks for the past 25 years, that's simply not true.
The answer is, it's complicated. JPMorganChase for instance has a $12 billion annual IT spend. They do a LOT of different things. Admittedly certain things are best left on prem for regulatory audit points (more with respect to resilience/business continuity rather than security.) But a substantial portion of it could be moved to the cloud at some cost savings.
Additionally the brittleness of the service and database infrastructure is a pathology of the on-prem environment rather than an argument for it. Cohorts of SA's and DBA's are wasting their time doing work which in a modern environment would be scripted and more flexible.
[deleted]
Its not as simple as that. Just because its the case of "we have lots of old stuff" doesn't mean you need to ignore the new stuff. Building solutions with traditional data centers and staff, even if you have a lot of it, is often a lot slower. You can spin up entire fleets of servers (or even use services such as AWS Lambda, API Gateway and DynamoDB so you never use servers) and get a solution out in much less the time. Its not just about responding to load but also to be fast enough to get new stuff out there. Traditional financial services organisations are notoriously slow to adapt to changes in the market. Using cloud resources alongside the legacy infrastructure is one way to try and remain competitive.
> Traditional financial services organisations are notoriously slow to adapt to changes in the market. Using cloud resources alongside the legacy infrastructure is one way to try and remain competitive.
Or, you could have a single executive action revamp IT resource acquisition policies. Simply mandate that internalized self-service resource provisioning capabilities be developed.
It is not rocket science to put a web dashboard around vmware or some other virtualization solution. Most of them already sell something like this as part of their feature set.
You could set something like this up in a week if you had enough buy-in. There are no excuses if you want to win at this kind of game. The bad guys are way more patient in aggregate.
A 2nd perspective - One of our customers has a "traditional" process for setting up new IT workloads, and we were still able to get 3 servers provisioned within 8 hours along with 3 new publicly-routable IPv4 addresses and matching DNS+TLS certs. Anything less than this in 2021 for any organization is indicative of sheer incompetence IMO. We do have some customers that are really slow, but they are also really small. I don't think any F500 is taking weeks to provision SQL Server anymore.
Or, you could have a single executive action revamp IT resource acquisition policies. Simply mandate that internalized self-service resource provisioning capabilities be developed.
It is not rocket science to put a web dashboard around vmware or some other virtualization solution. Most of them already sell something like this as part of their feature set.
You could set something like this up in a week if you had enough buy-in. There are no excuses if you want to win at this kind of game. The bad guys are way more patient in aggregate.
A 2nd perspective - One of our customers has a "traditional" process for setting up new IT workloads, and we were still able to get 3 servers provisioned within 8 hours along with 3 new publicly-routable IPv4 addresses and matching DNS+TLS certs. Anything less than this in 2021 for any organization is indicative of sheer incompetence IMO. We do have some customers that are really slow, but they are also really small. I don't think any F500 is taking weeks to provision SQL Server anymore.
Not to mention that that same infrastructure you say is in abundance is usually pretty heavily utilised already so there isn't just spare capacity lying around. Procuring new hardware can take months and if there is no rack space you are looking at more months to get that deployed.
The people that need to plan, coordinate and install all of this are also pretty heavily overworked at the moment because, believe it or not, there is a very large shortage of skilled sys admins in the world.
Using the cloud doesn't solve those problems, but it does help reduce their impact.
The people that need to plan, coordinate and install all of this are also pretty heavily overworked at the moment because, believe it or not, there is a very large shortage of skilled sys admins in the world.
Using the cloud doesn't solve those problems, but it does help reduce their impact.
Does "on prem" ever include (a) on your own hardware, but in a third-party colo and (b) on rented hardware, where only you have root? Where do those sit between full cloud and traditional servers-in-the-basement?
To look at it another way, how much of the risk is about the physical location of machines, and how much is about who operates them?
To look at it another way, how much of the risk is about the physical location of machines, and how much is about who operates them?
One of the (legal ?) requirements we had to meet back in the 00s was that financial software (quant code, options trading) was "reproducible" for some length of time (7 years?). That meant: version control (SCCS, CVS, svn), archiving OS versions (SunOS, Solaris), and kit (Sun workstations and servers).
I have no idea if the last 2 ("OS", hardware) is even achievable in 2021. I can always get my source code from git, and possibly the 'configuration', but can I expect to run cloudy code in 7 years time?
I have no idea if the last 2 ("OS", hardware) is even achievable in 2021. I can always get my source code from git, and possibly the 'configuration', but can I expect to run cloudy code in 7 years time?
I'd expect that if the cloud providers wish to acquire customers that have to comply with those regulations, the cloud providers will have clause in the contract that they are contractually obligated to keep the same services running for that period of time.
OS is reproducible with artifact repositories for OS and library packages. Hardware perhaps less so, but I guess a custom deal could be worked out with the right supplier. That, or go with emulation.
Emulation can technically have some frustrating difference quirks - which suggests a "stupid but works" source of consistency by using emulated environments from day one.
What about having regulations updated instead of bending the industry backward to fit the outdated regulations?
I can't testify to my statement being 100% true without a lot of tedious searching on the UK FSA/SFA/FCA/whatever they are now's website. But that's what I have a vague recollection of.
It's similar to HMRC's "we can come after you for unpaid tax for 7 years", but you can only go after the HMRC for 5 years for overpaid tax.
The rules are ultimately for the benefit of long term investigations such as the LIBOR rigging, and Guinness trials.
https://www.theguardian.com/business/2016/jul/04/libor-riggi...
https://en.wikipedia.org/wiki/Guinness_share-trading_fraud
It's similar to HMRC's "we can come after you for unpaid tax for 7 years", but you can only go after the HMRC for 5 years for overpaid tax.
The rules are ultimately for the benefit of long term investigations such as the LIBOR rigging, and Guinness trials.
https://www.theguardian.com/business/2016/jul/04/libor-riggi...
https://en.wikipedia.org/wiki/Guinness_share-trading_fraud
What's outdated about it?
Seems a bit like a nothing-article - doesn't make any real point. The FCA/FSA have already been noticing a lot of the financial providers placing 'all their eggs in one basket' and have been putting guidelines in place for banks to have a multi-'cloud' strategy.
[0] https://www.fca.org.uk/publication/finalised-guidance/fg16-5...
[0] https://www.fca.org.uk/publication/finalised-guidance/fg16-5...
That's what I thought but also, when a statement is made by a senior executive, I usually assume they don't really know what they are talking about anyway. Loads of "keynote" speeches are very hand-wavy and have a feel of importance with little of real value.
I worked in a brand-name bank, so I am familiar with the goings in the tech side of finance.
Call me crazy, but I would trust aws, microsoft, and google with my PII and finances before I would trust Wells, BofA, JPMC, Goldman, et al.
The cloud giants pay their engineers more and technologists are second-class citizens at the financial institutions - infer what you want from that.
Call me crazy, but I would trust aws, microsoft, and google with my PII and finances before I would trust Wells, BofA, JPMC, Goldman, et al.
The cloud giants pay their engineers more and technologists are second-class citizens at the financial institutions - infer what you want from that.
Definitely agree. Having worked extensively with some large banks I've found that they are absolutely cutting edge in technology in one area and one area only: credit card fraud analytics. That's because credit card fraud creates a material financial risk for the bank that they can't defer to someone else. Most other risks banks deal with are managed through complex financial hedging, government backstops, or insurance schemes and so they invest very little in doing anything "correctly". Rather, almost everything about banking compliance is driven by checkbox tickers and not by experts.
I think it is unfair to think that banks don't want to do things correctly but they suffer along with all other corporates of too many levels of management to be agile, systems that are far to risky to change/replace, legacy systems and a high turnover of staff. If I ran a bank, I suspect I would run it exactly the same way because I have to.
Perhaps I'm too close to the issue as I have worked with these businesses, but I disagree. This is intentional on their part, because banks first and foremost are about identifying and managing risk. If they can find a way to mitigate that risk or externalize it, then they no longer need to deal with it head-on. Most of the risks related to IT should be dealt with head-on because they grow over time as you stay still and technology moves further away from you, and it's impossible for anyone to accurately predict the future and therefore future risks.
Many banks and other financial institutions are finding this out now in their desperate bid to find competent COBOL programmers to continuing maintaining legacy critical applications running on mainframes when most COBOL programmers are retired or dead, and more are headed that way every day. It wouldn't shock me to find out that the effects of COVID being weighted towards worse outcomes for older people had a material effect on the human resource risk of using COBOL-based critical systems.
Banks will absolutely hold on to anything to avoid an unknown risk as long as they think they can hedge or mitigate known risks, and utterly fail to acknowledge the truth of unknown unknowns. This will ultimately be their downfall if governments ever let them follow standard business outcomes, otherwise they'll eventually absorb some upstart to keep hedging forever.
Many banks and other financial institutions are finding this out now in their desperate bid to find competent COBOL programmers to continuing maintaining legacy critical applications running on mainframes when most COBOL programmers are retired or dead, and more are headed that way every day. It wouldn't shock me to find out that the effects of COVID being weighted towards worse outcomes for older people had a material effect on the human resource risk of using COBOL-based critical systems.
Banks will absolutely hold on to anything to avoid an unknown risk as long as they think they can hedge or mitigate known risks, and utterly fail to acknowledge the truth of unknown unknowns. This will ultimately be their downfall if governments ever let them follow standard business outcomes, otherwise they'll eventually absorb some upstart to keep hedging forever.
>Banks will absolutely hold on to anything to avoid an unknown risk as long as they think they can hedge or mitigate known risks, and utterly fail to acknowledge the truth of unknown unknowns
Nassim Taleb has written entire books on this fact about banks. It's interesting that the same flaws in their financial thinking apply to their IT infrastructure.
Nassim Taleb has written entire books on this fact about banks. It's interesting that the same flaws in their financial thinking apply to their IT infrastructure.
I think technology is fundamentally a tool for organizations, and as such will always evolve to reflect the larger organization. You can see this in software design and it is also seen in the IT infrastructure architecture.
My understanding is that at least Chase, Blackrock, and Visa are rapidly raising their pay for new engineering / analyst hires. Is this not true?
I can speak for the bank I worked at - this is true, but the base pay level was very low to begin with and the increase is nowhere near competitive for the most talented engineers. Also, the experienced engineers at those companies are salty about new hires making more then they are.
Again, this stems from the culture of financial institutions where tech workers are viewed as support staff, not strategically critical, despite what financial institutions say publicly.
Again, this stems from the culture of financial institutions where tech workers are viewed as support staff, not strategically critical, despite what financial institutions say publicly.
I know for a fact that base pay for a first year SWE with a BS+MS in CS at Goldman just broke $100k a couple years ago. That probably sounds like a lot to most people, but if you're hiring staff to live and work in NYC, especially when there's an absolutely massive Google campus 10 minutes North of the GS HQ, that's pretty much table stakes. Never mind the glaring cultural differences.
Does that take bonuses into account? Goldman (and others in the industry) tend to have lower base pay with a much higher bonus. It's typical for low level employees to get a year-end bonus equal to about 50-60% of their salary. As they advance to mid-level their bonus will equal their salary and, if they're in a senior position, it will greatly exceed their salary.
This is strictly base pay, but the bonuses are simply not that good for the average SWE at the banks, because as others have said, technologists are seen as a supporting role. There are exceptions, but a much more senior coworker of mine that worked at a number of HF's and banks once told me a good rule of thumb is that the further away you are from executing a trade, the lower your comp. Of course, management doesn't really follow that rule.
This is all with worse work/life balance, low flexibility, (GS was the first big bank to recall everyone and enforce working on site I believe.), a 'conservative' tech stack if you're lucky or a simply unpleasant one if you're not, and generally not being valued the same way as you would be at a FAANG or typical tech company.
This is all with worse work/life balance, low flexibility, (GS was the first big bank to recall everyone and enforce working on site I believe.), a 'conservative' tech stack if you're lucky or a simply unpleasant one if you're not, and generally not being valued the same way as you would be at a FAANG or typical tech company.
"Back office" (which includes most programmers, with the exception of algo traders) bonuses are much much lower than "front office" (traders, ibankers, sales) bonuses.
Google/FB devs total comp (including RSUs) will easily beat Goldman devs by 150-200%, depending upon your performance.
Google/FB devs total comp (including RSUs) will easily beat Goldman devs by 150-200%, depending upon your performance.
Do banks not offer engineers RSUs?
If so, $100k plus a 15% bonus is laughably low. Even shitty startups pay new grads more than that.
My understanding is that GS's interview is actually harder to pass than a lot of FAANG companies. So you have to wonder why anyone would even interview there if the pay is that low...
If so, $100k plus a 15% bonus is laughably low. Even shitty startups pay new grads more than that.
My understanding is that GS's interview is actually harder to pass than a lot of FAANG companies. So you have to wonder why anyone would even interview there if the pay is that low...
Bonuses are not that high at Goldman and their comp still does not compare to FAANG, it never will.
Keep in mind it easily could at the stroke of a pen though. It's entirely their choice to not compensate competitively.
https://tipalti.com/profit-per-employee/
Visa regularly makes double the profit per employee of FAANG - so, yes, they could easily pay more.
Visa regularly makes double the profit per employee of FAANG - so, yes, they could easily pay more.
Don't all the smart companies avoid making any profit at all, since profit is taxed (but share buybacks are not)?
You have to buy your shares with FCF (usually profit) or debt.
They have profits. They just aren't in the US.
They have profits. They just aren't in the US.
Thanks for the source to backup that feeling.
Goldman’s payday is through bonuses, not the base salary.
Only if you're in a front office role, which the vast majority of SWEs aren't.
No? Even at entry level, you’re still getting $150k for base+bonus. You can expect at least 25% to be bonus if not more.
Maybe you and I have a different concept of payday.
What makes GS less than FAANG is the annual stock grant FAANGs throw at software engineers, and of course it Finance.
Maybe you and I have a different concept of payday.
What makes GS less than FAANG is the annual stock grant FAANGs throw at software engineers, and of course it Finance.
Definitely agree. In my experience, the financial industry tends to lean heavily on checklists and bureaucracy to enforce security. This requires additional headcount and prevents automation.
Technology companies lean the other way and enforce security through comprehensive automation of the controls they must follow.
Technology companies lean the other way and enforce security through comprehensive automation of the controls they must follow.
Yes, agreed. The choking bureaucracy has the unintended consequence of lowering risk - if its very time consuming to build thing, you build less things that break over time, and rely on old things that have worked for a long time.
One secretive group used to dictating their own terms unhappy about another secretive group used to dictating their own terms...
The Bank of England is under the democratic control of the people of England.
How exactly do you figure that? Like any other quango, they are almost entirely unaccountable to anyone except their own staff. A Politician might be able to cause a stink but that's assuming they knew enough about what was going on and they probably don't.
The elected representatives of the English people could shut down the Bank tomorrow, or today.
>The elected representatives of the English people could shut down the Bank tomorrow, or today.
There is a certain irony in this statement, given that such an act would require approval from the queen.
There is a certain irony in this statement, given that such an act would require approval from the queen.
Let's be serious.
*In theory. However, in practice...
The BoE is independent. That's a good thing generally - it doesn't have to bow to pressure from whatever party happens to be in power at a specific time.
Right, most central banks have some insulation from politics. BoE's independence is granted by Parliament, and can be taken away at any time.
I think a big risk is a cpu level security issue similar to meltdown or spectre that ends up weakening the hardware isolation between tenants to the point where it can be exploited on mass on the cloud providers to wreak havoc. The probability of something like this happening is very low but not zero, I would say same level of probability as datacenter fire or earthquake banks should be planning for how to handle this type of event.
Separate hardware for your stuff is a standard AWS product, for example - you can just buy this.
That’s one way of dealing with hardware isolation risk but not every bank is doing this on public cloud.
I don't see that this risk is any different than a similar apocolyptic failure happening to your on-prem equipment. There's not much you can do about it differently than the cloud just add some extra controls and hope for the best.
I very much doubt that anyone would not use the cloud because of a theoretical de-isolation bug.
Also, by the time you found out, it would probably already be too late anyway if you were a victim. If not, you just switch it off.
I very much doubt that anyone would not use the cloud because of a theoretical de-isolation bug.
Also, by the time you found out, it would probably already be too late anyway if you were a victim. If not, you just switch it off.
I am not saying that they should not use cloud just that it is important to have a plan in place to deal with a unlikely but high impact security event affecting a cloud provider. Just like companies have business continuity plans in case a data center disaster they need to have plans for evacuate a cloud provider should they need too.
I put on my seat belt when I drive on the highways even though a nasty crash at 120 kph would likely kill me. Not using a seat belt because you will be severely injured anyway is not wise.
Given the amount of profit banks make what is the Downside of having them be resilient against public cloud failures?
I put on my seat belt when I drive on the highways even though a nasty crash at 120 kph would likely kill me. Not using a seat belt because you will be severely injured anyway is not wise.
Given the amount of profit banks make what is the Downside of having them be resilient against public cloud failures?
We can be almost certain that there are sw and hw vulnerabilities that can be so exploited, given the rate of discovery and knowing what now-public hypervisor and cpu vulns a time traveler from today could exploit eg 5 or 10 years in the past.
If you're that size why not split the workload between a couple or all 3 of the big providers. Get them to work against each other.
All the concerns about vendor lock-in are ridiculous right now. Prices have consistently lowered for all cloud providers.
All the concerns about vendor lock-in are ridiculous right now. Prices have consistently lowered for all cloud providers.
[deleted]
Something to keep in mind here: The employees of the Bank of England are ultimately government employees.
They are subjected to the pay scales dictated by bureaucrats and politicians. Of course, they can't attract the caliber of engineers that works for large commercial cloud providers, so they have to settle for programmers that didn't make the cut. And the leadership is non-technical and from the financial and government worlds (you know how these "elites" see programmers and software engineers in the UK...).
The existing bureaucrats running the bank probably see the Cloud as something that will reduce their headcount (why would they have sysadmins and datacenter employees on their payroll when they can simply buy it from a reliable provider), thus making them look less important to other managers. The existing unionized employees see it as a threat to their stable jobs (they are now competing with engineers that are 10x their caliber). That's a pretty bad thing for both.
They are subjected to the pay scales dictated by bureaucrats and politicians. Of course, they can't attract the caliber of engineers that works for large commercial cloud providers, so they have to settle for programmers that didn't make the cut. And the leadership is non-technical and from the financial and government worlds (you know how these "elites" see programmers and software engineers in the UK...).
The existing bureaucrats running the bank probably see the Cloud as something that will reduce their headcount (why would they have sysadmins and datacenter employees on their payroll when they can simply buy it from a reliable provider), thus making them look less important to other managers. The existing unionized employees see it as a threat to their stable jobs (they are now competing with engineers that are 10x their caliber). That's a pretty bad thing for both.
From my understanding of the article, the BoE aren't concerned about their own infrastructure.
They're concerned about all the major retail banks using the same cloud provider and then that provider having a major outage.
Individual banks having an outage is an issue but one that can be handled. Three or four of the main banks all going down at once could be catastrophic.
They're concerned about all the major retail banks using the same cloud provider and then that provider having a major outage.
Individual banks having an outage is an issue but one that can be handled. Three or four of the main banks all going down at once could be catastrophic.
> They're concerned about all the major retail banks using the same cloud provider and then that provider having a major outage.
I wouldn't worry too much about that. Major outages are exceedingly rare. I can't say the same for on-prem infrastructure.
I wouldn't worry too much about that. Major outages are exceedingly rare. I can't say the same for on-prem infrastructure.
Does anyone know if CSPs can be managed via supplier agreements? I’ve always assumed it’s a commodity service and as such there are standard T&Cs, TOS etc. Certainly those in the financial sector wish to manage other types of supplier risk in that way where possible.
Isn't that the million dollar question (or probably billion now).
Theoretically, the Ts & Cs cover everything but clearly you cannot reasonably mitigate risk on the basis that the "supplier told me that it wouldn't happen".
You have a (sometimes legal) requirement to do due diligence and at least decide what additional controls you can have to help with compliance and what your BC/DR process is if everything really goes south.
Realistically, I don't really see AWS or Azure etc. being able to promise that their systems can never be hacked/broken. On the other hand, the assumption that doing things on-prem removes this risk is naiive since we can make just as many cock-ups even if we work for the company. Anyone ever forgotten to setup a firewall or vpn properly?
True story: When I worked at a security company, the sales team wanted to demo a digital camera-over-IP system and the IT Manager told them not to use our internal network. They ignored him and plugged it in, flooding the network with packets that took down the computers and phones for about 30 minutes until we worked out what had happened. That was inconvenient but imagine if they had plugged in something much worse "because sales"
Theoretically, the Ts & Cs cover everything but clearly you cannot reasonably mitigate risk on the basis that the "supplier told me that it wouldn't happen".
You have a (sometimes legal) requirement to do due diligence and at least decide what additional controls you can have to help with compliance and what your BC/DR process is if everything really goes south.
Realistically, I don't really see AWS or Azure etc. being able to promise that their systems can never be hacked/broken. On the other hand, the assumption that doing things on-prem removes this risk is naiive since we can make just as many cock-ups even if we work for the company. Anyone ever forgotten to setup a firewall or vpn properly?
True story: When I worked at a security company, the sales team wanted to demo a digital camera-over-IP system and the IT Manager told them not to use our internal network. They ignored him and plugged it in, flooding the network with packets that took down the computers and phones for about 30 minutes until we worked out what had happened. That was inconvenient but imagine if they had plugged in something much worse "because sales"
We need a better 'Open Stack' for cloud stuff.
So that you can just run a bunch of your own servers, lay over the 'Stack' and then get nice Lambdas, provisioning, and other things.
Imagine if you could just buy some hardware, and have some regular IT guys reproduce most of Amazon without the Amazon?
That would be a 'reverse revolution'.
National Regulators could also support strategic investment in the sector, i.e. 'financial ops have to be backed-up in-nation by a local provider' i.e. some kind of forced local diversity in the system by regulatory fiat. As an idea.
So that you can just run a bunch of your own servers, lay over the 'Stack' and then get nice Lambdas, provisioning, and other things.
Imagine if you could just buy some hardware, and have some regular IT guys reproduce most of Amazon without the Amazon?
That would be a 'reverse revolution'.
National Regulators could also support strategic investment in the sector, i.e. 'financial ops have to be backed-up in-nation by a local provider' i.e. some kind of forced local diversity in the system by regulatory fiat. As an idea.
original/source article on Reuters: https://www.reuters.com/business/retail-consumer/bank-englan... (https://news.ycombinator.com/item?id=27819016)
any other more detailed breakdown of this somewhere?
any other more detailed breakdown of this somewhere?
How reliant are banks and insurers on cloud outsourcing? https://www.bankofengland.co.uk/bank-overground/2020/how-rel...
(January 2020)
Cloud computing feels like one area we have fantastic competition in, even if there are only 3 major players (with one or two smaller ones). Feature and price competition between AWS and Azure is fierce, and if you’re dumb enough to have any trust at all in Google as a supplier then there’s even GCP.
TLDR "Secretive" means "big providers could dictate terms and conditions - as well as prices - to key financial firms."
Thanks, I only skimmed this headline and assumed “secretive” meant groups of developers set up their own AWS/GCP/Azure accounts because they were hitting walls with the company’s IT dept.
That is what I thought the article was about, too.
> But big providers could dictate terms and conditions - as well as prices - to key financial firms.
What exactly is the concern here? Cloud compute is becoming cheaper over time due to market forces. Its not like Amazon is cornering the market for CPUs.
What exactly is the concern here? Cloud compute is becoming cheaper over time due to market forces. Its not like Amazon is cornering the market for CPUs.
> What exactly is the concern here?
Top of the article:
> Concentration of compute could threaten financial stability.
If BigCloud goes down, so does banking, and banking doesn't seem to like that - and I'm with them.
Top of the article:
> Concentration of compute could threaten financial stability.
If BigCloud goes down, so does banking, and banking doesn't seem to like that - and I'm with them.
Also changing IT in a bank is slow, dead slow. A bank wouldn’t be able to cope with being booted overnight from its infrastructure Parler-style.
That FUD is unwarranted. No bank is going to be kicked off it’s infrastructure. Bringing up Parler seems completely irrelevant.
While Parler is somewhat irrelevant and the likelihood of any bank being taken down for similar reasons as Parler, it isn't completely incomprehensible.
There have been many horror stories of businesses being banned, blocked, or messed around by Google and Amazon. Their policy does not protect anyone but themselves. It is within the realms of reality for bank to be taken down by them in a matter of days.
There have been many horror stories of businesses being banned, blocked, or messed around by Google and Amazon. Their policy does not protect anyone but themselves. It is within the realms of reality for bank to be taken down by them in a matter of days.
I have my own horror story, and that's not even about the cloud, just how centralizing a service magnifies issues.
I work for near FAANG company who still sends important email information out to people. One day, many years ago, a new gmail feature landed: automatic smart tabs - and ALL of our email started landing in promotions. And those mails were send from IP addresses which were dedicated for these and only these kind of email, so we started to panic.
We started going through the correct channels for reporting this as a problem, at which point we got a "cheers, we'll get back to you in 2 weeks". At that point we were certain we were loosing money in the millions soon, so we walked over to marketing - Google PPC (pay per click) to be specific, and asked them to get hold of someone high enough at Google, we don't care how, or over what channel.
Within an hour, the change in gmail to put everything in promotions that was "noreply@" was rolled back, and our arses were saved.
If we were still in the early 2000s with countless small email servers and providers, if one of them done this, their customers would be angry at them. But since people believe Gmail is their saviour - and because nobody can get hold of anyone at google to report a problem - now it was our fault, despite the fact that we had absolutely no idea what went wrong, and there were no changes on our side.
My summary? Fuck the cloud and the centralized services; I want the small, loosely connected internet services back.
I work for near FAANG company who still sends important email information out to people. One day, many years ago, a new gmail feature landed: automatic smart tabs - and ALL of our email started landing in promotions. And those mails were send from IP addresses which were dedicated for these and only these kind of email, so we started to panic.
We started going through the correct channels for reporting this as a problem, at which point we got a "cheers, we'll get back to you in 2 weeks". At that point we were certain we were loosing money in the millions soon, so we walked over to marketing - Google PPC (pay per click) to be specific, and asked them to get hold of someone high enough at Google, we don't care how, or over what channel.
Within an hour, the change in gmail to put everything in promotions that was "noreply@" was rolled back, and our arses were saved.
If we were still in the early 2000s with countless small email servers and providers, if one of them done this, their customers would be angry at them. But since people believe Gmail is their saviour - and because nobody can get hold of anyone at google to report a problem - now it was our fault, despite the fact that we had absolutely no idea what went wrong, and there were no changes on our side.
My summary? Fuck the cloud and the centralized services; I want the small, loosely connected internet services back.
These things do happen, but I'd be willing to place money on it not happening to a major financial institution. The day AWS and friends summarily terminate the account of a major UK bank, causing people to lose the ability to access their money, would also be the day that every company in the country immediately pulls out the business continuity plan and digs into the section on dealing with having to migrate to a new provider.
I doubt the payment systems would run on a 3rd party cloud provider. But there are many other systems used by a bank, to calculate their risks, run their accounting, do their regulatory reporting, all the network drives that are used across the bank, emails, that could be killed in such a scenario. That alone is enough to threaten the stability of the banking system.
Any provider offering this kind of service is exceptionally unlikely to offboard any financial services firm without serious forethought- any outsourcing of critical processes like this would need pre-approval from the FCA and for a bank the PRA would take a hard look too.
Guess, as Bank you just need to make sure you have Amazon and Google as customer :)
Kick a UK bank off cloud infrastructure for no good reason and you will get rung up by the PM, who will make it clear that you will reinstate them or be forever banned from government procurement.
The issue is that the Cloud provider is not UK-based. If they were to ban a bank, it's probably due to pressure from their own government (US) who could be having some problems with their client's government (UK). It's not like it is not happening right now.
Sure, just like all the other people who have been called to parliament and suffered the consequences. Ah wait..
Our current leaders are useless in this regard. They'll be given a stern warning letter and nothing will come of it.
Our current leaders are useless in this regard. They'll be given a stern warning letter and nothing will come of it.
I have little doubt that the major cloud providers would be willing to contractually bind themselves to a notice and appeals period that would suffice for “because we don’t want your business anymore” reasons.
The cross-bank correlated technical outages and general “fear of the new (15 years now, but continually being enhanced and changed)” is harder to get beyond.
The cross-bank correlated technical outages and general “fear of the new (15 years now, but continually being enhanced and changed)” is harder to get beyond.
Notice and appeals period isn't good enough. It takes UK banks millions to migrate to a cloud provider. Being given a notice period just means they have less than x amount of time to potentially spend equally as much as it took for them to get onto the cloud provider in the first place.
A lot of people hate banks and want them to go down, just like how a lot of people hate Parler.
This is obvious, concentration of risk is a legitimate concern. I was specifically wondering about the "dictate terms and conditions - as well as prices" line, which sounds bullshit to me.
How is that different than the banks own systems? In the UK, we have seen about 4 or 5 really bad system failures in banks leading to unpaid salaries/bills etc. and I believe all of them were on-prem.
Sure, hedge your bets but I would hope that most people using cloud are smart enough to at least get it 95% cloud-agnostic so if price gouging occurs, they can leave.
Sure, hedge your bets but I would hope that most people using cloud are smart enough to at least get it 95% cloud-agnostic so if price gouging occurs, they can leave.
If the ATM stops working because of an AWS outage, I will very much be disappointed in my banking regulator.
This is a good conversation to have. There are no meaningful guarantees that cloud providers won’t scrape data or ideas.
Then there’s the cost: Cloud is extremely expensive over the long run; more-so than the equivalent on-prem.
The long term solution here is to add better Internet in all regions, similar to our highway system today.
Then any company of a large enough size can build out their own DCs on the cheap.
It would also inject much needed dollars into the non-coastal areas.
Then there’s the cost: Cloud is extremely expensive over the long run; more-so than the equivalent on-prem.
The long term solution here is to add better Internet in all regions, similar to our highway system today.
Then any company of a large enough size can build out their own DCs on the cheap.
It would also inject much needed dollars into the non-coastal areas.
Some ups and downs, but cloud’s value isn’t really for its costs.
The risk Bank of England doesn't like comes with the territory, and everybody who uses public cloud is subject to it. The only difference is that smaller companies don't have a big megaphone or the clout that they have, and must either accept the terms and shut up or find an alternate solution. If Bank of England wants to use their power constructively, they can literally go shopping for data center and IT companies to serve their needs.
> risk Bank of England doesn't like comes with the territory
The BoE is the central bank; to a great extent they get to define the territory. Especially with regard to risk. Basel III and that kind of thing.
> they can literally go shopping for data center and IT companies to serve their needs.
No, it's not about what they themselves buy, it's about what all the banks in the UK buy. A situation where all the banks are using the same cloud provider(s) is unacceptable because of the high risk of correlated failure. It's bad enough when one bank goes down and its ATMs stop working; if EU-west-1 goes down and takes out all the banks, that's a disaster, and the BoE is rightly taking steps to prevent it.
The BoE is the central bank; to a great extent they get to define the territory. Especially with regard to risk. Basel III and that kind of thing.
> they can literally go shopping for data center and IT companies to serve their needs.
No, it's not about what they themselves buy, it's about what all the banks in the UK buy. A situation where all the banks are using the same cloud provider(s) is unacceptable because of the high risk of correlated failure. It's bad enough when one bank goes down and its ATMs stop working; if EU-west-1 goes down and takes out all the banks, that's a disaster, and the BoE is rightly taking steps to prevent it.
Ahh, I stand corrected.
What BoE can say is that certain risks are unacceptable for UK banks, period. It's not BoE shopping for data centers, it would be BoE setting the rules for every bank shopping for data centers.
If BoE does not like some risk that really does come with the territory and everybody who uses public cloud is subject to that risk, oh well, then that would imply that UK banks have to keep out of that territory and can't use public cloud for their key services - it is a bit drastic, but certainly something that BoE has the right to rule if it really wanted.
On the other hand, if there's a way to use public cloud and meet the requirements, but current major vendors are simply not offering it, then BoE can make up arbitrary requirements, followed by the major UK banks going to the major cloud vendors with a proposal "meet these requirements or we won't use your services, because we'd be prohibited to".
If BoE does not like some risk that really does come with the territory and everybody who uses public cloud is subject to that risk, oh well, then that would imply that UK banks have to keep out of that territory and can't use public cloud for their key services - it is a bit drastic, but certainly something that BoE has the right to rule if it really wanted.
On the other hand, if there's a way to use public cloud and meet the requirements, but current major vendors are simply not offering it, then BoE can make up arbitrary requirements, followed by the major UK banks going to the major cloud vendors with a proposal "meet these requirements or we won't use your services, because we'd be prohibited to".
The risk that they're concerned with is if every bank is using AWS, then AWS goes down, the entire UK economy crashes until it's back up.
Even if AWS itself is more reliable than every bank's on-prem solution, that's no good if it goes down for everyone simultaneously and I can't just use a backup credit card.
Even if AWS itself is more reliable than every bank's on-prem solution, that's no good if it goes down for everyone simultaneously and I can't just use a backup credit card.
Exactly, they're concerned with the systemic risk. As an example, there's no visibility or processes in place to ensure that all these banks aren't sharing the same availability zone.
They're likely only looking for ways to mitigate these risks, not necessarily a complete return to self-hosting.
They're likely only looking for ways to mitigate these risks, not necessarily a complete return to self-hosting.
I would say it's nearly a 50/50 split until we have conversations about how our product actually works and the incredibly sticky problem that is PII...
Once the risks are reviewed in open and honest ways, we find that virtually all of our clients would prefer to keep our solution on-prem. Only those who have already made a full step into cloud compute have to continue to take exception for obvious practicality reasons.