Deutsche Bank's “dysfunctional” IT division (2018)(efinancialcareers.co.uk)
efinancialcareers.co.uk
Deutsche Bank's “dysfunctional” IT division (2018)
https://www.efinancialcareers.co.uk/news/2018/04/technology-at-deutsche-bank
175 comments
That just sounds like a normal day of interacting with basic German IT infrastructure. It's all the same.
My personal favorite is the website for reporting large international transactions, which can only be using during regular working hours on weekdays.
What exactly is that? Is it for moving savings from a non-eu country into Germany ?
This thingy: https://extranet.bundesbank.de/ams
How they describe it: https://www.bundesbank.de/en/service/reporting-systems/exter...
Any single transaction exceeding 12500€ entering (or leaving) Germany has to be manually reported here. So for example if you're living in Germany and working remotely as a contractor for a US based company you'll probably have to report your income every month.
There are many exceptions that don't need manual reporting though, like if you're buying physical goods from overseas. I guess those are already automatically tracked through other ways.
For fun, I just tried to login at ~9:30 PM local time on a Sunday, and it said:
"Access Denied"
"You cannot access the resource you have requested at this time."
"The resource you have requested is protected by a policy restricting access to specific time periods. Your request has occured outside of those permitted time periods."
How they describe it: https://www.bundesbank.de/en/service/reporting-systems/exter...
Any single transaction exceeding 12500€ entering (or leaving) Germany has to be manually reported here. So for example if you're living in Germany and working remotely as a contractor for a US based company you'll probably have to report your income every month.
There are many exceptions that don't need manual reporting though, like if you're buying physical goods from overseas. I guess those are already automatically tracked through other ways.
For fun, I just tried to login at ~9:30 PM local time on a Sunday, and it said:
"Access Denied"
"You cannot access the resource you have requested at this time."
"The resource you have requested is protected by a policy restricting access to specific time periods. Your request has occured outside of those permitted time periods."
> This thingy: https://extranet.bundesbank.de/ams
My favourite thing on that website is the script block inside a noscript tag that they actually expect to run. Things like that tend to suggest the developers have absolutely no idea how to code for a web browser.
My favourite thing on that website is the script block inside a noscript tag that they actually expect to run. Things like that tend to suggest the developers have absolutely no idea how to code for a web browser.
That's one way to implement a feature flag I guess, not the way many sane people would choose.
For some of us, using a middle company such as Deel might be a good idea. They take some money for that, but you get a real German contract and don't have to care dealing with the local bureaucracy.
To be honest, average American IT infrastructure is the same; the larger the business, the worse it seems to be. Large tech companies like Google are outliers.
Rather, the large tech companies are some of the worst offenders, but it's coupled with egregious arrogance and a self-held belief that they're better and smarter than everyone else. Google is in an even more unique position as they can bully their way around the internet by breaking and influencing standards at their whims. Ever tried to get support from Google? Go ahead, I'll wait.
That’s a conspiracy theory.
Reality is much mundane and boring: the person writing the web app has no vested interest in the app actually providing a good service for customers. His or her boss doesn’t care either.
It’s simple disinterest: the self-interests of the coders are too far removed from the interests of the customers.
If you’re a solo dev working on a startup and every lost customer is money gone from your pocket, you will pay attention to every little detail.
If you’re a cog in a giant soulless machine, you do the bare minimum and go home.
Reality is much mundane and boring: the person writing the web app has no vested interest in the app actually providing a good service for customers. His or her boss doesn’t care either.
It’s simple disinterest: the self-interests of the coders are too far removed from the interests of the customers.
If you’re a solo dev working on a startup and every lost customer is money gone from your pocket, you will pay attention to every little detail.
If you’re a cog in a giant soulless machine, you do the bare minimum and go home.
Not just business. My dad worked for the FBI in IT for about 15 years after he retired from the military. The stories he can tell….
I bank with Chase and the app is really good. Are there major US banks with crappy apps?
Can second this. I'm not even living in the US any longer, and yet I have never had an instance where the Chase website or app failed to provide a wanted service or work as expected. Their I.T, at least from the customer point of view, seems top notch.
I was pretty impressed with the internal IT situation working at JPMC. It's not perfect, but reading this about Deutsche Bank I can see what the alternative is and it's rightfully horrifying.
The US is not anywhere near as bad as Deutsche that's for sure. I think whether or not the app is crappy depends on your perspective in the US. I've never seen one try to give me a pin instead of a password, and like you said, some of them are pretty good like Chase.
PNC Bank’s mobile app and website are pretty bad. The only reason I haven’t switched is due to friction and because I have my mortgage with them. The website won’t work unless I allow 3rd party scripts from Adobe to run. The UX is terrible too.
Odd.. I use PNC website and app regularly and my one complaint is how difficult it is to find my account number. No need for third party scripts.
[deleted]
Last I heard, Google staff use Concur to handle travel... there's no escaping SAP (and similarly frustrating enterprise software) in large corporations, even FAANG ones.
Is SAP good at anything aside from locking customers in? I ask this seriously because I am an outsider on this, but I have never heard a single good thing about SAP. NOT ONE.
[deleted]
>Google staff use Concur to handle travel
I mean what else should they use?
I mean what else should they use?
Zoho? Airbase? Rydoo?
Nobody got fired for using Concur
maybe so but they took like a month to call me back about an implementation and I ended up using Certify at my company
works okay but the credit card integration is a complete sh*t show
works okay but the credit card integration is a complete sh*t show
Have you used Bank of America’s app? It is quite good.
They sent me a very phishing looking email.
From a domain like db-service.com, only text “there is a problem with your account, please call xxx”, with xxx a number that was nowhere to be found on their website.
Turned out to be a legit inquiry from some audit department.
From a domain like db-service.com, only text “there is a problem with your account, please call xxx”, with xxx a number that was nowhere to be found on their website.
Turned out to be a legit inquiry from some audit department.
German speaking financial institutions are a special kind of IT hell. Perhaps it's a cultural love for rules in combination with compliance that evolves in a hellish bureaucracy, perhaps it's something in the local beer. We can only speculate but I've never worked in an environment as dysfunctional as the ones I've seen at big German companies.
To give a few examples:
- Running all services under a single Linux account, then requiring vendors to support password encryption in the startup scripts (everyone can access our production passwords!), then proceeding to store the production password in plain text in GIT and Confluence where everyone could read it.
- Everyone (including contractors) had access to the production environment through the testing environments, including unrestricted access to the production database (without requiring as much as a password). This was not deemed nearly as important as the passwords in the point above.
- I received an official warning for disclosing to AWS/GCloud that we did not have CI/CD setup but would be interested in getting one up and running at some point. This was considered to be "sensitive" information.
- Ansible deployments taking 12 hours (on a 6 server environment) because as it turns out, some random "security" settings slowed down SSH, incompetent people not really understanding what they we're doing did the rest.
- Ansible being blocked because someone in the security department read that it only required HTTPs access, turned out that they were reading the documentation for the WebUI.
- Requiring manual deployment of PR's to test environments and manual testing, the "evidence" + logs of doing this was to be documented in a Word document and attached to the PR.
- Automation and DevOps was banned because "The law does not allow us to do this"
- No rsync due to security reasons, great fun when you had to sync 200k small files :)
- PRs were kept around for months, so "senior" developers could estimate the risk. This included fixes that were hotfixed in production so that it was guaranteed to break again at the next 12 hour deployment (thus giving the same developers even more reasons to be even more "cautious" with all future PRs)
- Three different database schema's, three different database technologies, microservices, a React frontend, 50 outsourced developers for an internal CRUD application with 2000 entities, to be used by 10-20 users
- All .exes not on a whitelist were banned, so in order to run your application you just had to rename it to word.exe :)
- At one point external developers were hired, we requested a desktop for them that would allow them to "write" code. They got a locked down VM with notepad installed and nothing else
To give a few examples:
- Running all services under a single Linux account, then requiring vendors to support password encryption in the startup scripts (everyone can access our production passwords!), then proceeding to store the production password in plain text in GIT and Confluence where everyone could read it.
- Everyone (including contractors) had access to the production environment through the testing environments, including unrestricted access to the production database (without requiring as much as a password). This was not deemed nearly as important as the passwords in the point above.
- I received an official warning for disclosing to AWS/GCloud that we did not have CI/CD setup but would be interested in getting one up and running at some point. This was considered to be "sensitive" information.
- Ansible deployments taking 12 hours (on a 6 server environment) because as it turns out, some random "security" settings slowed down SSH, incompetent people not really understanding what they we're doing did the rest.
- Ansible being blocked because someone in the security department read that it only required HTTPs access, turned out that they were reading the documentation for the WebUI.
- Requiring manual deployment of PR's to test environments and manual testing, the "evidence" + logs of doing this was to be documented in a Word document and attached to the PR.
- Automation and DevOps was banned because "The law does not allow us to do this"
- No rsync due to security reasons, great fun when you had to sync 200k small files :)
- PRs were kept around for months, so "senior" developers could estimate the risk. This included fixes that were hotfixed in production so that it was guaranteed to break again at the next 12 hour deployment (thus giving the same developers even more reasons to be even more "cautious" with all future PRs)
- Three different database schema's, three different database technologies, microservices, a React frontend, 50 outsourced developers for an internal CRUD application with 2000 entities, to be used by 10-20 users
- All .exes not on a whitelist were banned, so in order to run your application you just had to rename it to word.exe :)
- At one point external developers were hired, we requested a desktop for them that would allow them to "write" code. They got a locked down VM with notepad installed and nothing else
> - PRs were kept around for months, so "senior" developers could estimate the risk. This included fixes that were hotfixed in production so that it was guaranteed to break again at the next 12 hour deployment (thus giving the same developers even more reasons to be even more "cautious" with all future PRs)
Classic strategy :D, those aren't seniors: they're Gods.
Classic strategy :D, those aren't seniors: they're Gods.
I worked for a Swiss firm and never noticed anything strange.
Both founders are German speaking. Good guys.
Although that was a small firm not a giant
Both founders are German speaking. Good guys.
Although that was a small firm not a giant
I work for a small Swiss firm at the moment as well, the smaller firms are a lot better especially in Zurich where Google also has a big office (and Meta and Nvidia have smaller offices) so there's lots of cross pollination.
The firm in my post above is very much a Swiss firm though, you've probably heard of them, they're not a bank.
The firm in my post above is very much a Swiss firm though, you've probably heard of them, they're not a bank.
[deleted]
[deleted]
Most likely it's a lack of talent, and lack of political will to actually take IT seriously.
>Most likely it's a lack of talent
Lack of talent is not a national issue in Germany. German unis and schools produces great devs, along with the slef-taught "hackers" and tinkerers which are quite numerous there. The whole "talent shortage" in Germany is propaganda from shitty companies engaging in wage suppression, and actually a shortage of pay that keeps decent devs away from such companies.
It's more the lack of good pay/equity to motivate people, but the main issues are the immovable and outdated bureaucracy without any wish to address it, and the heavy top - down approach where top management is clueless and dysfunctional, scaring away the top talent over time and ensuring that only the equally clueless yes-men stay, and the result is what wee can see.
These large German "ships" are only good at bureaucracy, maintaining the jobs and status quo for those at the top, not innovation, efficiency(despite the stereotype) or customer satisfaction.
My favorite about German companies is project management done in Excel, and bosses who don't want to use Jira and demand their seniors send them emails explaining the current work status. You can't make this up.
Lack of talent is not a national issue in Germany. German unis and schools produces great devs, along with the slef-taught "hackers" and tinkerers which are quite numerous there. The whole "talent shortage" in Germany is propaganda from shitty companies engaging in wage suppression, and actually a shortage of pay that keeps decent devs away from such companies.
It's more the lack of good pay/equity to motivate people, but the main issues are the immovable and outdated bureaucracy without any wish to address it, and the heavy top - down approach where top management is clueless and dysfunctional, scaring away the top talent over time and ensuring that only the equally clueless yes-men stay, and the result is what wee can see.
These large German "ships" are only good at bureaucracy, maintaining the jobs and status quo for those at the top, not innovation, efficiency(despite the stereotype) or customer satisfaction.
My favorite about German companies is project management done in Excel, and bosses who don't want to use Jira and demand their seniors send them emails explaining the current work status. You can't make this up.
Developer hoping for management to use Jira? Now I've seen it all.
Just kidding, nightmarish as it is, beats emails and excel for sure.
Just kidding, nightmarish as it is, beats emails and excel for sure.
For some reason, I think forcing the use of a word document in a PR is the funniest part of these anecdotes. Thanks for sharing.
> DevOps was banned because "The law does not allow us to do this"
I want that on a shirt lol
I want that on a shirt lol
I have some of these too.
- technologies had to be evaluated to be allowed on the network. This process took years. Git (as a replacement for subversion) had been in evaluation for over 14 years at the time.
- network authentication mechanisms made building an http/s Webservice internally impossible without the security teams approval which takes months.
- ironically the 'rules' around 'security' were regularly broken and not audited when it was convenient for someone with the clout to make something happen.
- technologies had to be evaluated to be allowed on the network. This process took years. Git (as a replacement for subversion) had been in evaluation for over 14 years at the time.
- network authentication mechanisms made building an http/s Webservice internally impossible without the security teams approval which takes months.
- ironically the 'rules' around 'security' were regularly broken and not audited when it was convenient for someone with the clout to make something happen.
Achtung baby
Similar experience with Comdirect. They also have a 6-digit pin.
When I asked support why I have to pay for my depot (even though its free if I have a checking account) they said they signed me up under two separate customer IDs, but they cannot simply consolidate them. I have to open a new depot and transfer my stocks.
When I asked support why I have to pay for my depot (even though its free if I have a checking account) they said they signed me up under two separate customer IDs, but they cannot simply consolidate them. I have to open a new depot and transfer my stocks.
I would imagine a water fire to be a very interesting phenomenon, no?
This is not surprising at all, and is not specific to Deutsche Bank or even banking in general.
This is a property of large, bloated legacy companies in any field. IT is still seen as a cost center and a secondary concern rather than the enabler of their business. As a result, pay/resources and "political capital" (for the lack of a better word) are allocated accordingly.
IT folks there aren't given the pay nor recognition they deserve, so no good talent joins or stays for long enough. Junior talent that joins ends up just learning from the mess and has no chance of actually becoming "good", so the problem continues.
Furthermore, the messy and unefficient IT systems benefit many people there, from lower-level menial positions whose jobs would be obsoleted by good IT to managerial positions who have a large list of reports to manage which gives them prestige and justifies their salary. Third-party suppliers also benefit as a bad IT system requires constant attention while a good system would require less attention (and a competent in-house team can attend do it, requiring no third-party involvement). Bad IT can also serve as cover - problems can be blamed on it instead of incompetence.
Fixing it incrementally from inside is politically impossible as people who rely on the status-quo will fight you on every step of the way. The only potential way is the organizational equivalent of a "full rewrite" - set up a subsidiary, give it unlimited money and task it with building a competing product. Operate it like a startup with the appropriate culture (especially regarding tech). Once the product is competitive, migrate customers onto it over time. This should be feasible at least for retail banking as UK fintech startups proved it's not actually impossible to create a bank from scratch. Rinse and repeat for every vertical of the business.
This is a property of large, bloated legacy companies in any field. IT is still seen as a cost center and a secondary concern rather than the enabler of their business. As a result, pay/resources and "political capital" (for the lack of a better word) are allocated accordingly.
IT folks there aren't given the pay nor recognition they deserve, so no good talent joins or stays for long enough. Junior talent that joins ends up just learning from the mess and has no chance of actually becoming "good", so the problem continues.
Furthermore, the messy and unefficient IT systems benefit many people there, from lower-level menial positions whose jobs would be obsoleted by good IT to managerial positions who have a large list of reports to manage which gives them prestige and justifies their salary. Third-party suppliers also benefit as a bad IT system requires constant attention while a good system would require less attention (and a competent in-house team can attend do it, requiring no third-party involvement). Bad IT can also serve as cover - problems can be blamed on it instead of incompetence.
Fixing it incrementally from inside is politically impossible as people who rely on the status-quo will fight you on every step of the way. The only potential way is the organizational equivalent of a "full rewrite" - set up a subsidiary, give it unlimited money and task it with building a competing product. Operate it like a startup with the appropriate culture (especially regarding tech). Once the product is competitive, migrate customers onto it over time. This should be feasible at least for retail banking as UK fintech startups proved it's not actually impossible to create a bank from scratch. Rinse and repeat for every vertical of the business.
This is not surprising at all, and is not specific to Deutsche Bank or even banking in general. This is a property of large, bloated legacy companies in any field.
There is also a large cultural component. Having lived both in The Netherlands and Germany, dealing with banks or government institutions is a stark difference. The large Dutch banks (eg. ING) are efficient, you can arrange everything online and they have modern apps. With German banks for every small administrative change we had to go to a physical office and the apps/website looked as if they were literally from 1999 (and weird security practices like using a pin to log in). Similar with government institutions, in NL almost everything can be arranged online with a single authentication method (everything from requesting a parking permit to filing for taxes). In Germany it's a lot of literal paperwork, going to offices, waiting in line, and no integration between administration of different public institutions.
My wife (who is German) says that one factor is German history (nazis and the stasi), which makes people distrust any kind of central (digital) administration. So all databases are uncoupled from eachother, people prefer cash, etc. Another thing that is really stifling in Germany is the hierarchy, there is so much formalistic bullshit around company/government/... hierarchies. I experienced this a lot when working with the university administration (I was a researcher/lecturer at a German university). No one dares to make choices that can have some impact, afraid that higher-ups, the court, or whatever will punish them for making the wrong choice. So they hide behind rules and necessary decisions are not made/postponed. Almost no one stands up to their leadership, so even they have wrong ideas, there is rarely any counterbalance.
There is also a large cultural component. Having lived both in The Netherlands and Germany, dealing with banks or government institutions is a stark difference. The large Dutch banks (eg. ING) are efficient, you can arrange everything online and they have modern apps. With German banks for every small administrative change we had to go to a physical office and the apps/website looked as if they were literally from 1999 (and weird security practices like using a pin to log in). Similar with government institutions, in NL almost everything can be arranged online with a single authentication method (everything from requesting a parking permit to filing for taxes). In Germany it's a lot of literal paperwork, going to offices, waiting in line, and no integration between administration of different public institutions.
My wife (who is German) says that one factor is German history (nazis and the stasi), which makes people distrust any kind of central (digital) administration. So all databases are uncoupled from eachother, people prefer cash, etc. Another thing that is really stifling in Germany is the hierarchy, there is so much formalistic bullshit around company/government/... hierarchies. I experienced this a lot when working with the university administration (I was a researcher/lecturer at a German university). No one dares to make choices that can have some impact, afraid that higher-ups, the court, or whatever will punish them for making the wrong choice. So they hide behind rules and necessary decisions are not made/postponed. Almost no one stands up to their leadership, so even they have wrong ideas, there is rarely any counterbalance.
> My wife (who is German) says that one factor is German history (nazis and the stasi), which makes people distrust any kind of central (digital) administration. So all databases are uncoupled from eachother, people prefer cash, etc.
True.
> Another thing that is really stifling in Germany is the hierarchy, there is so much formalistic bullshit around company/government/... hierarchies. I experienced this a lot when working with the university administration (I was a researcher/lecturer at a German university). No one dares to make choices that can have some impact, afraid that higher-ups, the court, or whatever will punish them for making the wrong choice. So they hide behind rules and necessary decisions are not made/postponed. Almost no one stands up to their leadership, so even they have wrong ideas, there is rarely any counterbalance.
This depends a lot on the company, and is something that you, in my experience, mostly experience at big companies, government-run companies and government agencies (this includes university administrations, even though at many faculties, things are completely fine (even many professors prefer to keep away from the university administration as far as it is possible in their job)). So, if you really don't like this, attempt to look for a smaller company as your employer and/or find a job in the private sector.
True.
> Another thing that is really stifling in Germany is the hierarchy, there is so much formalistic bullshit around company/government/... hierarchies. I experienced this a lot when working with the university administration (I was a researcher/lecturer at a German university). No one dares to make choices that can have some impact, afraid that higher-ups, the court, or whatever will punish them for making the wrong choice. So they hide behind rules and necessary decisions are not made/postponed. Almost no one stands up to their leadership, so even they have wrong ideas, there is rarely any counterbalance.
This depends a lot on the company, and is something that you, in my experience, mostly experience at big companies, government-run companies and government agencies (this includes university administrations, even though at many faculties, things are completely fine (even many professors prefer to keep away from the university administration as far as it is possible in their job)). So, if you really don't like this, attempt to look for a smaller company as your employer and/or find a job in the private sector.
... but then you have two competing organizations: OLD versus NEW. This is how it plays out: NEW starts as your shiny, agile, start-up-ish org with all the young people with the latest ideas. A company nearby got rid of all their freelancers? Great, let´s hire all of them. Think big. Great meetings and people seem to really make progress happen. We´re getting rid of all the REST APIs and introduce Apache Kafka as a message bus for the company. Only then progress stalls and OLD slowly begins to crawl back in. Legal department audits the agile process and unfortunately German laws are quite tough on bogus self-employment. All the freelancers have to go and NEW loses all their expertise. Other employees follow as this whole thing does not pay that great and looks more and more like any other 9/5 job.
Fast forward, 5 years later: the startup spirit is long gone. Corporate culture at NEW mirrors OLD. And not only culture wise. The NEW, "-tech" company is not seen as an independent company any more but supposed to be merged with the parent company. Because why have two of the same, right? Meanwhile OLD made some progress and introduced some reform projects. So developers do not have to fill out a printed paper application to get new servers any more. Only now you have two Kafka clusters with completely different setups as OLD also started their one one at some point.
My learning here: new is not new if the culture stays the same. Also: never underestimate the power of old. People always talk about the new, shiny stuff. But old was there first. And is much more resilient than it seems.
My learning here: new is not new if the culture stays the same. Also: never underestimate the power of old. People always talk about the new, shiny stuff. But old was there first. And is much more resilient than it seems.
Paying good salaries (2x market rate) to 300 people will still be much cheaper than paying market rates to thousands of people (which is what legacy orgs currently do) and then the problem goes away. Disguised self-employment is not the only way to run a successful engineering team.
> paying market rates to thousands of people (which is what legacy orgs currently do)
* paying contractor rates to thousands of people
* paying contractor rates to thousands of people
there are a whole array of "neobanks" in Australia, a few of which have been bought out by large incumbents and their IT platforms subsumed into the colossus or whatever.
I can hazard a guess why this hasn't happened in Germany but I don't want to make assumptions based on stereotypes, lol.
I can hazard a guess why this hasn't happened in Germany but I don't want to make assumptions based on stereotypes, lol.
No this definitely happened in Germany too, Solaris has been around for a long time and a whole slew of German neobanks and trading apps have been built on top of it. You can open an account with most of them by downloading the app and doing a video identification process with your id.
Culturally though, older Germans mistrust then and want to use the older, established banks with physical branches.
Culturally though, older Germans mistrust then and want to use the older, established banks with physical branches.
The problem is that some of these neobanks suck too, e.g. N26 is a bit notorious for bad security (at least in the past).
> The problem is that some of these neobanks suck too, e.g. N26 is a bit notorious for bad security (at least in the past).
In the case of N26, the exploits that CCC found were disclosed and fixed before the presentation, this was also a while ago now, 2016-17 maybe? I'm not sure what the situation is now, though.
In the case of N26, the exploits that CCC found were disclosed and fixed before the presentation, this was also a while ago now, 2016-17 maybe? I'm not sure what the situation is now, though.
N26 is also infamous for being the go-to-bank for cybercriminals. That may be because they are easier to fool than your local bank branch where most financial stuff happens in Germany, but my impression is, that they don't care...
Interesting, makes sense. Deutsche Bank has a pretty bad reputation for money laundering, as does Germany in general and especially Berlin.
Or they were initially started by the big banks anyway. It's funny, when we moved to Australia we opened a joint NAB account. My wife still had a Citibank account. Shortly thereafter NAB bought Citibank, now we had two NAB accounts. Then we opened a Ubank account for better interest and turns out that one is also owned by NAB. Can't escape them...
I have so many good stories from developing software inside banking.
Every management team I have ever worked for in banking - I have worked 4 stints in 4 different banks as a contractor - has been so old school that its been impossible to apply any up to date constructs from the professional software world.
I once had a manager who had been with the bank for 40 years. He simply drawed 3 boxes on a board, drawed a few lines between them and asked "how long will this take ?"
Hmm...what is in the first box?
Old school, grey and stale. That's banking for you in Europe at least.
Every management team I have ever worked for in banking - I have worked 4 stints in 4 different banks as a contractor - has been so old school that its been impossible to apply any up to date constructs from the professional software world.
I once had a manager who had been with the bank for 40 years. He simply drawed 3 boxes on a board, drawed a few lines between them and asked "how long will this take ?"
Hmm...what is in the first box?
Old school, grey and stale. That's banking for you in Europe at least.
I have to run integrations with things those sort of people designed. We get EDI streams and error codes straight from some COBOL thing from the 70s which has been carefully ported through to whatever crap they're running today verbatim.
Recently there was an issue where a field overflowed which broke literally everything. Then they added a zero to the start as part of the specification which broke it even more because half the companies on the planet were treating it as a string and not a number.
If there's anything I've ever learned about working in the financial sector it is that the only thing that is protecting you is the legal framework around it, not the banks or financial companies which are universally a fucking shit show.
Recently there was an issue where a field overflowed which broke literally everything. Then they added a zero to the start as part of the specification which broke it even more because half the companies on the planet were treating it as a string and not a number.
If there's anything I've ever learned about working in the financial sector it is that the only thing that is protecting you is the legal framework around it, not the banks or financial companies which are universally a fucking shit show.
Most banks are still running COBOL at the core, I think. That's at least the case here in South Africa. Each bank then has large IT teams building interfaces and pretty things around the core, using languages such as C# and Java.
It varies vastly depending on the bank and what part. I know from my career some of the most advanced computing platforms I’ve ever seen across any industry were in leading banks with sophisticated quant trading divisions - but only in the trading front office. Goldman in the late 1990’s through 2010 or so was bar none extraordinary with some of the most brilliant human beings alive developing systems that are unmatched today. On the other hand in the same bank some of the crustiest most process laden software management was crushing the life out of human souls.
They acquired Postbank (where I have an account) some years ago. Postbank had a reputation for good IT. Now they are consolidating their online banking system... Of course, on the bad system of Deutsche Bank. It's a huge mess. I need to log into two different systems for credit card and bank account things. They mailed me account statements for the timeframe of one day once and for a few days another time. I never get printed statements otherwise. The switchover is also months behind schedule.
So, Deutsche Bank is still crap at IT.
So, Deutsche Bank is still crap at IT.
I have been in the process of being migrated from Postbank to Deutsche Bank since about a year now (no, not a typo). They occasionally send letters, emails, pop-ups etc. that at some weekend they will do a migration. There is always some online banking downtime for that and for some reason the CC migration seems to have broken something with my account. I just got a different CC and called it good, I need a reliable way to pay one-off things.
It has gotten that bad that the regulator is starting to get interested [1, German]. Luckily I was not hit by the more serious problems of missed payments. Really "funny" to see that nothing has changed since 2018, I would have thought this is a current article if it weren't for the year in the title...
[1] https://www.tagesschau.de/wirtschaft/verbraucher/umstellung-...
P.S. for the Germans looking for a privacy.com alternative (not available in Germany): Revolut took less than 30min to setup from scratch to first CC payment, not affiliated or anything, just really happy with a financial service not stuck in the past
It has gotten that bad that the regulator is starting to get interested [1, German]. Luckily I was not hit by the more serious problems of missed payments. Really "funny" to see that nothing has changed since 2018, I would have thought this is a current article if it weren't for the year in the title...
[1] https://www.tagesschau.de/wirtschaft/verbraucher/umstellung-...
P.S. for the Germans looking for a privacy.com alternative (not available in Germany): Revolut took less than 30min to setup from scratch to first CC payment, not affiliated or anything, just really happy with a financial service not stuck in the past
I've worked in IT (system Integration) for Postbank a number of years ago... Their IT was never good when you can see how the sausage is made
I can recommend Comdirect. Have been with them for a number of years. It's not a perfect all digital experience but largely a good offering. moved to a non-european country and was able to keep my account and stock portfolio with them (just have to tell them that you aren't a tax resident anymore so they don't deduct captial gains tax automatically). I am always a bit worried about using neo-anything for the core of my savings since I would like to not have to touch then for a long time. Having to sell or move all your assets because a startup shuts down isn't fun.
Uh oh, I was considering opening an account with them… actually they couldnt in July, because of their IT transition. Maybe I‘ll pick a different one. Any recommendations for a good Business Account in Germany?
I've been with Deutsche Bank for 14 years and it has been mostly smooth sailing. Some things require calling my account manager and/or driving to their local office with my ID card and their contract forms are ridiculously lengthy, but overall I can't remember any major issue. I use the Banking4 app for online access and import into Agenda bookkeeping. For Sepa deductions from my customers, I can import CSV files through Banking4 and then submit them with HBCI. Oh and don't be afraid to ask for rebates. Good behaviour in the past may allow you to get lower fees, because then you're lower risk for them.
Sadly, business accounts are kinda all over the place. I use Kontist, but it's pretty basic. You could also look into your local Sparkasse, it should have everything you need.
Revolut?
It's a UK bank with a Lithuanian subsidiary. Which is fine, but means you don't have a German account number, nor an easy way to sue/contact if legal problems arise. The latter is rather unlikely to be a concern, but the former might be a pain to deal with, depending on your customers. A lot of companies have rules which require double-checking every transfer to non-German accounts. Suddenly, your payments take a few weeks longer to arrive.
I’d say that’s an issue with those companies but of course that doesn’t help the situation
N26 is German fwiw similar to Revolut
N26 has a history of problems. I would recommend staying as far away as possible.
Could you elaborate on that? I've not been aware of any.
I'm just gonna link a few German articles. They seem to have reestablished their phone hotline by now, but I don't think they have changed their "move fast and break things" attitute.
[1] https://www.anwalt.de/rechtstipps/erneute-probleme-bei-n26-b... [2] https://www.businessinsider.de/wirtschaft/einem-n26-kunden-w... [3] https://www.businessinsider.de/gruenderszene/business/n26-er...
[1] https://www.anwalt.de/rechtstipps/erneute-probleme-bei-n26-b... [2] https://www.businessinsider.de/wirtschaft/einem-n26-kunden-w... [3] https://www.businessinsider.de/gruenderszene/business/n26-er...
Well, since we were just talking about how dysfunctional IT traditional banks are in germany, maybe in this context a -move fast and break things- attitude is appreciated?
I think the difference is, as a customer I don't care how dysfunctional their internal IT is, as long as the customer facing IT and support "works".
N26 is not for corporations (GmbH, UG) so cannot be used for most startups. they have poor ratings.
I tried Finom, but the experience was abysmal. They behave like some sort of shady porn subscription, not a place I would want to entrust my money with.
I’m not sure about giving money to these Neobanks. They’re not really cheaper, and have so many downsides.
I tried Finom, but the experience was abysmal. They behave like some sort of shady porn subscription, not a place I would want to entrust my money with.
I’m not sure about giving money to these Neobanks. They’re not really cheaper, and have so many downsides.
I use Volksbank and I’m really happy.
Volksbank, Raiffeisenbank, and Sparkasse are not the cheapest but very solid. They are typically too small and local to be host to the sort of high-powered CEO that optimizes shareholder value and then takes the golden parachute when it crashes. (Actually, some of them are legally cooperatives, too lazy to look up which is which.)
Deutsche Bank has made 5 acquisitions
Norisbank
1954, Bonn
Carron Energy 2004, Cardiff
Better Payment 2014, Berlin
United Financial Group 1981, Garden City
Quantiguous 2014, Mumbai
Given all of these needing to integrate together to some degree, and the bureaucracy of banks in general, add that to their resistence to change / improve something related to the tech if it still works, none of this surprises me.
You see these kind of problems in all conglomerates which have to absorb other companies into their ecosystem.
Edit :
Slightly unrelated but I have just seen that she has passed away last year from cancer.
https://www.fnlondon.com/articles/former-deutsche-bank-chief...
Edit 2 : my mistake of details
Carron Energy 2004, Cardiff
Better Payment 2014, Berlin
United Financial Group 1981, Garden City
Quantiguous 2014, Mumbai
Given all of these needing to integrate together to some degree, and the bureaucracy of banks in general, add that to their resistence to change / improve something related to the tech if it still works, none of this surprises me.
You see these kind of problems in all conglomerates which have to absorb other companies into their ecosystem.
Edit :
Slightly unrelated but I have just seen that she has passed away last year from cancer.
https://www.fnlondon.com/articles/former-deutsche-bank-chief...
Edit 2 : my mistake of details
My current consulting gig is at a government "superdepartment" that was formed by mashing together a few dozen small agencies that were previously already combined into large agencies.
Their IT was as diverse as it gets, and then they were told to move into one building and "make it work".
It's as messy and as broken as you imagine it is.
Their IT was as diverse as it gets, and then they were told to move into one building and "make it work".
It's as messy and as broken as you imagine it is.
I wonder if maybe AI could help with such an ungrateful job.
Yes because one division is sending the data in XML and a 3 letter county code and values in thousands of monetary unit and another system does things in a completely different way
Yes because one division is sending the data in XML and a 3 letter county code and values in thousands of monetary unit and another system does things in a completely different way
No, because the field that in one system is called "cash" is called "settled" in another; system A uses BCD for money, system B uses 32 bit unsigned ints reckoned in millicents, and system C uses a cash object which can define original currency, calculated currency value, guaranteed conversion preference, contract reference number, and precision.
Understanding semantics is required, not just form.
Understanding semantics is required, not just form.
Yes, you got it. But I still think AI can at least make it easier
If you mean deep learning type systems, you need to train them on vast training sets of similar data so they can learn very general patterns, similarities and connections. Like giving them 10,000 pictures of dogs, so they can learn to recognise pictures of dogs.
These banking systems are all massively complex one off systems, often with very different sets of functions or even definitions of the problem, and the implementation details are different in almost every way possible. They’re the exact opposite of the sorts of problems modern trained AI systems are good at dealing with.
These banking systems are all massively complex one off systems, often with very different sets of functions or even definitions of the problem, and the implementation details are different in almost every way possible. They’re the exact opposite of the sorts of problems modern trained AI systems are good at dealing with.
well said
IT guys are paid a lot not to be lazy.
Changing data types in an interface is a boring job (didnt Office Space joke at that?) but doing this via some broken AI will probably introduce even more spaghetti.
Reality is that those systems should be killed one by one and finally when there is one system, it should be upgraded to something better. What is very difficult.
Changing data types in an interface is a boring job (didnt Office Space joke at that?) but doing this via some broken AI will probably introduce even more spaghetti.
Reality is that those systems should be killed one by one and finally when there is one system, it should be upgraded to something better. What is very difficult.
20mil for the new system, 80mil + 5 years of implementing for the translation layers to communicate with the legacy systems.
> Reality is that those systems should be killed one by one and finally when there is one system
It take so long to replace each of the systems, by the time you get to the "one true solution" 40 years have passed and you are back where you started.
> Reality is that those systems should be killed one by one and finally when there is one system
It take so long to replace each of the systems, by the time you get to the "one true solution" 40 years have passed and you are back where you started.
If you can write one in COBOL and it can finish processing the whole set of transactions in less than 24h you may have a chance. /s
>I wonder if maybe AI could help with such an ungrateful job.
It's not necessarily ungrateful. It can even be interesting, but it all depends how everyone approaches it and the most important thing ever is: realistic timeliness and outcomes.
It's not necessarily ungrateful. It can even be interesting, but it all depends how everyone approaches it and the most important thing ever is: realistic timeliness and outcomes.
Yeah but it's hard not to feel it's ungrateful when you see the 5th "creative" way of doing something which is pretty much like running with a spare tire and thinking it's all good
One major acquisition that is missing from your list was the aquisition of Postbank in 2018.
This is part of the problem with banks with acquisitions and why they end up being so unwieldy. In the old days we used to create gateways to each by "joining" them in some short-term strategy to merge in the long-run. But as we know nothing in the long-run generally happens due to changing priorities and differing goals of teams.
Norisbank was bought 2006/2012.
https://en.m.wikipedia.org/wiki/Norisbank
https://en.m.wikipedia.org/wiki/Norisbank
Deutsche Bank looks like a gigantic organisation rampant with misaligned incentives and barely any oversight. Looking at their list of controversies (https://en.m.wikipedia.org/wiki/Deutsche_Bank#Controversies), it is absolutely unsurprising that their IT infrastructure is a disorganised and mismanaged mess. It looks like it's beyond saving, with its depressing history over the last decade and a half.
This is what I think a lot people don't understand when it comes to unruly IT systems in large organizations. Deutsche Bank does not have an IT or software engineering problem, they have a culture and corporate politics problem. Even if you have the world's best engineered systems, as soon as you have 30 different systems you'll have a giant mess, and the reason you have 30 different systems is because you have groups of people with different incentives not interested in using someone else's system.
This is really a non sequitur. It's perfectly easy to imagine a large organization which has a disorganized and mismanaged mess for their IT infrastructure, but a great compliance and regulatory culture, or vice versa.
DB happens to have both a broken engineering culture and (at least historically) an even more broken culture of mis-selling and fraud, but there's nothing to say that one always engenders the other.
DB happens to have both a broken engineering culture and (at least historically) an even more broken culture of mis-selling and fraud, but there's nothing to say that one always engenders the other.
Worked in a similar organisation also.
The culture in european banks is even more dull than I expected before working there.
The view on technology and infrastructure is very primitive... as you can also read from just the header "IT division". I initially thought that they were competent but greedy turns out the banks are really just greedy, naive and sometimes... plain stupid.
I don't know if it's unique to Europe, but I am never working for a non tech-first company again.
The view on technology and infrastructure is very primitive... as you can also read from just the header "IT division". I initially thought that they were competent but greedy turns out the banks are really just greedy, naive and sometimes... plain stupid.
I don't know if it's unique to Europe, but I am never working for a non tech-first company again.
Same experience, putting everything related to tech (printers, programming, Wifi, email) into the "IT department", which is seen as a support service to the core business of banking, instead of seeing automation as business critical in reducing overhead and managing risk.
This is why fintech (although a very overused word by now) is such a breath of fresh air in the banking industry, and desperately needed.
This is why fintech (although a very overused word by now) is such a breath of fresh air in the banking industry, and desperately needed.
> The culture in european banks is even more dull than I expected before working there.
> I don't know if it's unique to Europe, but I am never working for a non tech-first company again.
For what it's worth, Goldman Sachs is reasonably competent in tech, and used to be even more so (at least in relative terms).
Bloomberg is about half-way between the banks and the more competent tech companies.
(I worked at both, and a few other financial companies and also at Google, Facebook etc.)
> I don't know if it's unique to Europe, but I am never working for a non tech-first company again.
For what it's worth, Goldman Sachs is reasonably competent in tech, and used to be even more so (at least in relative terms).
Bloomberg is about half-way between the banks and the more competent tech companies.
(I worked at both, and a few other financial companies and also at Google, Facebook etc.)
Working on the buy side, I can tell you that GS is definitely not competent anymore, with all the cost cutting going on. Getting a bunch of CSV files on a SFTP server seems to be extremally challenging for them. Getting the numbers to match with what we're getting over email/on the website is impossible.
It's so bad that they're losing serious business on this despite being competitive on fees and service.
Bloomberg is great if you're big and bring a lot of money to the table. For any small/mid-size firm (think less than 5M ARR on Bloomberg licenses) the answer for even basic requests is often "we'll file an enhancement request" only for things to disappear and never get adjusted. To give an example, it's still not possible to get an hourly export of the trade history in CSV or XML on a SFTP server. There's workarounds but it's definitely not half-way competent.
It's so bad that they're losing serious business on this despite being competitive on fees and service.
Bloomberg is great if you're big and bring a lot of money to the table. For any small/mid-size firm (think less than 5M ARR on Bloomberg licenses) the answer for even basic requests is often "we'll file an enhancement request" only for things to disappear and never get adjusted. To give an example, it's still not possible to get an hourly export of the trade history in CSV or XML on a SFTP server. There's workarounds but it's definitely not half-way competent.
> Working on the buy side, I can tell you that GS is definitely not competent anymore, with all the cost cutting going on.
I last worked there in 2018. So my experience is a bit out-of-date.
> Bloomberg is great if you're big and bring a lot of money to the table.
My experience is from working inside of Bloomberg and trying to get stuff done. I don't know how it looks like from the outside.
Though I can tell you that eg Google was more competent on the inside when I was working there (in 2014 to 2016); but from the outside they are probably not easier to work with than Bloomberg?
I last worked there in 2018. So my experience is a bit out-of-date.
> Bloomberg is great if you're big and bring a lot of money to the table.
My experience is from working inside of Bloomberg and trying to get stuff done. I don't know how it looks like from the outside.
Though I can tell you that eg Google was more competent on the inside when I was working there (in 2014 to 2016); but from the outside they are probably not easier to work with than Bloomberg?
I almost wrote a non-US company.
In general I must say that working for tech in European firms vs US firms is night and day, but that is another discussion :)
Willing to elaborate a bit on this?
Not the person you asked but in my limited experience, tech is actually valued in the US. Sure, there is also a culture of MBAs and business types taking decisions, but they at least understand that tech is essential (regardless of the reason).
In Europe, tech is seen as "just IT" especially in non tech corporations. It's much less important than in the US.
It's a pretty minor thing but personally I think just the fact that I rarely see the distinction between "software engineering/development" and "IT" in Europe is pretty telling.
The salaries are also another indicator.
In Europe, tech is seen as "just IT" especially in non tech corporations. It's much less important than in the US.
It's a pretty minor thing but personally I think just the fact that I rarely see the distinction between "software engineering/development" and "IT" in Europe is pretty telling.
The salaries are also another indicator.
Perfectly sums up my experience dealing with the mess that is the IT of major German company in the automotive industry that I work for. Tons of mini-fiefdoms, each with its own strong-minded lord who has held that position for literal decades and who wants to expand their influence on other fiefdoms, an infrastructure where every single one of its components is incompatible and was never designed to work with anything else and that is held together with figurative duct tape and tons of depressed and overworked colleagues who would jump to another opportunity in a heartbeat but the money is actually good so we all just have to deal with it.
Yepp, we once spent about 1.5 hours in a meeting walking the org chart of such a company to figure out which 4 emails to 4 people (plus copious amounts of CCs just to add to the "urgency" and inbox bloat) I had to write to unblock the project after 2 or 3 unproductive weekly meetings. And I did that as an external supplier, I would have thought it was in their best interested to just collect what we need and get the ball rolling internally. After all, they are paying us to deliver something.
If you guys had 3 different town halls in the past couple of years where 3 completely different strategies and visions for IT were discussed, then we are probably colleagues lol
> Now that Cryan's gone, however, it's becoming apparent that his authority over some senior managers at Deutsche Bank was weak. He and Hammonds' ability to challenge Deutsche's most powerful tribal leaders appears to have been correspondingly limited. "It's going to be a far more uphill task to cut systems in future," says one senior insider. "The people who control these systems don't want them cut. It's all about politics."
> "Hammonds was innovative, but she couldn't do what she set out to do," says one DB insider. "So we're just sitting here, waiting to see what happens next."
Yikes! Sounds like an impossible uphill task.
Is there any examples of such turnarounds in the recent years?
> "Hammonds was innovative, but she couldn't do what she set out to do," says one DB insider. "So we're just sitting here, waiting to see what happens next."
Yikes! Sounds like an impossible uphill task.
Is there any examples of such turnarounds in the recent years?
As a firstborn American nerd with 100% German ancestors, it’s basically been frustrating my entire life to watch Germany whiff on basically any computer- or Internet-related technology. I can't even attribute it to the language barrier since they're all taught English at an early age. Perhaps it's that old German pride/hubris/NIH? “Stolz”… They’re a very smart people, not sure what gives. Perhaps a lack of respect for “nerd” aesthetic
If you ever visit Germany, prepare to not be impressed by their Internet access (but possibly impressed by everything else)
If you ever visit Germany, prepare to not be impressed by their Internet access (but possibly impressed by everything else)
Personal opinion as an outsider: Germany seems to be a very risk averse society. They love to plan everything a year in advance and hate surprises. Their lives are so ordered and predictable that they have an unconscious resistance to change, compared to other cultures.
> They love to plan everything a year in advance
Apart from energy independence and defense.
Apart from energy independence and defense.
It's not that Germany is uniquely bad at software. It's that the USA is uniquely good.
A culture in which software people are considered to be really low status is pretty much the global default and is still common today, even in the US. I worked for American finance types before. They couldn't understand why programmers earned so much and why their software team couldn't be completely outsourced to India. They took perverse pride in having absolutely no idea what their own software stack actually did. Last time I checked they had got rid of the only people who were any good and then outsourced the rest (to India, of course), and apparently lost the ability to ship new versions of their software in the process. Even in the US many investment banks have totally dysfunctional IT, with the possible exception of Goldman which is famous for actually being good at it (although from reading the comments in this thread maybe that has changed?).
Why is there such a thing as a "tech" firm when all firms use tech? It's because tech firm is really meaning a firm created and run by programmers, as that's the only environment in which they can get respect and a productive setup. If it weren't the case companies like Amazon could never have existed because they'd have just been crushed by other better established retailers doing the internet well.
A culture in which software people are considered to be really low status is pretty much the global default and is still common today, even in the US. I worked for American finance types before. They couldn't understand why programmers earned so much and why their software team couldn't be completely outsourced to India. They took perverse pride in having absolutely no idea what their own software stack actually did. Last time I checked they had got rid of the only people who were any good and then outsourced the rest (to India, of course), and apparently lost the ability to ship new versions of their software in the process. Even in the US many investment banks have totally dysfunctional IT, with the possible exception of Goldman which is famous for actually being good at it (although from reading the comments in this thread maybe that has changed?).
Why is there such a thing as a "tech" firm when all firms use tech? It's because tech firm is really meaning a firm created and run by programmers, as that's the only environment in which they can get respect and a productive setup. If it weren't the case companies like Amazon could never have existed because they'd have just been crushed by other better established retailers doing the internet well.
One thing to consider: Germany has a rapidly aging population so it naturally gets more unlikely to fund and get success with new technology. Imagine trying to develop some shiny new app in a country where over a quarter of the population is 60+ and most people refuse to pay cashless let alone accept any other new tech.
Even though this is from 2018, it was and still is completely applicable to every other investment bank out there.
Investment banking organizational structures are set up such that bureaucracy encourages 'silos' where teams can deliver quicker if they don't depend on other teams, and performance related pay dominates the managers (managing directors) thinking.
Having worked at Megabank, it ran at least one system on SunOS (not Solaris), well into the late 2010s. It had Windows XP, long after end-of-life.
Nothing about that story is peculiar to DB.
Investment banking organizational structures are set up such that bureaucracy encourages 'silos' where teams can deliver quicker if they don't depend on other teams, and performance related pay dominates the managers (managing directors) thinking.
Having worked at Megabank, it ran at least one system on SunOS (not Solaris), well into the late 2010s. It had Windows XP, long after end-of-life.
Nothing about that story is peculiar to DB.
DB was running LotusNotes instead of Outlook well into the 2010s.
This feels like crap-Investment-Bank poker.
I'll raise you a company-wide ticketing system written in COBOL, running on an IBM mainframe.
I'll raise you a company-wide ticketing system written in COBOL, running on an IBM mainframe.
That sounds kind of awesome though.
People build, sorry architect, incredibly complex systems and then try to stop them from being complex. I for one have learned to stop worrying and love the complexity. Maybe a naive question: why is it important that a company with thousands of people in the IT department all run on the same system? I mean I understand the many theoretical benefits of it, but in reality software is there to do something, and if you spend millions and millions and years and years on operating systems, what are you even doing for your customers? I am actually a Deutsche Bank customer and I do not care about operating systems they run. I anticipate people will tell me that surely this helps eventually fulfill business needs -- but does it really? In my experience and as others have pointed out, practically all other places are in the exact same mess, dreaming of holy grails of standardized infrastructure, clean code and engineering that will make other engineers pat you on the back. And some get it done and some can't, so basically at the end of the day you need people that will sort through the mess and get things done. It's never easy.
Once they are done with all this standardization, they will realize they just have new roadblocks, and will spend a half decade trying to fix those too.
Once they are done with all this standardization, they will realize they just have new roadblocks, and will spend a half decade trying to fix those too.
It does lead to real customer facing problems. A jungle of different systems won't have transactionality between them at all, it's all a mess of rec scripts, so customers will see a lot of weirdness like the bank being unable to decide what their balance actually is.
Deutsche Bank was one of the first places I worked at, closer to its heyday, I absolutely loved it. But it has been in decline for years, and has a lot of management issues, and fiefdoms.
One of my favorite stories to retell is my wife had a job there also and she had a medical issue from a car wreck. Her boss said the bank couldn’t function if she was doing physical therapy two hours a week, basically fired her, all the paperwork was drawn up, sent to a manager in London to sign. The manager in London was “too important” to sign things so he just leaves it for someone else to rubber stamp in his name the next day. Meanwhile my wife goes across the street (literally), gets a doctor to sign her disability paperwork, walks it into HR - and instead of doing physical therapy two hours a week she was on paid medical leave for almost a year. Her termination paperwork did get signed eventually but because she was on a protected leave they had to throw it out.
One of my favorite stories to retell is my wife had a job there also and she had a medical issue from a car wreck. Her boss said the bank couldn’t function if she was doing physical therapy two hours a week, basically fired her, all the paperwork was drawn up, sent to a manager in London to sign. The manager in London was “too important” to sign things so he just leaves it for someone else to rubber stamp in his name the next day. Meanwhile my wife goes across the street (literally), gets a doctor to sign her disability paperwork, walks it into HR - and instead of doing physical therapy two hours a week she was on paid medical leave for almost a year. Her termination paperwork did get signed eventually but because she was on a protected leave they had to throw it out.
Is getting disability for a year so easy? Afaik in Austria your GP is not enough for that. It needs to get approved by supervisors of the national insurance to get it approved to prevent cheaters with "friendly" getting fake disability claims.
Yeah, in AT you'll get referred to an Amtsarzt from the ÖGK before anything - Also pretty sure the employer is off the hook and you get the money straight from either the ÖGK (federal health insurance) or the AMS (unemployment office) for any sick / disability leave longer than 6 weeks.
I've worked as a contractor in a large bank leading a project to migrate from a plethora of backup systems into one newly built system. 30% of my time was spent on filling "paperwork". Implementation plans, risk assessments, fail back plans and so on. There was a lot of this, but people in general had positive attitude (no big difference between contractors and employees) so I think of the time as an interesting experience rather than a negative. We too had around 30 OSes to deal with, but in itself it wasn't a problem.
I suspect the problem in DB is not it's aging, disjointed IT infrastructure, but the culture.
I suspect the problem in DB is not it's aging, disjointed IT infrastructure, but the culture.
I did a short stint at BP and it was the same. The final straw for me was the 8 week process to get an internally signed TLS cert which required signoffs by like 6 VPs across every region (US, EMEA, etc.).
It paid well but fuck that noise.
It paid well but fuck that noise.
This is 5 year old scuttlebutt and even what's publicly known about DB's systems has superseded most of the details here.
Cool, can you share some of the publicly available information then?
I interviewed at DB around 2018, and from that experience (what's the worst thing about working here, I asked my interviewer. The politics, she replied) plus that of all my friends who spent time there, this article rings pretty true.
But I'd love to get some more up to date info, so would be happy if you could share.
I interviewed at DB around 2018, and from that experience (what's the worst thing about working here, I asked my interviewer. The politics, she replied) plus that of all my friends who spent time there, this article rings pretty true.
But I'd love to get some more up to date info, so would be happy if you could share.
Much fresher scuttlebutt: https://www.efinancialcareers.com/news/2022/05/eisler-capita...
Kannon isn't even mentioned in the older article.
Note that I am not claiming that Deutsche's IT department is no longer dysfunctional! Just nothing in TFA provides convincing evidence about the current state of affairs.
Kannon isn't even mentioned in the older article.
Note that I am not claiming that Deutsche's IT department is no longer dysfunctional! Just nothing in TFA provides convincing evidence about the current state of affairs.
banks are particularly low imagination, high control places. The "resources" they hire are particularly generic, safe and fungible/expendable. By definition, such "resources" are playing with themselves, creating "frameworks" and "platforms".
The low imagination management of banks cannot understand that they got "resources" they filtered for and results accordingly.
https://bitslap.it/blog/posts/playground-it.html
I very briefly worked in a similar organization a decade ago, and it checks out with how I remember things.
Though in that case half the mess were from mergers leaving the bank with multiple instances of similar yet incompatible IT systems. Another source was the wave of regulation and reporting requirements crashing onto finance after the subprime crisis.
Though in that case half the mess were from mergers leaving the bank with multiple instances of similar yet incompatible IT systems. Another source was the wave of regulation and reporting requirements crashing onto finance after the subprime crisis.
Don't think this is atypical of many organisations that have grown their IT over a long period. When I did some work for Network Rail (here in the UK) there were some absolutely hair-raising diagrams of umpteen systems all with lines between them, doubtless many going back to BR days.
This makes me excited for what current microservices architecture systems will look like in fifty years!
Microservices just means you get that kind of mess in months instead of years/decades, and that mess is actually the objective.
True, but at least those lines (you'd hope) will represent some json going over http, whereas with these legacy systems the lines could be more or less anything!
maintaining web frontends will also be an infinite source of employment
[deleted]
Multiplicity of systems was a feature of nearly all banks before 2008 and was a contributing factor to some of the inability to perceive and react to risk. After 2008, and forced to by regulators, most banks focussed a lot of resources and attention to consolidate and give themselves a strategic view. Deutsche seems to have found this task far too difficult.
These days I had big trouble to get a refund from an airline. I had their confirmation, via email, that the ticket would be reimbursed. But then, after some days, they alleged, absurdly, that I had flown the ticket and hence I wouldn't get the refund.
It turns out that they were mixing up that ticket with another one I bought, and I only managed to find that out after almost 1 month of trial and error talking with their staff, which pretty much resembled a chat-bot experience.
We put ourselves in a sort of quicksand. Technology's promises are tempting, but there are hidden costs. We didn't know that and now we're finding out what happens when complexity is not tamed. Lessons are being hopefully learned, but I'm afraid that what we currently see is just the beginning of the nightmare
It turns out that they were mixing up that ticket with another one I bought, and I only managed to find that out after almost 1 month of trial and error talking with their staff, which pretty much resembled a chat-bot experience.
We put ourselves in a sort of quicksand. Technology's promises are tempting, but there are hidden costs. We didn't know that and now we're finding out what happens when complexity is not tamed. Lessons are being hopefully learned, but I'm afraid that what we currently see is just the beginning of the nightmare
You should name and shame the airline
Not the same poster but as airlines have come up, this German IT dysfunction appears there too. At some point Lufthansa bought Swiss. Swiss previously had a pretty competent website. It worked fine, so of course, they ported it to the Lufthansa system. What a disaster. It regressed about 15 years overnight. Incredibly buggy, primitive ugly clunky UI. Ordering tickets was easy and never failed on the old system, on the new one there would often be just random failures halfway through the process and you'd get stuck unable to proceed.
Did NOT leave me with a good impression of German IT.
Did NOT leave me with a good impression of German IT.
Definitely agree. Lufthansa’s website and app are a complete mess.
I was working for another bank a few years ago, and a recruiter for Deutsche hit me up about a role that sounded interesting, something pretty hardcore whose details I forget other than it involved C++. We finally get to the part about what they're looking to pay. Massive pay cut over what I was already making. The recruiter assured me they'd see if they could maybe possibly get me a slightly less massive pay cut. Never heard back. I assume they either never filled that role, or eventually found someone to take it who couldn't get a job anywhere else and was about to be homeless. Hearing that they're a shitshow internally does not surprise. Sometimes you get what you pay for.
The branch of the company I used to work for got bought around 2013 by Lear corp. Our European headquarters shifted to a city near Munich. It took 6 months for us to get at laptop that was allowed to logon to the corporate network. After less then a year, suddenly our builds became very slow. After some investigations i sent some screendumps from task manager to one of our managers, showing that 90 percent of the cpu cycles was eaten by some security service scanning our running scripts each and every time they were invoked. It took several months before they bothered to fix the resource hog. Truly the most work-hampering IT I could imagine.
Having experienced working in a legacy financial institution for half a decade, the main takeaway for me was to never work for a legacy financial institution again.
They are horrible places if you take your craft seriously.
They are horrible places if you take your craft seriously.
45 different operating systems??
I had no idea 45 different operating systems still existed that were functional
I had no idea 45 different operating systems still existed that were functional
I had the same reaction, but I don’t think they mean “operating system” in the computer science sense. (E.g. Windows, Linux, etc.) I think they’re actually talking about 45 different systems they developed internally to run their business.
I think you’re correct - I worked there for about 5 years in the infrastructure engineering department. We had very rigorous processes around the underlying platform and keeping the operating systems and hardware up to date.
They’re talking about the “banking” operating systems.
Also note that Deutsche Bank (like many global financial institutions) is not really a single company, but more like 10 different companies all operating under one umbrella.
It’s all very well pointing at a company like Goldman and saying “look, they do IT well” when they don’t have the same range of commercial and consumer businesses as part of their company. If you’re just an investment bank, life is a bit simpler.
On a more general note I feel like in our industry there’s a tendency towards “perfection worship” where people think there’s some kind of immortal “perfect solution” that a company needs and anything less than that is terrible and must be chastised.
This is egoistical and forgets that IT and computing exists to enable companies and not the other way round. The world does not do computing for computing’s sake. Company change can be almost constant, changing systems to keep up is hard and expensive, being able to live with imperfection is critical to commercial success.
They’re talking about the “banking” operating systems.
Also note that Deutsche Bank (like many global financial institutions) is not really a single company, but more like 10 different companies all operating under one umbrella.
It’s all very well pointing at a company like Goldman and saying “look, they do IT well” when they don’t have the same range of commercial and consumer businesses as part of their company. If you’re just an investment bank, life is a bit simpler.
On a more general note I feel like in our industry there’s a tendency towards “perfection worship” where people think there’s some kind of immortal “perfect solution” that a company needs and anything less than that is terrible and must be chastised.
This is egoistical and forgets that IT and computing exists to enable companies and not the other way round. The world does not do computing for computing’s sake. Company change can be almost constant, changing systems to keep up is hard and expensive, being able to live with imperfection is critical to commercial success.
> [DB is] more like 10 different companies all operating under one umbrella.
A standard part of an acquisition is integrating and migrating IT over a defined time period, so you don't keep accruing and perpetuating technical debt. Who were the CIOs at DB?
A standard part of an acquisition is integrating and migrating IT over a defined time period, so you don't keep accruing and perpetuating technical debt. Who were the CIOs at DB?
Agreed - but only where it makes commercial sense to, and each of the layers of the company's IT platform may end up with different plans for integration and end goals for it.
You probably don't want 10 different desktop hardware providers and IT support systems, but it might be a bad idea to try and unify the trading systems for slightly different looking business units unless there's some competitive advantage to be had by doing it.
There wasn't a CIO at Deutsche when I was there, it all fell under the remit of the COO who was this guy: https://en.wikipedia.org/wiki/Hermann-Josef_Lamberti
You probably don't want 10 different desktop hardware providers and IT support systems, but it might be a bad idea to try and unify the trading systems for slightly different looking business units unless there's some competitive advantage to be had by doing it.
There wasn't a CIO at Deutsche when I was there, it all fell under the remit of the COO who was this guy: https://en.wikipedia.org/wiki/Hermann-Josef_Lamberti
Pascal Boillat - as CIO (Information Officer - not Investment officer ;) )
Think of all OSs, hypervisors, mainframes and their own platform, iterations, versions, flavors, builds, dependencies etc...
I personally once serviced what looked like a 2003 windows server in a DB office where it has a weird logo to it, just like it was a custom/special flavor built for DB, I beleive at the time 2003 was already EOL but they were probably paying MSFT millions to keep supporting it.
It's an article written by someone who doesn't know what an operating system is.
That website is nothing but glogspam and straight up AI generated material. I wouldn't trust anything they post.
[deleted]
People say that this is just like any big IT corporation. It's not IT in Germany does a lot of things manually because it's been done like that historically and they can't fire those people due to unions.
I gave a used copy of a book so someone from HR there once. It was about industrialization. I hope they had time to read through it, if it even helped. It was a story of us and not to be overly afraid of it; takes time and adjustments was the thought when given. Always make it better.
[deleted]
[deleted]
Note the article is from 2018.
Sounds like every other major bank's IT environment.
I can't choose a password for my account; instead they assigned me a 6-digit PIN (Not kidding. A PIN).
When I registered my account at a local branch I gave them my dutch phone number. Their website's transaction page wouldn't load due to a JavaScript parsing error over the dutch phone number format. I called customer support and they basically reacted as if water was burning. They would promise to get back to me. 3 weeks later I still couldn't access my online bank account's transaction page. I ended up eventually manually editing the HTML in inspector in order to be able to navigate to the page where I could update my phone number to a German one and I never heard back from support.
I tried opening a depot account for trading stocks. It just says "Internal server error. your account id is incompatible. Please open an additional account with Deutsche Bank". I then tried this; had to go to KYC all over again; but I can't because I can't register two accounts on the 2FA app that is mandatory to have on my phone. Customer support adviced to get a second phone (??).
and I'm just a normal customer. It's impossible to bank with this bank. Every time I have to interact with them it feels like i'm in some weird fever dream.