Why COBOL Isn't the Problem(lucidchart.com)
lucidchart.com
Why COBOL Isn't the Problem
https://www.lucidchart.com/techblog/2020/11/13/why-cobol-isnt-the-problem/
58 comments
> The problem is maintenance, it's a cost center, it doesn't directly generate revenue, so it's been eliminated.
It does (effectively), the problem is that it is on a long enough time span that the dividends don't appear until long after the decision makers are all long gone. Executives need to be compensated with annuities depending on the future performance of the company to disincentivise slash & burn profit farming.
It does (effectively), the problem is that it is on a long enough time span that the dividends don't appear until long after the decision makers are all long gone. Executives need to be compensated with annuities depending on the future performance of the company to disincentivise slash & burn profit farming.
This is about government, though. Nothing except tax collection and certain police activities are a source of revenue. That is the entire point of government. They're not doing things intended to make money or get rewarded by Wall Street.
It's state government. Everything is a cost centre.
There are a few things that the author forgot:
- COBOL programs have history. They were changed in '77 and then the change was reverted in '79 and then again in '84. Nobody really remembers why.
- COBOL programs do what they do. They do not have specs, but if that's the way we have been computing account interests for the last 50 years, they are right. It's lawyers documenting them, not the other way around.
- COBOL programmers were often meant to be "less" than real programmers. More like junior accountants automating boring procedures, than C wizards. Wizards would not touch COBOL code with a stick. This often shows.
This said, the kind of things you could do on a IBM mainframe in the '70s (virtualization, data safety and efficiency, disaster recovery, even uptime) still run rings around the 90-core boxes with Linux that are in your datacenter racks.
- COBOL programs have history. They were changed in '77 and then the change was reverted in '79 and then again in '84. Nobody really remembers why.
- COBOL programs do what they do. They do not have specs, but if that's the way we have been computing account interests for the last 50 years, they are right. It's lawyers documenting them, not the other way around.
- COBOL programmers were often meant to be "less" than real programmers. More like junior accountants automating boring procedures, than C wizards. Wizards would not touch COBOL code with a stick. This often shows.
This said, the kind of things you could do on a IBM mainframe in the '70s (virtualization, data safety and efficiency, disaster recovery, even uptime) still run rings around the 90-core boxes with Linux that are in your datacenter racks.
> COBOL programs do what they do. They do not have specs
And nothing around them is standard either. No database, no best practices, no comments in code. Combined with
> OBOL programmers were often meant to be "less" than real programmers.
you get programs that are completely unmaintainable. Contrast this with the first versions of UNIX or BSD (written around the same time) and you can still build and understand them pretty easily. That's the difference between programmers and software engineers.
> This said, the kind of things you could do on a IBM mainframe in the '70s (virtualization, data safety and efficiency, disaster recovery, even uptime) still run rings around the 90-core boxes with Linux that are in your datacenter racks.
All through obscure, proprietary APIs and along with a hefty support contract from IBM. Making sure nobody in academia or in SV could get within 500m of a mainframe sure helped them become more than a niche.
And nothing around them is standard either. No database, no best practices, no comments in code. Combined with
> OBOL programmers were often meant to be "less" than real programmers.
you get programs that are completely unmaintainable. Contrast this with the first versions of UNIX or BSD (written around the same time) and you can still build and understand them pretty easily. That's the difference between programmers and software engineers.
> This said, the kind of things you could do on a IBM mainframe in the '70s (virtualization, data safety and efficiency, disaster recovery, even uptime) still run rings around the 90-core boxes with Linux that are in your datacenter racks.
All through obscure, proprietary APIs and along with a hefty support contract from IBM. Making sure nobody in academia or in SV could get within 500m of a mainframe sure helped them become more than a niche.
> COBOL programs have history. They were changed in '77 and then the change was reverted in '79 and then again in '84. Nobody really remembers why.
I wonder how different it would be if they had a good source control system (and used it). Would you be able to look at the history and understand why the changes were made, or would you just get a bunch of commit messages like "made update" or "reverted earlier change"?
I wonder how different it would be if they had a good source control system (and used it). Would you be able to look at the history and understand why the changes were made, or would you just get a bunch of commit messages like "made update" or "reverted earlier change"?
Having worked on the PL/1 and Assembler that formed the core accounting systems of a bank: yes.
Not only did I have source control I had flow diagrams of the entire system for all points in the chain. My code reviews had me doing line-by-line justifications. I wrote tests.
Just because the technology and practitioners are old it doesn't mean they don't know what they're doing.
Generally they invented whatever "you" are reinventing the first time around.
Not only did I have source control I had flow diagrams of the entire system for all points in the chain. My code reviews had me doing line-by-line justifications. I wrote tests.
Just because the technology and practitioners are old it doesn't mean they don't know what they're doing.
Generally they invented whatever "you" are reinventing the first time around.
COBOL was used for more mundane tasks - a lot of data loading, formatting, batches of all kinds. Not all of which seemed so important when they were written, but that ended up being a plug nobody wants to take the risk of disconnecting.....
> Just because the technology and practitioners are old it doesn't mean they don't know what they're doing.
I certainly didn't mean to imply that. It's more that many people now don't do a good job with that stuff, so I was curious if it would have been better then.
I certainly didn't mean to imply that. It's more that many people now don't do a good job with that stuff, so I was curious if it would have been better then.
In my professional experience, code comments and commit messages only cursorily reflect the change in the business logic that prompted the modifications. I suspect this is because the knowledge of the client's business logic is concentrated in the product managers' domain and not in the programmers' domain. All of the back and forth emails with the customer don't get included in our SCM.
Imagine they used jira, confluence, and git. It would still be a huge no! Code archeology is a real necessity even on a <10 year old system. The best way to find out about something is to have a chat with a ‘historian’ who has been with the company for 10+ years.
It depends on team culture at a time.
There are teams which require all changes come via defined channel (like bug tracker or special email), and which put ticket # in each commit (like Gitlab -- see the "closes #..." at the commit message [0])
But many teams have no procedure, or the procedure is optional -- and there you can easily get "made update" commits.
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56794
There are teams which require all changes come via defined channel (like bug tracker or special email), and which put ticket # in each commit (like Gitlab -- see the "closes #..." at the commit message [0])
But many teams have no procedure, or the procedure is optional -- and there you can easily get "made update" commits.
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/56794
I worked at 3 different companies in the mid 1990s that had teams actively working on COBOL systems and they were all using mainframe based version control systems.
I think the issue at this point is whether those version control systems are still being maintained and those orgs are continuing to pay for the licenses for those systems, probably during years where you weren't even planning on using them.
My father has been working in COBOL since the 80s, and his “reference” books are literally a couple of 4” 3 ring binders that he’s assembled over 40 years now of coding.
Everytime the bank tries to switch away from COBOL they run into problems ranging from code latency to lack of support for certain features in the new language among other issues thus far.
I really don’t know what the banks that run cobol mainframes are going to do as most have ignored the Father Time problem where cobol programmers aren’t getting any younger and the language, in my opinion, is miserable compared to the “magic” of newer era languages.
My pops also has made less money then I did as an engineer at a tech co with 3 years experience so banks aren’t valuing the work these folks do very much either.
I’ve told my pops he can do consulting once he retires and make a killing because there’s gonna be a major shortage of cobol folks and a massive amount of mainframe code out there that needs maintenance or a transition into a newer era language.
Edit: - The author is wrong here
“If you need to change old programs, hiring experienced programmers and teaching them COBOL is the cheap part.”
There is not a surplus of developers or entry level folks willing to learn cobol over another entry level language, so it’s not cheap and it’s not “easy”.
Everytime the bank tries to switch away from COBOL they run into problems ranging from code latency to lack of support for certain features in the new language among other issues thus far.
I really don’t know what the banks that run cobol mainframes are going to do as most have ignored the Father Time problem where cobol programmers aren’t getting any younger and the language, in my opinion, is miserable compared to the “magic” of newer era languages.
My pops also has made less money then I did as an engineer at a tech co with 3 years experience so banks aren’t valuing the work these folks do very much either.
I’ve told my pops he can do consulting once he retires and make a killing because there’s gonna be a major shortage of cobol folks and a massive amount of mainframe code out there that needs maintenance or a transition into a newer era language.
Edit: - The author is wrong here
“If you need to change old programs, hiring experienced programmers and teaching them COBOL is the cheap part.”
There is not a surplus of developers or entry level folks willing to learn cobol over another entry level language, so it’s not cheap and it’s not “easy”.
>> There is not a surplus of developers or entry level folks willing to learn cobol over another entry level language, so it’s not cheap and it’s not “easy”.
Wrong, there are literally 10,000s of former cobol developers due to decades of offshoring. I know of several thousand in from one local company alone. Go look for these job listings, you won't find them because they're all offshore.
Wrong, there are literally 10,000s of former cobol developers due to decades of offshoring. I know of several thousand in from one local company alone. Go look for these job listings, you won't find them because they're all offshore.
Offshore isn’t a realistic solution, the quality & language barriers that come into play significantly slow down any sort of production cadence compared to in-house.
[deleted]
Also, the program written in COBOL often comes with an ecosystem that can be harder to port than the code itself. Record vs stream oriented filesystems, batch job subsystems, file transfers, block mode terminals, and so on.
> Actually providing the programmers the time and money to make sure
In my experience management sidesteps that problem by insisting that you're an incompetent fool if you need time and money to accomplish anything.
In my experience management sidesteps that problem by insisting that you're an incompetent fool if you need time and money to accomplish anything.
In my experience, telling them that hiring the wrong person for the job must have been their incompetence and walking out makes them reconsider their strategies quickly.
Rehiring talent costs - on average - an annual salary. If more of us grew a spine, we wouldn't see the management/IT-ops staging a recreation of the stereotypical high-school jocks/nerds conflict.
Rehiring talent costs - on average - an annual salary. If more of us grew a spine, we wouldn't see the management/IT-ops staging a recreation of the stereotypical high-school jocks/nerds conflict.
> Rehiring talent costs - on average - an annual salary.
That's best-case scenario - an ordinary cog in a big team supporting well-documented/tested and fairly generic, simple software. A friend of mine did a research paper and said it's more like 3 years annual salary in most companies because:
* Loss of productivity during notice period
* Onboarding time of new staff
* Negative impact on morale of rest of team
* Rest of team needing to support work
* Recruiters fees
* Time spent recruiting
* Customer frustration
* Replacing them with someone worse
* Extra mistakes
That's best-case scenario - an ordinary cog in a big team supporting well-documented/tested and fairly generic, simple software. A friend of mine did a research paper and said it's more like 3 years annual salary in most companies because:
* Loss of productivity during notice period
* Onboarding time of new staff
* Negative impact on morale of rest of team
* Rest of team needing to support work
* Recruiters fees
* Time spent recruiting
* Customer frustration
* Replacing them with someone worse
* Extra mistakes
Eh. Employers in the USA have very little incentive to worry about this, especially big employers.
Even if they manage to piss off an employee, it might take said employee a couple of months to find a new job (assuming they're in a field like programming where there are any jobs at all).
And the employee won't even start looking for a job after they've been unhappy for several months, because finding a new job is an exhausting, demoralizing timesink; it can be all but impossible to conduct an effective job search while handling other "real life" demands. Not to mention that it's "bad" to leave a job too soon.
And the employee can't just quit because their access to healthcare is dependent on having a full-time job with benefits. Individual market healthcare plans are all but unaffordable. Not to mention that it's "bad" to have a gap in your resume.
So even if the employee starts hating their job by month 2, if they aren't some kind of super-hot commodity in the job market, they won't leave until month 12 at the earliest. Heck, they might not ever leave at all.
Adjusted for the probability that the employee might leave in any given year and amortized over the lifteime of the employee at the company, the costs of making an employee unhappy are trivial compared to the value that can be extracted from them before they quit.
And that's all assuming that employees are somewhat non-fungible. For a job that doesn't require specific technical skills, employees are essentially fungible. This is kind of a good thing for society, because it means that there is a large number of educated, thoughtful, high-functioning people out there with good communication skills. But it's a bad thing if you are one of those people and you need to pay rent and go to the doctor.
People on HN (and in tech generally) seem to forget that they are tremendously fortunate to have any seller power at all in the labor market.
Even if they manage to piss off an employee, it might take said employee a couple of months to find a new job (assuming they're in a field like programming where there are any jobs at all).
And the employee won't even start looking for a job after they've been unhappy for several months, because finding a new job is an exhausting, demoralizing timesink; it can be all but impossible to conduct an effective job search while handling other "real life" demands. Not to mention that it's "bad" to leave a job too soon.
And the employee can't just quit because their access to healthcare is dependent on having a full-time job with benefits. Individual market healthcare plans are all but unaffordable. Not to mention that it's "bad" to have a gap in your resume.
So even if the employee starts hating their job by month 2, if they aren't some kind of super-hot commodity in the job market, they won't leave until month 12 at the earliest. Heck, they might not ever leave at all.
Adjusted for the probability that the employee might leave in any given year and amortized over the lifteime of the employee at the company, the costs of making an employee unhappy are trivial compared to the value that can be extracted from them before they quit.
And that's all assuming that employees are somewhat non-fungible. For a job that doesn't require specific technical skills, employees are essentially fungible. This is kind of a good thing for society, because it means that there is a large number of educated, thoughtful, high-functioning people out there with good communication skills. But it's a bad thing if you are one of those people and you need to pay rent and go to the doctor.
People on HN (and in tech generally) seem to forget that they are tremendously fortunate to have any seller power at all in the labor market.
> If more of us grew a spine, we wouldn't see the management/IT-ops staging a recreation of the stereotypical high-school jocks/nerds conflict.
I cannot upvote this enough.
I haven't seen a company change mind so fast until 3/4 of a huge team (25-30 people) team left the company within 6-8 months.
I cannot upvote this enough.
I haven't seen a company change mind so fast until 3/4 of a huge team (25-30 people) team left the company within 6-8 months.
What made them leave? It's different if they leave randomly or because a well known set of reasons.
Everybody lies on exit interviews. You mostly don't want to burn bridges and frequently will need some references. So HR departments have 0 useful information as to why people are leaving and even less about the costs of having developers go.
> Everybody lies on exit interviews. You mostly don't want to burn bridges
Hold on - not everyone does (I offer myself as anecdata). And I am more than happy to burn bridges if I'm leaving a toxic workplace because, well, it's toxic.
Hold on - not everyone does (I offer myself as anecdata). And I am more than happy to burn bridges if I'm leaving a toxic workplace because, well, it's toxic.
It really depends. where the scene is not that big, it does happen to meet again a former coworker (or a former boss) at another company.
I've seen that happen.
I've seen that happen.
you're pretty much right.
Finding a job with higher pay or a better work environment makes the rest of that team (and possibly members of other teams) reconsider staying.
Personal anecdata: one person leaving a team of 10-20 caused 3-5 other people to either leave or re-negotiate in private. In extreme cases, this triggers a chain reaction.
Personal anecdata: one person leaving a team of 10-20 caused 3-5 other people to either leave or re-negotiate in private. In extreme cases, this triggers a chain reaction.
This is also true.
a long streak of dumb decisions made by upper management (think C-level execs, not direct managers).
The article gets it: the problem isn't COBOL, the problem is lack of maintenance. The analogy is apt, too: if you never put oil in a car & it fails, the problem is not how the car the built.
There are materials for learning COBOL. Here are some materials from the Linux Foundation's Open Mainframe project:
https://www.openmainframeproject.org/projects/coboltrainingc...
And when there was a call last year for COBOL programmers to help some of these aging systems, a lot of people immediately popped up: https://www.techrepublic.com/article/ibm-linux-foundation-se...
While COBOL has its quirks, it's not that hard to learn. It even has some advantages, for example, it has built-in support for fixed-point decimal arithmetic.
In general, COBOL is the scapegoat, not the actual problem.
There are materials for learning COBOL. Here are some materials from the Linux Foundation's Open Mainframe project:
https://www.openmainframeproject.org/projects/coboltrainingc...
And when there was a call last year for COBOL programmers to help some of these aging systems, a lot of people immediately popped up: https://www.techrepublic.com/article/ibm-linux-foundation-se...
While COBOL has its quirks, it's not that hard to learn. It even has some advantages, for example, it has built-in support for fixed-point decimal arithmetic.
In general, COBOL is the scapegoat, not the actual problem.
> if you never put oil in a car & it fails, the problem is not how the car the built
No, the analogy is perfectly apt. Electric vehicles are rapidly replacing gasoline vehicles. There is no oil to change in an electric car! Better languages have long since surpassed COBOL.
No, the analogy is perfectly apt. Electric vehicles are rapidly replacing gasoline vehicles. There is no oil to change in an electric car! Better languages have long since surpassed COBOL.
My takeaway: if New Jersey had decided they could save money by just never changing the oil in all their state-owned vehicles they would have been in the same situation as they were when they chose to save money by not maintaining their state-owned software.
I agree to some degree that learning a new language is easy, but learning a new language and understanding its intricacies that could cause issues in a program takes a whole lot longer.
It takes a whole lot less time for an experienced COBOL dev to understand a program than it takes for an experienced programmer who just learned COBOL to understand it in my experience.
It takes a whole lot less time for an experienced COBOL dev to understand a program than it takes for an experienced programmer who just learned COBOL to understand it in my experience.
Not just the language, but IBM mainframes and transactions and that while ecosystem. It's not like having your Python script query a SQL database. I'm sure I could learn to write sample programs like "FizzBuzz" or "hello world" in a very short amount of time. I'm not going to be comfortable writing bank software for years.
Yes, I think that this article missed the point to some extent. Knowing C quite well and C++ not very well, learning the Java 'language' was quite easy. Learning the Java 'model'/'runtime' was much harder and ultimately more important.
I don't know almost anything about cobol, but from what I understand, there are more intricacies than language in cobol, so yes, it takes long to learn it.
Also hiring people for a language that nobody knows anymore will be much tougher.
As I see it that this was a organizational culture failure where he tries to explain it with a technical debt problem.
I see the fingerprints of almost feudal organization structure the other aspects when I first read about the New Jerseys problem with COBOL.
I see the fingerprints of almost feudal organization structure the other aspects when I first read about the New Jerseys problem with COBOL.
He's trying to explain that the reason you end up with so much technical debt in the first place (after explaining to a layperson what that is) is /because/ of organizational cultural failures.
There is a difference between accruing technical debt on a legacy system is one thing, that is something that almost never can avoid. The other is to panic hire someone when the proverbial shit hits the fan is another story. That is the biggest dysfunction here that someone realized they didn't have the competence to maintain their COBOL code in house and it was their responsibility, but only after it stopped working.
> Programming languages are easy to learn
From our experience with VBA users, this is only true for non-technical (as in STEM background) users when it comes to languages with a close-to-English syntax together with tightly integrated IDE/editor. Lua, Python (and occasionally Ruby) are two other languages that are quite "easy" to learn. The curly braces languages? Not so much. Java is especially bad due to its poor error messages and opaque package management/build tools. However being easy to learn for the users doesn't mean the users are capable of writing good code. We found that code written by amateur users tend to be rather unstructured and incoherent. On the other hand some of the cleanest codebases I have read (from those without formal instruction or experience in software engineering) are by mathematicians and electrical engineers.
In terms of tooling Glitch.com and Repl.it are best in for zero-config workflows.
From our experience with VBA users, this is only true for non-technical (as in STEM background) users when it comes to languages with a close-to-English syntax together with tightly integrated IDE/editor. Lua, Python (and occasionally Ruby) are two other languages that are quite "easy" to learn. The curly braces languages? Not so much. Java is especially bad due to its poor error messages and opaque package management/build tools. However being easy to learn for the users doesn't mean the users are capable of writing good code. We found that code written by amateur users tend to be rather unstructured and incoherent. On the other hand some of the cleanest codebases I have read (from those without formal instruction or experience in software engineering) are by mathematicians and electrical engineers.
In terms of tooling Glitch.com and Repl.it are best in for zero-config workflows.
> programming languages are really easy to learn
> For evidence of how easy it is to learn programming languages, one of the authors of the current paper has, over the course of his life, learned 18 distinct programming languages. His father, a physics professor, claims to have used 20 and 30 programming languages. A quick poll of engineers at Lucid Software showed that this was not unique, with all of the engineers having used at least two programming languages, and the majority having used somewhere between 5 to 20.
I'm curious as to how different the languages that they're referring to are.
Python, Ruby, Perl, PHP, C, C++, D, and Java are all really similar to each other relative to the differences between them and Coq, Haskell, Lisp, Erlang, Prolog, Io, Spiral, FORTH, or Chapel.
> For evidence of how easy it is to learn programming languages, one of the authors of the current paper has, over the course of his life, learned 18 distinct programming languages. His father, a physics professor, claims to have used 20 and 30 programming languages. A quick poll of engineers at Lucid Software showed that this was not unique, with all of the engineers having used at least two programming languages, and the majority having used somewhere between 5 to 20.
I'm curious as to how different the languages that they're referring to are.
Python, Ruby, Perl, PHP, C, C++, D, and Java are all really similar to each other relative to the differences between them and Coq, Haskell, Lisp, Erlang, Prolog, Io, Spiral, FORTH, or Chapel.
I mean, the similarities between languages are a big part of why they're easy to learn. Obviously if you only learn within one cluster, you operate best in that cluster, but one or two archetypical languages in each are probably enough to use any of them.
Loose | Elixir, Lisp? | Ruby, Python
No idea if that rendered right at all, such is life. Seems like Lisp-esq languages are kind of a category on their own, but what's an oversimplified taxonomy without some really forced placements?
|Functional|Prodedural
Loose | Elixir, Lisp? | Ruby, Python
No idea if that rendered right at all, such is life. Seems like Lisp-esq languages are kind of a category on their own, but what's an oversimplified taxonomy without some really forced placements?
I still wonder what would happen if IBM sold a IBM i for $499 only licensed for developers. It's not like the thing really needs high end specs. I wonder how many programmers would go for it on a fluke. Would be good for schools too.
Version control is COBOL's problem. Testing mainframe COBOL not in a modern version control is YOLO PITA.
Are the cobol devs from the past the equivalent of the modern day vbs and excel script persons?
Are the cobol devs from the past the equivalent of the modern day vbs and excel script persons?
No, they were writing the typical corporate applications - an interface to enter data into a database, or to run a query and generate some nicely formatted output. They were the equivalent of a modern day webdev.
No, they were writing the typical corporate applications - an interface to enter data into a database, or to run a query and generate some nicely formatted output. They were the equivalent of a modern day webdev.
You can still get a lot done being a scripting guru for an organization. I view it less as a specialized role and more of something nearly all employees should be able to do now of a technical company.
To COBOL's defence, the compiler generates very fast machine code, so COBOL is all right in my book id est screwing around with the intricacies of that language is okay since the end result is a fast, small program. Based on these two criteria, I conclude that it's well worth the effort on the programmer's part.
it's well worth the effort on the programmer's part
It's worth it if you have all the supporting pieces in place. For example if your OS provides record-oriented I/O and your hardware platform provides dedicated processors for accelerating I/O, then you can write a very small, efficient program to, say, calculate today's interest rate and apply it to millions of accounts. But if you don't have those things then you need to write a lot more code and/or have another program (e.g. a database server) to try and emulate them in software, and it will be slower because it has the overhead of calling between components rather than being tightly integrated.
It's worth it if you have all the supporting pieces in place. For example if your OS provides record-oriented I/O and your hardware platform provides dedicated processors for accelerating I/O, then you can write a very small, efficient program to, say, calculate today's interest rate and apply it to millions of accounts. But if you don't have those things then you need to write a lot more code and/or have another program (e.g. a database server) to try and emulate them in software, and it will be slower because it has the overhead of calling between components rather than being tightly integrated.
COBOL I learned on ran under MS-DOS, which does not have record oriented I/O, and it was fast. Very fast.
NOBODY is willing to spend money for maintenance on ANYTHING, look at our infrastructure.
If it wasn't for the FAA making it illegal to not maintain planes, we'd be seeing the same thing with airlines and their planes.
We are seeing this with the Air Force right now, where they are now trying to reverse engineer parts for planes and then 3d print those parts, because no one was willing to spend the money to maintain documentation, maintain vendors, and parts inventory, etc.
Software is especially vulnerable to this phenomenon, because it's not physical so it's considered to be completely fungible, but in fact it's even harder to maintain than physical systems.
The same way Wall Street rewarded companies that closed factories, and outsourced physical locations, and staff, they've also rewarded companies that have eliminated maintenance.
That's just the way it is.