The governor of NJ went live on TV to request COBOL programmers
46 comments
Discussed at: https://news.ycombinator.com/item?id=22782097
I'm a younger engineer, so I say this with a degree of humility, but is trying to modify these monstrous legacy systems on the fly during a worldwide catastrophe really the most responsible or intelligent solution? Is there no way to authorize some kind of emergency system built with modern tooling to offload the system burden and then process it later through the system of record? If I were trying to deal with this problem, my approach would be the following:
- Organize the legislature(s) to pass some kind of emergency act that gives protection around PCI compliance and that sort of thing to new engineering teams
- Hire business and technical experts, fast
- Build a system on AWS that can handle massive scale and start offloading the most important business functionality to that system (things like registering for aid, fraud prevention)
- when things begin to return to normal, start processing requests through the original system, deal with problems, whatever
I know all of this is not easy. Legal and compliance stuff is tough, and I know the domain itself must not exactly be simple. But this is an emergency, and it calls for emergency measures. Trying to hack your way through an ancient code base on the fly seems like it might be doomed to failure.
- Organize the legislature(s) to pass some kind of emergency act that gives protection around PCI compliance and that sort of thing to new engineering teams
- Hire business and technical experts, fast
- Build a system on AWS that can handle massive scale and start offloading the most important business functionality to that system (things like registering for aid, fraud prevention)
- when things begin to return to normal, start processing requests through the original system, deal with problems, whatever
I know all of this is not easy. Legal and compliance stuff is tough, and I know the domain itself must not exactly be simple. But this is an emergency, and it calls for emergency measures. Trying to hack your way through an ancient code base on the fly seems like it might be doomed to failure.
I think you underestimate the scope and scale of systems like this. And maybe overestimate the success rate of rewrites, made worse when the developers have no domain experience and a rushed schedule. Under the best conditions rewrites fail more often than not.
Maintaining legacy code isn’t always easy, but it has a big advantage over a rewrite: it works and satisfies requirements to some non-zero percentage, whereas imaginary code built with “modern tooling” does not work or meet any requirements. COBOL and old business systems aren’t that hard to understand.
Maintaining legacy code isn’t always easy, but it has a big advantage over a rewrite: it works and satisfies requirements to some non-zero percentage, whereas imaginary code built with “modern tooling” does not work or meet any requirements. COBOL and old business systems aren’t that hard to understand.
I completely agree. Rewriting a system on the fly is absolutely going to fail. There are no requirements and without the domain knowledge, there is no way that anything could be completed correctly in any reasonable time. This is a project that should have been scheduled years ago.
Some level of technical debt will exist in every organization. The problem is that NJ never had prioritized this work and now they have an emergency and they are looking for volunteers. They should absolutely should be paying for this work.
Some level of technical debt will exist in every organization. The problem is that NJ never had prioritized this work and now they have an emergency and they are looking for volunteers. They should absolutely should be paying for this work.
I wasn't imagining a full rewrite, more like a "holding area" that could defer some of the load. Again I may be totally wrong about this—I have never written a COBOL system, and if there is some plausible way to make it effective for this problem in a reasonable timeframe, then I understand why that would make sense. I understand all of your reasoning and agree with it in general. I know rewrites are very expensive and often ineffective. I'm just skeptical that the system can be modified quickly to deal with this new load. But again I am not an expert and could be totally wrong, and I'm open to being convinced.
The big problems likely have very little to do with COBOL, which is just another programming language, not an alien artifact. The problem is legacy systems like this that pre-date relational databases (mid-1980s) have huge volumes of data stored in application-specific formats that modern languages don't have drop-in packages or libraries for. The data is always a bigger problem than the code in large systems.
For the record, I wrote enterprise software in COBOL back in the late 70s and early 80s. COBOL itself is no big deal to learn or read. Working with huge volumes of data stored in proprietary binary formats (which COBOL was designed for) is not something I want to go back to -- relational databases rule the world for a good reason.
As for shunting some work to another system, imagine how you would do that with something much simpler. Suppose Excel suddenly stopped working for you with some big spreadsheets. How would you shunt that to another application? How would you do it without risking the important data? It's not a simple problem of just spinning up more servers and replacing some old data management code with MySQL or Redis.
For the record, I wrote enterprise software in COBOL back in the late 70s and early 80s. COBOL itself is no big deal to learn or read. Working with huge volumes of data stored in proprietary binary formats (which COBOL was designed for) is not something I want to go back to -- relational databases rule the world for a good reason.
As for shunting some work to another system, imagine how you would do that with something much simpler. Suppose Excel suddenly stopped working for you with some big spreadsheets. How would you shunt that to another application? How would you do it without risking the important data? It's not a simple problem of just spinning up more servers and replacing some old data management code with MySQL or Redis.
> Universities stopped training because it was not—glamorous.
It also doesn't pay very well.
It also doesn't pay very well.
If it’s anything like legacy public service code in my country it’s not just that, it’s also dealing with terrible organisational culture, d-grade devs, c-grade contractors and corrupt management that let it get that way (as millions are dumped on the contractors, as it turns out).
Some days I wish an ultra virus would take out computers as we know it so we’d be forced to start again with what we know now, namely not creating these monolithic nasty monsters.
Some days I wish an ultra virus would take out computers as we know it so we’d be forced to start again with what we know now, namely not creating these monolithic nasty monsters.
Cool we'd have a bunch of new monsters based on a super old version of nodejs instead.
I’m not saying you can’t write a monster in node (nor would I recommend writing complex apps in it) but modern systems are somewhat less monolithic, if a component is garbage rebuild it, run unit/integration tests and push to business to verify. You can’t do that when your application is such a black box even the devs don’t truly know how it works, which often seems to be the case with legacy cobol.
Cool you missed "actually understand the legacy system" and went right to the grand rewrite into microservices. If you can't be bothered to understand one service from 30 years ago, how is someone 30 years from now going to understand your 45 service masterpiece let alone run the out of date kube cluster you deployed it to.
You’re putting words into my mouth I never mentioned microservices and think that largely they’re a mistake (except SSO). I also never mentioned node. At a glance I assume C# or Java would be the best fit but I’m not a system architect so my POV is probably mute.
Policy should define how a system works so go read that. It’s not like you build an entire system, delete the old one, drop in the new turn it on and really really hope that it just works as intended.
But you’re right, lets just pay IBM, SAP, Oracle billions every yeah because rebuilding is too hard.
But you keep projecting on me with opinions I don’t even agree with lol. I’d hate to ever work with you.
Edit: but like k8s or not, code defined infra is a good idea - even if k8s suffers a terrible death enough detail should be documented that migration into the next big thing won’t be terribly hard. but you shouldn’t worry about that, your legacy java fat client app has so many undocumented legacy dependencies that containerisation will never be an option lol.
Policy should define how a system works so go read that. It’s not like you build an entire system, delete the old one, drop in the new turn it on and really really hope that it just works as intended.
But you’re right, lets just pay IBM, SAP, Oracle billions every yeah because rebuilding is too hard.
But you keep projecting on me with opinions I don’t even agree with lol. I’d hate to ever work with you.
Edit: but like k8s or not, code defined infra is a good idea - even if k8s suffers a terrible death enough detail should be documented that migration into the next big thing won’t be terribly hard. but you shouldn’t worry about that, your legacy java fat client app has so many undocumented legacy dependencies that containerisation will never be an option lol.
Yeah, did my time working in consulting for state gov bureaus. Some of the worst working environments I've ever been in. Asshole, down-right criminal managers. IT actively pirating software. Computers literally falling apart.
Not sure you could pay me enough to go back. Maybe for something ludicrous, like $3mil/yr and mutual termination at any time. I could probably put up with it for 4 months.
These places that "can't find COBOL programmers" need to stop blaming universities and start taking a harder look at themselves.
Not sure you could pay me enough to go back. Maybe for something ludicrous, like $3mil/yr and mutual termination at any time. I could probably put up with it for 4 months.
These places that "can't find COBOL programmers" need to stop blaming universities and start taking a harder look at themselves.
Yes and no remote work as far as I am aware of these things. It just sucks. If it wouldn't suck, I would've went into it: I like retro and COBOL is not a hard language to learn, really, it is not.
This. HN and other places love to regurgitate this was the reason people didn't learn cobol.
The real reason is that most companies that don't bother to move away from either physical or emulated cobol systems just do not care.
They certainly do not care enough to pay market wages.
The real reason is that most companies that don't bother to move away from either physical or emulated cobol systems just do not care.
They certainly do not care enough to pay market wages.
My dad, when I was in elementary school, started a small commercial janitorial business that he runs to this day, even though he’s mostly retired.
He always told me how it was helpful to have an unsexy but useful skill set, and basically do the work that nobody else wanted to do to ensure a long career.
He always told me how it was helpful to have an unsexy but useful skill set, and basically do the work that nobody else wanted to do to ensure a long career.
Curious, how does a small commercial janitorial business survive for so long? My naive understanding as a non-business-owner is that small entities can survive or even do well if they offer a product that's hard or not worth it for big entities to replicate at scale. Some examples might be art, or a nice feeling of "buying local" from people you know and like, or a niche product whose market is just not big enough for giants to bother with.
A janitorial business doesn't seem to be any of those things. So it seems like an area where a big company would dominate (because, say, a lot of 1000 janitorial carts is way cheaper than a lot of 10). But, no?
A janitorial business doesn't seem to be any of those things. So it seems like an area where a big company would dominate (because, say, a lot of 1000 janitorial carts is way cheaper than a lot of 10). But, no?
I don’t know why^, but the largest janitorial/cleaning companies that serve ordinary businesses^^ are franchise companies. Local independents just have to compete against branded local companies.
^I suspect it’s because janitors have access so trust of the individuals involved matters more than cost or having access to a pool of interchangeable employees.
^^There are large contractors that serve really large facilities, but that business isn’t necessary to win to be viable.
^I suspect it’s because janitors have access so trust of the individuals involved matters more than cost or having access to a pool of interchangeable employees.
^^There are large contractors that serve really large facilities, but that business isn’t necessary to win to be viable.
I honestly think it has everything to do with having started it in the early 90s and just keeping his customers happy for 30ish years. He has several big local commercial clients from his early years that still use him because they know he or one of his staff will always show up.
Also I think the bigger players in the space, ABM-types, aren’t interested in commercial space smaller than XX,000 sqft where my dad has been able to fit in nicely.
Also I think the bigger players in the space, ABM-types, aren’t interested in commercial space smaller than XX,000 sqft where my dad has been able to fit in nicely.
I guess like with many job. You can put a little heart into it and care about the customers. This is something that is hard to replicate in scale for big cleaning company.
Not many cleaning companies do that. And those who do are loved deeply by their customers.
There aren't very meaningful benefits to scale in that business. It's labor intensive, and no critical central infrastructure is needed. Capital costs (those janitorial carts) and supplies are a lot smaller than labor costs.
>>needs 100's of programmers - TODAY
If you haven't done it already, please read Fred Brook's timeless classic 'The Mythic Man-Month' (1975) https://en.wikipedia.org/wiki/The_Mythical_Man-Month
Nine women can't make a baby in one month - classic wisdom that should be understood by anyone trying to organize and plan (like a governor).
If you haven't done it already, please read Fred Brook's timeless classic 'The Mythic Man-Month' (1975) https://en.wikipedia.org/wiki/The_Mythical_Man-Month
Nine women can't make a baby in one month - classic wisdom that should be understood by anyone trying to organize and plan (like a governor).
Generally speaking how long would it take to mobilize any competent senior software developer as a COBOL developer?
And could you imagine the DPA being invoked to govern what kind of _software_ output a company produces?
I know I'm being far fetched but I'm thinking about the hypothetical of the government just saying, "Apple, you're doing government payment software now. Here's your contact. Get to work."
And could you imagine the DPA being invoked to govern what kind of _software_ output a company produces?
I know I'm being far fetched but I'm thinking about the hypothetical of the government just saying, "Apple, you're doing government payment software now. Here's your contact. Get to work."
I learned COBOL from the (then) standard text by McCracken back in the early 1980s. It’s not hard, a week or two for someone with programming experience.
The bigger problem is domain expertise (or lack of), and coming up to speed on the standards and idioms used in projects like this. Requirements and documentation likely out of date.
The bigger problem is domain expertise (or lack of), and coming up to speed on the standards and idioms used in projects like this. Requirements and documentation likely out of date.
In the YouTube clip, the URL http://covid19.nj.gov/ but there is still as of yet no mention of COBOL as far as I can see.
If anyone has the actual links that include the job description, etc, I'd be curious.
If anyone has the actual links that include the job description, etc, I'd be curious.
8 seconds in he says "we need Cobalt programmers" and repeats at 27 seconds https://www.youtube.com/watch?v=HSVgHlSTPYQ
https://www.youtube.com/watch?v=HSVgHlSTPYQ
8 seconds in he mentionds "Cobalt" & also at 27s
8 seconds in he mentionds "Cobalt" & also at 27s
I don’t have a direct link for you, but this topic has popped up several times in the past few days and you might find something more substantive.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
URL? What does he need them for?
How hard would it be to automatically convert that COBOL to e.g. C++ or Rust? I suspect not that hard, since COBOL does not even have a heap afaik.
And then you have a bunch of code that's even harder to understand than the original, working in ways you don't expect from the language it's now in.
And not working with the normal build or debug tools on the mainframe it runs on.
And not working with the normal build or debug tools on the mainframe it runs on.
A high-fidelity translation between almost any two serious programming languages sound like a very difficult task. The first 95% might indeed be quite easy, but the last 0.1% would often turn out to be extremely difficult. And if the software "matters", you have to achieve 100%.
Lots of things don't have a 1:1 match. For example Cobol has a PICTURE clause for describing a record. It uses 1 character symbols like in fscanf() but there are different options. Like binary coded decimal.
It can be automatically translated. But then all you've done is convert a bunch of IF A < B THEN GO TO STEPTWO into if (a < b) { goto steptwo; }
And with a lot of extra unreadable stuff to support things COBOL does but the target language doesn't.
The biggest problem with COBOL codebases is that COBOL (at least back in the day) doesn't offer much in the way of structured programming or standard libraries. Horrifying spaghetti code with one-off local solutions for common tasks is the norm, especially on some ancient endlessly patched code.
And with a lot of extra unreadable stuff to support things COBOL does but the target language doesn't.
The biggest problem with COBOL codebases is that COBOL (at least back in the day) doesn't offer much in the way of structured programming or standard libraries. Horrifying spaghetti code with one-off local solutions for common tasks is the norm, especially on some ancient endlessly patched code.
Not true, COBOL has as much structure as any other language, if the programmers wrote their code that way.
The main problem with a direct conversion is COBOL has database operations in the language, mapping data directly to variables (records). All modern languages you might translate COBOL to use libraries to perform database operations, and may not even have an off-the-shelf implementation of the pre-relational data structures COBOL was originally designed to work with.
The main problem with a direct conversion is COBOL has database operations in the language, mapping data directly to variables (records). All modern languages you might translate COBOL to use libraries to perform database operations, and may not even have an off-the-shelf implementation of the pre-relational data structures COBOL was originally designed to work with.
The ignorance of cobol is not the blocking point.
Ignorance of the details of the NJ system is the great challenge. Its likely a huge legacy system with all of the attendant warts, histories, strange choices, and odd limitations that come with that. Given that these systems are built up through years of teams of people that come and go - it will likely take a long time to address the real issues.
Ignorance of the details of the NJ system is the great challenge. Its likely a huge legacy system with all of the attendant warts, histories, strange choices, and odd limitations that come with that. Given that these systems are built up through years of teams of people that come and go - it will likely take a long time to address the real issues.
> The ignorance of cobol is not the blocking point.
> Ignorance of the details of the NJ system is the great challenge.
This seems very plausible. But on this analysis, what's the point of trying to hire "COBOL programmers"? We've just stipulated that the COBOL is easy. Onboarding an experienced COBOL programmer is only a trivial savings over onboarding someone who's never heard of COBOL; they'll both have to learn the system -- but the pool of experienced COBOL programmers is much smaller than the pool of "everyone".
> Ignorance of the details of the NJ system is the great challenge.
This seems very plausible. But on this analysis, what's the point of trying to hire "COBOL programmers"? We've just stipulated that the COBOL is easy. Onboarding an experienced COBOL programmer is only a trivial savings over onboarding someone who's never heard of COBOL; they'll both have to learn the system -- but the pool of experienced COBOL programmers is much smaller than the pool of "everyone".
I don't disagree - but its a government agency and so the job requirements are always going to come out like that. Between that and the below market pay - there is a long road ahead for them.
From what I gather, it's an issue of floating point precision. See more here: https://medium.com/@bellmar/is-cobol-holding-you-hostage-wit....
Must not be willing to pay enough or be hard to work with. At the right price, 100 people should not be hard to find. Of course, they probably would not be in this position if they were willing to put money into the system or keep people on retainer at least.
I recommend taking a look at the other discussions linked. Unfortunately the situation is more nuanced then "just pay them more."
I tried reading the other threads but they are not very information-dense. It's mostly people arguing about whether governments are good employers.
Would you mind sharing a quick summary if you've synthesized the position?
Would you mind sharing a quick summary if you've synthesized the position?
Universities stopped training because it was not—glamorous.
Now NJ needs 100s of programmers—TODAY.