Inside an IBM AS/400 eSeries Factory [video](youtube.com)
youtube.com
Inside an IBM AS/400 eSeries Factory [video]
https://www.youtube.com/watch?v=m53H0pO2FD4
57 comments
They are so slow, though. DB2 and API performances are atrocious. This is a machine designed to run batch jobs, and nothing else.
> They are so slow, though.
I guess that depends?
I used to go to an AS/400 ERP software conference every year. The number of users dwindled and each year was a game of "Who's missing this year..."
Rightfully, everyone was going to SaaS products or Windows-based systems.
One year, I didn't see a group that was usually there (they were missed as always bought tons of company swag to give away). Emailed the CTO after the conference to ask what was up, and they were 75% complete switching over to one of the big-name, Windows-based on-prem ERP systems.
They bought the software, 4/5 Windows licenses for SQL, IIS, backup server, etc. The next year they were back at the conference! They could never get the new software to process their daily shipping fast enough for it to work.
I guess that depends?
I used to go to an AS/400 ERP software conference every year. The number of users dwindled and each year was a game of "Who's missing this year..."
Rightfully, everyone was going to SaaS products or Windows-based systems.
One year, I didn't see a group that was usually there (they were missed as always bought tons of company swag to give away). Emailed the CTO after the conference to ask what was up, and they were 75% complete switching over to one of the big-name, Windows-based on-prem ERP systems.
They bought the software, 4/5 Windows licenses for SQL, IIS, backup server, etc. The next year they were back at the conference! They could never get the new software to process their daily shipping fast enough for it to work.
I haven't seen that its slow, except when the RAID battery is off. Generally the IBM rep shows up when that happens.
I worked at a company for 3 months in 1998 and the AS/400 there crashed once. It required IBM personnel to come and fix it. So, I like the narrow scope of the machine and certain concepts in the OS, but “rock solid” might as well be an urban legend.
My sample is of a couple of hundred AS/400 machines.
This was prior to the rebranding to iSeries
They were used for accounting, logistics, and point of sales systems at different companies.
Common for nearly all was that they lacked any IT department or even a sole tech geek.
Nothing is perfect. it makes sense there might be some lemons around but from my experience they were the most reliable platform I have ever been in contact with.
in different companies used mostly for accounting, logistics, and some POS systems that were entirely lacking any IT department or even the odd tech geek.
They were used for accounting, logistics, and point of sales systems at different companies.
Common for nearly all was that they lacked any IT department or even a sole tech geek.
Nothing is perfect. it makes sense there might be some lemons around but from my experience they were the most reliable platform I have ever been in contact with.
in different companies used mostly for accounting, logistics, and some POS systems that were entirely lacking any IT department or even the odd tech geek.
>in different companies used mostly for accounting, logistics, and some POS systems that were entirely lacking any IT department or even the odd tech geek.
I saw this as well, at least in a way. As a contractor a few weeks out of college, I worked at a regional telco provider that had an AS/400 used for accounting etc. While a reasonably tech-ey firm (they were an ISP as well as providing pager service, frame relay, and other fun relics of a another era), nobody really knew how the AS/400 worked so they were a little wary of letting the new guy use it. It hadn't ever gone down to anyone's recollection, but they were pretty sure that with the staff they had they wouldn't know how to quickly bring it back up if things went pear-shaped.
I saw this as well, at least in a way. As a contractor a few weeks out of college, I worked at a regional telco provider that had an AS/400 used for accounting etc. While a reasonably tech-ey firm (they were an ISP as well as providing pager service, frame relay, and other fun relics of a another era), nobody really knew how the AS/400 worked so they were a little wary of letting the new guy use it. It hadn't ever gone down to anyone's recollection, but they were pretty sure that with the staff they had they wouldn't know how to quickly bring it back up if things went pear-shaped.
We've has one and its successors (iSeries) since the 90's and we have never had a crash of any sort. We've had hardware failures, but the IBM rep fixes those and the machine never crashed because of any of them. The damn thing is rock solid and we bought a new one (Power9) last year.
Probably around the same time, we had a client where IBM convinced them to install Lotus Notes/Domino on their iSeries, which ran their factory assembly line. This caused extended downtime and cost them tons of money. IIRC they ultimately installed a Pentium II board into the iSeries running NT and ran Lotus off that... silly
> but “rock solid” might as well be an urban legend.
I can indeed confirm that "rock solid" is an urban legend. I swore I'd never have anything to do with the AS/400 or IBM after a certain incident.
I need to be careful what I say, but basically there was a certain fault with a certain chip on a certain board.
The so-called IBM technicians we called out ultimately proved to be completely incompetent. Our engineers were collecting hard-facts data on the fault(s) we were seeing. But the IBM guys could never find the problem.
In the end, because IBM engineer after IBM engineer continued to sign-off the machine, we had little choice but to allow it to continue in production albeit with close monitoring.
In the end, it ended up with an almighty climax event. Let's just say its the first (and hopefully only) time I've witnessed an IT company's Professional Indemnity insurance being used in anger to pay for recovery (and indeed why if you are involved with the management of an IT company you should ensure such insurance is in place).
TL;DR The client's AS/400 was thrown onto the scrap heap as soon as humanly possible after an all-hands-on-deck accelerated migration away took place.
I can indeed confirm that "rock solid" is an urban legend. I swore I'd never have anything to do with the AS/400 or IBM after a certain incident.
I need to be careful what I say, but basically there was a certain fault with a certain chip on a certain board.
The so-called IBM technicians we called out ultimately proved to be completely incompetent. Our engineers were collecting hard-facts data on the fault(s) we were seeing. But the IBM guys could never find the problem.
In the end, because IBM engineer after IBM engineer continued to sign-off the machine, we had little choice but to allow it to continue in production albeit with close monitoring.
In the end, it ended up with an almighty climax event. Let's just say its the first (and hopefully only) time I've witnessed an IT company's Professional Indemnity insurance being used in anger to pay for recovery (and indeed why if you are involved with the management of an IT company you should ensure such insurance is in place).
TL;DR The client's AS/400 was thrown onto the scrap heap as soon as humanly possible after an all-hands-on-deck accelerated migration away took place.
> They were not sexy, they didnt have games (actually a couple)
I'm not aware of a Doom port, but there are some console-based games that would be put to shame by a ZX81
http://www.jcrcmds.com/jcrgames2.html
I'm not aware of a Doom port, but there are some console-based games that would be put to shame by a ZX81
http://www.jcrcmds.com/jcrgames2.html
This is basically what the Admin of the AS/400 systems at my current place told me. He says he has never seen one crash.
One of the comments on the video:
> Randy Schnier
> 8 hours ago (https://www.youtube.com/watch?v=m53H0pO2FD4&lc=Ugzs2gCWlsh1_...)
> I worked at this IBM site as an engineer for almost 30 years, leaving in 2012. Several divisions were based there besides AS/400, including IBM Software Group that was responsible for the WebSphere Application Server family of products. As the video mentions, at its peak around the year 2000, the site had about 8,000 employees. It is now down to significantly less than 1,000 and most of the site is leased out to other companies for office and warehouse space. Actually IBM sold the site several years ago to an external real estate firm and leases back the space they currently use! You might notice in the video that the AS/400 cabinets were rounded at the rear -- this was to prevent customers from pushing the unit right up against a wall and blocking the cooling vents.
> Randy Schnier
> 8 hours ago (https://www.youtube.com/watch?v=m53H0pO2FD4&lc=Ugzs2gCWlsh1_...)
> I worked at this IBM site as an engineer for almost 30 years, leaving in 2012. Several divisions were based there besides AS/400, including IBM Software Group that was responsible for the WebSphere Application Server family of products. As the video mentions, at its peak around the year 2000, the site had about 8,000 employees. It is now down to significantly less than 1,000 and most of the site is leased out to other companies for office and warehouse space. Actually IBM sold the site several years ago to an external real estate firm and leases back the space they currently use! You might notice in the video that the AS/400 cabinets were rounded at the rear -- this was to prevent customers from pushing the unit right up against a wall and blocking the cooling vents.
Once you notice the "things in the rear to prevent the customer from pushing it up against the wall and blocking the cooling system" you notice this everywhere. Refrigerators, test equipment, microwave ovens.
This thing is still operating the ERP of many brick and mortar french businesses. A few software companies have the guys able to maintain it, so they are well off, charging big bucks for every maintenance operation.
As a developer I have been asked a few times to build websites that feed on flat files exported from AS400 (don't bother asking about APIs). It was a pain every single time.
Zend provides PHP that runs on the i, so you should be able to get pretty much any information you want out of and exchange information with SOAP or build an API just like you would on Linux
https://www.zend.com/products/zend-server/ibmi
https://www.zend.com/products/zend-server/ibmi
Ouch, you brought up nightmares about flat files from a previous job. I’m betting it was an AS400 or something similar providing the data I needed to ingest. It was painful.
file drop apis weren't so bad. drop a file in the input directory with a unique name, wait for the response to appear in the output directory. with samba these became easy to work with.
i wrapped a couple of these with soap webservices back in the day and any reliability issues almost always came from the soap server.
ups used to be a gigantic as/400 shop... i wonder if they're still using them...
i wrapped a couple of these with soap webservices back in the day and any reliability issues almost always came from the soap server.
ups used to be a gigantic as/400 shop... i wonder if they're still using them...
Thanks, this brought back some memories. I did an internship / co-op in software engineering at that site in Rochester while a student. They needed extra help so I worked a few evenings at 1.5 pay alongside the regular factory workers.
About three years id my software development career were detonated to AS/400 RPG programming. It was fun but I’m glad I moved on.
Gasp! You poor soul :o) RPG (Report Program Generator) is the worst programming language on the face of the Earth. Its paradigm is the IBM 402 tabulator.
Thank you for clearing up my cognitive dissonance. I mean, how many people had jobs programming RPGs? How many people possibly played role playing games on AS/400?
Clearly, someone need to build/port an RP game builder (EasyRPG?) to AS/400 RPG.
> detonated
That bad, huh?
That bad, huh?
I wonder if this is how Oxide Computer is building their systems in 2022.
Interesting to read some of the hype around 0xide, it seems like its just some software ontop of a blade system?
From listening to their podcasts / twitter spaces it sounds like they're building machines from the ground up. I get the impression they'd hate the analogy, but an apple style take on owning the hardware and software together.
For a dip into how deep in the weeds they are, https://www.youtube.com/watch?v=HCkuCkp3Zoo (they also do this as "oxide and friends" wherever you find podcasts, which is a nuisance to link to but works better)
For a dip into how deep in the weeds they are, https://www.youtube.com/watch?v=HCkuCkp3Zoo (they also do this as "oxide and friends" wherever you find podcasts, which is a nuisance to link to but works better)
For what it’s worth, I often use an Apple analogy. Sun works too. Both share that same idea: if you want to build a great computer, you should write software as well as build hardware.
Not a blade. They're selling whole racks because that's the level that the hyperscalers work at, and Oxide are trying to bring the same class of systems to on-prem deployments. The closest comparison would be to something like modern mainframes, though the provided featureset might be subtly different. (For one thing, Oxide systems are ordinary x86 systems at the CPU architecture level; it's just "everything else" that's potentially custom.)
Hyperscalers do their own boards, chassis, BMCs, racks, etc., and can do their firmware (or modifications to it) though. Several are even getting into their own CPU designs. They don't want any boutique designs controlled by other companies there. And they don't want to pay premiums for it. They certainly don't want to run a proprietary hypervisor / host OS stack on it.
You're assuming hyperscalars would be our target buyer. That's not the case; in fact, it's almost backwards. Hyperscalars build these things for themselves, but do not sell them to others. There are other organizations that could take advantage of this technology, but literally cannot buy it. That's where Oxide fits in.
True, I thought that's what OP was implying. If not then my points don't necessarily apply of course.
They may have been. Regardless, that’s not the case. No harm no foul either way! I’d absolutely agree with you that trying to sell to them wouldn’t make sense.
Typical blade servers are around 10RU in size and have similar function of shared powersupplies, backplane, network etc.
so 0xide is like a massive full rack sized blade enclosure so all blades can talk to each other over fast interconnects?
so 0xide is like a massive full rack sized blade enclosure so all blades can talk to each other over fast interconnects?
No dude. They are reimagining the firmware and ROMs on the mother board, and so on. I recommend looking further into it.
I did have a read of their website, but it reads a bit like a kickstarter campaign with lots of vague statements and no details.
At this stage I'm unsure what benefits it has over a blade server managed by traditional api like snmp.
At this stage I'm unsure what benefits it has over a blade server managed by traditional api like snmp.
This was my take as well; bold claims with minimal details or demos on their website.
Expect more details in the future. Right now we're focused on building the product.
This is sarcasm, right?
It is not. I listened to their podcast and got my information from there, straight from Cantrill.
It just doesn't seem to offer much, for all the grandiose claims. Who is going to pay a premium to have "reimagined" firmware and ROMs? Or yet another proprietary hypervisor? Or a stack that seems to be built around Solaris?
It looks like a vanity or nostalgia project started by old Solaris folk, as far as I've seen.
It looks like a vanity or nostalgia project started by old Solaris folk, as far as I've seen.
Here is a recent thread where someone asks the same questions, and gets a bunch of answers from various people: https://news.ycombinator.com/item?id=32039943
It's mostly the same sort of handwaving though, which I still don't understand.
"anyone who has at least a rack full of Dell/HPE hardware"
Big enterprise and government customers want white glove support, 10+ year continuity, ISV certification for their hardware. They will pay a premium, but they expect a premium and not in reimagined firmware, but really support. I can't see how Oxide could compete there or why it would want to.
"If Oxide computers had existed when I was working on sorting out the infrastructure for in-country domestic payments at Visa, I'd have lobbied pretty hard to adopt them. Assembling basically the same thing from a bunch of random vendors and terribly integrated software was an infuriating, expensive process that only achieved great results because we invested the time and money into it that I'd have rather budget allocated to writing core payments & settlement applications."
I'm not doubting this person's story, but perhaps he doesn't know exactly what Oxide would give either. This kind of thing is exactly what you can write a check to Dell or HPE for and they've already assembled those things with their boxes and their hardware partners they've got the firmware and software working, and the solution is all certified by SAP and Oracle and Microsoft and whoever else.
But Visa didn't go that way, they instead paid the premium for experts on staff or contracted to do more of it in house. And they got a great result.
Between the two, I'm having a hard time seeing where Oxide would fit. Pay a premium for the hardware solution and for the in house IT team?
"anyone who has at least a rack full of Dell/HPE hardware"
Big enterprise and government customers want white glove support, 10+ year continuity, ISV certification for their hardware. They will pay a premium, but they expect a premium and not in reimagined firmware, but really support. I can't see how Oxide could compete there or why it would want to.
"If Oxide computers had existed when I was working on sorting out the infrastructure for in-country domestic payments at Visa, I'd have lobbied pretty hard to adopt them. Assembling basically the same thing from a bunch of random vendors and terribly integrated software was an infuriating, expensive process that only achieved great results because we invested the time and money into it that I'd have rather budget allocated to writing core payments & settlement applications."
I'm not doubting this person's story, but perhaps he doesn't know exactly what Oxide would give either. This kind of thing is exactly what you can write a check to Dell or HPE for and they've already assembled those things with their boxes and their hardware partners they've got the firmware and software working, and the solution is all certified by SAP and Oracle and Microsoft and whoever else.
But Visa didn't go that way, they instead paid the premium for experts on staff or contracted to do more of it in house. And they got a great result.
Between the two, I'm having a hard time seeing where Oxide would fit. Pay a premium for the hardware solution and for the in house IT team?
> they expect a premium and not in reimagined firmware, but really support. I can't see how Oxide could compete there or why it would want to.
One aspect of being responsible for the entire stack is that we can give excellent support. It’s never some other company’s problem.
> This kind of thing is exactly what you can write a check to Dell or HPE for
See my other link elsewhere in the thread for why we’re different than writing that check to Dell.
But even then, this means you do know there is a market for this kind of server. It’s now “why Oxide instead of Dell” instead of “why does Oxide exist.” I’m sure lots of people will still buy servers from other folks for a very long time, but some will also choose us. That’s the joy of startups, we’ll all just have to wait and see!
One aspect of being responsible for the entire stack is that we can give excellent support. It’s never some other company’s problem.
> This kind of thing is exactly what you can write a check to Dell or HPE for
See my other link elsewhere in the thread for why we’re different than writing that check to Dell.
But even then, this means you do know there is a market for this kind of server. It’s now “why Oxide instead of Dell” instead of “why does Oxide exist.” I’m sure lots of people will still buy servers from other folks for a very long time, but some will also choose us. That’s the joy of startups, we’ll all just have to wait and see!
> One aspect of being responsible for the entire stack is that we can give excellent support. It’s never some other company’s problem.
You aren't responsible for the entire stack though. You have Intel, Mellanox, Microsoft, Redhat, Oracle, SAP, Intersystems, etc.
And this type of "support" is surely a huge burden in terms of manpower and infrastructure required. You can't scale this and have the core developers on call solving customer problems. You could have a hypervisor architecture which is technically better and requires less support overhead than KVM for example, but can you really compete with Redhat for governments and enterprise white glove support just with that advantage?
> See my other link elsewhere in the thread for why we’re different than writing that check to Dell.
This one?
https://news.ycombinator.com/item?id=30678324
The problem is once you're talking a real solution, the system management and provisioning and BMC and partitioning stuff is not the really hard part.
Plug in and power on another rack and the systems show up in the management console and you can partition and provision them, sure. IBM and other "enterprise" vendors have that. They're clunky but they have procedure manuals and training and they just work for the most part.
It's what you actually do with those racks is the difficult thing. Enterprise isn't as smooth or scalable as "hyperscale" here. It's a lot of old crufty legacy in house and ISV software.
You aren't responsible for the entire stack though. You have Intel, Mellanox, Microsoft, Redhat, Oracle, SAP, Intersystems, etc.
And this type of "support" is surely a huge burden in terms of manpower and infrastructure required. You can't scale this and have the core developers on call solving customer problems. You could have a hypervisor architecture which is technically better and requires less support overhead than KVM for example, but can you really compete with Redhat for governments and enterprise white glove support just with that advantage?
> See my other link elsewhere in the thread for why we’re different than writing that check to Dell.
This one?
https://news.ycombinator.com/item?id=30678324
The problem is once you're talking a real solution, the system management and provisioning and BMC and partitioning stuff is not the really hard part.
Plug in and power on another rack and the systems show up in the management console and you can partition and provision them, sure. IBM and other "enterprise" vendors have that. They're clunky but they have procedure manuals and training and they just work for the most part.
It's what you actually do with those racks is the difficult thing. Enterprise isn't as smooth or scalable as "hyperscale" here. It's a lot of old crufty legacy in house and ISV software.
Perhaps, but if their secret sauce produces a much better product, I'm not terribly concerned that the value-add is a software-based and not hardware-based.
Yeah I'm not arguing for or against their product, just genuinely curious to know what differentiates them from existing blade servers hosting kubernetes or something.
I wrote about this a while back: https://news.ycombinator.com/item?id=30678324
hmm not sure how this benefits over something like vxrail that's a supported turnkey solution, but I guess will wait for the 3rd party reviews to come out once it ships.
I knew without opening the link that it's about Rochester MN. I have a love/hate relationship with AS/400 and I think they're interesting in their own unique way. Systems like SUN/SPARC, Power, 68k etc all have become parts of history, but AS/400 (iSeries) and s360 (Mainframes) keep chugging on.
I recently visited Poco in Berlin and was pleasantly surprised to see them using AS/400 terminals. I also know it is used in several finance companies like Societe General.
I recently visited Poco in Berlin and was pleasantly surprised to see them using AS/400 terminals. I also know it is used in several finance companies like Societe General.
[deleted]
I would have enjoyed that a lot more if it had any commentary about what those components being installed were, and a ton less "and this is how we boxed them up" ... "ooo, conveyor belts!"
Thanks for trip down memory lane. These machines were rock solid.
Still working with an as400 today, it’s my company’s main db system.
We needed AS400 Opel Sourced. IBM are you listening?
Pff it's ibm, they don't even give free emulated developer licenses for z/os. It's like they don't care...give no access to mainframe programming an being late to cloud (even with redhat) and only big-bet on quantum computing leaves IBM in such a bad spot it's not even funny anymore.
But hey MVS3.8,VM/370 and DOS/VS is "opensource/public-domain".
MVS:
http://www.prince-webdesign.nl/index.php/software/update-on-...
VM/370:
http://www.vm370.org/vm
DOS/VS:
http://www.vm370.org/dos
But hey MVS3.8,VM/370 and DOS/VS is "opensource/public-domain".
MVS:
http://www.prince-webdesign.nl/index.php/software/update-on-...
VM/370:
http://www.vm370.org/vm
DOS/VS:
http://www.vm370.org/dos
IBM AS/400s are now Lindy.
In some companies these things were in a closet, under piles of other things and everyone had forgotten about its existence.
Then comes the IBM rep to fix it and there is confusion about its existence.
The operating system was genuinely interesting. I wish they had open sourced it.
They were not sexy, they didnt have games (actually a couple) did run much sofwtware you knwo about but that it did it was very good at.