Old software keeping space missions alive(bbc.com)
bbc.com
Old software keeping space missions alive
https://www.bbc.com/future/article/20230815-the-ancient-tech-keeping-space-missions-alive
66 comments
Agreed. The older I get, the saltier I get when I read anything that refers to something from the 90s as "ancient".
> Agreed. The older I get, the saltier I get when I read anything that refers to something from the 90s as "ancient"
As someone who started programming in 1980, I feel your pain. My oldest spreadsheet goes back to 1995.
As someone who started programming in 1980, I feel your pain. My oldest spreadsheet goes back to 1995.
There's a little more to it. If your original team retires because of old age, that's a lot of wisdom lost. Not everyone wants to learn COBOL to keep planes in the air and the IRS running, imagine needing to figure out the low level code on something millions of miles away with half the hardware broken, the battery cracked and leaking – and if the software update doesn't work, the entire mission is over?
It's altogether amazing.
It's altogether amazing.
COBOL is really quite straightforward and readable. I don't think a developer who's already familiar with a modern language or two would have a hard time picking it up. The system it's operating is domain-specific knowledge anyway, and would be something a newcomer would have to learn (or be familiar with) regardless of language.
>I don't think a developer who's already familiar with a modern language or two would have a hard time picking it up.
You're probably correct, but it's more about if they would want to pick up a language that has no future, except maintenance of already existing systems. (see: Perl).
You're probably correct, but it's more about if they would want to pick up a language that has no future, except maintenance of already existing systems. (see: Perl).
The article calls software from the 90s "old". Nobody used COBOL for mission-critical software at that time. There is embedded software I wrote in the 80s and 90s still running in the field, mostly written in C.
I didn't wrote that COBOL was used in aerospace, but used it as an example in different industries.
> Not everyone wants to learn COBOL to keep planes in the air and the IRS running
But there's excellent money in being a COBOL programmer. Some might be interested in that.
But there's excellent money in being a COBOL programmer. Some might be interested in that.
Not to mention that they confuse RAM with storage. The iPhone 14 certainly doesn't have a minimum of 128 GB of RAM.
Depending on context and the audience, it could be fair to use "RAM" to refer to SSD storage.
SSDs are random access, just not volatile like DRAM.
SSDs are random access, just not volatile like DRAM.
The only context in which the technical difference between memory and disk can be glossed over is one where the audience is made of laypersons that wouldn’t understand the first thing about seekable vs random-access memory in the first place.
A tech-illiterate author/editor is the much more plausible explanation.
A tech-illiterate author/editor is the much more plausible explanation.
On the other hand, a modern SSD is probably faster than RAM on a spaceship from the 90s. (Don’t quote me on this, I didn’t double check).
All of a sudden I had an urge to get MS-DOS to run with a SSD as memory. Unfortunately I know DOS can’t address 1TB of memory, but the mental image of it is hilarious.
All of a sudden I had an urge to get MS-DOS to run with a SSD as memory. Unfortunately I know DOS can’t address 1TB of memory, but the mental image of it is hilarious.
I'm not sure you'll see this, but just in case: the Apollo Guidance Computer used RAM called "core rope memory", and the cycle time was 11.7 microseconds[1]. This isn't too far off from what the best SSDs can achieve for random reads at 8-10 microseconds[2], although these numbers are likely for their minimum block size of 512 bytes.
[1] https://history.nasa.gov/afj/compessay.html
[2] https://www.tomshardware.com/news/micron-finally-rolls-3d-xp...
[1] https://history.nasa.gov/afj/compessay.html
[2] https://www.tomshardware.com/news/micron-finally-rolls-3d-xp...
I find one of the subtitles for one of the images strangely worded too:
“The Voyager space missions were maintained by technology that would today be artefacts in a museum”
The technology either is or isn’t in a museum right now. Why say “would today be”. If it is you can say “technology that by now you can see in a museum” and if it isn’t you could say “expect to be in a museum”.
“The Voyager space missions were maintained by technology that would today be artefacts in a museum”
The technology either is or isn’t in a museum right now. Why say “would today be”. If it is you can say “technology that by now you can see in a museum” and if it isn’t you could say “expect to be in a museum”.
Ok, I've taken the ancient technology out and replaced it with old software in the title above. Thanks!
The article is bad but the premise is not wrong. The things that fly and a lot of the things that stay on the ground are analyzed with NASTRAN whose solver kernel dates back to 1964. At its heart it is still the old FORTRAN code because reproducibility is king.
> it's just some old software from the 90s.
What nonsense. the 90s is recent, not ancient.
What nonsense. the 90s is recent, not ancient.
[deleted]
It may be true with embedded applications or with Windows. Not on a Mac.
Software does degrade with age, though. Simply the fact that vulnerabilities are found over time means that our understanding of a piece of software and how it works changes. But it's not just vulnerabilities being found, it's protocols used that may be unsupported etc. That change is a kind of a rot like people call 'bit rot' but it's a specific software rot that does happen. Systems are complicated.
Most software used not to be connected to the outside world. It wasn't vulnerable at all unless the attacker already had control of the host.
In the thirty years I spent developing software I spent hardly any time on anything that was accessible through the network and I'm confident that I'm not the only one. The embedded control software that I wrote for 6502s would still work today and would be completely invulnerable to attacks other than from someone standing right in front of the machine pushing buttons.
Of course now that everything has to be networked so that your fridge can advertise special offers to you the situation is changing for the worse.
In the thirty years I spent developing software I spent hardly any time on anything that was accessible through the network and I'm confident that I'm not the only one. The embedded control software that I wrote for 6502s would still work today and would be completely invulnerable to attacks other than from someone standing right in front of the machine pushing buttons.
Of course now that everything has to be networked so that your fridge can advertise special offers to you the situation is changing for the worse.
An example is SSL root certificates, which expire by design. Also ABI. But if you use containers or virtual machines you can basically have a program run forever.
You can, but you shouldn't, because at some point you should patch the vulnerabilities you inevitably find on any non-trivial piece of code after a while.
Certain code can run with bugs forever, even vulnerabilities, because it will never interact with anything (see the famous "the missile" bug - a counter would overflow at some point, but by then the warhead would have detonated, so who cares?).
Sure, but none of that applies to non-networked embedded systems.
Brand new software made yesterday is full of bugs and will kill or maim you if you try to use it for safety critical systems. I trust 10+ years old code that has been proven in combat. Not crappy new code written by some arrogant young “Ninja” who has no clue on how little he/she/it knows.
How about traveling-wave vacuum tubes from the pre-transistor era. Doesn't Voyager use them to generate its RF transmission power?
It does! But the really interesting thing is that travelling wave tubes are still being used for quite a few modern satellite designs. For example, SiriusXM used them in SXM-9/10, and is probably using them in SXM-11/12.
Yes travelling wave tubes are still very common on modern satellites. They tend to be used most on high power (>50W RF power)/high frequency (K-band and higher) systems. I don't know the exact rationale, but for space systems in the higher frequency and power regimes travelling wave tubes are still more reliable than solid state power amplifiers. Traveling wave tubes are still in the 40-50% efficiency range so they're not out of range with efficiency of some solid state devices as well.
Right, although I've worked with TWTs, klystrons, etc. in terrestrial equipment and have some familiarity with them, I've only casually thought about the TWTs in the Voyagers and in space use generally.
Given the remarkable longevity of the Voyagers, what I'm surprised about is that there hasn't been much discussion about their componentry and why they've been so reliable. For example I've seen nothing written about the engineering involved in Voyagers' TWTs and why they have been so reliable.
For instance, what is the cathode material, barium, strontium, thorium oxide, etc. used in these TWTs? Was its selection criteria based on emitters with the lowest work function/highest emission at the lowest temperature with preservation of the heater life foremost in mind, and or was it based on oxides with highest ruggedness—least affected by cathode poisoning, etc. Discussions about Richardson's laws and cathode emitters is something I almost never come across these days let alone how they've played a role in the engineering of Voyagers' longevity.
Whilst component manufacturers consider these matters, terrestrial users generally don't, we just reach for replacement parts when components fail. Perhaps I'm just not reading the right material but given the remarkable performance of these spacecraft, I'm surprised we're not focusing on the science and engineering that's made that all possible.
No doubt those who're involved in space engineering are focused on these issues but it seems to me not much information has filtered down to even people like me who have some limited knowledge of the technology let alone the general science-reading public.
Using the Voyagers' history and notoriety would be an excellent way to interest students in the physics of TWTs not to mention the material science and the engineering used in their design and manufacture.
When one considers it, there's a lot of fascinating science and engineering involved in making this 'relic' from the vacuum tube era function and keeping it so.
Given the remarkable longevity of the Voyagers, what I'm surprised about is that there hasn't been much discussion about their componentry and why they've been so reliable. For example I've seen nothing written about the engineering involved in Voyagers' TWTs and why they have been so reliable.
For instance, what is the cathode material, barium, strontium, thorium oxide, etc. used in these TWTs? Was its selection criteria based on emitters with the lowest work function/highest emission at the lowest temperature with preservation of the heater life foremost in mind, and or was it based on oxides with highest ruggedness—least affected by cathode poisoning, etc. Discussions about Richardson's laws and cathode emitters is something I almost never come across these days let alone how they've played a role in the engineering of Voyagers' longevity.
Whilst component manufacturers consider these matters, terrestrial users generally don't, we just reach for replacement parts when components fail. Perhaps I'm just not reading the right material but given the remarkable performance of these spacecraft, I'm surprised we're not focusing on the science and engineering that's made that all possible.
No doubt those who're involved in space engineering are focused on these issues but it seems to me not much information has filtered down to even people like me who have some limited knowledge of the technology let alone the general science-reading public.
Using the Voyagers' history and notoriety would be an excellent way to interest students in the physics of TWTs not to mention the material science and the engineering used in their design and manufacture.
When one considers it, there's a lot of fascinating science and engineering involved in making this 'relic' from the vacuum tube era function and keeping it so.
> I'm surprised about is that there hasn't been much discussion about their componentry and why they've been so reliable
I think part of this might be because it is still considered proprietary information. Kind of crazy, but there is really only one company in the US that makes space qualified TWTs which is Stellant systems (formerly L3, formerly Hughes microwave) who made the Voyager TWTs as well.
I think part of this might be because it is still considered proprietary information. Kind of crazy, but there is really only one company in the US that makes space qualified TWTs which is Stellant systems (formerly L3, formerly Hughes microwave) who made the Voyager TWTs as well.
I'd forgotten about Hughes, Voyager and TWTs, but now that you mention it, it does ring a bell.
You're likely right about the proprietary nature of such manufacturing but perhaps I'm reading too much into this. As I mentioned in an earlier post the emission in the CRT of my 43-year-old Sharp TV is still OK—at least as far as the quality of the image is concerned.
The technology used in the Wehnelt cylinder in my TV's CRT is of a similar vintage to Voyager (I'm pretty sure the TV was manufactured around 1979), and given the millions of CRTs made to similar a quality around that time it's likely that manufacturering techniques and reliability figures were widely known throughout the industry by then.
Anecdotally, I've noticed that CRTs made from the early '70s onwards had much better longevities than their earlier '50s counterparts, whether this was because the formulation of cathode emitters had changed or manufacturing techniques had improved or both is an open question.
Of course, such comparisons are tenuous but that's all I've got to go on, for starters, the Wehnelt in the Voyager's TWT would have been deigned to carry more current. Perhaps NASA has some spare TWTs that one day someone will reverse engineer and we'll know for sure.
Nevertheless, it seems to me that knowledge about component reliability is critical to NASA, so it's likely the answer already exists somewhere in the depths of NASA's archives.
You're likely right about the proprietary nature of such manufacturing but perhaps I'm reading too much into this. As I mentioned in an earlier post the emission in the CRT of my 43-year-old Sharp TV is still OK—at least as far as the quality of the image is concerned.
The technology used in the Wehnelt cylinder in my TV's CRT is of a similar vintage to Voyager (I'm pretty sure the TV was manufactured around 1979), and given the millions of CRTs made to similar a quality around that time it's likely that manufacturering techniques and reliability figures were widely known throughout the industry by then.
Anecdotally, I've noticed that CRTs made from the early '70s onwards had much better longevities than their earlier '50s counterparts, whether this was because the formulation of cathode emitters had changed or manufacturing techniques had improved or both is an open question.
Of course, such comparisons are tenuous but that's all I've got to go on, for starters, the Wehnelt in the Voyager's TWT would have been deigned to carry more current. Perhaps NASA has some spare TWTs that one day someone will reverse engineer and we'll know for sure.
Nevertheless, it seems to me that knowledge about component reliability is critical to NASA, so it's likely the answer already exists somewhere in the depths of NASA's archives.
In recent years I've occasionally wondered what the type/part number of that TWT is. Given its history, its heater and cathode emission seems to have held up remarkably well.
Another question I'd like answered, do the Voyagers cycle down or turn off the heater during TX downtime? Turning it off risks metal/cycling fatigue, leaving it on risks reducing the cathode life—or even poisoning it from stray ions.
A final point, presumably the TWT's output has dropped over the decades, how is this monitored and do we know the percentage drop in power output (from the TWT not the Pu power source)?
The fact the TWT is still working seems quite remarkable—but then perhaps I shouldn't be overly surprised, one of my TV sets, a 23" Sharp, is 43 years old and the CRT still works well (and it's switched on for several hours every day).
Another question I'd like answered, do the Voyagers cycle down or turn off the heater during TX downtime? Turning it off risks metal/cycling fatigue, leaving it on risks reducing the cathode life—or even poisoning it from stray ions.
A final point, presumably the TWT's output has dropped over the decades, how is this monitored and do we know the percentage drop in power output (from the TWT not the Pu power source)?
The fact the TWT is still working seems quite remarkable—but then perhaps I shouldn't be overly surprised, one of my TV sets, a 23" Sharp, is 43 years old and the CRT still works well (and it's switched on for several hours every day).
> Another question I'd like answered, do the Voyagers cycle down or turn off the heater during TX downtime? Turning it off risks metal/cycling fatigue, leaving it on risks reducing the cathode life—or even poisoning it from stray ions.
I suspect that now the heaters are turned off during Tx downtime, but I think this is probably because they Pu power is much lower and so thy want to conserve as much power as possible. Even if they did have more power, Voyager only makes contact once per day so I think the increased thermal cycling is not much compared to reduction in cathode life so they probably go this route anyway. I do know that other missions I have worked on that have much more frequent contacts, we do keep the Tx heaters on so there is probably some number of cycles where one becomes more dominant.
> A final point, presumably the TWT's output has dropped over the decades, how is this monitored and do we know the percentage drop in power output (from the TWT not the Pu power source)?
This is telemetry from the TWTAs on the helix current and the anode voltage. Along with the DC power draw of the TWTAs I think you can interpret some of what the power output is and how it changed over life. Some of this might be curve fitting to ground-based life tests so there could be some estimation. The ground is also certainly measuring the received power so they can estimate power output from the spacecraft from that as well.
> The fact the TWT is still working seems quite remarkable
You're not the only one. From this article [0] a JPL engineer says "Nobody can explain why the Voyager TWT is still working".
[0] https://spinoff.nasa.gov/Traveling-Wave-Tubes-Travel-Far
I suspect that now the heaters are turned off during Tx downtime, but I think this is probably because they Pu power is much lower and so thy want to conserve as much power as possible. Even if they did have more power, Voyager only makes contact once per day so I think the increased thermal cycling is not much compared to reduction in cathode life so they probably go this route anyway. I do know that other missions I have worked on that have much more frequent contacts, we do keep the Tx heaters on so there is probably some number of cycles where one becomes more dominant.
> A final point, presumably the TWT's output has dropped over the decades, how is this monitored and do we know the percentage drop in power output (from the TWT not the Pu power source)?
This is telemetry from the TWTAs on the helix current and the anode voltage. Along with the DC power draw of the TWTAs I think you can interpret some of what the power output is and how it changed over life. Some of this might be curve fitting to ground-based life tests so there could be some estimation. The ground is also certainly measuring the received power so they can estimate power output from the spacecraft from that as well.
> The fact the TWT is still working seems quite remarkable
You're not the only one. From this article [0] a JPL engineer says "Nobody can explain why the Voyager TWT is still working".
[0] https://spinoff.nasa.gov/Traveling-Wave-Tubes-Travel-Far
Yes, they're used in the rf power amplifiers. Both the X-band amps use tubes, and one of the pair of S-band amps does. The other is solid state. I don't know what drove this decision back then - they're presumably mission-critical so perhaps they were separately sourced.
Slightly tangential, but if money were no object, do vacuum tubes or solid-state amps have a longer service life in this kind of application?
Theoretically, solid state amplifiers would have a longer service life. Vacuum tubes will always have cathode depletion causing them to degrade eventually (although this is pretty well controlled in most cases for long life). In space though the answer might become more complicated. Because the vacuum tubes are more tolerant of high voltages, they are probably also more tolerant of charge build-up and discharge from free electrons in space. Significant amounts of free electrons trapped in the second Van Allen belt that affects satellites in geosynchronous orbit (like a lot of comm satellites with big TWTAs). Less of an effect in deep space though.
That makes sense, also it's likely solid state devices would be more susceptible to cosmic rays, high velocity protons, which could cause catastrophic (instant) failure whereas TWTs would be unaffected.
I wonder if you may know any answers to questions I put to AllanYx ?
I wonder if you may know any answers to questions I put to AllanYx ?
It also has a vidicon tube for gathering images, although that can no longer be switched on.
Do we know the reason? Heater/cathode failure or HV supply would be likely reasons, or is there now insufficient power from the Pu source to run it?
(The HV supply on a vidicons is usually about 700V—that's high enough to be quite stressful on P/S components.)
(The HV supply on a vidicons is usually about 700V—that's high enough to be quite stressful on P/S components.)
There is a heater for the vidicon which had to be shut off due to power budget. This heater had been on since launch. After being switched off, turning on this heater back on, or the vidicon's cathode filament (itself a separate heater) would have cracked the tube due to the cold of space.
Yes, to save power. But there's nothing to photograph out there.
Makes sense, nice to know it wasn't due to failure.
maybe not.
https://buy.hpe.com/us/en/compute/apollo-systems/apollo-80-s...
502 Bad Gateway
https://buy.hpe.com/us/en/compute/apollo-systems/apollo-80-s...
502 Bad Gateway
From rumors I heard, for a while there before 2005, NASA was one of the biggest buyers of salvage sun4 hardware. they had multiple missions running on it and there were some particular high rate data input cards on mbus that were hard to duplicate on other platforms.
"we have a closet full of sparc 10s, we figure that'll be enough to last as long as the satellite"
"we have a closet full of sparc 10s, we figure that'll be enough to last as long as the satellite"
The spacecraft doesn't have a fuel gauge ...
Zoom call with 100 Airbus engineers ...
The original software for part of the onboard navigation system was running on a Windows 98 PC that no-one could find the password to and ended up using bolt cutters to extract the hard drives.
Talk about a deep space horror story!
Zoom call with 100 Airbus engineers ...
The original software for part of the onboard navigation system was running on a Windows 98 PC that no-one could find the password to and ended up using bolt cutters to extract the hard drives.
Talk about a deep space horror story!
No spacecraft has a "fuel gauge" in the traditional sense. You don't have gravity to for a float to work like any fuel gauge in a car or airplane. Fuel is driven by surface tension and pressure rather than any buoyancy/gravity forces. These leads to the development of diaphragm type tanks where the pressurized bladder pushes fuel out of the tank or propellant management device (PMD) type tanks [0/1] that use the surface tension to guide pressurant-free fuel to the thrusters.
So determining how much fuel you have left is done by a combination of integrating how much time you had thrusters firing, coupled with what pressure the tanks/lines were at while the thrusters were firing. Errors in these measurements accumulate over time which is why there is a lot of effort in to determining how much fuel is left in a spacecraft. Especially critical for things like big comm birds in GEO where fuel can be limiting in operation and the more fuel you have the longer you can keep station and get revenue from the satellite. But you need to still be conservative enough to have enough fuel to get out of the GEO belt for decommissioning your satellite.
[0] https://en.wikipedia.org/wiki/Propellant_management_device [1] https://www.northropgrumman.com/space/pmd-tanks/
So determining how much fuel you have left is done by a combination of integrating how much time you had thrusters firing, coupled with what pressure the tanks/lines were at while the thrusters were firing. Errors in these measurements accumulate over time which is why there is a lot of effort in to determining how much fuel is left in a spacecraft. Especially critical for things like big comm birds in GEO where fuel can be limiting in operation and the more fuel you have the longer you can keep station and get revenue from the satellite. But you need to still be conservative enough to have enough fuel to get out of the GEO belt for decommissioning your satellite.
[0] https://en.wikipedia.org/wiki/Propellant_management_device [1] https://www.northropgrumman.com/space/pmd-tanks/
I'm sure we'll get there eventually. But there is a fun idea in there somewhere about a future where the only need for manned spaceflight is so we can get pinged by the computer once every few days and go and hug/wrap a tape measure around the fuel bladder
The Apollo capsule didn't have a fuel gauge, either, because nobody could figure out how to make a fuel gauge work in weightless conditions. Various schemes failed. The solution turned out to be to have a reserve tank with enough propellant in it to do reentry. Then when the main tank ran dry, you knew just what you could do with what was left.
Since you know how much fuel you start with, can't you integrate something like the time spent firing the engine and use that as gauge?
Yes, and that is what most satellites due to estimate fuel. But any errors in that measurement can accumulate. See my other response on this thread for more details.
I'm sure they thought of that, so there must be some reason why it is inaccurate.
No, since engines aren't perfect and wastes propellant. Also fuel boils off in space.
Why would anyone need bolt cutters to extract the drives? I have a couple Win98 PCs in the basement, and you just unscrew the drives. The password on it didn't encrypt the drives, either.
As for the password, this is the old days. Could probably google how to crack the password for it. Or just try the classic 123456, querty, password, or letmein.
As for the password, this is the old days. Could probably google how to crack the password for it. Or just try the classic 123456, querty, password, or letmein.
Why would anyone need bolt cutters to extract the drives?
I've had PC cases that had eyelets for a lock. You'd have to get through that before "unscrew(ing) the drives".
I've had PC cases that had eyelets for a lock. You'd have to get through that before "unscrew(ing) the drives".
The drives aren't password protected. Why would one need to take the drives apart? Besides, the text was about extracting the drives, i.e. taking them out of the computer box.
Maybe to access the drive, they have to go through a lockable case. If someone lost the keys to that case, they might have had to pop the locks with a bolt cutter
[deleted]
That was exactly what I was driving at, thanks. In order to take the drives "out of the computer box", one needs to get through the lock on that computer box first.
It's not immediatley on point, but the title reminds me that many here may enjoy the documentary about the Voyager team as it works today called It's Quieter in the Twilight.
> Launched almost 46 years ago in 1977, the twin Voyager probes continue to send back data from beyond the Solar System.
> I checked with Nasa, which has assured me that the spacecraft are still being controlled from the same beige cubicle in an annex of its Jet Propulsion Laboratory (JPL) that I visited in 2017, marked with a homemade cardboard sign reading: "Mission critical hardware – PLEASE DO NOT TOUCH".
I throw around the term "mission critical" maybe too easily. Because that beige cubicle with the homemade cardboard sign sounds humbling.
> I checked with Nasa, which has assured me that the spacecraft are still being controlled from the same beige cubicle in an annex of its Jet Propulsion Laboratory (JPL) that I visited in 2017, marked with a homemade cardboard sign reading: "Mission critical hardware – PLEASE DO NOT TOUCH".
I throw around the term "mission critical" maybe too easily. Because that beige cubicle with the homemade cardboard sign sounds humbling.
Also NASA's infrastructure is crumbling, and 25% of workers are eligible for retirement.
https://arstechnica.com/space/2023/08/nasas-buildings-are-ev...
https://arstechnica.com/space/2023/08/nasas-buildings-are-ev...
A lot of software wraps old code. There is no reason to replace what works and works well. Besides, modern doesn’t generally mean more reliable or performant. It’s often the other way around.
Over the last few decades, embedded programming hasn’t changed much, for example. The rate of change is much higher with the ultra-abstracted languages.
Over the last few decades, embedded programming hasn’t changed much, for example. The rate of change is much higher with the ultra-abstracted languages.
space-x has taken over Nasa to some degree, i think
anyways, for those who don't have the patience, the title is misleading. it's just some old software from the 90s. It's only ancient if you're one of those people who completely rewrites their entire code base from the ground up every two years because to pad out your resume with whatever bullshit new "framework" is in vogue. Contrary to popular belief you actually can just keep using the same software indefinitely, it doesn't degrade with age, and for the most part the hardware lasts pretty long too as long as you take care of it (with a few exceptions like optical drives and SSDs).