New Horizons enters safe mode 10 days before Pluto flyby(planetary.org)
planetary.org
New Horizons enters safe mode 10 days before Pluto flyby
http://www.planetary.org/blogs/emily-lakdawalla/2015/07042044-new-horizons-enters-safe-mode.html
47 コメント
The idea that we have an ongoing digital comms with something this far from Earth is completely fascinating to me. Are the encodings used documented anywhere? All I can find out is that it's X-band, 1kb/s.
"...encodes block frame data from the spacecraft Command and
Data Handling (C&DH) system into rate 1/6, CCSDS Turbo-coded
blocks."
Cooooool. ["CCSDS" "standard"] was the Google search I was looking for.
If you are curious about reading more, CCSDS is an overloaded term in the space industry for an onion of different OSI layers that are changing all the time. My guess is that the telemetry coding standard CCSDS 101.0-B-6 is likely to have been used on New Horizons. Even though it is now a "historical document", some new space missions still use it.
Here is a copy: http://public.ccsds.org/publications/archive/101x0b6s.pdf
Here is a copy: http://public.ccsds.org/publications/archive/101x0b6s.pdf
Sadly, I'm guessing a scavenged satellite tv dish and a $12 SDR with a home built antenna is not going to let me listen in...
I wonder just how badly TCP would cope with 32million millisecond round trip ping times?
I wonder just how badly TCP would cope with 32million millisecond round trip ping times?
You could eavesdrop on some of the probes with a setup like that, just not anything as far out as New Horizons. Google "amateur DSN" for more.
> Alan Stern, the mission's P.I., dispelled rumors that contact had been lost on the USF forum
Not sure what you mean? He was responding to the (supposed) rumor that contact had "been lost again or was never regained".
That Deep Space Network Now page is great. Thanks for sharing.
Not sure what you mean? He was responding to the (supposed) rumor that contact had "been lost again or was never regained".
That Deep Space Network Now page is great. Thanks for sharing.
We're saying the same thing. He dispelled rumors that contact had been lost [permanently]. Sorry if that was unclear - the article clearly states it was certainly lost the first time.
"NASA’s New Horizons mission is returning to normal science operations after a July 4 anomaly and remains on track for its July 14 flyby of Pluto.
The investigation into the anomaly that caused New Horizons to enter “safe mode” on July 4 has concluded that no hardware or software fault occurred on the spacecraft. The underlying cause of the incident was a hard-to-detect timing flaw in the spacecraft command sequence that occurred during an operation to prepare for the close flyby. No similar operations are planned for the remainder of the Pluto encounter."
http://www.nasa.gov/nh/new-horizons-plans-july-7-return-to-n...
The investigation into the anomaly that caused New Horizons to enter “safe mode” on July 4 has concluded that no hardware or software fault occurred on the spacecraft. The underlying cause of the incident was a hard-to-detect timing flaw in the spacecraft command sequence that occurred during an operation to prepare for the close flyby. No similar operations are planned for the remainder of the Pluto encounter."
http://www.nasa.gov/nh/new-horizons-plans-july-7-return-to-n...
I wouldn't want to be the one debugging this. Talk about pressure to deliver and difficult constraints!
On the contrary, this is where you can really prove that you are worth your salt. There is no better arena to prove yourself than a real-live production-down situation.
It's a shame really, as the engineers fighting the crisis' gets a lot of attention, but very often those crisis' are caused by themselves.
It is the engineers who's projects run smoothly who is ultimately worth more, as they can predict and prevent problems before they become a crisis, but get no recognition for it.
It is the engineers who's projects run smoothly who is ultimately worth more, as they can predict and prevent problems before they become a crisis, but get no recognition for it.
Yes, this. It reminds me of goalies in hockey, and how people are in awe when a goalie makes some ridiculous save when in fact the goalie would have never had to have made such a save if they hadn't been out of position in the first place. The best goalies are pretty boring to watch.
Yes! The Phillies used to have a fan-favorite outfielder who played hard and often made spectacular catches - but he was actually a pretty bad outfielder; the reason he made spectacular catches is because he turned routine plays into adventures.
I remembered now how some people considered the US goalie one of the best world cup players...
The thing is, he was considered one of the best world cup players because the US defense was so bad, but so bad, that without him US would have ended the cup losing all games outright.
The thing is, he was considered one of the best world cup players because the US defense was so bad, but so bad, that without him US would have ended the cup losing all games outright.
In any sufficiently complex system, chaos can always rear its ugly head, no matter how good the engineers. The quality of engineers is then reflected in how well they respond to those cases. With space, you add other variables like micrometeors taking out a component or an errant cosmic ray flipping a bit at a really initiative time.
And on the other hand there are things even a number of engineers working together can't predict. Everybody makes mistakes, and having people who can work well under pressure is important regardless of whose fault caused the problem to begin with.
And may even get the boot, because the beancounters wonder what said engineer is really doing for the paycheck...
I guess it can be seen as some variant of the Lucas Critique.
https://en.wikipedia.org/wiki/Lucas_critique
I guess it can be seen as some variant of the Lucas Critique.
https://en.wikipedia.org/wiki/Lucas_critique
Yep, I think the ops / system admin saying should be "all my successes happened in the darkness and all my failures occurred in the harsh light of day".
This is a pretty common sentiment among firefighters/paramedics. You don't wish for something bad to happen to someone, but you absolutely want to be there if it does.
It might be a shame when your real-live production-down situation is a multimillion dollar space probe nine years into a mission to capture the first high-resolution imagery as well as collect mounds of scientific data about one of the (ex-)planets in our solar system we know the least about, right after discovering surface anomalies including four suspicious dark spots on one of the body's faces.
This isn't about proving you're worth your salt. It's about "let's get this thing working, ASAP, because who knows when we'll have another chance, since there's not exactly a plan B."
This isn't about proving you're worth your salt. It's about "let's get this thing working, ASAP, because who knows when we'll have another chance, since there's not exactly a plan B."
"Proving oneself" is not a thing everyone values.
That assumes you're proving yourself to a rational engineer - rather more likely one would be attempting to prove oneself to a non-tech manager, and what happens there is that typically the fixer is so deeply associated with the problem that they end up being seen as the problem.
I mean, how many times have you fixed a downed server through no fault of your own to then have your head blown off by a client?
I mean, how many times have you fixed a downed server through no fault of your own to then have your head blown off by a client?
I beg to differ, a situation with a lot of factors you can't control is rarely a way to prove that you're worth your salt, It's the way you communicate your advancements what makes a difference.
Being "worth your salt" definitely includes situations out of your control, because you need to be able to handle issues that probably aren't in your control. You're only useful if you can help with the current issues...
Yeah, it's bad enough when there's latency and my SSH session is delayed several seconds... 3 billion miles is 4h at light speed.
Well hey, at least they're probably not using SSH ;)
Well hey, at least they're probably not using SSH ;)
Does anyone know which language(s) the NASA engineers write the spacecraft's software in?
The main command and data handling processor runs code written in C, as does the guidance and control computer. However, the G&C algorithms were 'written' in Simulink, and autocoded to C. I believe the instruments also use C, unless they are too resource constrained even for that, in which case, I'd guess Forth.
Pretty interesting maybe-answer to that, from some searching:
http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit
> Which language is used for programming New Horizon's flight software? How long is code? How do you make sure that it's gonna work?
We don't do the coding for the flight software, that's SciOps.
However here's what I know, each set of spacecraft commands is put up as a "command load", which has a name that's the year plus the day of year. So the 15188 load runs on July 7, and has commands for nine days.
Each load has a multi week coding and vetting process and is simulated on the ground, on a system called "NHOPS", pronounced "nops".
-AZ
I emailed some folk, here's an answer to #3, from Jillian, one of the members of our SciOps team.
3: We have a program called Statesim that checks to make sure constraints are not violated. For instance the Alice aperture door shouldn't be open within 20 degrees of the Sun. We also have something called NHOps which is a software EM of New Horizons and the commands are executed on it and data is downloaded and reviewed by the instruments. We also had a rehearsal in 2013 for the Pluto Closest Approach load.
-AZ
Okay, here's an answer from Helen Hart at Johns Hopkins University Applied Physics Lab (AZ):
We do not program New Horizons flight software. We store commands, packaged into macros, in the portion of C&DH memory designated for command storage, and execute the macros via a Time Tag.
See #1.
How are we sure that it¹s gonna work? That¹s a really, really big question, involving multiple layers, multiple platforms, and a lengthy, intensive Load Build and Review process. For more details on the process, talk to the Science Sequencers.
The command load is simulated in SEQGEN, which is also where it gets built. The Seqgen Modeling File contains hundreds of checks for problems with commands vs. the state of the spacecraft/instrument. The Seqgen modeling file also contains the Mission Operations Playback Data Volume models.
We have a SOFTWARE SIMULATOR called stateSim which models the response of the spacecraft and payload to the command sequence. stateSim generates the file that Mission Operations Command Sequencer turns into the Command Sequence Timeline - the excel spreadsheet that is distributed as part of the load review process.
We have HARDWARE SIMULATORS, called NHOPS-1 and NHOPS-2, which contains HARDWARE modules for the onboard processors, including C&DH-1, C&DH-2, SSR-2, SSR-2, G&C-1, G&C-2, P-Alice, LORRI, PEPSSI, Ralph, REX-1, REX-2, SDC, and SWAP.
The NHOPS and stateSim simulators have different weaknesses; for example: a: NHOPS gets all the details of slews correct; stateSim gets the slew start time and duration correct, but does not track the exact path of the track because that part of the CG&C cannot be modeled in software. b: stateSim correctly predicts C&DH Thermal Control, but NHOPS does not predict this at all c: SSR usage: stateSim does not model SSR usage. NHOPS does, but gets the wrong answer for anything that gets Compressed or Packetized. SEQGEN is tied directly to commanded data downlinks, and cannot be manipulated to produce interim data volumes; it correctly models downlink data volume TO THE ACCURACY OF the Compression Ratios specified by the Science Sequencers (lately, DataTrack).
-- Helen
http://interviewly.com/i/nasas-new-horizons-oct-2014-reddit
> Which language is used for programming New Horizon's flight software? How long is code? How do you make sure that it's gonna work?
We don't do the coding for the flight software, that's SciOps.
However here's what I know, each set of spacecraft commands is put up as a "command load", which has a name that's the year plus the day of year. So the 15188 load runs on July 7, and has commands for nine days.
Each load has a multi week coding and vetting process and is simulated on the ground, on a system called "NHOPS", pronounced "nops".
-AZ
I emailed some folk, here's an answer to #3, from Jillian, one of the members of our SciOps team.
3: We have a program called Statesim that checks to make sure constraints are not violated. For instance the Alice aperture door shouldn't be open within 20 degrees of the Sun. We also have something called NHOps which is a software EM of New Horizons and the commands are executed on it and data is downloaded and reviewed by the instruments. We also had a rehearsal in 2013 for the Pluto Closest Approach load.
-AZ
Okay, here's an answer from Helen Hart at Johns Hopkins University Applied Physics Lab (AZ):
We do not program New Horizons flight software. We store commands, packaged into macros, in the portion of C&DH memory designated for command storage, and execute the macros via a Time Tag.
See #1.
How are we sure that it¹s gonna work? That¹s a really, really big question, involving multiple layers, multiple platforms, and a lengthy, intensive Load Build and Review process. For more details on the process, talk to the Science Sequencers.
The command load is simulated in SEQGEN, which is also where it gets built. The Seqgen Modeling File contains hundreds of checks for problems with commands vs. the state of the spacecraft/instrument. The Seqgen modeling file also contains the Mission Operations Playback Data Volume models.
We have a SOFTWARE SIMULATOR called stateSim which models the response of the spacecraft and payload to the command sequence. stateSim generates the file that Mission Operations Command Sequencer turns into the Command Sequence Timeline - the excel spreadsheet that is distributed as part of the load review process.
We have HARDWARE SIMULATORS, called NHOPS-1 and NHOPS-2, which contains HARDWARE modules for the onboard processors, including C&DH-1, C&DH-2, SSR-2, SSR-2, G&C-1, G&C-2, P-Alice, LORRI, PEPSSI, Ralph, REX-1, REX-2, SDC, and SWAP.
The NHOPS and stateSim simulators have different weaknesses; for example: a: NHOPS gets all the details of slews correct; stateSim gets the slew start time and duration correct, but does not track the exact path of the track because that part of the CG&C cannot be modeled in software. b: stateSim correctly predicts C&DH Thermal Control, but NHOPS does not predict this at all c: SSR usage: stateSim does not model SSR usage. NHOPS does, but gets the wrong answer for anything that gets Compressed or Packetized. SEQGEN is tied directly to commanded data downlinks, and cannot be manipulated to produce interim data volumes; it correctly models downlink data volume TO THE ACCURACY OF the Compression Ratios specified by the Science Sequencers (lately, DataTrack).
-- Helen
Very likely it's cosmic rays causing a memory error
I think I figured it out. Someone put their book on the F8 key during a restart.
That is a really well-written article. It explains the problem, explains what the remediation plans are, and puts a human face on it...while remaining positive and optimistic.
Looking forward to seeing what the problem turned out to be and how they solve it!
Looking forward to seeing what the problem turned out to be and how they solve it!
I highly recommend the planetary.org blogs for space news, especially Emily Lakdawalla's - they are always excellent.
Does anyone know how good of a quality the cameras are on New Horizons?
Here's some detailed description[1] of New Horizon's science payload. Also technical details link on the bottom of the page. I guess not all the information is in the form of a layman (like me too) asking "just how good they are?", so might need some "science->masses" translation.
There's also a writeup on ArXiv[2], that seems to be even more detailed, though just found it and skimmed so far.
[1]: http://pluto.jhuapl.edu/Mission/Spacecraft/Payload.php
[2]: http://arxiv.org/abs/0709.4261
There's also a writeup on ArXiv[2], that seems to be even more detailed, though just found it and skimmed so far.
[1]: http://pluto.jhuapl.edu/Mission/Spacecraft/Payload.php
[2]: http://arxiv.org/abs/0709.4261
Wikipedia says the camera is 1024x1024. There are a bunch of other non-optical sensors on board, as well.
https://en.wikipedia.org/wiki/New_Horizons
https://en.wikipedia.org/wiki/New_Horizons
1024x1024 seems rather low res even by 2006 standards. Any idea what special/expensive/practical reason it needed to be that low res?
One of the team members had an excellent answer to this question a few days ago in a Reddit IAmA:
https://www.reddit.com/r/IAmA/comments/3bnjhe/hi_i_am_alan_s...
>We’re limited in other ways, weirdly. For example, LORRI, our high resolution imager, has an 8-inch (20cm) aperture. The diffraction limit (how much an 8” telescope can magnify) is 3.05 mircorad. which is just over half the size of single pixel 4.95 microrad. So if we swapped out the current sensor with a higher res one, we couldn’t do much better because of the laws of physics. A bigger telescope would solve that problem, but then it would make the spacecraft heavier, which require more fuel to send to Pluto AND a longer time to get there, because the spacecraft is more massive. We launched Pluto on the largest, most powerful rocket available at the time (the Atlas V, with extra boosters), so again we’re limited by physics: “At the time” doesn’t mean best ever. The Saturn V rocket, which sent astronauts to the moon, was actually more powerful.
>More megapixels also means more memory. For example, LORRI images are made up of a header and then the 1024x1024 array of numbers that make up our image and go from 0 to 65535 (216). There’s not really a way to make that info smaller if we went to 2048x2048. We could downlink a compressed version, but we want the full info eventually.
tl;dr
1. Between optical physics and balancing different costs to launch mass, it was the sound engineering choice.
2. Higher resolution would take even longer to retrieve the captured data.
>We’re limited in other ways, weirdly. For example, LORRI, our high resolution imager, has an 8-inch (20cm) aperture. The diffraction limit (how much an 8” telescope can magnify) is 3.05 mircorad. which is just over half the size of single pixel 4.95 microrad. So if we swapped out the current sensor with a higher res one, we couldn’t do much better because of the laws of physics. A bigger telescope would solve that problem, but then it would make the spacecraft heavier, which require more fuel to send to Pluto AND a longer time to get there, because the spacecraft is more massive. We launched Pluto on the largest, most powerful rocket available at the time (the Atlas V, with extra boosters), so again we’re limited by physics: “At the time” doesn’t mean best ever. The Saturn V rocket, which sent astronauts to the moon, was actually more powerful.
>More megapixels also means more memory. For example, LORRI images are made up of a header and then the 1024x1024 array of numbers that make up our image and go from 0 to 65535 (216). There’s not really a way to make that info smaller if we went to 2048x2048. We could downlink a compressed version, but we want the full info eventually.
tl;dr
1. Between optical physics and balancing different costs to launch mass, it was the sound engineering choice.
2. Higher resolution would take even longer to retrieve the captured data.
I'm not an engineer, but it's probably a combination of mission specs being drawn in 2001 and the fact that adapting technology for use in space is a multi-year process.
The Curiosity rover, which was commissioned in 2004 and launched in 2011, has a primary camera with 1600x1200 resolution.
The Curiosity rover, which was commissioned in 2004 and launched in 2011, has a primary camera with 1600x1200 resolution.
> and a less educational (but not catastrophic) gap in our
> light curves for Nix and Hydra.
You can figure out the rotational period, and get an idea of what the surface looks like, from a light curve. For example, if half of it reflects a lot more light than the other half, then you'll see a repeating brighter/dimmer/brighter/dimmer pattern with a period equaling the rotational period.
If you have a gap in the light curve, it increases your uncertainty.
If you have a gap in the light curve, it increases your uncertainty.
Edge. Of. Seat.
Edge. Of. Solar System.
It won't reach the edge of the solar system for many decades, at the very least (if the edge is defined as interstellar space).
However, it truly won't leave the solar system entirely for at least 30 000 years: http://spectrum.ieee.org/tech-talk/aerospace/astrophysics/vo...
However, it truly won't leave the solar system entirely for at least 30 000 years: http://spectrum.ieee.org/tech-talk/aerospace/astrophysics/vo...
I was being metaphorical and poetic. :)
>Not Implemented Tor IP not allowed
Come on! What's the point of blocking tor for them?
Come on! What's the point of blocking tor for them?
Also the Deep Space Network[1] page shows an ongoing 1kbps downlink from New Horizons; during the safe mode event it was at only 9bps. So that's a good sign! I'm sure they are still panicking a bit about what went wrong, but hopefully we're out of the woods on this particular anomaly.
[0] http://www.unmannedspaceflight.com/index.php?showtopic=8047&... [1] http://eyes.nasa.gov/dsn/dsn.html