|
This is for my fellow geeks and nerds.
TL;DR
The best tech job in the world may be supporting two 47 year-old spacecraft which are approximately one (count’m 1) light day away from Earth.
A partial transcript from the Security Now Podcast:
NASA's Voyager 1 Resumes Regular Operations After Communications Pause." And I'm going to share it because, as I said, it contains a bunch of interesting and amazing science and engineering information. And then we're going to even dig down a little deeper.
So they wrote: "NASA's Voyager 1 has resumed regular operations following a pause in communication last month."
LEO: Geez.
STEVE: Yeah. "The probe had unexpectedly turned off its primary radio transmitter, called an X-band transmitter, and turned on the much weaker S-band transmitter. Due to the spacecraft's distance from Earth about 15.4 billion miles, 24.9 billion kilometers this switch prevented the mission team from downloading science data and information about the spacecraft's engineering status.
"Earlier this month, the team reactivated the X-band transmitter and then resumed collecting data the week of Nov. 18 from the four operating science instruments. Now engineers are completing a few remaining tasks to return Voyager 1 to the state it was in before the issue arose, such as resetting the system that synchronizes its three onboard computers. The X-band transmitter had been shut off by the spacecraft's fault protection system when engineers activated a heater on the spacecraft." Whoops.
LEO: Okay.
STEVE: "Historically, if the fault protection system sensed that the probe had too little power available, it would automatically turn off systems not essential for keeping the spacecraft flying in order to keep power flowing to the critical systems. But the probes have already turned off all nonessential systems except for the science instruments. So the fault protection system turned off the X-band transmitter and turned on the S-band transmitter because it uses lower power." Unfortunately, it also means it transmits at lower power, which means you can't get the data through, which is why they had stopped collecting data.
They said: "The mission is working with extremely small power margins on both Voyager probes. Powered by heat from decaying plutonium that is converted into electricity, the spacecraft lose about four watts of power each year. About five years ago, some 41 years after the Voyager spacecraft launched, the team began turning off any remaining systems not critical to keeping the probes flying, including heaters for some of the science instruments. To the mission team's surprise, all of those instruments continued to operate despite reaching temperatures lower than what they'd been tested for.
"The team has computer models designed to predict how much power various systems, such as heaters and instruments, are expected to use. But a variety of factors contribute to uncertainty in those models, including the age of the components and the fact that the hardware doesn't always behave as expected.
"With power levels being measured to fractions of a watt, the team also adjusted how both probes monitor voltage. But earlier this year, the declining power supply required the team to turn off a science instrument on Voyager 2. The mission shut off multiple instruments on Voyager 1 in 1990 to conserve energy, but those instruments were no longer in use after the probe flew past Jupiter and Saturn. Of the 10 science instruments on each spacecraft, four are now being used to study the particles, plasma, and magnetic fields in interstellar space," which is where both probes are.
"Voyagers 1 and 2 have been flying for more than 47 years and are the only two spacecraft to operate in interstellar space. Their advanced age has meant an increase in the frequency and complexity of technical issues and new challenges for the mission engineering team."
Okay. So reading that, the article said: "The X-band transmitter had been shut off by the spacecraft's fault protection system when engineers activated a heater on the spacecraft." What it didn't tell us is why the JPL engineers turned on that heater. And there's even more fascinating information about that.
Our listener Jeff Root in San Diego supplied the link to a story in The Register, of all places, titled "Best Job at JPL: What it's like to be an engineer on the Voyager project." This was posted two days later on the U.S.'s Thanksgiving Thursday. And it, too, is chock full of interesting science and engineering insight.
So the Register wrote: "The Voyager probes have entered a new phase of operations. As recent events have shown, keeping the venerable spacecraft running is a challenge as the end of their mission nears." And of course "end of the mission" just means we don't know what happened; right? I mean, it's like, it's way past its design end of mission, and it keeps getting extended.
So they wrote: "As with much of the Voyager team nowadays, Kareem Badaruddin, a 30-year veteran of NASA's Jet Propulsion Laboratory, divides his time between the twin Voyager spacecraft and other flight projects. He describes himself as a supervisor of chief engineers, but leaped at the chance to fill the role on the Voyager project. Suzanne Dodd, JPL Director for the Interplanetary Network Directorate, is the Project Manager for the Voyager Interstellar Mission.
"Badaruddin told The Register: 'She knew that the project was sort of entering a new phase where there was likely to be a lot of technical problems. And so chief engineers, that's what they do. They solve problems for different flight projects.'
"Dodd needed that support for Voyager. Badaruddin would typically have found someone from his group, but he said: 'I was just so excited about Voyager, I said, you know, look no further; right? I'm the person for the job.'" In other words, this was one he did not want to delegate. He said: 'I'm your engineer. You know, please pick me.'
"So Badaruddin has spent the past two years on the Voyager project. After decades of relatively routine operation, following plans laid out earlier in the mission when the team was much larger, the twin Voyager spacecraft have begun presenting more technical challenges to overcome as the vehicles age and power dwindles.
"The latest problem occurred when engineers warmed up part of the spacecraft, hoping that some degraded circuits might be 'healed' by an annealing process. Badaruddin explained that 'There's these junction field effect transistors (JFETs) in a particular circuit that have become degraded through radiation. We don't have much protection from radiation in an interstellar medium'" - remember, where this thing was never designed to function, right, because it wasn't expected to live this long.
"'We don't have much protection in an interstellar medium because we're outside the heliosphere, where a lot of that stuff gets blocked. So we've got this degradation in these electronic parts, and it's been proven that they can heal themselves if you get them warm enough, long enough. And so we knew we had some power margin, and we were hopeful that we had enough power margin to operate this heater. And as it turned out, we didn't. It was a risk we took to try to ameliorate a problem that we have with our electronics. So now the problem is still there, and we realize that we can't solve it this way. And so we're going to have to come up with another creative solution.'"
So The Register says: "The problem was that more power was demanded than the system could supply. A voltage regulator might have smoothed things out, but the Voyagers no longer have that luxury. Instead, engineers took a calculated risk and ran afoul of the then-innovative software onboard the spacecraft. The under-voltage routine of the fault protection software shuts down loads on the power supply; but since the Voyager team had already shut down anything that's not essential, there isn't much left for it to shut down.
"Badaruddin explained. He said: 'So the under-voltage response doesn't do much except turn off the X-band transmitter and turn on the S-band transmitter. And that's because the S-band transmitter uses less power, making it the last safety net to save you.' He said: 'And save the mission it did. While the S-band is great for operations near Earth, such as the Moon, it's almost useless at the distance of the Voyager spacecraft. However, by detecting the faint carrier signal of the S-band transmission, the team was able to pinpoint that the problem had been the act of turning on the heater, even without X-band telemetry from the spacecraft.
"'The challenge for engineers isn't just the time it takes to get a command to the Voyagers and receive a response, but also checking and rechecking every command that gets sent to the spacecraft.' He said: 'The waiting is apparently not as frustrating as we might think.' Badaruddin said: 'This is the rhythm we work in. We've grown accustomed to it. It used to be a very small time delay, and it's gradually grown longer and longer through the years.'
"With duplicate physical hardware long gone, the team now works with an array of simulators. Badaruddin said: 'We have a very clear understanding of the hardware. We know exactly what the circuitry is, what the computers are, and where the software runs. And as for the software? It's complicated. There have been so many tweaks and changes over the years'" - remember, 47 years - "'that working out the exact revision of every part of Voyager's code has become tricky.' Badaruddin said: 'It's usually easier to just get a memory readout from the spacecraft to find out what's going on out there.'
"The challenge for the Voyager team is that the spacecraft are nearing the half-century mark, as is the documentation. He said: 'We have documents that were typewritten in the '70s that describe the software, but there are revisions. And so building the simulators, we feel really good about the hardware, but we feel a little less good about understanding exactly what each instruction does.' The latest bit of recoding occurred with the failure of one of Voyager's integrated circuits, which manifested itself as meaningless data last year." And of course we talked about that on the podcast at the time.
"Badaruddin reminds us: 'The basic problem was figuring out what was wrong with no information. We could see a carrier signal; we knew we were transmitting in the X-band; we knew we could command the spacecraft because we could tweak that signal slightly with commands. So we knew the spacecraft was listening to us, and we knew the spacecraft was pointing at Earth because otherwise we wouldn't get a signal at all.'
"The engineers went further down the fault tree, and eventually managed to get a minimum program to the spacecraft to get a memory readout. That readout could be compared to one retrieved when the spacecraft was healthy. 256 words were corrupted, indicating a specific integrated circuit. Code was then written to relocate instructions around that failed area." And remember, this is almost a light-day away at that point, a year ago. "The problem there is the code was very compact. There was no free space that we could take advantage of. So we had to sacrifice something." So they're patching on the fly on an operating machine, what is it, 15 billion miles away.
That something that needed sacrificing was one of the Voyager's higher data rate modes, used during planetary flybys. And that makes sense; right? It's like, hey, what don't we need? Well, we don't need the high data rate mode used during planetary flybys because we're not going to be flying by any planets.
So now back to the present. "The current challenge" - if you'll pardon the pun - "involves dealing with the probes' thrusters." And here's the problem, Leo. Silicon from bladders inside the fuel tanks has begun to leach into hydrazine propellant. Since silicon doesn't ignite like hydrazine, meaning it doesn't get burned off, a tiny amount gets deposited in the thrusters and slowly builds up in the thruster capillaries. Badaruddin uses the analogy of clogging arteries. Eventually, the blockage will prevent the spacecraft from firing its thrusters to keep it pointed at Earth.
"However, the pitch and yaw thrusters, each of which have three branches, are clogging at different rates. The current software works on the basis that branch 1, 2, or 3 will be used. But could it be operated in mixed mode, where branch 2 is used for the pitch thruster, but branch 3 is used for yaw?
"Badaruddin notes: 'So that's a creative solution. It would be very complicated. This would be another modification in interstellar space to the software.' And getting it right the first time is not just nice to have, it's almost essential. By the time the results of a command come back from the Voyager spacecraft, it might be impossible to deal with the fallout of a failure."
LEO: Wow. What do they write it in? Is it assembly language? What is it?
STEVE: Oh, yeah. It's all individual, like, they have - they invented their own processor.
LEO: Oh, of course.
STEVE: They're not using any commercial processor. They invented a computer that reads this code. And that's where he's saying sometimes we're not sure what an instruction does, because somebody typed it in 1970 and may have said, oh, it's lunchtime, I'll get back to you later.
LEO: Wow. Wow. This is amazing.
STEVE: It is just incredible.
LEO: Oh, my god, good stories, yeah.
STEVE: He said: "The Voyager spacecraft are unlikely to survive another decade. The power will eventually dwindle to the point where operations will become impossible."
LEO: Is it a nuclear power plant on that?
STEVE: Yeah, yeah. It is a nuclear power. It is using decaying plutonium, the heat generated from the particle decay, to heat a thermocouple which generates the electric current to drive all of this.
LEO: Oh. So it's a tiny bit of...
STEVE: And it's been exponentially decaying for 47 years.
LEO: Pretty good.
STEVE: Since this thing was first launched.
LEO: That's a long time, wow.
STEVE: Yeah. So he says: "High data rates, which is to say 1.4 kilobits per second, will only be supported by the current Deep Space Network until 2027 or 2028. After that, some more creativity will be needed to operate Voyager 1's digital tape recorder. Badaruddin speculates that shutting off another heater (the Bay One heater) used for the computers would free up power for the recorder." I should mention that we're only able - the Deep Space Network, as I recall, is only out of Australia. And so it's only during a brief time window once a day as the Earth rotates that the Deep Space Network antenna is able to point at Voyager 1. And so Voyager 1 records its data during the dark period and then dumps it to us when it knows we're able to receive it.
So he says: "Turning off the Bay One heater used for the computers would free up power for the recorder, according to the thermal model, but it'll be a delicate balancing act. And, of course, the recent annealing attempt demonstrated the limitations of modeling and simulations on Earth.
"So does Badaruddin have a favorite out of the two spacecraft? He replies: 'Well, Voyager 2 is the one that's been flying the longest, and Voyager 1 is the one that's furthest from Earth. So they both have a claim to fame.' He said: 'To use another analogy, they're essentially twins. They're basically the same person, but they live different lives, and they have different medical histories and different experiences.'"
LEO: What a great line.
STEVE: "Badaruddin hopes to stick with the mission until the final transmission from the spacecraft. He said: 'I love Voyager. I love this work. I love what I'm doing. It's so cool. It just feels like I've got the best job at JPL.'"
LEO: And he's, I'm sure, in his 60s if not 70s; right?
STEVE: Yeah.
LEO: He's been with it for 30 years with JPL.
STEVE: Yeah.
LEO: Wow.
STEVE: So I just checked on the Voyager 1 mission status, which is what gave me the title for today's podcast. That intrepid little spacecraft is now so far away that light and radio signals take more than 23 hours to travel in each direction. Not round trip. Each direction. So two days round trip. So it's nearly an entire light-day distant. Yet Voyager 1 - and this is what boggles my mind - is managing to keep itself pointed at our Earth across all that distance, and we still have working bi-directional communication with it. This entire endeavor has been an astonishing example of incredible engineering. The original design - and this, too. The original design was flexible enough and software controlled enough that even though it was designed in the 1970s and launched on September 5th, 1977, all well before the Internet and all of the technology we now take for granted, this machine has endured and has exceeded everyone's expectations many times over.
The story does make one principle absolutely clear: No pure hardware solution could have ever done this. No pure hardware solution would still be alive, functioning, and communicating after 47 years of space flight. Nor even could any fixed firmware hybrid hardware/software solution. The reason is that none of what has transpired since Voyager 1's original mission was redefined and extended, after it continued to perform so brilliantly, could have been anticipated by NASA's brilliant engineers in the mid-'70s. The sole key to Voyager 1's success today is that to an extremely large degree the original designers of the spacecraft put the machine's hardware under software control. The reason they did that way back in the '70s was different from the reason they're now glad they did that. They created a deeply software-based control system back then because software doesn't weigh anything, and the spacecraft didn't have an ounce of weight to spare.
So the engineers of the '70s put their faith in software. And that faith, and the inherent dynamic redesign flexibility it enabled, has given the spacecraft a far longer life than it could have ever otherwise enjoyed because software doesn't weigh anything.
LEO: Isn't that amazing.
STEVE: And all of that said, yesterday's and today's software is ultimately at the mercy of hardware. You know? If the attitude control systems' capillaries ultimately become clogged with leached and deposited silicon, the spacecraft's ability to maneuver and keep itself pointing at the Earth will eventually be lost. At some point in the not too distant future it will still be alive out there, but we'll have lost contact with one another. You know, what an amazing accomplishment, Leo.
LEO: It's a great story.
STEVE: I mean, it makes you proud.
LEO: It also - there's another lesson which is sometimes constraints force a kind of creativity that's better than if you have unlimited hardware and software, unlimited memory, unlimited storage.
STEVE: It's why I'm pointing at that PDP-8 behind me. It came with 4K words of memory. And it was expandable to 16, I think, or 12. It's what I miss about the old days where you really - there was creativity and engineering instead of just asking ChatGPT for a program.
LEO: Right.
STEVE: You know, which it spits out from having ingested the Internet.
LEO: Right.
STEVE: It is a different world.
LEO: Yeah. Fascinating. Well. You know, we've covered this story for a couple of years now, and it's...
STEVE: As it's been - that intrepid little probe has been out there.
LEO: And there are, I've mentioned already, there are some documentaries. There's one fairly recent one that covers the old folks.
STEVE: And I watched it after your recommendation. It was fantastic. Really fun.
_________________
Holoholo …
|
