Boeing's 2nd Starliner software glitch could have led to an in-space collision - Space.com

Boeing's CST-100 Starliner spacecraft is pictured back home at the company's Commercial Crew and Cargo Processing Facility, where it is undergoing inspection after its Orbital Flight Test mission in December 2019. (Image credit: Frank Michaux/NASA)

Seven weeks after Boeing's CST-100 Starliner failed to reach the International Space Station as planned during its first orbital flight test, NASA and Boeing officials disclosed the preliminary results of an investigation into what went wrong.

The uncrewed Starliner spacecraft launched atop an Atlas V rocket on Dec. 20, 2019, on a mission to dock with the International Space Station (ISS). The mission was designed to demonstrate the new spacecraft's ability to safely transport astronauts to and from the orbiting laboratory. However, Starliner failed to reach the correct orbit and instead spent the next two days circling Earth alone before executing a picture-perfect landing in New Mexico's White Sands Missile Range on Dec. 22.

In a teleconference with reporters on Friday (Feb. 7), NASA Administrator Jim Bridenstine said that an independent review team has identified several issues during the Orbital Flight Test (OFT) mission, particularly when it comes to the spacecraft's software. Along with the previously disclosed error with Starliner's onboard timer, a second software issue could have potentially led to a slight but problematic collision of two of the spacecraft's components, investigators determined.

The first software issue was identified shortly after Starliner launched, when the vehicle failed to execute an orbit-insertion burn. Boeing soon discovered that the spacecraft's onboard timing system, called a "mission elapsed timer," had erroneously pulled an incorrect time from the Atlas V rocket nearly 11 hours before liftoff. Starliner was supposed to retrieve that information from the rocket during the final countdown period before liftoff, and because this step happened prematurely, Starliner's internal clock had the incorrect time.

Then a second software problem was discovered before the spacecraft's return to Earth, and this glitch could have potentially led to a collision between Starliner's crew module and its service module, which are designed to separate in orbit before the crew module lands.

This problem, which Boeing officials called a "valve mapping error," had to do with the software that tells Starliner's crew module and service module to separate before landing. The service module's thrusters are responsible for conducting the deorbit burn that takes Starliner out of orbit, sending it back to Earth for landing. After the deorbit burn, the service module detaches from the crew module and safely splashes down into the Pacific Ocean.

"During what we call 'free flight,' when the crew module is attached to the service module, there's a certain valve mapping, and the flight computers on the crew module command all of the individual thruster firings. But after you separate the launch vehicle from the crew module, the propulsion controllers on the service module have to conduct those thruster firings to get the proper separation and disposal burn," John Mulholland, vice president and program manager of Boeing's Starliner program, said during the teleconference.

"That valve mapping is different in those two cases, and the software unfortunately had the same valve mapping for both of those conditions, so we had an incorrect valve mapping for the separation and disposal burn," Mulholland said.

Thankfully, the team detected that software error before Starliner began its descent procedure. "The team very quickly recoded the software, reverified it in the labs, and we were able to upload that software correction and safely complete the mission," Mulholland said.

An artist's illustration of Boeing's CST-100 Starliner spacecraft in orbit. (Image credit: Boeing)

If Boeing hadn't caught that glitch before Starliner came back to Earth, the two modules could have "bumped" into each other after separation, which could have destabilized the crew module during a critical point in its descent.

"The thrusters' uneven firing would cause the service module, which is a piece of a cylinder, to come away from the crew module and recontact, or bump back into it," Jim Chilton, senior vice president for Boeing Space and Launch, said during the teleconference, adding that "bad things" can happen as a result of that eventuality.

"That means you go poke the crew module, and it gets unstable and it has to go stabilize itself," Chilton said. "So, one question is, do you hit it hard enough to where it tumbles or you have a problem? Another thing is, you don't want to damage that heat shield, because you need the heat shield to come back in."

"It's hard to say where the service module would have bumped, but nothing good can come from those two spacecraft bumping back in, so we sent the software to make sure that couldn't happen," he said.

Communication problems

A third major problem that Starliner encountered during its orbital flight test was not related to the spacecraft's software, but rather had to do with interference that disrupted communications between ground controllers and Starliner.

When ground controllers tried to manually command Starliner to conduct an orbit-insertion burn after the timing error prevented that from happening automatically, they were unable to communicate with the spacecraft in a timely manner. Communications between Starliner and ground control are relayed via NASA's network of Tracking and Data Relay Satellites, but "high noise" interfered with those signals, Mulholland said.

The investigative review team is still working to determine the exact cause of this interference, but it appears to be associated with cell phone towers, he added.

While the independent review of Starliner's OFT issues is still underway, NASA expects to have the final report by the end of February, Bridenstine said. He added that NASA has not yet decided whether to conduct a second orbital flight test before using Starliner to fly astronauts to the International Space Station.

"This flight test taught us a lot," Chilton said. "What we wish we had done better is software."

Doug Loverro, the director of NASA's Human Spaceflight Operations directorate, said during the teleconference that the software issues "are likely only symptoms. They are not the real problem."

"The real problem is that we had numerous process escapes in the design, development and test cycle for software," Loverro said, "and as we go forward that is what we're going to be concentrating on ... how do we assure ourselves that all of the software that we've delivered, not just the two routines that were affected by these issues, are fixed?"

Email Hanneke Weitering at hweitering@space.com or follow her @hannekescience. Follow us on Twitter @Spacedotcom and on Facebook.