An OIG video accompanying the release of the audit. Video source: NASA OIG YouTube channel.
The NASA Office of Inspector General issued a report today blasting NASA’s Ground Systems Development and Operations (GSDO) Program at Kennedy Space Center for delays and cost overruns in its Spaceport Command And Control System software.
The report describes SCCS as “a software system that will control pumps, motors, valves, power supplies, and other ground equipment; record and retrieve data from systems before and during launch; and monitor the health and status of spacecraft as they prepare for and launch.”
The audit states:
The SCCS development effort has significantly exceeded initial cost and schedule estimates. Compared to fiscal year 2012 projections, development costs have increased approximately 77 percent to $207.4 million and the release of a fully operational version has slipped by 14 months from July 2016 to September 2017. In addition, several planned capabilities have been deferred because of cost and timing pressures, including the ability to automatically detect the root cause of specific equipment and system failures. Without this information, it will be more difficult for controllers and engineers to quickly diagnose and resolve issues. Although NASA officials believe the SCCS will operate safely without these capabilities, they acknowledge the reduced capability could affect the ability to react to unexpected issues during launch operations and potentially impact the launch schedule for the combined SLS-Orion system ...
In the past, NASA has encountered difficulties with large and complex command and control software development efforts, failing on two occasions to meet expected requirements despite spending more than $500 million. In something of a repeat of this pattern, the SCCS development effort is more than 1 year behind schedule and significantly over cost, and several planned software capabilities have been deferred.
NASA made its decision regarding the SCCS software architecture nearly 10 years ago, but in our view this may no longer be the most prudent course of action given significant advances in commercial command and control software over that time. For example, the two companies under contract with NASA to deliver supplies to the International Space Station — Orbital Sciences Corporation and Space Exploration Technologies — both use commercial software products to accomplish their missions. In our judgment, the GSDO Program’s reluctance to change course reflects a cultural legacy at NASA of over-optimism and over-promising what the Agency can achieve in a specific timeframe.
Criticism of the NASA culture is a recurring theme over the years in various internal audits and external reviews.
The Columbia Accident Investigation Board report released in August 2003 showed a distinct lack of faith in NASA, citing a cultural arrogance resistant to change or external advice. The report stated on Page 102:
External criticism and doubt, rather than spurring NASA to change for the better, instead reinforced the will to “impose the party line vision on the environment, not to reconsider it,” according to one authority on organizational behavior. This in turn led to “flawed decision making, self deception, introversion and a diminished curiosity about the world outside the perfect place.” The NASA human space flight culture the Board found during its investigation manifested many of these characteristics, in particular a self-confidence about NASA possessing unique knowledge about how to safely launch people into space.
Space journalist James Dean of Florida Today reports on the release of the audit.
The initiative managed by KSC’s Ground Systems Development and Operations program faces ongoing challenges that could delay launches, prompting auditors to suggest that NASA consider scrapping it in favor of commercially available alternatives ...
In a response, Bill Gerstenmaier, head of NASA’s human spaceflight systems, acknowledged the work has proven more challenging than estimated and agreed to conduct an independent assessment after the first software system is ready.
The agency has “instigated a series of process improvements that are bearing positive results,” Gerstenmaier wrote last week.
A 2011 introductory film about Spaceport Command and Control System. Video source: ExploreKSCresources's channel YouTube channel.
UPDATE March 29, 2016 — Despite yesterday's events, the GSDO Program office which received the bad audit issued a press release today declaring all is well on the “Journey to Mars.”
“The team is working hard and we are making remarkable progress transforming our facilities,” said Mike Bolger, GSDO Program Manager. “As we are preparing for NASA's journey to Mars, the outstanding team at the Kennedy Space Center is ensuring that we will be ready to receive SLS and Orion flight hardware and process the vehicle for the first flight in 2018.”