NASA Messup Leaves Hundreds VoIP-less

Several hundred NASA employees had to rely on personal phones and PDAs Wednesday when a new VoIP system shut down at the Washington DC headquarters.


The system outage, which occurred at around 1:30 pm, April 12th, disabled the phone system and all computer networking for several hours.  While technicians were able to reinstate network connectivity by 3pm, VoIP phone service wasn’t repaired until nearly 7:30pm.

“The technicians were doing standard configurations and adding new users when the system shut down.” Said Sonja Alexander, spokesperson for NASA.  “But, during the outage, key NASA employees were able to continue working by using their cell phones.”

According to the internal NASA memo circulating on Thursday, the shutdown was caused when a contracted technician inadvertently deleted the entire NASA Headquarters VoIP user database while adding 19 new VoIP user accounts to the system.  Somehow, when the technician encountered the warning question advising against the action, “the technician answered the question incorrectly,” the memo said.  NASA expects to repair the database and continue using the VoIP system, but with a series of safety measures against future malfunctions, as well as reduce recovery time should another one take place.  “I know that we are creating additional procedural changes so that, if something happened again, the system would be restored more quickly,” Sonja Alexander said.  Too bad it wasn’t April 1st.

Related Links
No related posts.
Summary

Several hundred NASA employees had to rely on personal phones and PDAs Wednesday when a new VoIP system shut down at the Washington DC headquarters.


Contact the author.
Comments

Who was the vendor’s VOIP System was NASA using ? We have had so many problems using CISCO VOIP infrastructure.

Posted by David Samter on April 18th, 2006 at 10:26 am

Our product architecture (at FreelineUSA, Inc.) specifically addresses this type of failure in company / government VoIP communications infrastructure. Fully distributed calling databases, located in each active node, are specifically “not centrally updatable” in real-time. This avoids a single point of failure, and provides a level of survivability that is truly unique and resistant to man-made, natural, and all too frequent IP WAN network failures. The FL-USA 800 VoIP Series in use 12/7/365 world-wide today, in networks with zero downtime to date. We have to prove this new technology is not only as good as the technology we replace, we have to prove we can do better.

Posted by Robert Simkavitz on April 18th, 2006 at 12:06 pm

Where were their backups? Why couldn’t they just replace the DB? Or did it take 6 hours because it was so big?

Posted by J. Wade on April 18th, 2006 at 12:51 pm

they used cisco infrustructure. a lot of issues with that product

Posted by George on April 25th, 2006 at 1:46 pm

Oh come on George, “they used cisco infrustructure. a lot of issues with that product”. I guess 8+ million phones shipped and #1 position in the IP telephony market really shows “a lot of issues”. Let me guess, you work for Avaya? I’m a contractor who has some colleagues working at NASA, and someone stated that the engineer simply did not follow protocol. Don’t confuse pilot error with product function. By the way, “infrustructure” is spelled “infrastructure”.

Posted by Joe on July 24th, 2006 at 7:15 am
Post your Comments
Please ensure your comment is appropriate before posting.