VoIP provider Ooma recovers from complete service outage

12:45 pm EDT April 14, 2009 - This afternoon, a spokesperson for the data center co-location service Internap -- a name brought up in connection with both the Ooma and RIM service failures yesterday, which took place at approximately the same time -- denied any service outage, though admitted to some routine router replacements and maintenance.

Ooma's single data center, mentioned by its technical VP in his blog post yesterday, is located on the West Coast -- Internap, meanwhile, is located in Atlanta. Any service issues related to both services may have had to have been incidental.

11:00 am EDT April 14, 2009 - From the very beginning, its value proposition has been to enable businesses to rapidly create its Internet telecommunications infrastructure through the deployment of network hubs, sold for a one-time fee of $250. But yesterday, the complete hub infrastructure of the Ooma network failed completely for as much as four hours during lunchtime on the West Coast, in what its VP and technical chief late yesterday called an "event."

"Between 2PM and 3PM [PDT Monday], Internet connectivity was slowly being restored to our service," wrote Dennis Peng on the company's blog. "However, the flood of ooma Hubs coming back online created an immense amount of load on our provisioning systems. We rushed to add capacity to the system, but the nature of the network outage had interfered with the system's ability to recover by itself."

While VoIP competitors such as Skype rely on the variable P2P capacity of users' PCs to provide the network at large with the bandwidth it requires, Palo Alto-based Ooma's system relies on separate VoIP hardware clients -- the Ooma hubs. They're devices sold through outlets such as Amazon, and after paying the $250 and installing it themselves, customers pay no other fees for the lifetime of the product. Calls may be placed through Ooma to any telephone number -- not just Ooma users.

As the company's sales pitch puts it, traditional phone companies "limit your choices and charge exorbitant fees for the luxury of using outdated services. Instead of answers, the phone company gives you dead ends. They have invested hundreds of billions of dollars into a system meant to lock you in."
As Peng described yesterday, however, in a rather honest mea culpa, yesterday's outage appeared to be something related to the general Internet -- something that was out of Ooma's control. Though service engineers rushed to mitigate damage, he says something about this particular Internet outage made it impossible for the self-healing nature of the service's hub networks to repair their own damage. The long-term solution to prevent a reoccurrence of this problem, he added, may be to invest some money into the system.

"Discussions have already started on how to make the service resilient to a similar event in the future. Ooma currently has one data center located in west coast," wrote Peng. "We have planned to light up a second data center in the midwest or east coast this year, and this outage has served as a stark reminder for us to get moving on that. This has also served as a good opportunity for us to re-evaluate our contingency and business continuity plans."

Participants in Ooma's ongoing Twitter discussion yesterday blamed the service outage on a bigger service failure at Internap, a data center, co-location, and content delivery services provider based in Atlanta. They said such an outage affected multiple VoIP services and other major businesses, including Google's Gmail and RIM's BlackBerry service. Indeed, BlackBerry users did report a service backlog, during roughly the same timeframe yesterday, though no mail ended up being lost.
Internap has not made any public statements regarding a service outage, and Betanews has contacted Internap for clarification.