news Telstra has confirmed that it suffered a major outage in its high-end corporate cloud computing platform last week that left a number of its most high-profile customers without some of their services for a period as long as 24 hours.
According to information provided to Delimiter, the company’s Exhibition St datacentre in the Melbourne central business district suffered a severity one outage starting on Monday night last week at about 7pm Eastern Standard Time. The issue related to its storage layer knocked offline services belonging to a number of major customers reputedly including VISY, Real Insurance, Hollard Financial Services, Oz Minerals and others.
Responding to the issue this week, a spokesperson for the telco confirmed it had suffered an outage on its cloud computing infrastructure. “Last week we had an intermittent service outage on our cloud platform that affected a small number, around 20, of our business customers,” the spokesperson said. “The issue started on Monday 25 March when we identified a failure in the data storage equipment that supported the customers that we affected.”
“When the failure was identified we immediately engaged our storage partner and started restoring services. By Tuesday (26 March) afternoon the majority of services had been restored, though restoration activities for a small number of customers continued into Wednesday. We continue to closely monitor the dedicated hosting services of all customers affected by this issue and apologise for the impact on their services.”
The news has the potential to knock the facility’s uptime capability almost out of the top high-availability tables for IT infrastructure. Under commonly used guidelines, technology services are classified by the amount of downtime they suffer per year. If substantial parts of Telstra’s cloud computing infrastructure was offline for the majority of one day last week, the company’s cloud platform may no longer be able to be classed as having uptime above 99.9 percent (‘three nines’), as this would entail suffering downtime limited to 8.76 hours per year.
It is common for enterprise cloud computing platforms to enjoy uptime higher than this figure. For example, cloud computing vendor Salesforce.com states on its website that its Force.com platform has had a proven 99.9+ uptime “for years”. The company publishes uptime statistics on its site.
The news comes as Telstra continues to push the case that its cloud computing platform reflects a sizable revenue opportunity for it going forward. In mid-2011 the company stated that it would spend $800 million in the space to develop its infrastructure and target new customers.
At the time, the telco said the $800 million spend will go on a range of areas over the succeeding five years, but would especially be focused on the construction of a new Melbourne datacentre, slated to go live this year, which would bolster Telstra’s hosting capability by more than 40 percent, with an extra 2000 square metres of space.
The money was also to go on modernising existing Telstra datacentres, expanding the range of enterprise applications the company provides, building a new integrated online account management portal, increasing the automation of utility computing services, and enhancing the capabilities of T-Suite. The company’s key partners for the next phase of its strategy will be Cisco on the hardware front, VMware for virtualization, Accenture for the company’s systems integration skills and Microsoft for software.
In December last year, Telstra announced it would construct four new datacentres to meet demand.
Paul Geason, Group Managing Director, Telstra Enterprise and Government, said Telstra was committed to providing localised infrastructure to cater for the needs of its customers around the country. “We continue to see tremendous growth in our cloud business with positive customer take up of the technology across a range of industries,” Geason said at the time. “With many organisations moving into the cloud, the feedback has been clear – the option to use local data centres is important either because their applications are sensitive to latency or they require data to be hosted within their state.”
Look, I’m sure not all of Telstra’s cloud computing infrastructure went down, and that much of the services being provided to customers remained up and functioning fine. In this situation, Telstra would throw considerable resources at the issue to get it fixed as soon as possible.
However, to enterprise IT customers — and Telstra has some big manufacturers and so on in its customer ranks for cloud computing — 24 hours is a very, very long outage for this kind of enterprise-grade cloud computing. Frankly, I would be very surprised if Telstra hadn’t broken some of its service level agreements here, and I wouldn’t be surprised to hear that there’s compensation in the works here and there.
You cannot … you absolutely cannot, as an enterprise cloud computing player, let your services stay down for a whole day. That’s the kind of situation which encourages customers to start looking elsewhere for these kinds of services. These sorts of outages aren’t supposed to happen with a provider as large as Telstra — that’s why you go with Telstra in the first place.
Image credits: Telstra