• Great articles on other sites
  • RSS Great articles on other sites

  • Renai's other site: Sci-fi + fantasy book news and reviews
  • RSS Renai LeMay

  • Enterprise IT, Featured, News - Written by on Wednesday, April 3, 2013 14:51 - 19 Comments

    Telstra’s cloud computing suffers 24 hour outage

    news Telstra has confirmed that it suffered a major outage in its high-end corporate cloud computing platform last week that left a number of its most high-profile customers without some of their services for a period as long as 24 hours.

    According to information provided to Delimiter, the company’s Exhibition St datacentre in the Melbourne central business district suffered a severity one outage starting on Monday night last week at about 7pm Eastern Standard Time. The issue related to its storage layer knocked offline services belonging to a number of major customers reputedly including VISY, Real Insurance, Hollard Financial Services, Oz Minerals and others.

    Responding to the issue this week, a spokesperson for the telco confirmed it had suffered an outage on its cloud computing infrastructure. “Last week we had an intermittent service outage on our cloud platform that affected a small number, around 20, of our business customers,” the spokesperson said. “The issue started on Monday 25 March when we identified a failure in the data storage equipment that supported the customers that we affected.”

    “When the failure was identified we immediately engaged our storage partner and started restoring services. By Tuesday (26 March) afternoon the majority of services had been restored, though restoration activities for a small number of customers continued into Wednesday. We continue to closely monitor the dedicated hosting services of all customers affected by this issue and apologise for the impact on their services.”

    The news has the potential to knock the facility’s uptime capability almost out of the top high-availability tables for IT infrastructure. Under commonly used guidelines, technology services are classified by the amount of downtime they suffer per year. If substantial parts of Telstra’s cloud computing infrastructure was offline for the majority of one day last week, the company’s cloud platform may no longer be able to be classed as having uptime above 99.9 percent (‘three nines’), as this would entail suffering downtime limited to 8.76 hours per year.

    It is common for enterprise cloud computing platforms to enjoy uptime higher than this figure. For example, cloud computing vendor Salesforce.com states on its website that its Force.com platform has had a proven 99.9+ uptime “for years”. The company publishes uptime statistics on its site.

    The news comes as Telstra continues to push the case that its cloud computing platform reflects a sizable revenue opportunity for it going forward. In mid-2011 the company stated that it would spend $800 million in the space to develop its infrastructure and target new customers.

    At the time, the telco said the $800 million spend will go on a range of areas over the succeeding five years, but would especially be focused on the construction of a new Melbourne datacentre, slated to go live this year, which would bolster Telstra’s hosting capability by more than 40 percent, with an extra 2000 square metres of space.

    The money was also to go on modernising existing Telstra datacentres, expanding the range of enterprise applications the company provides, building a new integrated online account management portal, increasing the automation of utility computing services, and enhancing the capabilities of T-Suite. The company’s key partners for the next phase of its strategy will be Cisco on the hardware front, VMware for virtualization, Accenture for the company’s systems integration skills and Microsoft for software.

    In December last year, Telstra announced it would construct four new datacentres to meet demand.

    Paul Geason, Group Managing Director, Telstra Enterprise and Government, said Telstra was committed to providing localised infrastructure to cater for the needs of its customers around the country. “We continue to see tremendous growth in our cloud business with positive customer take up of the technology across a range of industries,” Geason said at the time. “With many organisations moving into the cloud, the feedback has been clear – the option to use local data centres is important either because their applications are sensitive to latency or they require data to be hosted within their state.”

    Look, I’m sure not all of Telstra’s cloud computing infrastructure went down, and that much of the services being provided to customers remained up and functioning fine. In this situation, Telstra would throw considerable resources at the issue to get it fixed as soon as possible.

    However, to enterprise IT customers — and Telstra has some big manufacturers and so on in its customer ranks for cloud computing — 24 hours is a very, very long outage for this kind of enterprise-grade cloud computing. Frankly, I would be very surprised if Telstra hadn’t broken some of its service level agreements here, and I wouldn’t be surprised to hear that there’s compensation in the works here and there.

    You cannot … you absolutely cannot, as an enterprise cloud computing player, let your services stay down for a whole day. That’s the kind of situation which encourages customers to start looking elsewhere for these kinds of services. These sorts of outages aren’t supposed to happen with a provider as large as Telstra — that’s why you go with Telstra in the first place.

    Image credits: Telstra

    Print Friendly


    You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.

    1. Posted 03/04/2013 at 3:05 pm | Permalink |

      Yep, that’s pretty bad.

    2. Frank
      Posted 03/04/2013 at 7:42 pm | Permalink |

      If the outage was caused by a file server going down then I would not describe Telstra’s network as cloud computing. If it was cloud computing then the data would have been spread between multiple machines with multiple redundancy same as Google and Amazon

    3. Daniel
      Posted 03/04/2013 at 9:51 pm | Permalink |

      That’s what you get for putting your eggs in the “cloud” basket.

      Presumably the storage devices that failed were EMC, as they have a reputation for taking down customers environments. I can recall 2 similar stories with EMC SAN failures in the past couple of years.

      • Craig
        Posted 04/04/2013 at 2:41 pm | Permalink |

        Why do you say EMC? I believe Telstra are now a NetApp shop

      • Robert
        Posted 08/04/2013 at 2:48 pm | Permalink |

        NO Daniel, actually it Was HDS….

    4. Phil from outer brisbane
      Posted 04/04/2013 at 9:11 am | Permalink |

      I really laugh at this stuff. Why does absolute crap like free to air television , which uses sophisticated SYNCRONOUS data networks rarely go down for as little as 5 minutes every 5 years . Yet important stuff like banking and now “the cloud” which is “the business” goes down for 24 hours or more !.

    5. Douglas
      Posted 04/04/2013 at 9:22 am | Permalink |

      Coogans 1, cloud nil ;)

    6. Zwan
      Posted 04/04/2013 at 11:59 am | Permalink |

      So telstra… Hows all the out sourcing going for you? Saving money yet?

    7. Steve Hodgkinson, Research Director IT Asia/Pacific, Ovum
      Posted 04/04/2013 at 12:05 pm | Permalink |

      Hmmm … s**t has always happened and will continue to do so I guess … as much as we would wish it otherwise. Cloud skeptics tend to jump on these outage stories with glee, but the organisations that are actually involved will take a more measured approach.

      For Telstra, this is a ‘rich learning experience’ … which will reinforce their commitment (and investment prioritization) to ensuring that the experience is not repeated. For the customer organisations affected, this will reinforce the need for appropriate risk management and business continuity strategies for their cloud sourced services. The net result from a strategic perspective is better for both parties … despite the short term pain.

      The reality is that cloud services adoption is all about trade-offs. Of course the best option is gold plated fully in-house world-class assets, applications and staff … but this is just not affordable or sustainable for many organisations … so cloud services have a valid place in the ICT mix. Occasional outages are just a fact of life, and don’t change this reality IMHO … unless of course they become a pattern and are evidence that the wrong cloud services provider has been chosen or cloud services is not an appropriate sourcing option for a particular workload or application.

      • Woolfe
        Posted 04/04/2013 at 2:07 pm | Permalink |

        Indeed, the concern I have regarding cloud, is that often the companies making the decisions don’t truly understand the potential risk. As with most outsourcing of services, it is driven by accountants looking at the dollars.

        I see no reason why the cloud isn’t a viable option. But you have to be aware that you are giving up an element of control and relying on a service provider. And the key thing with service providers, is they don’t really care about your business. So if you don’t build appropriate SLA’s and redundancies then you are shooting yourself in the foot. And most organisations I have been in, don’t build appropriate SLA’s.

        This is just an example of the risk. Like all things don’t throw the baby out with the bathwater, but take it as a warning, and make sure the cloud is appropriate to your core business.

        • Steve Hodgkinson, Research Director IT Asia/Pacific, Ovum
          Posted 04/04/2013 at 4:40 pm | Permalink |

          Hi Woolfe,

          Responding to your comment “the key thing with service providers, is they don’t really care about your business” …

          This is true of outsourcing and managed service deals because you require the service provider to provide dedicated services customized for your needs … hence if they don’t ‘care about your business’ enough the service will be poor.

          I actually think, however, that one of the value propositions of cloud services, as massive scale shared services, is that the service provider doesn’t actually need to (and shouldn’t) “care about your business” in the hand-crafted, personalized, way of traditional outsourcing or managed services.

          The whole deal is premised on the fact that the cloud service provider has constructed an industrialized facility that delivers a high quality standardized (but somewhat configurable) service offering. The provider should care a great deal about THEIR BUSINESS (i.e. about quality, reliability, sustainability, relevance, functionality etc. of the service offering) and of course should care about the collective needs and satisfaction of customers … but they are not in the business of trying to care for each customer individually in a traditional dedicated/customized service provider manner.

          We can’t have it both ways … the benefits of cloud services stem from their industrialization … so we need to learn how to be intelligent consumers of generic industrialized services in order to become happy with this model … and to apply it to appropriate use cases.

          If you buy dedicated/customized managed services then you need a trusted service provider that understands your individual needs and ‘cares’ about you … and you need to ‘work the relationship’. When things go badly wrong you phone the lawyers because you are locked in to a long term contract.

          If you buy cloud services then you need a service provider that offers high quality standardized shared service that leverages economies of scale in development and operations across a large customer base … and cares deeply about their operational performance metrics for all customers as a group … but may not really even know much about you as an individual customer at all. When things go wrong you rely on the ‘wrath of the crowd’ to discipline the provider … and if that fails to improve the situation then you may need to terminate the service and exercise your tested plan B or find a replacement service.

          This market is still maturing, and many so-called cloud services are really just an evolved form of managed services (in terms of both the capabilities of the providers and the buying behaviors of the customers) … so we are on a journey. More trials and tribulations are sure to follow …

          • Woolfe
            Posted 05/04/2013 at 1:21 pm | Permalink |

            Oh I agree with you completely. That is essentially what I meant. The Consumer of the Service needs to ensure that they understand what it is they are getting into. So not just the up front cost benefits, but also the potential pitfalls etc.
            And as well as that, as you say they need to work with and manage the relationship with the service provider.

            But if your business relies upon an element being available, then that needs to be understood and taken into account when looking at cloud based services. Because if said services are unable or unwilling to ensure your service(and by that I mean keeping it going, not simply paying rebates if it goes down) is always on, then it may be that the cloud is not appropriate for your business yet.
            As you say it is still maturing, so they are really targeting the majority 80% of business at the moment, the more complex 20% exceptional business will come as additional controls and guarantees get bolted on.

    8. mike zane-thomas
      Posted 04/04/2013 at 1:24 pm | Permalink |

      24 hours? Try 6 days. Rebuilds and verifications rendered the Hitachi platform unusable until the following Monday.

    9. James
      Posted 04/04/2013 at 3:19 pm | Permalink |

      In response to Steve’s comment, your words are too kind. For customers choosing to pay up with Telstra with all its marketing jingles and premium rates, there will be an expectation of premier service and reliability.

      From my one on one experience with Telstra Cloud services, there is a gap between what is being marketed and the real world practice.

      The negative comments are more than valid, customers are not dealing with a geek who just setup a cloud service with little experience.

      • Zwan
        Posted 04/04/2013 at 3:52 pm | Permalink |

        no the jumped out and completely source solutions out to vendors at great expense to get it up and running as quickly as possible without sitting down and thinking it out and choosing to skill I.T. locally.

        The premium is the speed in which the delivered their projects, not the quality of the service.

      • Steve Hodgkinson, Research Director IT Asia/Pacific, Ovum
        Posted 04/04/2013 at 3:55 pm | Permalink |

        Hi James, yes, fair enough. The customers concerned would seem to have legitimate reason to be very annoyed with Telstra if the reporting is accurate. I suppose I was talking more in generalities about what this means for cloud services as a way of sourcing ICT capabilities.

        The problem we have in Australia is that even the largest cloud service providers are still scrambling up the learning curve in terms of their ability to build/implement/operate large scale high reliability cloud services and to turn around their existing process, service, marketing and sales models. Occasional dropping of the ball is just a reality … hence …

        Caveat emptor … don’t forget the seven basics of cloud services procurement:

        (1) Contract – make sure it covers off your business risk exposures.
        (2) SLA – make sure it meets your business needs.
        (3) Information categorisation and management – make sure you understand what data can and cannot be in a cloud service and how these decisions are made.
        (4) Data prenup – make sure you know how to get your data back … with local periodic replication/backup … daily if needs be.
        (5) Tested Plan B – make sure you know what do do if the cloud service goes down … and have a tested business continuity strategy if needs be.
        (6) Staff training – make sure your people understand how to use the service, how to cope with an outage and what their obligations are in regard to security practices etc.
        (7) Quality certification – make sure the provider has the requisite quality & security certifications.

        Beyond that … given that there is an overall business value benefit/risk proposition in favor of cloud adoption … cross your fingers!

    10. Alan Lee
      Posted 06/04/2013 at 5:42 pm | Permalink |

      There will always be problems with every single vendor out there. Even them NASA guys have problems and I think their budget would be all of the providers put together and then some more.

      What becomes important and probably one of the biggest things to look at these days would be the amount of time it takes to recover services after an event.

      There was another provider the other day who had a storage issue. They seemed to take twice as long as Telstra to get their systems back up and running.

    11. Steve McCabe
      Posted 06/04/2013 at 6:01 pm | Permalink |

      Most cloud providers are like low cost airlines. You get what you paid for.

    12. Jim
      Posted 10/05/2013 at 1:47 pm | Permalink |

      They should use Vblocks…this would never happen on a Vblock!

  • Get our weekly newsletter

    All our stories, just one email a week.

    Email address:

    Follow us on social media

    Use your RSS reader to subscribe to our articles feed or to our comments feed.

  • Enterprise IT stories

    • Govt blows $14.4m on Windows XP, Server 2003 support nadella-1

      The Federal Government has paid Microsoft more than $14.4 million for custom support of the outdated Windows XP and Windows Server 2003 operating systems, in a costly move which further demonstrates the extreme cost of running operating systems which are no longer formally supported by their vendors.

    • [ad] Tatts Group optimises business performance with StruxureWare software schneider

      Tatts Group is one of Australia’s largest lottery and gaming companies, operating the majority of lotteries within Australia, as well as wagering, telephone betting and online transactions. Their online transactions make them one of the highest trafficked sites in the country.

    • Legacy health software lands SA Govt in court doctor

      In which the South Australian Government comes up with complex legal arguments as to why it should be able to continue to use a 1980’s software package.

    • Microsoft wants to win you back with Windows 10 windows-10

      The latest version of Microsoft’s Windows operating system will begin rolling out from Wednesday (July 29). And remarkably, Windows 10 will be offered as a free upgrade to those users who already have Windows 7 and 8.1 installed.

    • Qld Govt Depts have no disaster recovery plan brisvegas2

      Two sizable Queensland Government departments have no central disaster recovery plan, the state’s Auditor-General has found, despite the region’s ongoing struggles with extreme weather conditions that have previously knocked out telecommunications and data centre infrastructure.

    • ASD releases Windows 8 hardening guide windows-8-1

      The Australian Signals Directorate appears to have released a guide to hardening Microsoft’s Windows 8 operating system, three years after the software was released for use by corporate customers, and as Microsoft is slated to release its next upgrade, Windows 10.

    • ASG picks up $35m CIMIC IT services deal money

      Perth-headquartered IT services group ASG this week revealed it had picked up a deal worth at least $35 million over five years with CIMIC Group — the massive construction and contracting group previously known as Leighton Holdings.

  • News, Policy + Politics - Aug 4, 2015 16:12 - 10 Comments

    Turnbull revises history on NBN satellite demand

    More In Policy + Politics

    Enterprise IT, News - Aug 3, 2015 16:03 - 3 Comments

    Govt blows $14.4m on Windows XP, Server 2003 support

    More In Enterprise IT

    Industry, News - Aug 4, 2015 16:52 - 1 Comment

    Posse group picks up $5m in funding

    More In Industry

    Consumer Tech, News - Jul 29, 2015 17:14 - 11 Comments

    Telstra integrates Netflix, Stan, Presto into re-badged Roku box

    More In Consumer Tech