• Great articles on other sites
  • RSS Great articles on other sites

  • Enterprise IT, Featured, News - Written by on Thursday, August 2, 2012 15:44 - 12 Comments

    CBA outage took down CommSee

    news The Commonwealth Bank’s wide-ranging outage also took down its customer relationship management platform CommSee, one of its main unions has revealed, in a move which further illustrates how extensive the technology-related problems suffered by the bank over the past week truly have been.

    On Thursday last week, according to sources, a patch was issued using Microsoft’s System Center Configuration Manager (SCCM) remote deployment tool within CommBank. It appears as if the patch was intended to be distributed to a small number of the bank’s desktop PCs as a disaster recovery exercise only, but it was mistakenly applied to a much wider swathe of the bank’s desktop and server fleet than was intended. Late last week, sources said that some 9,000 desktop PCs, hundreds of mid-range Windows servers (sources said as high as 490) and even iPads had been rendered unusable due to software corruption issues associated with the patch, with HP (one of the bank’s IT outsourcing partners) and its internal team scrambling to restore systems.

    The issues had not previously been reported to have effected any of the bank’s top-level technology systems. However, in a statement issued this week, the Financial Services Union revealed the crash also took the bank’s Commsee customer relationship management system offline. CommSee is a critical system for CommBank. Developed in the early years after 2000 and deployed through the middle years of the decade, the system enables the bank’s staff to get a single view of each customers’ information, drawn from various internal banking resources.

    “While the efforts of staff throughout the bank meant that most customers would have been oblivious to the problems, the bank’s reliance on the system meant that most staff were working without access to critical information and facilities,” the FSU wrote. “It was effectively impossible to complete much of the work.”

    “The clean-up and backlog caused by the crash has meant that it would have been impossible for many staff to complete planned work for up to five days and it will now be difficult to catch up over the coming weeks. On this basis FSU believes that it would be unreasonable to hold staff accountable for targets set for July, where between three and five of the last working days of the month were lost and August, where an unclear number of days will be spent catching up on July’s unfinished work.”

    The FSU noted in its statement that it had met with CommBank yesterday, and had called on the bank to suspend employee targets for July and August in the wake of what it called the “disastrous” crash. In response to its request, the FSU wrote, representatives of the bank acknowledged the problems caused by the system crash and said that an announcement would be made soon about targets. The bank said, according to the FSU, that sales meetings should not have taken place on Monday 30 July.

    The FSU said it had been informed that the broader outage had occurred because of the actions of “an outsourced provider”. Delimiter believes that the rogue SCCM deployment originated within a HP team in New Zealand, but neither CommBank nor HP has publicly confirmed that allegation. A HP spokesperson declined to comment on the issue today, while a Commonwealth Bank spokesperson has not yet returned a call requesting further comment this afternoon.

    In a statement last week, the bank didn’t provide any details of what it said was “a problem with an internal software upgrade”, and played down the issue. It noted that while the vast majority of its over 1,000 branches were offering full services and its ATM, and Internet and phone banking services were unaffacted, about 95 branches were only offering limited services, such as access to automatic teller machines.

    “Our branch staff in each affected branch are available as usual to assist our customers with their enquiries,” the bank said in a statement issued on Friday. “Customers may experience some increased wait times in some of our branches and when calling our call centres. Our priority is on restoring all services as quickly as possible and we apologise for the inconvenience.”

    “It is unclear whether the problems would have been avoided if the work had remained in-house but the problem underpins the FSU criticism of outsourcing where the bank loses direct control over end to end processes,” the union wrote, pointing out that CBA had recently announced its IT help desk facilities at Sydney Olympic Park would be outsourced to HP, resulting in 50 CBA jobs being lost.

    HP, is believed to have allocated additional resources in an emergency effort to re-image the servers and desktop PCs from scratch with the bank’s standard operating environment and other platforms where appropriate, with the bank lodging a ‘P1′ highest priority incident notice with the company. Internally, some staff at HP were told to throw every resource possible at the situation. CommBank’s own backup and restore teams were also believed to be throwing resources at the issue wholesale last week and over the past few days.

    With the union statement issued this week and other bits and pieces of information which we have received, we’ve now got several new pieces of information with respect to CommBank’s outage last week (which is still affecting the bank’s operations to some degree).

    Firstly, it appears pretty clear now that CommBank’s issues originated within HP, and likely within one of the company’s New Zealand teams. I don’t think the FSU’s statement linking this particular issue to the issue of IT outsourcing in general is that legitimate (this sort of human error could just have easily have occurred if CommBank’s IT operations were completely insourced), but it does seem clear that this looks like a stuffup on the part of a major CommBank IT services provider. There will be questions raised about to what extent governance procedures were in place both within that provider as well as within CommBank itself, to oversee the services provided.

    Secondly, we have the new and somewhat disturbing information that not only were thousands of the bank’s PCs and servers taken down due to the outage, but that one of its main critical systems was also affected. For CommSee to be taken down in CommBank is no laughing matter. This is a “heads will roll” kind of situation. CommSee is the kind of system which thousands upon thousands of CommBank staff rely on daily to get basic stuff done. As the union says, if it doesn’t work, the bank doesn’t work, and that’s a huge issue.

    What all of this adds up to is the sort of situation which will likely have had CommBank’s chief executive Ian Narev and its chief information officer Michael Harte stampeding around its head office in Sydney’s Commonwealth Bank Place with looks of burning fury writ large upon their faces. This sort of thing isn’t supposed to happen at CommBank anymore — with all of the huge improvements CommBank has made to its internal IT over the past decade, this is one bank which is supposed to be beyond this kind of outage.

    To think that a simple configuration mistake by one or a small handful of staff at an outsourcer could bring down so many critical pieces of IT infrastructure at one of Australia’s largest IT shops is just staggering. It starkly illustrates that for all of their advances over the past decade in terms of reliability, capability and service assurance, Australia’s banks really do have a long way to go to get their IT systems to the state of stability which the next few decades will demand of them.

    Because if this kind of thing could happen to CommBank, which in almost every way is the technology leader in Australia’s financial services sector, then it could also happen to anyone else — at almost any time.

    For me personally, this issue is also a reminder that as much as the global technology industry believes itself to be mature in many areas, the truth is that it’s not. The truth is that we are still right at the dawn of humanity’s understanding of technology in general, and how to keep technology working all the time, come what may. Today may be a good time to reflect on the fact that we’ve only had modern computer-based systems as we know them for little over half a century. No doubt it will take another half a century or more until they become stable enough to be truly described as “reliable”.

    In 2012, you can plough billions of dollars into technology infrastructure if you so desire. But that doesn’t mean that a human can’t take much of it down with the accidental flick of a switch.

    Image credit: megawatts86, Creative Commons

    submit to reddit


    You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

    1. Adam Nelson
      Posted 02/08/2012 at 8:19 pm | Permalink | Reply

      It sounds like someone designed SCCM corrections incorrectly and placed most of their infrastucture inside a collection. Which would of been a forced pushed an advertisement out to all PCs and services inside the collection

      • Adam Nelson
        Posted 02/08/2012 at 8:20 pm | Permalink | Reply

        However not an expert on this. So feel free to correct me

    2. Tony Healy
      Posted 02/08/2012 at 9:30 pm | Permalink | Reply

      Actually, it’s quite legitimate to blame outsourcing. As a process, outsourcing generally involves replacing experienced bank staff with cheap plug-in staff at the outsourcer.

      Typically some MBA has gone along and said: “this job just needs an SCCM operator. We can get them for $35 an hour.” So the outsourcer puts out an ad that’s more concerned about what version of SCCM the candidate has used.

      Meanwhile the guys who used to do the job in the bank probably had safe procedures that always worked, knew which branches had dodgy PCs, which days were bad for updates (because managers were at golf) and so on. The MBAs lacked the expertise to perceive the real value the old staff provided.

      Almost all the strange bank problems over the past two years have involved outsourcers.

    3. Harry
      Posted 02/08/2012 at 10:38 pm | Permalink | Reply

      Some history for the young ones


    4. John
      Posted 02/08/2012 at 11:23 pm | Permalink | Reply

      Word has it (from an extremely well placed source) that HP has flown its most senior executives, including CEO Meg Whitman in from the US for damage control. One can only assume that CBA may be looking for blood… and that blood could be cancellation of the impending $700mill HP contract extension!

      • Mike
        Posted 03/08/2012 at 8:42 am | Permalink | Reply

        Not suprised. I worked for an IT company that was providing the CBA with a contracted service worth over $50 million. 1 year into the 5 year contract the CommBank CIO asked our CEO to give him a good reason why he shouldn’t cancel the contract there and then for poor delivery of the service. Needless to say our CEO kicked a lot of butts and salavaged the contract and relationship with the CBA.

        It just goes to show the CBA are big enough and ugly enough to demand the best. And good on them, I like to see the big outsourcers on their knees crying for mercy, from time to time.

    5. faster and cheaper
      Posted 03/08/2012 at 1:08 am | Permalink | Reply

      When will companies learn that outsourcing always costs in the end.

    6. bob
      Posted 03/08/2012 at 1:12 am | Permalink | Reply


    7. J. Mathers
      Posted 03/08/2012 at 5:39 am | Permalink | Reply

      I’ve been an SCCM Admin for many years. As a direct hire employee, as well as a third party provider. I’ve seen huge targeting mistakes like this one and it has often been by direct hires. (Also keep in mind that when a company outsources, they outsourcer often hires employees from the outsourcing company..)

      This is not just a question of outsourcing. It’s a question of losing focus on the task at hand AND not having enough checks and balances in place to mitigate these types of risks.

      The patch package could have been written with logic to also limit the target audience rather than to rely solely on collection criteria. As well, the Servers and Workstations should have been on different sites entirely.

      • socrates
        Posted 03/08/2012 at 3:57 pm | Permalink | Reply

        Those are some good comments. There’s another interesting aspect to this report.

        It seems to have become standard practice for unions to publicly trash any company they have been in dispute with. It is surprising that unions with a lot of members at a company appear blind to the fact that such attacks will affect the reputation of the company (as intended), which will reduce the amount of business they do, which will reduce the number of employees they have.

        One of the best (worst) examples of this union foot-shooting is Qantas, which has been under sustained attack from some unions for years. The resulting loss of public confidence and consequent reduction in the size of the company has caused a significant loss of jobs.

        • Clytie Siddall
          Posted 05/08/2012 at 3:22 pm | Permalink | Reply

          QANTAS have shot _themselves_ in the foot, repeatedly and deliberately, since acquiring Joyce as CEO. They have refused to work with their own people, preferring to strand many thousands of their own customers. They have given away lucrative overseas routes, made grandiose “plans” which don’t materialize, got rid of many of their most experienced pilots and made really bad choices in fleet purchases. See Crikey’s “Plane Talking” aviation blog for more details.

    8. Jack's Bleeding Heart
      Posted 07/08/2012 at 11:55 am | Permalink | Reply

      I wonder how much the hiring freeze, redundancies and culling of contingent workforce in the past six months has impacted HP.

      From personal experience and from talking to others who used to work there, it seems that experienced local engineers / sysadmins have been walked. This puts extra pressure on the poor FTE folk left. I’ve spoken to admins that regularly put in 60-70 hour weeks trying to keep their heads above water, so to speak. Add to that, the loss of accumulated knowledge and having admins thrust into unfamiliar environments…. I’d suggest it is a miracle this kind of thing hasn’t happened sooner (and talk on the inside is that similar, yet obviously smaller, stuff does happen).

      It’s hard to sympathise with big companies who cut their IT staff to save a dollar. If your business is absolutely reliant on your IT systems, why would you trust it in the hands of big global companies like HP?

    Leave a Comment


  • Get our 'Best of the Week' newsletter on Fridays

    Just the most important stories, one email a week.

    Email address:

    Follow us on social media

    Use your RSS reader to subscribe to our articles feed or to our comments feed.

  • Most Popular Content

  • Enterprise IT stories

    • Super funds close to dumping $250m IT revamp facepalm2

      If you have even a skin deep awareness of the structure of Australia’s superannuation industry, you’ll be aware that much of the underlying infrastructure used by many of the nation’s major funds is provided by a centralised group, Superpartners. One of the group’s main projects in recent years has been to dramatically update and modernise its IT platform — its version of a core banking platform overhaul. Unfortunately, the $250 million project has not precisely been going well.

    • Qld’s Grant joins analyst firm IBRS peter-grant

      This week it emerged that Peter Grant, the two-time former Queensland Whole of Government CIO (pictured), has joined well-regarded analyst firm Intelligent Business Research Services (IBRS). We’ve long had a high regard for IBRS, and so it’s fantastic to see such an experienced executive join its ranks.

    • Westpac dumps desk phones for Samsung Android mobiles samsung-galaxy-ace-3

      The era of troublesome desk phones tied to physical locations is gradually coming to an end in many workplaces, with mobile phones becoming increasingly popular as organisations’ main method of voice telecommunications. But some groups are more advanced than others when it comes to adoption of the trend. One of those is Westpac.

    • Ministers’ cloud approval lasted just a year reverse

      Remember how twelve months ago, the Federal Government released a new cloud computing security and privacy directive which required departments and agencies to explicitly acquire the approval of the Attorney-General and the relevant portfolio minister before government data containing private information could be stored in offshore facilities? Remember how the policy was strongly criticised by Microsoft, Government CIOs and Delimiter? Well, it looks like the policy is about to be reversed.

    • WA Govt can’t fund school IT upgrades oops key

      In news from The Department of Disturbing Facts, iTNews revealed late last week that Western Australia’s Department of Education has run out of money halfway through the deployment of new fundamental IT infrastructure to the state’s schools.

    • Turnbull outlines Govt ICT vision turnbull-5

      Communications Minister Malcolm Turnbull has published an extensive article arguing that the Federal Government needed to do a better job of connecting with Australians via digital channels and that public sector IT projects needn’t cost the huge amounts that some have in the past.

    • NZ Govt pushes hard into cloud zealand

      New Zealand’s national Government announced a whole of government contract this morning for what it terms ‘Office Productivity as a Service’ services. This includes email and calendaring services, as well as file-sharing, mobility, instant messaging and collaboration services. The contract complements two existing contracts — Desktop as a Service and Enterprise Content Management as a Service.

    • CommBank reveals Harte’s replacement whiteing

      The Commonwealth Bank of Australia has promoted an internal executive who joined the bank in September after a lengthy career at petroleum giant VP and IT services group Accenture to replace its outgoing chief information officer Michael Harte, who announced in early May that he would leave the bank.

    • Jeff Smith quits Suncorp for IBM jeffsmith4

      Second-tier Australian bank and financial services group Suncorp today announced that its long-serving top technology executive Jeff Smith would leave to take up a senior role with IBM in the United States, in an announcement which marks the end of an era for the nation’s banking IT sector.

    • Small business missing the mobile, social, cloud revolution iphone-stock

      Most companies that live and breathe the online revolution are not tech startups, but smart smaller firms that use online tools to run their core business better: to cut costs, reach customers and suppliers, innovate and get more control. Many others, however, are falling behind, according to a new Grattan Institute discussion paper.

  • Blog, Enterprise IT - Jul 5, 2014 13:53 - 0 Comments

    Super funds close to dumping $250m IT revamp

    More In Enterprise IT

    Blog, Telecommunications - Jul 5, 2014 12:12 - 0 Comments

    What should the ACCC’s role be in guiding infrastructure spending?

    More In Telecommunications

    Analysis, Industry, Internet - Jun 23, 2014 10:33 - 0 Comments

    ‘Google Schmoogle’ – how Yellow Pages got it so wrong

    More In Industry

    Blog, Digital Rights - Jun 30, 2014 22:24 - 0 Comments

    Will Netflix launch in Australia, or not?

    More In Digital Rights