[ad] The service leader for Cloud is now in Australia. Secure, reliable cloud and managed hosting all backed by 24x7x365 Fanatical Support. Create your free account now.
Buy an Seagate Business Storage NAS for your chance to win a holiday
[ad] Purchase a selected Seagate Business Storage NAS to receive a $20 cash-back AND go into the draw to win a $1,000 Flight Centre voucher so you can holiday in the destination of your choice. T&Cs apply.
Great articles on other sites
- Xbox One smashes sales records
- Tech leaders call for speed, ubiquity in NBN rollout
- AIIA urges Hockey to tackle taxes
- IBM accuses Qld govt of trying to ‘rewrite history’
- Newlease undergoes reverse takeover to score ASX listing
- Australia Post loses battle | The Australian
- Start-ups leap at Telstra's accelerator
- Labor won't hand over NBN advice to Turnbull
- Adelaide Uni on hiring blitz for tech transformation
- Human Services to cut 56 IT jobs
How mobile and social media affect your Customer Experience strategy
[ad] How will the adoption of mobile devices and social media affect your Customer Experience strategy? Are you reaching your organisation's customers through these touch points? Click here to download a whitepaper by Fifth Quadrant examining consumer and business attitudes to these new contact channels.
50 things top IT pros need to know
[ad] This 18 page TechRepublic whitepaper explores 10 things you should know to become an epic IT manager, 40 other essential tips to advance your IT career and practical guidance for starting an IT consulting business. Click here to access the whitepaper.
Enterprise IT, Featured, News - Written by Renai LeMay on Thursday, August 2, 2012 15:44 - 12 Comments
CBA outage took down CommSee
news The Commonwealth Bank’s wide-ranging outage also took down its customer relationship management platform CommSee, one of its main unions has revealed, in a move which further illustrates how extensive the technology-related problems suffered by the bank over the past week truly have been.
On Thursday last week, according to sources, a patch was issued using Microsoft’s System Center Configuration Manager (SCCM) remote deployment tool within CommBank. It appears as if the patch was intended to be distributed to a small number of the bank’s desktop PCs as a disaster recovery exercise only, but it was mistakenly applied to a much wider swathe of the bank’s desktop and server fleet than was intended. Late last week, sources said that some 9,000 desktop PCs, hundreds of mid-range Windows servers (sources said as high as 490) and even iPads had been rendered unusable due to software corruption issues associated with the patch, with HP (one of the bank’s IT outsourcing partners) and its internal team scrambling to restore systems.
The issues had not previously been reported to have effected any of the bank’s top-level technology systems. However, in a statement issued this week, the Financial Services Union revealed the crash also took the bank’s Commsee customer relationship management system offline. CommSee is a critical system for CommBank. Developed in the early years after 2000 and deployed through the middle years of the decade, the system enables the bank’s staff to get a single view of each customers’ information, drawn from various internal banking resources.
“While the efforts of staff throughout the bank meant that most customers would have been oblivious to the problems, the bank’s reliance on the system meant that most staff were working without access to critical information and facilities,” the FSU wrote. “It was effectively impossible to complete much of the work.”
“The clean-up and backlog caused by the crash has meant that it would have been impossible for many staff to complete planned work for up to five days and it will now be difficult to catch up over the coming weeks. On this basis FSU believes that it would be unreasonable to hold staff accountable for targets set for July, where between three and five of the last working days of the month were lost and August, where an unclear number of days will be spent catching up on July’s unfinished work.”
The FSU noted in its statement that it had met with CommBank yesterday, and had called on the bank to suspend employee targets for July and August in the wake of what it called the “disastrous” crash. In response to its request, the FSU wrote, representatives of the bank acknowledged the problems caused by the system crash and said that an announcement would be made soon about targets. The bank said, according to the FSU, that sales meetings should not have taken place on Monday 30 July.
The FSU said it had been informed that the broader outage had occurred because of the actions of “an outsourced provider”. Delimiter believes that the rogue SCCM deployment originated within a HP team in New Zealand, but neither CommBank nor HP has publicly confirmed that allegation. A HP spokesperson declined to comment on the issue today, while a Commonwealth Bank spokesperson has not yet returned a call requesting further comment this afternoon.
In a statement last week, the bank didn’t provide any details of what it said was “a problem with an internal software upgrade”, and played down the issue. It noted that while the vast majority of its over 1,000 branches were offering full services and its ATM, and Internet and phone banking services were unaffacted, about 95 branches were only offering limited services, such as access to automatic teller machines.
“Our branch staff in each affected branch are available as usual to assist our customers with their enquiries,” the bank said in a statement issued on Friday. “Customers may experience some increased wait times in some of our branches and when calling our call centres. Our priority is on restoring all services as quickly as possible and we apologise for the inconvenience.”
“It is unclear whether the problems would have been avoided if the work had remained in-house but the problem underpins the FSU criticism of outsourcing where the bank loses direct control over end to end processes,” the union wrote, pointing out that CBA had recently announced its IT help desk facilities at Sydney Olympic Park would be outsourced to HP, resulting in 50 CBA jobs being lost.
HP, is believed to have allocated additional resources in an emergency effort to re-image the servers and desktop PCs from scratch with the bank’s standard operating environment and other platforms where appropriate, with the bank lodging a ‘P1′ highest priority incident notice with the company. Internally, some staff at HP were told to throw every resource possible at the situation. CommBank’s own backup and restore teams were also believed to be throwing resources at the issue wholesale last week and over the past few days.
With the union statement issued this week and other bits and pieces of information which we have received, we’ve now got several new pieces of information with respect to CommBank’s outage last week (which is still affecting the bank’s operations to some degree).
Firstly, it appears pretty clear now that CommBank’s issues originated within HP, and likely within one of the company’s New Zealand teams. I don’t think the FSU’s statement linking this particular issue to the issue of IT outsourcing in general is that legitimate (this sort of human error could just have easily have occurred if CommBank’s IT operations were completely insourced), but it does seem clear that this looks like a stuffup on the part of a major CommBank IT services provider. There will be questions raised about to what extent governance procedures were in place both within that provider as well as within CommBank itself, to oversee the services provided.
Secondly, we have the new and somewhat disturbing information that not only were thousands of the bank’s PCs and servers taken down due to the outage, but that one of its main critical systems was also affected. For CommSee to be taken down in CommBank is no laughing matter. This is a “heads will roll” kind of situation. CommSee is the kind of system which thousands upon thousands of CommBank staff rely on daily to get basic stuff done. As the union says, if it doesn’t work, the bank doesn’t work, and that’s a huge issue.
What all of this adds up to is the sort of situation which will likely have had CommBank’s chief executive Ian Narev and its chief information officer Michael Harte stampeding around its head office in Sydney’s Commonwealth Bank Place with looks of burning fury writ large upon their faces. This sort of thing isn’t supposed to happen at CommBank anymore — with all of the huge improvements CommBank has made to its internal IT over the past decade, this is one bank which is supposed to be beyond this kind of outage.
To think that a simple configuration mistake by one or a small handful of staff at an outsourcer could bring down so many critical pieces of IT infrastructure at one of Australia’s largest IT shops is just staggering. It starkly illustrates that for all of their advances over the past decade in terms of reliability, capability and service assurance, Australia’s banks really do have a long way to go to get their IT systems to the state of stability which the next few decades will demand of them.
Because if this kind of thing could happen to CommBank, which in almost every way is the technology leader in Australia’s financial services sector, then it could also happen to anyone else — at almost any time.
For me personally, this issue is also a reminder that as much as the global technology industry believes itself to be mature in many areas, the truth is that it’s not. The truth is that we are still right at the dawn of humanity’s understanding of technology in general, and how to keep technology working all the time, come what may. Today may be a good time to reflect on the fact that we’ve only had modern computer-based systems as we know them for little over half a century. No doubt it will take another half a century or more until they become stable enough to be truly described as “reliable”.
In 2012, you can plough billions of dollars into technology infrastructure if you so desire. But that doesn’t mean that a human can’t take much of it down with the accidental flick of a switch.
Latest Delimiter 2.0 articles (subscriber content)
|Politicians from Australia’s major parties need to stop issuing ludicrous blanket pardons for the intelligence community’s ongoing misdemeanours and start applying a basic modicum of transparency and accountability to this important national security function.|
|The independent pro-fibre National Broadband Network movement is doing a far better job of promoting Labor’s Fibre to the Premises-based NBN policy than Labor itself. When is Labor going to wake from its slumber and start supporting this scrappy but energetic grassroots network of activists?|
|Ziggy Switkowski's first substantial public appearance since being appointed NBN Co chief executive has starkly demonstrated just how different he is from his predecessor, Mike Quigley, and just how strictly he will adhere to the guidelines which his patron, Communications Minister Malcolm Turnbull, has set for him.|
|Australian technology companies have been virtually absent from the the nation’s public stockmarket over the past decade as the stigma of the dot com bust took its toll on investor confidence. But a clutch of new listings planned for the closing months of 2013 shows renewed interest in the sector and that local entrepreneurs are smelling money in the air once again.|
|NBN Co’s Strategic Review process gives the company an unmissable opportunity to re-evaluate the early decision to deploy its FTTP network primarily through Telstra’s underground ducts. The company and its new Coalition masters must now seriously consider deploying more fibre aerially on power poles in an effort to speed up its rollout substantially.|
|That moment which many Australian technologists fervently hoped for but never expected to see has come to pass: Simon Hackett has been appointed to the board of the National Broadband Network Company. But what questions should the Internode founder be asking NBN Co’s executive management team? Here’s five ideas to start with.|
|The rapid replacement of respected NBN Co chief operating officer Ralph Steffens with a Telstra executive who appears less experienced with fibre rollouts but better politically connected represents a key signal that NBN Co’s senior executive hiring process has now become completely politicised and is no longer independent from the Federal Government.|
Enterprise IT, News - Dec 10, 2013 17:23 - 0 Comments
More In Enterprise IT
- David Boyle appointed NAB CIO
- Qld payroll lawsuit ‘rewriting history’, says IBM
- Harbour City Ferries goes Microsoft across the board
- Payroll disaster: Queensland sues IBM
- End of an era: Oracle Australia’s ‘safe hands’ leaves
News, Telecommunications - Dec 10, 2013 18:16 - 0 Comments
More In Telecommunications
- Telstra 4G trials hit 300Mbps
- “Captain of the Titanic”: Turnbull mocks Quigley’s NBN tenure
- NBN Co still has 1Gbps on way
- Delimiter appeals Turnbull Blue Book censorship
- Final closure: TPG buys AAPT for $450m
Blog, Industry, Startups - Dec 10, 2013 10:19 - 0 Comments
More In Industry
- Telstra shares millions with Box
- The Australian IT sector needs a stronger voice
- Xbox One goes off with a bang … but will the PS4 launch eclipse it?
- It’s not just Freelancer: Aussie tech IPOs are back in general
- Freelancer’s IPO: A billion reasons to care
Digital Rights, News - Dec 10, 2013 13:05 - 2 Comments
More In Digital Rights
- Labor, Coalition reject Intelligence committee reformation
- Screwed: Australian PS4, Xbox One lack basic functionality
- Censored: Appeal for AG’s Blue Book fails
- Senate to force TPP publication
- Global privacy group files formal ASD complaint