[ad] The service leader for Cloud is now in Australia. Secure, reliable cloud and managed hosting all backed by 24x7x365 Fanatical Support. Create your free account now.
Buy an Seagate Business Storage NAS for your chance to win a holiday
[ad] Purchase a selected Seagate Business Storage NAS to receive a $20 cash-back AND go into the draw to win a $1,000 Flight Centre voucher so you can holiday in the destination of your choice. T&Cs apply.
How mobile and social media affect your Customer Experience strategy
[ad] How will the adoption of mobile devices and social media affect your Customer Experience strategy? Are you reaching your organisation's customers through these touch points? Click here to download a whitepaper by Fifth Quadrant examining consumer and business attitudes to these new contact channels.
Great articles on other sites
- Turnbull to release NBN review next week
- Canberra blitzes states with NBN take-up rates
- War on whistleblowers from Abbott, Turnbull as ICJ case arrives
- Stockland tech revamp at centre of growth plans
- Clare warns of Gonski-like backflips on the NBN
- Victoria seeks early buy-in to avoid past disasters
- Vtalk bucks the China trend with plan for Aussie build
- Booksellers bristle at Amazon's arrival
- Australian customers upbeat on Dell going private
- FTTP NBN supporters lobby Turnbull
50 things top IT pros need to know
[ad] This 18 page TechRepublic whitepaper explores 10 things you should know to become an epic IT manager, 40 other essential tips to advance your IT career and practical guidance for starting an IT consulting business. Click here to access the whitepaper.
The new IT manager: Trends affecting IT in business
[ad] The tables have turned for IT managers. IT used to be able to dictate which computing assets would be used by employees and how they would be used. No longer. This free GigaOM Pro research paper (click here to download it) gives a solid, fact-based perspective on how IT consumerisation, mobile computing and cloud delivery trends are changing the paradigm.
Enterprise IT, Featured, News - Written by Renai LeMay on Monday, July 30, 2012 13:00 - 73 Comments
Disastrous patch cripples CommBank
news The Commonwealth Bank of Australia is currently reeling with internal chaos and some service delivery problems, following what appears to be a disastrous misapplication of an operating system patch to thousands of desktop PCs and hundreds of servers last week.
According to sources, on Thursday last week a patch was issued using Microsoft’s System Center Configuration Manager (SCCM) remote deployment tool. It appears as if the patch was intended to be distributed to a number of the bank’s desktop PCs only, but it was mistakenly applied to a much wider swathe of the bank’s desktop and server fleet than was intended.
In a statement last week, the bank didn’t provide any details of what it said was “a problem with an internal software upgrade”, and played down the issue. It noted that while the vast majority of its over 1,000 branches were offering full services and its ATM, and Internet and phone banking services were unaffacted, about 95 branches were only offering limited services, such as access to automatic teller machines.
“Our branch staff in each affected branch are available as usual to assist our customers with their enquiries,” the bank said in a statement issued on Friday. “Customers may experience some increased wait times in some of our branches and when calling our call centres. Our priority is on restoring all services as quickly as possible and we apologise for the inconvenience.”
However, internally at CommBank, the situation appears to be far more dramatic than it appears to be from an external customer point of view. Late last week, sources said that some 9,000 desktop PCs, hundreds of mid-range Windows servers (sources said as high as 490) and even iPads had been rendered unusable due to software corruption issues associated with the patch.
One of the bank’s IT services partners, HP, is believed to have allocated additional resources in an emergency effort to re-image the servers and desktop PCs from scratch with the bank’s standard operating environment and other platforms where appropriate, with the bank lodging a ‘P1′ highest priority incident notice with the company. Internally, some staff at HP have been told to throw every resource possible at the situation. CommBank’s own backup and restore teams are also believed to be throwing resources at the issue wholesale.
Over the past several days Delimiter has received a number of tips from CommBank staff and the customer’s partners with respect to the issue. “I work at Commonwealth Bank Place and in the last two hours several colleagues have had SOE patches pushed out which have totally killed the machines resulting in No Operating System found messages,” one unverified tip stated.
“Execs have received emails advised that all machines at [Commonwealth Bank Place] are to be logged out and removed from the network immediately,” wrote another. “Apparently a virus has been mentioned, from what I have seen it would perhaps be in the SOE update?” And a third comment was: “Heard the one about the CBA losing 20% of their laptops this afternoon after a mysterious system update rebooted machines without notice and into a system error screen?”
One industry source with knowledge of the situation said they had never seen a situation quite like it in Australia. The problem is believed to have affected around a quarter of the bank’s desktop PC machines.
Thousands of desktop PCs down, hundreds of mid-range Windows servers offline and even iPads being rendered inoperable, with an outsourcer throwing every available resource at the situation and even some bank branches unable to serve customers? Well, I wouldn’t say it’s the biggest IT disaster Australia has ever suffered (most would say Customs’ Integrated Cargo System catastrophe was the biggest), but in terms of one-off enterprise IT outages that interrupt normal business, it doesn’t get much worse than this. I would say CommBank will still be fixing this situation for months to come.
The damage to the desktop machines is bad enough. But it’s in the damage to the mid-range Windows servers that the real pain will be felt at the bank. With email usually stored remotely and backed up pretty rigorously, and most documents these days stored on network drives, it’s normally not that huge a deal to have a desktop PC fleet suffering problems (although to see 9,000-odd machines go down in one go really beggars belief). Even if data is lost, the desktop PCs can be re-imaged and providing basic services to staff pretty quickly — ideally using the same remote deployment tools which caused this headache in the first place.
But servers are a completely different beast altogether. Server admins know that most servers are pretty customised to fit the specific need they’re serving. That configuration usually isn’t something you can easily re-image from a base install package — and meanwhile, although the desktop PCs might be back up soon, the servers providing services across CommBank will still be down, helping to further cripple its operations. I really don’t envy the task of the admins at CommBank and HP for the next few months. I wouldn’t be surprised to find out that quite a few people had their annual leave cancelled as all hands will be needed to resolve these issues.
And all, or so it would appear, from one little operating system patch issues mistakenly to the wrong list of machines. Centralised software deployment and administration is fantastic, of course, but when it goes wrong, it really goes wrong. There will be more than a few red faces and angry meetings about this one before all is said and done. Let’s just hope the bank didn’t lose too much data along with the downtime. And I look forward to the inevitable internal investigation and report allocating blame for this little issue.
Latest Delimiter 2.0 articles (subscriber content)
|Politicians from Australia’s major parties need to stop issuing ludicrous blanket pardons for the intelligence community’s ongoing misdemeanours and start applying a basic modicum of transparency and accountability to this important national security function.|
|The independent pro-fibre National Broadband Network movement is doing a far better job of promoting Labor’s Fibre to the Premises-based NBN policy than Labor itself. When is Labor going to wake from its slumber and start supporting this scrappy but energetic grassroots network of activists?|
|Ziggy Switkowski's first substantial public appearance since being appointed NBN Co chief executive has starkly demonstrated just how different he is from his predecessor, Mike Quigley, and just how strictly he will adhere to the guidelines which his patron, Communications Minister Malcolm Turnbull, has set for him.|
|Australian technology companies have been virtually absent from the the nation’s public stockmarket over the past decade as the stigma of the dot com bust took its toll on investor confidence. But a clutch of new listings planned for the closing months of 2013 shows renewed interest in the sector and that local entrepreneurs are smelling money in the air once again.|
|NBN Co’s Strategic Review process gives the company an unmissable opportunity to re-evaluate the early decision to deploy its FTTP network primarily through Telstra’s underground ducts. The company and its new Coalition masters must now seriously consider deploying more fibre aerially on power poles in an effort to speed up its rollout substantially.|
|That moment which many Australian technologists fervently hoped for but never expected to see has come to pass: Simon Hackett has been appointed to the board of the National Broadband Network Company. But what questions should the Internode founder be asking NBN Co’s executive management team? Here’s five ideas to start with.|
|The rapid replacement of respected NBN Co chief operating officer Ralph Steffens with a Telstra executive who appears less experienced with fibre rollouts but better politically connected represents a key signal that NBN Co’s senior executive hiring process has now become completely politicised and is no longer independent from the Federal Government.|
Enterprise IT, News - Dec 6, 2013 12:50 - 0 Comments
More In Enterprise IT
- Payroll disaster: Queensland sues IBM
- End of an era: Oracle Australia’s ‘safe hands’ leaves
- Qld launches whole of government IaaS panel
- Defence finally allows staff iPhones, iPads
- NSW Govt refreshes ICT Advisory Panel
News, Telecommunications - Dec 6, 2013 11:54 - 36 Comments
More In Telecommunications
- NBN Co internal FTTN analysis: Turnbull refuses to retract inaccurate claim
- Defying the Senate: Turnbull to release NBN Review by end of 2013
- Senate to force Turnbull to publish NBN Review
- Get on with FTTN job, Quigley tells NBN Co
- Senate circus shows politics has no place in NBN
More In Industry
- Xbox One goes off with a bang … but will the PS4 launch eclipse it?
- It’s not just Freelancer: Aussie tech IPOs are back in general
- Freelancer’s IPO: A billion reasons to care
- Australian retailers online: Late to the party and much to do
- DesignCrowd picks up another $3m
Digital Rights, News - Dec 5, 2013 14:08 - 23 Comments
More In Digital Rights
- Global privacy group files formal ASD complaint
- Labor open to surveillance discussion
- Snowden an “American traitor”, says Australia’s Attorney-General
- ASD goes rogue with Aussie metadata
- It’s live: Delimiter publishes AGD FoI mirror