Great articles on other sites
- iiNet founder Michael Malone finally backs TPG Telecom takeover
- How and why the public sector must make friends with artificial intelligence
- Second anniversary of IT pricing report approaches - Computerworld
- Doctors spend 15 mins opening Fiona Stanley Hospital software
- What to expect from Abbott's national cyber security strategy
- ISPs need more time for data retention compliance
- TPG iiNet bid: major shareholders complain
- Qld emergency services payroll replacement on the rocks
- Victoria to wait another eight months for public IT dashboard
- Superloop CEO slams Australian govt tech policies
Renai's other site: Sci-fi + fantasy book news and reviews
- Kim Stanley Robinson’s new book Aurora is due in July
- What’s the future of “Grimdark” fantasy?
- An epic rant from Richard Morgan about nuance in writing
- Brandon Sanderson’s Firefight: Review
- Get into Jeff VanderMeer’s head as he writes the Southern Reach trilogy
- George R. R. Martin’s next book The Winds of Winter won’t arrive in 2015
- Alastair Reynolds’ Poseidon’s Wake launches 16 April
- Ann Leckie’s Ancillary Sword: Review
- Ann Leckie finishes Ancillary Mercy
- Hannu Rajaniemi’s The Fractal Prince: Review
Enterprise IT, Featured, News - Written by Renai LeMay on Monday, July 30, 2012 13:00 - 73 Comments
Disastrous patch cripples CommBank
news The Commonwealth Bank of Australia is currently reeling with internal chaos and some service delivery problems, following what appears to be a disastrous misapplication of an operating system patch to thousands of desktop PCs and hundreds of servers last week.
According to sources, on Thursday last week a patch was issued using Microsoft’s System Center Configuration Manager (SCCM) remote deployment tool. It appears as if the patch was intended to be distributed to a number of the bank’s desktop PCs only, but it was mistakenly applied to a much wider swathe of the bank’s desktop and server fleet than was intended.
In a statement last week, the bank didn’t provide any details of what it said was “a problem with an internal software upgrade”, and played down the issue. It noted that while the vast majority of its over 1,000 branches were offering full services and its ATM, and Internet and phone banking services were unaffacted, about 95 branches were only offering limited services, such as access to automatic teller machines.
“Our branch staff in each affected branch are available as usual to assist our customers with their enquiries,” the bank said in a statement issued on Friday. “Customers may experience some increased wait times in some of our branches and when calling our call centres. Our priority is on restoring all services as quickly as possible and we apologise for the inconvenience.”
However, internally at CommBank, the situation appears to be far more dramatic than it appears to be from an external customer point of view. Late last week, sources said that some 9,000 desktop PCs, hundreds of mid-range Windows servers (sources said as high as 490) and even iPads had been rendered unusable due to software corruption issues associated with the patch.
One of the bank’s IT services partners, HP, is believed to have allocated additional resources in an emergency effort to re-image the servers and desktop PCs from scratch with the bank’s standard operating environment and other platforms where appropriate, with the bank lodging a ‘P1′ highest priority incident notice with the company. Internally, some staff at HP have been told to throw every resource possible at the situation. CommBank’s own backup and restore teams are also believed to be throwing resources at the issue wholesale.
Over the past several days Delimiter has received a number of tips from CommBank staff and the customer’s partners with respect to the issue. “I work at Commonwealth Bank Place and in the last two hours several colleagues have had SOE patches pushed out which have totally killed the machines resulting in No Operating System found messages,” one unverified tip stated.
“Execs have received emails advised that all machines at [Commonwealth Bank Place] are to be logged out and removed from the network immediately,” wrote another. “Apparently a virus has been mentioned, from what I have seen it would perhaps be in the SOE update?” And a third comment was: “Heard the one about the CBA losing 20% of their laptops this afternoon after a mysterious system update rebooted machines without notice and into a system error screen?”
One industry source with knowledge of the situation said they had never seen a situation quite like it in Australia. The problem is believed to have affected around a quarter of the bank’s desktop PC machines.
Thousands of desktop PCs down, hundreds of mid-range Windows servers offline and even iPads being rendered inoperable, with an outsourcer throwing every available resource at the situation and even some bank branches unable to serve customers? Well, I wouldn’t say it’s the biggest IT disaster Australia has ever suffered (most would say Customs’ Integrated Cargo System catastrophe was the biggest), but in terms of one-off enterprise IT outages that interrupt normal business, it doesn’t get much worse than this. I would say CommBank will still be fixing this situation for months to come.
The damage to the desktop machines is bad enough. But it’s in the damage to the mid-range Windows servers that the real pain will be felt at the bank. With email usually stored remotely and backed up pretty rigorously, and most documents these days stored on network drives, it’s normally not that huge a deal to have a desktop PC fleet suffering problems (although to see 9,000-odd machines go down in one go really beggars belief). Even if data is lost, the desktop PCs can be re-imaged and providing basic services to staff pretty quickly — ideally using the same remote deployment tools which caused this headache in the first place.
But servers are a completely different beast altogether. Server admins know that most servers are pretty customised to fit the specific need they’re serving. That configuration usually isn’t something you can easily re-image from a base install package — and meanwhile, although the desktop PCs might be back up soon, the servers providing services across CommBank will still be down, helping to further cripple its operations. I really don’t envy the task of the admins at CommBank and HP for the next few months. I wouldn’t be surprised to find out that quite a few people had their annual leave cancelled as all hands will be needed to resolve these issues.
And all, or so it would appear, from one little operating system patch issues mistakenly to the wrong list of machines. Centralised software deployment and administration is fantastic, of course, but when it goes wrong, it really goes wrong. There will be more than a few red faces and angry meetings about this one before all is said and done. Let’s just hope the bank didn’t lose too much data along with the downtime. And I look forward to the inevitable internal investigation and report allocating blame for this little issue.
Blog, Policy + Politics - Jul 31, 2015 12:43 - 0 Comments
More In Policy + Politics
- Four months later, data retention funding model still incomplete
- Less talk, more action: Entrepreneur tells ‘Labor for Innovation’
- Bronny Copter is here to save us from Bishop’s Choppergate
- 7:30 exposes Aussie Hacking Team industry
- Hypocrisy? Fletcher pushs tech exports to China while TSSR bill looms
Blog, Enterprise IT - Jul 31, 2015 14:16 - 1 Comment
More In Enterprise IT
- Microsoft wants to win you back with Windows 10
- Qld Govt Depts have no disaster recovery plan
- ASD releases Windows 8 hardening guide
- ASG picks up $35m CIMIC IT services deal
- Datacom completes mammoth Health ICT takeover
Industry, News - Jul 28, 2015 12:37 - 0 Comments
More In Industry
- iiNet shareholders vote ‘yes’ for TPG buyout
- iiNet chairman “proud” as TPG sell-out looms
- Kotaku alleges abuse, gross staff neglect at retailer EB Games
- Aussie software firm Marketplacer grabs $10m
- Expert360 pulls in $4.1m for consultancy 2.0
Consumer Tech, News - Jul 29, 2015 17:14 - 11 Comments
More In Consumer Tech
- Older Australians embracing video games
- Gasp … Qld will fuel electric vehicle charging stations with solar
- Oops … Tesla enthusiast charges car on Qld windfarm
- Netflix Australia: Review
- RAC builds electric vehicle highway in WA