news The Federal Department of Health and Aging has accused technology giant IBM of causing a “catastrophic failure” in its IT systems stemming from an update to its storage environment that took down a number of services for a period of time this week.
The department’s secretary Jane Halton sent a staff-wide email (first reported by Crikey) to the department’s employees yesterday acknowledging the issue. “As you are no doubt painfully aware, changes made by IBM to the department’s IT storage environment over the weekend have resulted in a catastrophic failure in our IT systems that has significantly affected everyone,” she wrote.
“I spoke to the managing director of IBM yesterday and this morning and am writing to him today to express my extreme dissatisfaction with this completely unacceptable event on top of the delay to the implementation of our new desktop hard and software systems. This letter follows recent meetings that I, Paul Madden, the chief information and knowledge officer and deputy secretary Andrew Stuart have had with senior IBM executives on the desktop delay.”
IBM issued a statement this afternoon noting that “a regular update” to the department’s storage environment had resulted in IT system performance issues on Monday. “IBM and DoHA worked closely together and this issue is now resolved,” the company said. “IBM is committed to supporting DoHA to deliver high quality services to its customers.”
Halton said she acknowledged the department’s communication with staff on the issue had been inadequate and had asked that communication processes be reviewed to ensure staff are “more appropriately advised of any future IT problems and attempts to fix them in future” – “hoping and planning, of course, that we do not have another issue like this again”.
“Our own departmental IT staff worked very hard with IBM on resolving the issue yesterday, overnight and continue today to attempt to fix the problem and restore the system as soon as possible,” said Halton. “I want to thank our IT people for their efforts and energy, and acknowledge the unfortunate disruption to all staff which we are working as hard as possible to resolve.”
“Once the system is restored a joint review will be undertaken to ensure a similar failure does not recur for staff and any potential financial penalties are applied.”
IBM has had a long-running IT outsourcing relationship with the department, dating back to 1999. The pair recently renewed their vows in the closing months of 2010, inking a deal worth some $109 million over four years which will see IBM continue to provide a range of services to the department, including mainframe, mid-range, storage, help desk and end user computing services. In addition, Big Blue said at the time that it would also provide new security compliance solutions and will conduct mainframe and storage upgrades.
It will also include the rollout of a desktop virtualisation platform to all of DOHA’s 4,500-odd staff.
Things just never seem to go well for poor IBM when it comes to the Department of Health and Aging. The pair’s relationship was under fire late last year and early in 2011 due to what some saw as a lack of proper tendering processes in their new contract. And now this storage upgrade – which appears to be part of the new deal – has suffered what appears to be a substantial hiccup.
One also wonders just what Halton was referring to in her email when she wrote of other delays in IBM’s implementation of desktop systems in particular. Desktop virtualisation projects are notoriously difficult to get right. It would be interesting to know what kind of problems are happening in that arena in Health. This is what we wrote when the desktop virtualisation deal was signed last year:
Perhaps the most interesting part of the new agreement will be the included rollout of a desktop virtualisation platform to all of DOHA’s 4,500-odd staff.
“Commencing in the first half of 2011, IBM will deploy a virtualised thin client desktop solution across the entire department, enabling the movement of staff and workloads at a moment’s notice, whilst lowering technology support costs and increasing network performance,” IBM’s statement said. It is not immediately clear what technology the virtual desktops will be based on, but one possibility is VMware’s View platform, which has recently won a number of small customers around Australia.
The thing about desktop virtualisation projects, of course, is that datacentre-related outages, which is what appears to have happened to DoHA on the weekend, also have the potential to take down desktop systems across the whole organisation.
We must also, however, give a degree of credit to IBM and Health’s own IT staff for the speed at which it resolved the outage. “Catastrophic” is a term usually applied to outages which last for a week or more … you know, like they have at Victorian shared services agency CenITex. A few days’ outage is more of a substantial inconvenience. The speed of resolution especially appears surprising given the involvement in storage systems in the outage.
In this writer’s experience, any time a storage system suffers serious problems in a corporate IT environment, the organisation’s IT staff will be in for a hairy few weeks indeed.