Heartbleed, internal outages: CBA’s horror 24 hours

6

commbankatm

news The Commonwealth Bank’s IT division has suffered something of a nightmare 24 hours, with a catastrophic internal IT outage taking down multiple systems and resulting in physical branches being offline, and the bank separately suffering public opprobrium stemming from contradictory statements it made with respect to potential vulnerabilities stemming from the Heartbleed OpenSSL bug.

This morning Delimiter received an anonymous tip stating that the bank was suffering a major outage affecting its Internet banking, EFTPOS, telephone and branch banking and even its CommSec trading platform. The problem is is rapidly becoming visible externally due especially to the fact that some of CommBank’s branches have been unable to open.

The source of the outage, according to unverified internal information, is that the bank is suffering problems with a number of groupwide systems, including its CommSee customer platform and even systems used by its Colonial First State brand. The issue is described as a “massive system outage affecting all CommSee users”, and has been listed internally as a priority 1 outage.

Asked about the issue, the bank firstly issued a statement apologising to customers and noting that it was working to restore services as a priority. Shortly after, it issued an updated statement noting that its technical issue “has been resolved”, with full services progressively being restored.

The outage is not the first time in recent memory that CommBank has suffered an extensive internal outage with its IT systems which has resulted in significant chaos for staff and customers. In late July 2012, for instance, the bank was plunged into chaos following what appeared to be a disastrous misapplication of an operating system patch to thousands of desktop PCs and hundreds of servers at the time.

That outage similarly took down CommSee, effectively making it impossible for staff to complete much of their work. At the time, blame for the outage was pinned on outsourced supplier HP, leading to a direct visit by the vendor’s global chief executive Meg Whitman to Australia to hold talks directly with CommBank on the issue.

Separately, CommBank has also been suffering a high degree of criticism over the past 24 hours especially about its handling of the Heartbleed bug.

Heartbleed is a security bug in the open source OpenSSL cryptography library, which is widely used to secure Internet communications. It was recently discovered but has been in existence for several years, and has reportedly been used by government agencies such as the US National Security Agency to penetrate supposedly secure systems. The bug affects a huge percentage of Internet services.

In a post on the bank’s blog last week, CommBank general manager of digital channels and online banking, Drew Unsworth, had initially stated that the bank was “patched” against the bug and that customers did not need to update their Internet banking passwords.

However, customers quickly pointed out that the bug had been active for several years, making Unsworth’s statement likely to be factually incorrect. If the bug had affected CommBank, customer details could theoretically have been stolen during that time before any patches were applied.

Unsworth successively updated his post stating that the bank’s NetBank platform “does not (and did not) use OpenSSL. “All customer data is safe, so customers do not need to change their NetBank passwords or take any action,” he wrote. “We have multiple layers of security in place to protect our customer sites and services. Our security teams constantly monitor and stay abreast of the latest security vulnerabilities and are quick to take any action required to protect our customers.”

Unsworth also reminded customers that the bank offered a 100 percent guarantee, provided they kept their NetBank client number and password secure and notified the bank immediately of any suspicious activity on their account.

Despite the post, however, many customers have posted comments on the blog pointing out the inherent contradictions in Unsworth’s comments, in that he first stated that the bank had patched its systems, but secondly that it never used OpenSSL.

“This does not compute. Which is it?” asked one commenter. “This is really concerning, you need to get a security expert to actually rewrite this post,” another added. “1. You used OpenSSL in the last 2 years, therefore everyone MUST change their password. No ifs, or buts. 2. You didn’t use OpenSSL and this entire thing was just a media stunt. Either way, you need to address people’s concerns quickly. Copying generic responses will not answer the question.”

A third wrote: “Wow. I’m a CBA customer and a software developer. I’m amazed by the poor level of support by the CommBank moderator on this thread. ither you don’t understand the issue, or you’re trying to obfuscate. Could you *please* escalate this to someone who is qualified to comment?”

opinion/analysis
Pretty terrible 24 hours for CommBank, but I think the bank is gradually getting things under control :) Priority One outages (affecting fundamental service availability) are pretty nasty, but at least most things are back online now, and the Heartbleed issue looks mostly like a bit of a public relations gaffe rather than any kind of actual IT security issue.

I don’t personally really believe the statement that CommBank doesn’t use OpenSSL at all; I’m sure OpenSSL is indeed used somewhere in the bank’s vast operations. CBA uses virtually every technology known to man in some capacity; it’s that big. But I’m also pretty sure that the bank wouldn’t have been vulnerable in the normal way that smaller companies or platforms have been to the Heartbleed issue. I would say CommBank’s systems are too complex and its security too tight for that. I think this one was just communicated poorly. Happy to hear divergent views, however.

Image credit: megawatts86, Creative Commons

6 COMMENTS

  1. There is another bitey in this saga.

    Affected systems need not only an updated OpenSSL but they also need new certificates which should be generated on a computer which is NOT connected to the ‘net.

    Deutsche Bank’s main consumer portal checked out OK on the HeartBleat test app but was still vulnerable because no new cert had been applied.

    /R/

  2. How can a physical branch be “offline”?

    Maybe you mean closed, but how can that be? Surely humans can still unlock doors and interact with people without a computer?

    • But what can they say to those customers?
      They can’t withdraw, they can’t deposit, they can’t verify balances.

  3. So, was the massive outage caused by an authentication error when they attempted to update their key certificates following HeartBlead? That’s happened to a few companies: I was getting certificate errors for a few days as companies cycled through short-term certificates in their efforts to make quick fixes

  4. How many other holes or hidden passages exist in open-sourced code?
    Seems like the free ride is over for OSS freeloaders.

    • Actually, the free ride is still on. Heartbleed was/is not a bug in OpenSSL code, it is a bug in the TLS Heartbeat extension, nothing at all to do with OpenSSL.

      Other holes or hidden passages? Probably a lot fewer than those in closed source OSes.

      Gordon.

Comments are closed.