opinion I’ve been extremely surprised at the ongoing nature of the National Australia Bank’s technical difficulties over the past few weeks. Normally we don’t bother reporting the odd banking outage here and there; after all, they happen all the time, especially in the sensitive load-balanced area of internet banking.

However it’s hard to escape the impression that this isn’t just another ‘Severity 1′ incident for the bank. A problem that impacts other banks? That leads to a highly embarassing public apology by the CEO? With problems stretching out into the best part of a month now? Sounds pretty drastic.

The key question here for me is what is really going on from a technical perspective. So far, all we really know is that somebody loaded the wrong piece of code. Quoth the AustralianIT:

It is understood someone with access to NAB’s mainframe systems, either an internal staff member or one of the bank’s IT outsourcing partners in India, inadvertently bypassed a piece of code that checks BSBs against addresses. This happened during the batch transaction cycle and disrupted the bank’s ability to process the files.

However, speaking with technology sector insiders over the past few weeks, it’s plain that many are surprised that a simple bit of rogue code was able to cause so many problems at the bank.

It’s understandable, after all, that an issue would halt a batch transaction; large organisations like banks, the Australian Taxation Office and Centrelink often pause batch processing for small periods to deal with other things going on. It’s often an ongoing battle to keep the batch jobs up to date. However, what remains a mystery is why — as it appears to have done — the rogue code affected NAB’s ability to process batches in general, rather than just causing a short-term issue.

Now there’s been a fair degree of hype around the issue. I don’t personally agree with IBRS’ Jorn Bettin, who made some fairly hyped statements about future likely disasters at other banks because of their crumbling core banking platforms. Frankly, these systems have been stable for decades, and banks aren’t gradually migrating from them because of some kind of instability; it’s more the long-term cost and flexibility benefits that is spurring them to do so.

No, what the NAB situation feels like to me is a classic case of long-term underinvestment.

The NAB’s technology management team currently does not have a strong seat at the executive table as its counterpart at the Commonwealth Bank or Westpac have had for some years now in the persons of Michael Harte and Bob McKinnon. Instead, its technology leadership structure has remained unclear for more than a year since its high-profile chief information officer Michelle Tredenick was ousted by incoming CEO Cameron Clyne.

And it wasn’t just Tredenick that departed the bank last year; her second in command, Craig Bright, also left around the same time.

With Tredenick gone, it was new chief information officer Adam Bennett who took the technology reins at the NAB. However Bennett has never had a high public profile, and in practice it appears to have been the bank’s chief executive of Group Business Services, Gavin Slater, who has had overarching responsibility for the bank’s technology operations.

Slater’s an accountant by training and a CFO and general manager by trade; he appears to have zilch experience in technology. Yet his oversight of the bank’s tech team comes at a time when the NAB has been going through major technology initiatives; offshoring, a toe dipped in the water on core banking transformation and more.

To me, it sounds like the NAB has been underinvesting in its technology operation for some time (the normal strategy when a CFO-type oversees a technology department) and that this months’ events have exposed the weakness of such a strategy. It feels like the underinvestment is going on in several areas; governance — which is usually the problem in these cases — and possibly even at the application layer or even infrastructure.

The bank’s problems over the past month smack of what happens when you squash a cockroach in your bathroom. You might have fixed the immediate problem; but you know there is probably an absolute nest of the buggers hiding just below the floorboards. In places where you haven’t looked for years and messing with stuff that you have always assumed will “just work”.

In this case, NAB’s backup systems should have “just worked” to resolve its ongoing problems. They should have rolled back whatever code was introduced and then restarted processes. The fact that they couldn’t do that easily is a massive cause for concern.

In addition, NAB’s high-profile IT problems could not have come at a worse time for the bank. Not only are the nation’s politicians already riled up about what they see as unjustified interest rate rises and thus paying more attention right now, but both Westpac and CommBank currently have a lot to crow about when it comes to the performance of their technology systems.

The pair have been at pains over the past year to demonstrate that they have virtually eliminated the sort of “Severity 1″ incidents from their technology operations that the NAB is experiencing, continually, right now.

One wonders how long it will be before NAB’s board starts to realise the depths of the hole which the bank has placed itself in.

The Westpac experience has delivered Australia’s banking sector a stern lesson when it comes to reducing Severity 1 incidents in banks. You can cut down the problems; but it will take time, serious investment, strong leadership and a commitment to change. I’m not sure that the NAB has all the elements it needs right now; it may take a change of CEO for the bank to understand that, as it did at CommBank and Westpac before.

Image credit: Delimiter

Related posts:

1. iiNet and Netspace? Hell, it’s about time
2. Lotus fans: Show me the money or shut the hell up
3. Core banking overhaul is a false dichotomy
4. CBA’s happy Harte: ‘We’re years ahead’
5. ‘Keen’ NAB joins global cloud alliance

Tweet

1. Carly
   Posted 09/12/2010 at 3:28 pm | Permalink | Reply

   Wow! As if a major bank could have such technology issues that they hang onto your money. Amazing!
   Personally I think it might have a liquidity issue.
   Secretly borrowing $4.5b from the US Federal Reserve (America’s RBA) in 2008 and 2009 because it didn’t have enough cash to keep it running seems to have finally caught up with them.
   Am I the only one to hear the ominous whispers of “BANK RUN”?

   • Renai LeMay
     Posted 09/12/2010 at 3:32 pm | Permalink | Reply

     This sounds like a conspiracy theory ;)

   • Aj Faj
     Posted 11/12/2010 at 3:00 pm | Permalink | Reply

     I agree on the liquidity point. Perhaps it is in the early stages of insolvency with ‘cash flow’ problems. Try this:
     . replace the euphemism “customer”, with “creditor” – the money NAB is withholding from its ‘customers’ is their money, owed to them on demand by the NAB
     . consider what is happening in a business that is repeatedly unable to pay its creditors
     . consider NAB’s poor record of disclosure regarding its exposure to derivatives
     . consider evidence of its lack of understanding of derivatives shown by the poor internal controls that led to the ‘rogure trader’ problems a few years ago
     . consider that NAB may still be grappling to understand its exposure, particularly in light of the events in Ireland and the foreclosure-gate balck swan in the US

     Perhaps computer ‘gltch’ is a nicer euphemism than ‘bank holiday’ when a failing bank is desperately trying to buy time.

     Time to get as far away from the NAB as possible.

2. snerd
   Posted 09/12/2010 at 3:48 pm | Permalink | Reply

   Ah yes. Maybe they were making an end-run around OFAC, or they just managed to get BIC and account codes mismatched. You can do a lot of damage with SWIFT if you’re not careful, and it’s a nightmare to undo. Corporate Compliance must be having fun.

   • Renai LeMay
     Posted 09/12/2010 at 11:44 pm | Permalink | Reply

     Heh I bet there are quite a few departments inside NAB right now having kittens about this ;)

3. Dean
   Posted 09/12/2010 at 4:55 pm | Permalink | Reply

   When people think “mainframe” they usually think of some 40 year old hulking machine that’s not had the covers cracked in the last decade and is slowly rusting and gathering cobwebs. But it’s not really true, mainframe servers are in fact still being manufactured and sold today (see, for example, Unisys ClearPath).

   Of course, what’s happening with NAB makes it look like they actually HAVEN’T cracked their mainframe’s covers in the last 10 years and they actually ARE rusting and falling apart… if not literally, then certainly figuratively. You still need people maintaining and updating those servers…

   • Renai LeMay
     Posted 09/12/2010 at 11:49 pm | Permalink | Reply

     True, mainframes are being sold today, but not many. I recently heard that there were about 200 mainframes still in use in Australia, which rings about right.

     I suspect NAB actually has a crack squad of lifers doing the mainframe work; it’s the sort of thing you need to have a really good team on internally, and have training and succession planning so the skills never really leave the business etc.

     However, I think likely the problem here came in somewhere in the middleware layer which interfaces between the mainframe and the upper layers of software. Problems at this stage really would have the potential to screw a lot of things up. If it was just one upper layer application that had a problem (for example, the stuff that runs the ATMs or internet banking), then the issue wouldn’t have been so wide-reaching and stuck around for so long.

     If the problem really was (is?) in the core, then that is something I would be really worried about in NAB management, because it would mean the quality control around the core stability of their systems is really not there.

4. Jumbo_Kane
   Posted 10/12/2010 at 5:04 pm | Permalink | Reply

   My NAB accounts have been in ‘flux’ for two weeks now, nobody at NAB can fix it. I don’t understand what is going on. I am in the process of moving ALL banking away from NAB. Seems to me like the tip of an oncoming iceberg.

5. Michael Wyres
   Posted 13/12/2010 at 6:31 am | Permalink | Reply

   Having been a Customer Service boss at an ASX-listed company, I’ve had to “spin” outages in the past.

   I would have taken the situation at the NAB over the last month and come up with something like:

   “Thank you for your patience during our recent system test procedures.

   The first stage of the test involved deliberately imparing the input of funds into accounts to allow we at the NAB to activate and test our funds security procedures. Our test was highly successful, with some funds not being able to be moved for almost an entire week. The safety of your accounts was proven.

   The second stage involved the testing of procedures for situations where funds were unable to be output from accounts, and allowed us to understand the nature of the interaction between ourselves, and you – our valued customers. Once again, the security of our systems held tight, with no funds being able to be removed from accounts for some hours.

   Again, we thank you for your patience while we demonstrated the value of being an account holder with the NAB – your money is more “secure” than at any other bank.”