The Change Program is a failure … and that’s not all

opinion I’m ropeable. The curtain is rising on the drama that is the ATO Change Program and soon the plot will reveal the Australian IT industry as incompetent shysters, scamming the taxpayers of millions — if not billions.

There is absolutely no question that the Change Program is a failure. It’s over budget by hundreds of millions of dollars, it’s late, and by late I mean publicly announced “delayed” so many times that it’s hard to remember what decade it’s meant to be delivered. And now we hear that the first instalment of the software has miserably failed the very customers who paid for it.

We’re close to a billion dollars spent already, and we’ve barely switched the thing on. Anyone who’s done project management 101 will know that we’ve only scratched the surface of the cost: “The last 10 percent costs 90 percent”, and “50 percent to 90 percent of the cost of software is maintenance”. So the taxpayer is conservatively looking at a two billion dollar bill, and for what? Something that arguably we don’t need, and for certain, no man on the street gives a shit about.

Until now, the ATO has managed to avoid exposure of its blatant incompetence, but when the average Joe who’s counting on his refund cheque gets a “Hey there! We owe you money, but we’ve forgotten to attach the cheque” fuck you in the mail, he’s going to get a bit pissy about it. Maybe now while he’s listening, we can tell him how much he’s personally paying for his new “core processing” engine: $180. As he shrugs off the figure, it might be worthwhile mentioning that he’s paying $54 for research and innovation investment, $72 for community infrastructure, $290 for health and hospitals, and $409 for clean energy.

Ask him if he wouldn’t mind adding his share of “Change Program” money to the clean energy or health and hospitals budget? After all, his electricity bill is going up and the stress of not receiving a refund cheque is affecting his health.

It wont be long before Joe starts to point fingers. And the evidence just keeps building: in the last fortnight alone we’ve had two more enterprise software bungles: no pay for Queensland Health workers, as their HR system migration was a dismal failure, and the “Medicare IT bungle” which risked the health of thousands of people making front page in the AustralianIT. It’s never been a better time to be in software, particularly the enterprise.

It’s about time we took a good hard reflective look at our industry, conclude that enterprise software engineering is an abject failure, and collectively identify the problems and look to solve them. Having worked in this industry (government, private enterprise, projects both small and large) as a contractor and as a higher-up in one of the bigger software development houses, I figured I’d recount my tour of duty on the well trodden path of software crises:

Failure of accountability
We’ve all heard the anecdotes of “being promoted to your level of incompetence” and we smile as we know most anecdotes are anchored in truth. It seems though that with every major announcement of failure, the owners of those failures get off on a “it was him, sir” defence. The bigger the “partner” you can point to, the better. Nothing sounds better than “oh, if Multinational Conglomerate X can’t build the system, what hope did we have?”. This defence seems to work right down to even the leaf nodes in an organisation. It still surprises me when a failure gets announced, those responsible still keep their job.

Failure to invest in maintenance
Maintenance is arguably the most important part of the software development lifecycle. Over time, the cost of maintenance can eclipse the cost of development, yet we don’t foreshadow maintenance requirements in the development cycle. The adopted “get it over the line quickly” mantra displaces engineering effort to make the systems flexible, adaptable, and easily maintainable. My biggest gripe is the failure to invest in documented evidence of system changes. How often have we seen a “hot patch” go in to production without even a dash of change control, or impact analysis?

Failure to invest in definition
And the second most important part of the software dev lifecycle: modelling of the system. The dissonance between the requirements definition and the system that actually gets built is incredibly large. You’d wonder if the business analysts for Queensland Health wrote requirements for pretty payslip generation, and the engineers used a random number generator for the dollar amount to show in the bottom right corner.

The reality is that we don’t invest enough time in system modelling (and by that I mean modelling the truth and relationships of the system), and planning. Anyone who’s on an “agile” project will know in their hearts that they’re not truly following the agile manifesto, but rather a botched version designed to fool the ignorant. Same goes with system definition: we write orders of magnitude more code than the text we use to define, model and architect the requirements of the system. Think about the next refresh the system will need years out: if it’s natural to say “just look at the code” when someone asks you what the system does, it’s a failure.

Programmer skills
It really is tough to be a programmer these days. It’s hard enough dealing with the garbage in garbage out problems of bad requirements and architecture, but these days we’re dealing with deep, systematic platform issues that drive up complexity beyond the capabilities of our wetware.

Most development stacks these days have bi-yearly major version drops, with new libraries, new language features, new patterns, and sometimes even new development practices along with them. The latest, most vocal software crisis is concurrency: how do we scale our apps across multiple cores without blowing them up with race conditions, deadlocks and broken invariants.

A new drop of concurrent abstractions arrive on our favourite platform, we attempt to learn and change our habits of development to adapt, and we naively push the result out to production. Concurrency is an incredibly complex requirement, where you need specialty skills to do it reliably and safely. And we’re surprised when we see blowups in production later on in life. Determinism and scale are expensive feature these days.

And that’s just one example. There are literally dozens of new features designed to solve the world’s problems, but that ultimately only seem to be successful in increasing complexity. The adoption of small, lightweight, feature lean frameworks like Ruby on Rails comes at no surprise. methodMissing makes sense, co and contravariant type arguments in C# 4.0 doesn’t.

Developers shouldn’t get off easy though, as while the systems we’re building, and the platforms we’re building on get more complex, we’re struggling to represent truthfully the level of skill we have to cope.

Every programmer should know about the Drefus model (where you are in your level of expertise for a given skillset), and accurately represent that. And the consulting organisations that market the devs should also wake up and accurately represent that skillset to their clients. We’ve all seen egomaniacal “expert” consultants come in, only to realise their magic bullets aren’t all that magical after all.

Large Teams
The natural way we like to solve complexity issues like those above are to throw more resources (hopefully experts of course!) at it. It’s about time we flipped the bit on any software project manager who hasn’t heard of Brooks Law (adding manpower to a late software project makes it later) and who hasn’t read the Mythical Man Month. And while a statement like that is an oversimplification of the problem, the reality isn’t too far from it: we have project managers who don’t have the skills necessary to run large projects.

They simply don’t understand the complexities and skeletons that live in the phases of the software development life cycle, have no idea about the intricacies of methodology (ask your project manager what waterfall or agile really means), and often fails to provide the most important glue for software development: effective communication. Adding people to fill the project managers gaps isn’t the answer, in fact it’s the very antithesis of running a lean effective and successful software development project.

Testing
Probably the most underrated job on a software development project, and without cause. These guys are the last line of defence and typically clean up the crap before it’s delivered. If we had sufficient bars for all that we talked about above, we could only then start to think about reducing testing headcount, but these days, if you’re not running 1 to 1 test to dev, then you should be worried.

We have terrible practices for all of the above, and it’s only these obsessive compulsive bunch that have the mandate to point that out to us. We’re all offense, and little defence, and I think it’s about time we ramped our defence force. Think twice about joining a software development project that has little or no testing resources.

And Back to Reality
67 percent of software projects either fail, are late, are over budget or under featured. 35 percent of those projects identified lack of user input, incomplete requirements and lack of change control as the problem. So it should be no surprise that the ATO Change Program is massively over budget, or that the Medibank and Queensland Health systems are a black hole of failure, but what ticks me off is that while we can admit the fact that we’re terrible at building software, we still continue to pour mammoth amounts of money in the hope that it will just “get better”.

The harshest reality of all is that the cost of the problem is going to scale non-linearly with the size of the system. And I, like all of you, don’t want us pissing good money away when we the yield curve is so blatantly unattractive. It’s a bad deal for Joe. And inevitably a bad deal for us.

The author is an Australian IT industry sub-contractor.

Image credit: Simon Stratford, royalty free