Future IT project fail?
NSW Police gets COPS replacement funding

33

blog If you’ve been following state government IT in Australia for as long as I have, it starts to get easier and easier to see major IT project failures before they even happen. And NSW Police just popped up a doozy.

Those of you with some familiarity with how the NSW Police functions will know that for most of the past seventeen years, much of its core policing work has been been done with the aid of a text-only data entry and retrieval system dubbed ‘COPS’, which is a typical police database system in that it allows officers to enter and retrieve information about crimes, suspects etc. Every modern police force has one of these systems, and usually there are many satellite links into such platforms so that associated systems can pull data out.

The COPS system in use by the NSW Police is particularly ancient, dating back about 15 years. We know this because in September 2011, the agency noted that it (with the assistance of Fujitsu) had completed a major upgrade to the then 15-year-old platform that had layered a web-based interface on top of the system. The COPS platform is drastically simpler, but you could draw an analogy with the way modern banks tend to work, where their Internet banking platforms accesses by consumers are layered on top of multiple systems, underneath which often sits an archaic core banking platform dating back decades.

Dubbed ‘WebCOPS’, the Fujitsu upgrade has provided a much-needed lease of life to the COPS platform. But as with those tricky core banking platforms, it appears as though the database itself still needs to be upgraded or replaced. It appears that the NSW Government has this week approved funding for that project as part of its budget. Intermedium tells us (we recommend you click here for the full article):

“In 2013-14 the NSW Police Force will embark upon a $44.8 million technical migration of its Computerised Operational Policing System (COPS) to a modern database, to be completed by 2016.”

Now, why is this a problem? It’s a problem because the replacement of these kinds of platforms has proven a major, major problem for other states. Take Victoria, for example. The state’s own police force has a virtually identical core policing system to NSW’s. It’s dubbed LEAP, and Victoria has been trying to replace it for quite a few years now. It’s also close to several decades old.

However, various attempts at this effort have been aborted, with one landmark investigation into Victoria Police finding in March 2012 that the agency had no ability to delivery major IT projects. “Victoria Police spent $59 million on LINK over four years, only for it to be cancelled,” the Victorian Ombudsman wrote in a separate report in November 2011. Seems like an eerily similar amount of money to the amount which NSW wants to spend on replacing COPS.

I’m not quite sure what’s happening with LEAP right now in Victoria, but I’m sure there have been yet more plans put forward to remediate the situation, on which millions have already been spent and years of time been wasted.

Look, what am I trying to say here? In general, what I am trying to say is that we are seeing a massive, systemic, degree of IT project and service delivery failure across several Australian states right now — especially Queensland and Victoria, but also to a degree in NSW. We haven’t seen any large IT projects successfully delivered in NSW for some time.

In this context, and with the direct experience of the LEAP/LINK project failure in Victoria, the COPS replacement program in NSW must be regarded as extremely high-risk. It may be that this is only a first small step which has been budgeted this week in NSW, and that NSW Police is aware of the dangers they are tinkering with here; that they have a very conservative, long-term plan. I have to say I doubt that, given that NSW Police doesn’t even appear to be capable of auditing how many software licences it’s using.

If this what it appears to be — a total replacement of the underlying COPS infrastructure for the paltry cost of $44.8 million, over a period of just a couple of years, then I would have to say I predict that the project will not be delivered for that amount, that it will not be delivered on time, that it may fail, and that NSW Police may end up in protracted legal disputes with whichever partners they are getting to help them implement this project. If this is what it appears to be, then my opinion is that NSW Police has seriously underestimated what it will take to replace COPS.

This cannot be treated as a normal IT project, within normal IT parameters. Nothing about state government IT in Australia is normal right now. The norm is that every major project is going over budget, over time, and sometimes just failing completely. And especially so, when we’re talking about core platforms which are almost two decades old. I hate to be the one to say it: But all the warnings signs are here for another state government IT disaster, 4-5 years in the future. Am I wrong? Ill-informed? Happy to hear why. I’d rather discuss this now than be writing another article about a damning state government ICT audit report into a failed IT project, which is what I normally do.

Image credit: Back the Future movie promotional image

33 COMMENTS

  1. CPS – 15 years old? Bahahahaha! Built in the 80s by Arthur Andersen (Accenture). It’s over 25 years old now. Get your facts right! And it’s actually a good example of a large IT project that succeeded because it has been in operation for all that time and has been extended and grown over that time and has continued to do a reasonable job. OK, it’s old technology and looks dated, but its ability to manage information has held NSW Police in good stead for many years.

    • “its ability to manage information has held NSW Police in good stead for many years”

      Which is precisely why I am worried about it being replaced. Experience tells us state governments always underestimate the funding and time required for this. I would rather COPS run for another ten years than this get fucked up and NSW Police not be able to pull information on criminals for months on end.

      • So your solution is do nothing. Don’t take advantage of improved technology that could lead to better outcomes in bringing offenders to justice. So you are happy to see the kind of thing that allowed a convicted jihadist transferred to a minimum security facility until the Police heard about it on the radio!

        That kind of risk averse thinking leads us nowhere. You are not helping. Let’s accept risk and come up with some positive ways to improve the delivery of software development projects.

        BTW – I think you will find that the strategy NSWPF have been following is an incremental, staged upgrading of COPS. This stage only involves some of the functionality of COPS rather than a big-bang implementation. They are actually being very smart and if you had spent some time asking the people involved you might find something positive to write about.

        • My solution is not to “do nothing”. My solution is to treat this thing as if it is nuclear radiation dangerous. Take it in very small stages. Make sure you over-allocate capital. Expect a 5-7 year project cycle. Put hugely strong governance controls around it. Have several back-up plans, in fact develop several alternatives simultaneously — one path will no doubt prove itself better as the development process goes on. Bring in the best experts involved in such upgrades from countries like the US, which have stacks of police departments everywhere. Spend two years getting the upgrade plan right. Make the executives accountable publicly. Trial everything continually. Agile development constantly — no ‘big bang’ implementations.

          We need to treat this kind of thing as EXTREMELY HIGH RISK, DANGER WILL ROBINSON. Take every safety precaution possible.

          Think I’m wrong? Look at Queensland right now:

          http://delimiter.com.au/2013/06/07/systemic-business-risk-90-of-qld-govts-ict-needs-to-be-replaced-total-cost-7-4-billion/

          Look at Victoria right now:

          http://delimiter.com.au/2011/11/23/vic-government-it-in-flames-1-4-billion-over-budget-all-projects-late-or-failed/

          I have followed this area in depth in the past decade, and this project has all the warning signs.

          • If you have followed this in depth you should know that the US policing model does not apply here at all. Their police departments are mostly small, tiny in fact, by comparison with Australia. Their experts know nothing of the challenges faced by a large police organisation employing over 15,000 people, covering a geographical area equal to dozens of their states. If you know the technology they have then you should know that New York Police have just as outdated systems. NSW Police considered using their despatch systems a few years ago and rejected it on the basis of its outdated technology and poor information structures. Having “stacks of police departments everywhere” does not mean they understand the technology needed. That is a very naive statement.

            The involvement of large, expert firms was one of the reasons that QLD Health failed so abysmally. They will ensure that a budget of $44m gets blown out to epic proportions. I well know the experience in QLD and VIC – you are not the only one who reads technology reports. But some of us actually have some insight into what went on inside those projects.

            Meanwhile smaller teams and smaller local firms and internal software development teams are delivering high quality solutions to Australian organisations every day.

            If you want to point the finger then it is with the governance practices of government organisations, not the ability of teams to deliver software.

            I said in my reply that NSWPF are doing this project as an incremental build, but you haven’t taken the time to find that out and you’re being a chicken little.

            An attitude that says we need to treat this as dangerous as radiation will just paralyse people and lead to no investment to the point that our systems are so outdated that the IBMs of the world will jsut end up taking over all our publicly funded technology and then we’ll be paying much more than $44m.

          • No worries.

            OK, you are making some very good points, and I appreciate you contributing this information — you’re adding to my understanding of the situation.

            Some points:

            -I know the US is different — I just threw them in as an example. I think its critical that those responsible for this kind of upgrade examine similar systems and upgrades in countries such as the US, Canada, the UK and possibly even the non-English-speaking blocs as well, to understand how this is happening internationally. The best IT solutions are usually found in these kinds of areas where best practices and standardised software platforms evolve out of the morass of competing and complex systems. This is where I want to drive towards — not using “the best platform”, but using the most popular one, the most-well-understood one, the most supported one. Being a part of the crowd is where safety lies in IT. The NSW Police dept should not try to reinvent the wheel.

            It is for this reason that I am 100% against custom software development efforts within government. Integrating off the shelf software and bending actual business processes to fit that in as bog standard a way as possible, is clearly the answer. In the 3-7 year time frame in Australia, it will also go one step further — the widespread adoption of ‘as a service’ solutions to these kinds of IT service needs will see many of these IT systems started to be provided as SaaS deployments, requiring even heavier business process standardisation. Sounds like a pipe dream? It’s starting to happen in other areas in the NSW Govt, and both Qld and Victoria are also discussing this.

            -In the Qld Health case it was primarily the poor governance of the government itself, especially the departmental execs overseeing the project, that caused the issue, rather than the external parties. You can see this here:

            http://delimiter.com.au/2010/06/29/woeful-scope-definition-caused-qld-payroll-disaster/

            IBM did play a part, but most audits of this have found most of the blame lies with government agencies — CopTech, Queensland Health itself etc. There was also a huge factor regarding business processes (complex award structures etc) not being able to be bent at all to meet the software.

            -You are 100% right. The problem is with governance. State Government agencies in Australia simply cannot govern IT projects. This is my primary concern as well. It is the question I ask repeatedly of state IT ministers and CIOs whenever I talk to them, for example:

            http://delimiter.com.au/2012/10/10/nsw-govt-confident-it-can-avoid-it-disasters/

            I actually think we are pretty much on the same page about all this, but possibly coming at it from different directions :)

          • I guess I get frustrated at seeing only the bad press of IT project failure and no one ever talks about the successful implementation of software projects that goes on in both the private and public sector all the time. There have been some spectacular failures – no doubt about that – and they have wasted a serious amount of public money. But there are lots of good stories out there as well. They just don’t seem to get reported at all.

            I am always wary of blanket statements like, “100% against custom software developments within government”. Often the cost of bending business processes or configuring software to do what we want it to is greater than building a capability ourselves. There are lots of cases where an organisation can buy something off the shelf and should modify business practices to fit in with it. Payroll ought to be one of them! Makes me cranky that QLD Health botched it so badly. But there are also lots of situations where you can’t avoid custom software development. Where would you go buy a tax system for instance?

            Basically, the industry isn’t mature enough. You ought to be able to buy a system to manage clients, one to manage accounts, one to manage payments, one to manage debt, etc. They should all be able to plug together and you would get an integrated solution that could be configured. But those things don’t exist yet at that level of sophistication.

            I do agree with your comment that it should be about continuous, incremental change and improvement. If there is a well managed architectural capability an organisation ought to be able to acquire modular solutions as they become available and fit them into their IT landscape, adding whatever custom solutions are needed. Unfortunately, that is only too rare.

          • I agree with pretty much all of this. Even for the ATO’s Change Program, they had to do substantial customisation of Siebel to get it to be fit for purpose.

            I do do positive stories where I can find them — for example here:

            http://delimiter.com.au/2012/06/27/review-brands-atos-change-program-a-success/

            And there’s great stuff being done in NSW atm:

            http://delimiter.com.au/2013/02/05/nsw-kickstarts-cloud-email-virtual-desktop-trial/

            Another (quirky) one here:

            http://delimiter.com.au/2013/02/15/the-nsw-rtas-imacs-lasted-a-full-decade/

            I’d much rather highlight new and interesting uses of technology in government rather than project failures. The problem is usually that there are just way more failures than there are successes, and the failures are often the subject of publicly available audit reports. The successes usually require a lot more digging — and as just one journalist with a lot on his plate, I’d rather spend my investigative time holding the government to account for its Internet filtering/surveillance/data retention bullshit than on finding successful IT projects ;)

            However, if you know of any … let me know through the anonymous tips form.

            http://delimiter.com.au/anonymous-tips/

        • That kind of risk averse thinking leads us nowhere. You are not helping. Let’s accept risk and come up with some positive ways to improve the delivery of software development projects.

          Sure, let’s just accept the risk that police can’t be everywhere solving all problems for all people.

          Once we have accepted that risk, a bigger and more deluxe way to suck away the last crumbs of our privacy doesn’t seem so important any more. Australia is a low violence country anyhow (not because our police are significantly better but because political violence is pretty much non-existent here), and if it ain’t broke don’t spend millions of dollars fixing it.

          • The fact is that the platform that old systems like COPS run on costs lots of money each year. Newer technology platforms are much cheaper to run, so the replacement of these old mainframe based systems will result in lower spending by government.

            There are other benefits to the community as well. For example, it’s not possible with the current technology that NSW Police has, to provide mobile apps so that Police can spend less time keying in data at a Police station and more time in the field.

            I can’t see how you would think that giving modern technology to Police is going to lead to loss of your privacy. Conspiracy theory?

          • Wasn’t this an evidence-based site once upon a time?

            What platform does COPS run on?

            Show how much it costs and why there would be a saving.

            Show my how software gets more expensive to run as it gets older, like an example of how the BSD kernel (you know, the heart of Apple’s OS, the thing that runs inside every iPhone) is now much more expensive to run than back in the 1970’s when it was first running. From 1977 to 2013 is 36 years, and that’s how old BSD is. So on with the evidence shall we?

            As for waving away government invasion of privacy with the old “conspiracy theory” line, I suppose you also think that the only crime Al Capone ever committed was tax evasion, hmm? Seriously, where have you been? Don’t you read the news? Even on this site barely a few weeks goes by without more disclosure of governments spying on their own citizens. Don’t you lift your head when you walk? There’s a camera on every street corner, half a dozen in every train carriage.

            My point that never even got a response though was that life does involve risk. That’s something we agree on. The cops save some people, but mostly they clean up after the fact and try to be a bit of a deterrent. Fine to keep crime statistics so we can employ plenty of researchers to study them later on and hum and har over it, but I have yet to see evidence presented demonstrating that police become more effective, or that crime rates go down, as citizens lose their privacy.

            For example, those compute-collected statistics demonstrate that assault, and sexual assault have been steadily increasing in Australia. The cops simply don’t wait around in back alleys behind the pub just in case someone gets dragged into there. Even with their new computer system, they still don’t.

          • I get the feeling I could write.a book of evidence and you wouldn’t be satisfied, but here goes. COPS runs on an adabas/natural platform on an IBM z/os operating system. That means expensive licenses from Software AG and from IBM. Newer technology gives you the same processing power for much less cost. Browser based apps running over http are much cheaper to run than terminals on dedicated networks.

            If you are coming from a base of thinking about the apple os then I am not sure that’s the right frame of reference for the kind of systems we are talking about.

            Forget that. Surely you can imagine that maintaining older technology can be more expensive than moving to newer technology. I used to work for a merchant bank and a one gb db required expensive Dec alpha servers. I can now run larger dbs on my laptop. Need more evidence?

          • OK, thanks for mentioning the platform, I couldn’t find that on any newspaper site.

            Let’s go with the idea that they don’t need to expand their level of surveillance, and just need to keep the existing system going, given an expanding population and a proportionally expanding police force. So you have brought up a number of issues here.

            Starting with computing hardware (maybe presume the OS is lumped in with hardware for convenience), yes we have seen significant improvements over the last decade, and those improvements also include drastically lower costs. A quick check on Wikipedia says that adabas runs on Linux, Microsoft Windows, OpenVMS, and various other Unix variations. That sounds like they have a pretty broad selection of platforms they could choose. Half a million bucks easily buys a rack full of servers these days, and as you rightly point out even a laptop can run a decent size database, so let’s call that a whole $1 million spent on hardware, with redundancy. That gives a lot of options for hardware upgrade.

            Yet these guys want to spend 50 times as much. That totally doesn’t make sense for just a hardware upgrade. Even when you throw in some labour for testing, project management, and all the other fudgy stuff.

            So now you want to talk about software licenses. Well I would have to presume they knew those licenses were going to cost money right on day one, and I would have to presume that the licenses come with support, and regardless of what they buy they will need support. It is possible they are going to move to Open Source and do away with ongoing license fees forever, that might be a saving in the long run.

            Is a total rewrite a sensible option in that case?

            Adabas has SQL connectivity and with SQL you get Microsoft, Linux, Oracle, PostgreSQL, DB2, plenty of others. Keep the existing system, and gradually move a bit at a time away from the proprietary model onto a standard SQL platform. This gives you vendor independence, still doesn’t sound like $50 million worth of work there, and anyway there’s no need to do it in one big hit either. Managed migration almost always beats the big bang approach (but the “big bang” is easier to sell, especially to government).

            I also have trouble believing that the software license fees are so massive that moving away from an existing and well supported platform is cheaper in a short space of time. Why would the vendor set their license fees at punitive levels to drive clients away?

            Now you want to talk about front end user interface.

            Browser based apps running over http are much cheaper to run than terminals on dedicated networks.

            Hmmm, I keep looking at the article above and I keep seeing this word WebCOPS along with an explanation that they already have browser based apps running over http. Gosh darn, they can spend $50 million getting what they already have, that sure makes sense.

            As for “terminals running on dedicated networks”, let’s ignore the fact that they already support Web based access, and ask how many platforms currently require these “dedicated networks” you are talking about? Well I don’t frankly believe that adabas is locked to just serial lines, I kind of think that IBM have enough of a clue to be able to support IP, even for their older software. Just because I’m a stickler for details I decided to go to the SoftwareAG website and blow me down, they already support web interfaces as part of their standard package. They guys must be smart, just like you; they can adapt to newer technology by adding features to their existing product. If it supports WWW it kind of must be supporting IP as well.

            So you think that terminals are expensive? Hey, you have come to the right place, let me tell you about these guys called Wyse and they have their own website full of cheap terminals. You will be really impressed because they also support IP. Yup, Wyse went and updated their products, how about that? They support Citrix, they support VMWare, they support Microsoft’s RDP, and VNC, and X11 and every type of terminal emulation you ever heard of… and they might even offer a web browser if you feel lucky. These things are typically less than the cost of a laptop or desktop PC, and have less moving parts to go wrong.

            Don’t like Wyse? Maybe you want to support existing desktops? Well there are terminal emulators for Microsoft desktops that you can download for free (e.g. putty, and tera-term) with encryption (SSH) or you can use a whole bunch of options on a Linux desktop (possibly zero-cost or not, depending on the level of support required), or even Mac, or even ConnectBot for android tablets if you want mobility. I have no doubt there are several competing products that do the same thing for iPhones. Terminal emulation would have to be the most widespread and well supported application in computing history, right after the text editor.

            Spending $50 million because terminal emulators are too expensive? Really?!?

  2. I believe the WA Police spent tens of millions on development of their own system for decades (failure after failure) before they finally got it up in around 2006/7. Having not worked on such a system personally my opinion as to why is worthless, but I will say I am surprised at the level of incompetence that seems to be systemic in provision of these systems to police forces throughout Australia. It’s interesting that they are all custom builds and never off-the-shelf products, despite there being much larger police forces with working systems all over the planet. What am I missing here? What precludes deployment of a tweaked off-the-shelf product for Australian police?

    • No, no, no. The Victoria Police was an implementation of an off-the-shelf solution called Niche, from Canada. COPS by comparison was a custom built solution from the ground up, with a very good internal team who did all the data modelling. Not me BTW, but I have met them.

      Don’t assume or be quick to make judgements on this stuff unless you really know.

        • I notice that you didn’t try and address the fact that an “off the shelf” implementation failed miserably. Now that is a massive warning sign.

          • Customised “off the shelf” is the thing that should be the biggest flashing light of impending doom!
            COTS are one of those things sold by salesmen who never have to implement or support. They have no longevity because end of life is now under the control of an external vendor who is always going to be chasing the latest thing and only looking at real levels of support as an on-cost to be minimised.

            Custom solutions are proper solutions when done well. If you’re not doing it well that’s a management failure, not an IT failure.

      • Wow you seriously wrote that in response to my comment?

        1) I specifically said I wasn’t an expert with no direct experience in the sector.
        2) I passed no judgement, I asked (what I thought were pertinent) questions.
        3) If you can’t tell the difference between questions and statements you have flawed comprehension
        4) Your association (or lack thereof) with people who may or may not have been instrumental in the design and development of COPS is immaterial. The only material concerns are the quality of the system and the likelihood for success of the replacement/upgrade.
        5) Your knee-jerk reaction combined with your apparent inability to comprehend the difference between a statement and a relevant question casts an unfavourable light on the rest of your comments on the subject. Given that just about everything you’ve said is untestably anecdotal you have really undermined your whole argument simply by being unreasonable. Maybe try taking the time to read comments thoroughly and address what was actually written before dismissing them out of hand.

        • Oh come on. You said, “It’s interesting that they are ALL custom builds and never off-the-shelf products”. I simply pointed out you had made an uninformed assumption and that the Victoria Police implementation was an off-the-shelf product. If you think that is anecdotal, then I can’t help you improve your understanding.

          You also said there are “much larger police forces with working systems all over the planet”. That is wrong. I would say that New York Police, LAPD, the Met in the UK and the Canadian Mounties are the only forces of a size comparable to NSW Police Force. Many police forces do not have working systems, but rely on officers using notebooks – paper ones.

          You asked what you are missing. I have made other comments here about what “precludes deployment of a tweaked off-the-shelf product”. There are many factors, not the least of which is that there are only a couple of such products. One being the Niche product from Canada, which was at the centre of the failed project in Victoria. It is used in QLD, however it has all the problems of a COTS. Every improvement they want to make has to be supported by all customers around the world, which goes into a list of desired improvements and they have to wait for it to be funded and implemented. This includes legislative changes, which is obviously undesirable. The market across the world for police systems is very fragmented. In the States police forces are very small, apart from the couple I mentioned. Also in Europe, there are police forces for a city or province. Then there are large jurisdictional differences. Laws are very different. For example, in NSW when a person is taken into custody there are lots of rules and processes that must be followed to manage the person, mostly as a result of the inquiry into deaths in custody. These laws don’t apply in lots of jurisdictions around the world.

          Let me ask you to think about who builds COTS solutions. These companies employ people who are no smarter than you or me. They build it on technology that by the time they have sold it to a few customers is out of date. They do it often on smaller budgets than a government department has, resulting in short cuts. They also need significant “tweaking” to make them fit for purpose. When you add the initial payment for those solutions to the cost of tweaking, plus the ongoing licence and support fees, you get to pretty much the same numbers that it would have cost to build something. Then you have to deal with vendor lock-in. For example, the Niche product uses a proprietary database technology. So, when, in the future, you want to move to a new solution, you can’t. Or it costs a bomb to do so. That includes the needed hardware. You also lose a lot of your capability in understanding how your systems work, because you now depend on the vendor.

          You also expressed surprise at the ‘systemic level of incompetence’ in provision of these systems. That’s pretty harsh. There are competent IT people in all the police forces around the country who deliver good solutions all the time. COPS is not the only system used in NSW. LEAP is not the only system used in Victoria.

          I’m sorry if my direct approach offended you.

    • Only an idiot or the particularly brave would touched that, the requirements were excessive. Thankfully the government did learn something from that and simplified the fare structures to be far more reasonable.

  3. I think the debate in this discussion thread might be put under the heading of the industry saying – death by customisation – (of a COTs or built from the ground up system)!!! Warning Warning..

    PS – look at the massive QH Payroll fiasco – who would have thought anything could go wrong when you turn off a massively customised COTS payroll system that has been embedded into the organisation over a long duration and then cut over to a vanilla SAP HR/Payroll system – all hell breaks loose – and $1.3B of recustomisation later – who would have thought .

  4. Seems like an eerily similar amount of money to the amount which NSW wants to spend on replacing COPS.

    This says nothing about the project, state governments have their money spooned out to them from Canberra so they will consistently spend whatever they get. You never, ever in a you life will find a government department discovering its budget is bigger than it needed to be and handing back the rest.

    If there’s a similarity between states, is suggests that some guy in Canberra is being even handed with the money buckets.

  5. Don’t forget that contractors are also parties to blame for cost blowouts. It’s not in a contractors best interest to deliver a product or meet deadlines on time and within cost. As soon as the project ends so too the money.

  6. I would be very interested to see how this all concluded (or has it) nearly two years later on.
    Lots of interesting commentary and some from folks with a view to the internal workings of the NSWPF

Comments are closed.