news Australia’s telco industry will formally write to the Government to request a formal delay in enforcing the controversial new data retention legislation, while Internet service providers around Australia struggle to meet the October 13 deadline for enacting its requirements within their operations.
The data retention legislation passed in late March. Since that time, Australia’s broadband industry has been struggling to understand how it will apply to their operations. The issue is particularly fraught for smaller providers who may not have already been storing much of their customers’ telecommunications data, and who may not have spare funding to implement the bill’s requirements.
And the timeframes for the implementation are tight by industry standards. The legislation formally comes into effect on 13 October. ISPs who are not able to comply with the regime at this point are able to submit an implementation plan to the Attorney-General’s Department, but the soft deadline for this plan is 13 August — just two weeks away.
Speaking at the ACOMM Awards earlier this month, Communications Alliance John Stanton told members of the Government listening that there was “angst and uncertainty” gripping many carriers and service providers (CSPs) (and not only the hundreds of small ones) across Australia as they saw the August 13 effective deadline for lodging implementation plans approach, and didn’t know how they would be able to meet it.
“Time is short, the task is complex, the myriad uncertainties and legislative ambiguities are being gradually run to ground (but many still remain) and there is no advice on how and to what extent individual CSPs will be compensated for the capital costs they will need to commit to,” said Stanton at the time.
“Minister – industry seeks from Government more time to comply – or at least an unequivocal assurance that CSPs that are genuinely trying to complete [implementation plans] during the 6 months post October 13 will not find themselves being penalised or prosecuted.”
There is a potential avenue which could allow the telco industry to avoid being prosecuted for not meeting its requirements under the scheme.
It emerged yesterday that the Communications Alliance was planning to submit a formal request to the Government seeking a six month window of ‘regulatory forebearance’ where CSPs who were submitting data retention implementation plans would not be prosecuted by the Government for meeting their obligations, as long as they could show they were planning to.
The issue is being actively discussed within the membership of the Communications Alliance and Internet Australia, with smaller companies being particularly worried about the issue.
The Government has not yet made any public response to the telco industry’s ongoing overtures on the issue.
opinion/analysis
I have a number of different views on what’s happening here.
On the one hand, of course you would expect the telco industry to be seeking more time to comply with the data retention bill. The legislation is complex, confusing and extremely detailed. Every telco will struggle to ensure they fully comply with its requirements. Smaller telcos, with little in the way of legal resources, will particularly struggle. You can understand their particular frustration, as this legislation will be just another way in which large telcos like Telstra and Optus — who were already storing most of the data required under the bill and have huge legal teams — will retain an advantage over smaller companies.
On the other hand, the data retention legislation has been out there for quite some time now. This has been coming for years, the legislation itself has been public for quite a long time, and it has been pretty obvious that it was likely to pass. The telco industry should have been planning for this for years now.
Should the Government show a bit of regulatory forebearance and cut telcos a legal break while they implement the legislation? Yes, certainly it should, given the onerous nature of the legislation, and I believe it will. I don’t think either the Government or the Australian Communications and Media Authority are going to be out there with a big stick suing telcos for not complying, at least in the first few years.
In a wider sense, of course, what we’re seeing here is the consequences of successive governments to properly consult on this issue. The fact that the telco industry is struggling to understand the data retention legislation at this point — well after it passed Parliament — is indication that the legislation is too complex and overly burdensome on the industry. Let’s hope the Government at least does a good job of collaborating with the industry to help them implement it. It certainly owes the industry that much.
Renai,
I think you are being too generous to the government and agencies like the AGD’s who are tasked with ensuring implementation and correct governance of data retention.
To be frank, I simply believe that in their eyes wide open, mouth’s agape, foaming at the mouth, almost cult like determination in passing the legislation – they completely ignored the very real concerns surrounding implementation, how vague the legislation is (Brandis on many occasions spoke about how proud he was of how technologically neutral it was!!) and how unworkable it will be. It’s a sledgehammer in a watchmakers’ workshop and the blame lays with the two major parties and the agencies who begged for it to be passed.
I don’t think it will EVER work properly and I do think that it is only a matter of time before it is used incorrectly and causes embarrassment for the agencies themselves, the government of the day or an ISP who the government will go after with EVERYTHING to save their own face.
I believe you’re missing the word ‘failing’ after ‘successive governments’ in the last paragraph: ‘the consequences of successive governments to properly consult on this issue’
As a professional working in telco I work daily with “metadata” – although that fancy name is just another way of saying ‘billing records’.
I don’t think anyone, even in the Telco’s themselves understand how bloody hard it is to find records. I used to be naive and think that surely my employers would have robust and functioning usage viewing systems. Nup. For the most part I’ve had to build my own.
Take a simple questions. Say how about that 5GB of data that you downloaded say on 20/07/2015 via your ADSL. That is a really complex and difficult question to answer. I’ve been doing this for 20 years and I can tell you know that there are a handful of people at any particular telco who can answer that question.
The idiots in the liberal party must have looked at and thought golly they can just pull a report without any thought of what the raw records look like.
Well unlike 99% of most telco employees I’m one the few people who have ever looked at raw records cut on switches and big IP collectors. The amount of clean up (mediation) work that goes into making these records term into lines on a invoice is mind bogglingly complex.
Raw records contain a huge amount of information for every unit of usage. A phone call (CDR), a 5 minute log of source/destination IP data traffic, mobile phone switch records, 13/1300/1800 records and so on. They contain geography of the source and destination. For records that traverse multiple switches, or AGVCs there also sorts of additional information. Geolocation data is captured. start times, end-times and dates.For long duration records they get split.
Now all of that data has to be cut into a record that the billing system can rate, bill/charge and invoice. So all of that is feed through the mediation system. The mediation team at a telco is probably the most unheard of team in the entire company. Seriously probably only a handful of people know of their existence.
So lets take the 5GB data scenario. The cops want to see what that usage was.
So the map is like this
Core Router > Collector > Mediation > Billing
but sometimes there is also > Archive system/IP usage server.
Now in the major telcos as I know it, which why I’m really confused about this law, they are already taking mediation records and giving them to the Government. I know this because I was told by the people who are doing it that they are doing it. Voice records for millions of calls are being sent to the government (which is dodgy because 283 of the Telco Act says those records can only be given in relation to a crime).
Anyway we’d log into the archive/IP usage log server and ask it for the records. These servers would traditionally only have enough storage for 6 months worth of data and even then by the 5th month the software is design to delete/degrade the logs so they are less dense compared to when they were first written to the server.
Worse because of the constant read/write cycle and the fact that no one cares about the server they tend to lose information and records simply because they’re unreliable and need to be rebooted. Especially when some new person sends in a requests for 50GB of records
But lets pretend we pull a log of every 5 minutes of data flow on the 20/7/2015. These reports don’t resolve the URL. Just SOURCE and DESTINATION IP. PORT. PROTOCOL and a few other bits and pieces. Usually designated about whether the traffic was within the intranet or internet. Also flow indicators.
Now think of it. There could be over 2 million rows in this CSV file. now up until recently how would we have even opened the bloody spreadsheet? Most telcos are running Office 2003 and outside of a hardcore group of tech hands most staff have trouble barely know how to open excel let alone use it.
Open it in Access? Please most people barely know its the name of a program.
But lets pretended we’ve opened it. What next. Usually there is only say a couple of people who even know of these logs and they’re so busy on business critical stuff you wouldn’t pull it off to help you or worse they’ve left and the documentation for the system is a decade old.
So lets pretend you know how to read the logs. You run a series of sorts/filters/aggregation steps over the data to divorce the useless (like the millions of ICMP packets your system sends on a daily basis). Then you’d look at the top three IP addresses that your system communicated with.
Now due to layer 4 switching a single IP can be hosting dozens of websites/services. Its the payload in the packet that tells you what the URL that they were after. These systems don’t capture it and in some ways unless the core routers can transcode the payload into its log files I really don’t think the telcos have the capability to capture it en-mass.
Now in 99% of cases I have found the following with large amounts of usage.
1. Akamai
2. Microsoft updates (watch out for month of July – be a lottttta of bill disputes after the Win10 updates screw up). especially for all those SME that pay excess per MB rates
3. Some data hoster
4. mail server (looping errors).
5. porn (if your lucky that the IP resolves back to an identifiable host)
So imagine this problem but across Mobile, ADSL, Cable, Ethernet, ATM, Dialup data products. Some, especially mobile don’t even bother capturing logs of IP data. Others its almost impossible to get the logs (because finding who/where/how to run them is next to impossible.
Then imagine this problem for voice, but over Fixed Line, VoIP, ISDN and Mobile. Raw switch records are extremely incomprehensible.
And we haven’t gotten into all the stuff like mobile geolocation data or the huge amount of switch data that gets deleted at the mediation process.
What the government is asking for is for the capture of insanely massive amounts of data. Sickeningly large amounts of data that will never ever be looked at by Law Enforcement.
The only systems that collects massive amounts of data are A telco’s billing systems. The billing systems don’t want to know the payload of the communications. Just the geography of the source and destination.
And what this means is a massive redesign of the software that runs the processes/systems that are used for billing capture.
See that means altering the mediation process and producing logs after mediation. All of this is increadibly dangerous. Changing production systems that are at the heart of a telco for such a stupid purpose is frought with massive risk.
Worse the software is not design to capture the payload of data packets (URL etc). The cost of rewriting the core routers and collectors to capture this information, in real time would be absolutely massive.
And that’s at the big four service provider / carrier level. Imagine at the Business Service Provider end where they run a few VPN servers, asterisk server and so on. They have no capability whatsoever to capture any of this data. They are at the mercy of their equipment vendors having the feature to capture it. Just to get say a 10 or 20 TB raid array installed will be costly.
About $2000 a month for the rack. About $10k upfront for the server. The interconnect to their other equipment/racks would thousands per month to the bill. Remember they have no infrastructure onsite with them. Its all in their co-lo. So everything is charged for.
And this doesn’t get into the software, architecture work and project management/implementation costs. For a big telco I’d estimate we’re looking at least 7 figures. Probably $20-50m at least. Think about a billing system because in a way this is what the government wants. AAPT spent what $100m on a billing system about 10 years ago. Optus bought Kenan and spent a good couple hundred million on it and Telstra has been talking about getting rid of their main billing system for decades….and because of delays is now estimated to be in the billions.
Now of course if the cops want to intercept someone’s data in real time there are tools and resources for them to capture everything. And that’s how it should work. The cops get a warrant, intercept is requested via the Interception Act and bingo the cops get the data.
What the government has asked for just utterly insane
I don’t think the government has given much thought to it but there is just so much machine to machine data out there. And with predictions of billions of connected devices in just 6 years time the cost of this harebrained scheme is going to spiral out of control.
Finally the worst part of this entire scheme is that Only like 0.0000000000000001% of all that data will ever be looked at by a law enforcement official.
This is one of the most useless ideas to have ever emerged from government. Personally I think Brandis is punishing the telcos for putting up such a huge fight with Hollywood and Village Roadshow. For refusing to accept the cost of the notice system.
Makes sense seeing how petty liberals are.
interesting read 1984, has anyone out there got a letter about downloading Dallas Buyers Club movie yet ???
No letter MikeL coz the judge hasn’t approved the draft.
Zdnet released a copy of the draft. Terrible business. DBC have utterly trapped the judge. Such a poor judgement renders the rights of downloaders just ripped to shreds.
I plan, if I get the letter, to launch a kickstarter asking every Australian downloader to contribute $50.
Five million Australian by such a contribution would create a fund of mythical proportions.
I would do three things.
1. Create a legal team to fight all attempts by DBC to sue
2. Fund a variety of offensive actions that would see the utter destruction of DBC. When we destroy them we’ll blast a hole into Hollywood’s ego that they wont come within a 1,000 clicks of Australia ever again.
3. Create a fourth force in Australian politics, not named the pirate party, that will buy the balance of power in the senate.
It can be done with enough money. Look at Palmer etc etc.
With political power we will rewrite Australia’s copyright and media laws ensuring that exclusive deals by fox, TCN 9, Seven and ten would never be allowed again.
I would tariff the crap out of hardware and software companies that fail to make their RRP equivalent to their local markets.
I would happily throw $50 your way for a kickstarter of this type.
Thanks very much for posting that 1984. Obviously you’re pretty frustrated by the legislation/requirements but still put forward a very interesting and cogent case.
I think this would actually make a good article on its own here in delimiter – what do you think Renai?
Thanks for a thought-provoking insight into the metadata problem, 1984.
Looks like as real nightmare!
It seems almost as though it would be cheaper to intercept all the data in real time and extract the required info into a brand new system, although 1984’s post makes it clear that this wouldn’t be all that easy either.
Mind you, one way to help fund the data retention system would be if the government legislated that this retained data would be legally unusable for piracy investigations and prosecutions UNLESS the anti-piracy organizations paid a very significant portion of the system cost. You would get the anti-piracy organizations coughing up cash and/or screaming blue murder. Either way the public would get to see who the REAL BENEFICIARIES of this data retention are!
I wonder if it would be legal and cheaper for the ISPs to store (and surrender when required) the metadata in a relatively raw form, leaving much of the mediation mentioned by 1984 to the requesting agencies. After all, analysis of the metadata for security purposes is the job of the security agencies, NOT the ISPs. This could also reduce the total cost of the system, as instead of umpteen ISPs each sourcing mediation software, the security agencies could use a pooled processing centre running just one mediation software package.
Thanks Hicks.
This is the really screwed up thing. I was told by senior engineers at multiple telcos that the intelligence agencies were given core access, years ago. Presumably they meant Australian Signals Directorate but they just would say “oh ASIO and those guys”.
So if the federal agencies already get voice record enmass and have core access then this legislation has nothing to do with national security because that’s being meet.
Of course the government claim that local and state law enforcement need the records stored to win court cases against bad guys.
Well in the debate they failed to supply comprehensive lists of cases/prosecutions that failed due to the lack of historical usage record.
I listened to a great deal of question time and apart from sounded like utter dullards they didn’t give any detail except for really high level hyperbole about paedophiles and terrorists.
Like you said the obvious option would have been for the government to have had all the teclos connect to a massive government run exobyte sized data centre.
But like I said the government doesn’t need that coz they already have core access to the networks.
I reckon the government is doing this to punish the telcos for daring to stand-up against Hollywood and the three strikes rubbish.
Maybe the data retention data isn’t needed for national security, but it will sure come in handy for anti-piracy!
1984s comments just proves how idiotic and naive the Abbott Government is about how the internet works. They just seem to think you can just push a button and hey presto everyone internet habits , all 25 million persons daily browsing habits and other internet actions can be recorded down and delivered on a hard drive. Tony Abbott has proven to be a lame duck PM. The legislation is inept.
Brandis is a fool.
Comments are closed.