Data retention confusion could send “many” small ISPs broke, says Internet Australia


news Australia’s peak body representing Internet users has warned that “some, perhaps many” of Australia’s smaller Internet service providers could be forced out of business in the near term as a result of the lack of clarity over the Federal Government’s plans to reimburse ISPs for part of the cost of implementing its controversial data retention policy.

The Data Retention legislation was passed in March this year under the stewardship of Attorney-General George Brandis, but there has been a six month window where the Attorney-General’s Department has been working with the telecommunications industry to ensure compliance with its requirements of storing ‘metadata’ pertaining to emails, telephone calls, SMS messages and more, regarding all Australians’ telecommunications habits.

The Act is due to apply from October 13th, but the Attorney-General’s Department has so far failed to provide its proposed guidelines for the distribution of financial compensation to Internet Service Providers required to comply with the Act.

Computerworld reported last week that the funding model would be revealed in November, but the Communications Alliance — the body representing most of Australia’s telecommunications companies, including smaller providers as well as large telcos such as Telstra and Optus — has publicly criticised the Government for the lack of clarity around how the $131 million the Government has allocated to reimbursements will be divided up.

Meanwhile, iTnews has reported that ISPs granted certain exemptions under the scheme — for example, if they are unable to get their systems in order in time — are being silenced by the Government due to the risk that those seeking to avoid having their data retained (for example, criminals) may use their systems.

Over the weekend, Internet Australia added its voice to criticism of the Government over the issue, calling for “urgent clarification” on how reimbursements would take place.

Chief executive Laurie Patton warned that there was a “very real prospect of ISPs going out of business if they are not adequately reimbursed for the costs of implementation and the ongoing operating costs incurred in complying with this questionable law”.

Internet Australia has also recently called on new Prime Minister Malcolm Turnbull to reconsider the Data Retention Act. Overseas, there are moves away from this form of community surveillance amid concerns for personal privacy rights and limited evidence that data retention actually works in the fight against terrorism.

“Since the enactment of this legislation it has become clear that the Abbott Government significantly underestimated the costs that would be incurred by ISPs and failed to allocate sufficient funding in the Budget,” Patton added.

“There is a risk that some, perhaps many, of the smaller ISPs will simply go out of business as a result of this new law. This is especially unfortunate for rural Internet consumers who rely on local ISPs because they offer a specialised and personalised service”.

“At the very least the Government needs to commit to funding the costs incurred by ISPs if it insists on retaining this onerous law”, Patton concluded.

Australia’s telecommunications industry has been seeking urgent clarification on the costs of the scheme from the Government since at least March, when the nation’s major telcos penned an unprecedented joint letter to the Government on the issue.

Australia’s telecommunications industry has been seeking urgent clarity on this issue for at least six months. The lack of ability which the Attorney-General’s Department appears to have to provide that clarity speaks volumes about the viability of the Data Retention scheme as a whole.

Image credit: CeBIT, Creative Commons


  1. I sincerely hope that the bulk of ISP’s will simply ignore the laws in an act of civil disobedience – In all seriousness, if Telstra, Optus, TPG etc flatly refused to implement it, what could the gov really do?

    Surely the ISP’s could take it to court and show the lack of clarity etc and prove it is unworkable?

    • The major ISPs will not have a problem implementing the law. The smaller ones won’t have a budget for a lawsuit against the largest organisation in Australia … the Federal Government.

  2. I reckon “criminals” would sooner adopt a VPN and secure messaging services than change ISP…

  3. similar story on zdnet today made me laugh, it was frickin hilarious with IA claiming that DNS is required to be kept, FFS I wish journos would stop being so gullible, DNS is not required to be captured letalone stored

    during an INDUSTRY EVENT “During the event, it was clarified that any Australian carrier or carriage service provider that offers on-shore DNS to its customers would be required to store the metadata of the DNS requests”

    confirmed by who? the same people who got together to denounce DR, thats who.
    Anyone who takes the time to read the legislation can see very clearly it does not cover it.

    What rot… and they know it too!

    This industry has valid arguments over this BS law, but FFS keep it real and leave the deliberately misleading FUD (like DNS) out of it, it does no good to exaggerate and make shit up, it only leads to showing certain people as complete BS’ing unstrustworthy liars which causes extreme damage to the ISP industry when it comes to future input to Govts

  4. Feel like its complaining for complaints sake here. Storing most of this type of data is relatively trivial and low cost for most small businesses these days.

    All of our functions are already systemized through a veriaty of different methods. Retaining that data would just be a by product of that. Computerized phone systems and support systems and yes dns database records can be extracted easily with little work.

    My issue would be more how would one ensure they do comply. I find all the information a little confusing and being confused more by crap like this.

    • “Storing most of this type of data is relatively trivial and low cost for most small businesses these days.”

      That is not the information I have received from a number of those companies involved — most telcos consider this an unreasonable burden.

      • Is there a possibility of sharing what part of the Act changes are such a burden?
        If anyone, small ISP’s should have the least amount of issues applying the Data Retention changes.
        If they are reselling a service, most of the information they are obliged to store is billing information. They probably already have recording for this anyway and the amount of data needing to be stored for the customer base is already quite low.

  5. Based on radius detail log entries for start/stop/hrly_acct_pkts, 2 years worth of radius logs for an ISP with 1 million users could be stored in under 3TB compressed
    Email logs, under 1TB compressed.
    Account info, yep, most ISP databases keep closed accounts as well for a few years, either way, well under 1TB.

    A couple of raid large capacity sata disks in a 350 dollar desktop pc, and your done.

    Telco’s – those providing mobile voice will have much higher storage requirements of course because they have to keep all cell info as well as call info, but for your average ISP who even resells voice services, add up to another couple TB with billing info.

    Remember, you have to keep the data, they are not telling you how, as in they are not making you go out and buy a 100 TB EMC storage system (although Telstra, Optus, and vodafone might need it, the like sof exetel, tpg, iinet etc wont since they dont have all that extra cell id data). They are not making anyone have gold plated hardware with raid 10, with multiple backup locations.

    Most small to medium sized ISP’s can do this for little more than 1K, plus labour time to retrieve it when required.

  6. Actually, my calcs were a bit out for radius, I underestimated gzips abilities with duping all those radius attribute headers, you’ll need *substantially* less than 3 TB for radius…
    I’ve just compressed a 100M radius detail log file
    -rw-r–r– 1 root root 100M Sep 29 10:21 radact.log
    gzip shrinks that to
    -rw-r–r– 1 root root 768K Sep 29 10:21 radact.log.gz

    100M into 770K, for my example above with a million users you’ll need about 10TB UNcompressed for two years worth, which would likely fit into less than a GB given the above compression ratios. (I dont have 10TB uncompressed rad logs to test damit heh), So again adding in Email logs and account DB, an average ISP could get out of this very cheap indeed, less than a grand – if only they stopped listening to the FUD by the regular vocal opponents.

    • Nobby, you have the straight metrics worked out IF you didn’t have to respond to the several hundred thousand requests for data across the industry each year. There is no way you will be keeping these logs compressed all the time.

      So not only will you have storage issues and a serious need for grunt to go searching through the logs (unless you have a magically cheap database and indexing engine installed) but also the cost of the people required in processing the requests. And these can’t be Joe Monkeys off the street, there is a need to consider the evidential nature of the information as well as maintaining Cth security controls (ASD IMS etc) and those controls need to be applied to individuals as well as the system holding the information.

      While I see tons of FUD, I also see, from my experience actually planning these systems (Lawful Intercept), many people seriously under-estimate the operational and compliance costs required to maintain these systems appropriately and maintain the legally required integrity and response parameters.

      • @BruceH
        I don’t disagree, I have had to search through radius logs for s262 requests, which is why as I mentioned earlier + labour :) (thats not really all that time consuming anyway, the most 262’s I’ve done in one day was 3)

        The example I gave would be close to TPG (of old) size, even a medium size ISP like Exetel would be considerably less (since they have nowhere near one million users) than my example data storage requirements, so a small ISP/vISP could shit this in with little costs (again, yes, plus labour time).

        You of course don’t store radius or email server logs in one big archive, the archives would be compressed in date ranges, set a reasonable limit, logrotate weekly, monthly, or at $mb_limit, you have narrowed your time down, however as you know the smaller you keep the files, the more overall disk space required due to not utilising the maximum compression with dupes in content, so I’d never store radius in less than 100meg blocks, it’s all quite manageable.

        As for interceptions, thats entirely different since thats separate requirements and is of a targeted nature, not all encompassing.

  7. I don’t care about these attack on privacy for me, my family and all Australian citizens,
    I have VPNized all devices since since more than one year with a VPN provider (ActiVPN ).
    No interception for me, no attack on my privacy !

  8. I don’t care about this law because I use a VPN (ActiVPN )since one year , it’s more useful than bypass these data retention law, because you can watch every content, protect your privacy everywhere , secure your your computer against hacker …

  9. I’m curious as to why you guys think a VPN is better? There is no tracking of web history, there is no requirement to log or store DNS requests, proxy requests, or netflow data, so ISP’s wont know where youve been on the Net, so why bother. If your data is sought that stuff isnt part of the handover, and if you think your VPN is more secure from spooks, you havnt been following the Snowden files.

    (apart from the fact most of your VPS/VPN traffic will go via the U.S. or U.K. anyway and the spooks over there will have access and pass on your data :) )

Comments are closed.