Photos: Rain floods NBN nodes in Bowral

89

news Heavy rain appears to have flooded Fibre to the Node infrastructure in the rural New South Wales town of Bowral, potentially causing a dangerous situation for local residents and causing outages with the local National Broadband Network.

The original version of the NBN as envisioned by the previous Labor Government called for most Australian premises to be covered by a full Fibre to the Premises rollout, with the remainder to be covered by satellite and fixed wireless technology.

However, the Coalition’s controversial Multi-Technology Mix instituted by Malcolm Turnbull as Communications Minister has seen the company switch to a technically inferior model re-using and upgrading the legacy copper (Fibre to the Node) and HFC cable networks owned by Telstra and Optus.

Most of the FTTP infrastructure being deployed by Labor is what is known as “passive” infrastructure. This infrastructure does not require electricity and can even survive being wet.

However, most of the FTTN infrastructure being deployed under the Coalition, in particular, is “active” infrastructure. It requires an active electrical power connection. The 30,000 FTTN cabinets, or ‘nodes’ currently being deployed around Australia also come with battery backup capability within the cabinet.

Such infrastructure is easily disabled by water ingress.

Delimiter has been sent the above photo by a local resident. It shows a street in the rural NSW town of Bowral, where the NBN company has been deploying FTTN infrastructure since at least November 2015.

The photo shows that the node appears to be at least several centimeters under water.

big

The resident who sent Delimiter this photo stated that the area had received 285mm of rain in 36 hours. The situation has caused havoc with local roads. However, Bowral is known to be a relatively ‘wet’ area in the Southern Highlands of NSW and often does receive substantial rainfall.

The resident also forwarded the following map which they have plotted of other similar FTTN ‘nodes’ throughout the Bowral area. It is not clear whether they, too, would be under water at this point.

bowral

It is not clear whether the power to this FTTN infrastructure has been disabled, or whether the water situation has created a potential risk whereby the power from the node could be passing into the water around it.

The situation in Bowral mimics a similar situation seen in Tasmania earlier this month. At the time, it was revealed that the NBN company had deployed a FTTN cabinet on the banks of the Tamar River in Tasmania, ignoring advice from local residents that the infrastructure would be sure to be breached by water during periods of excess rain.

As at the time of publication, the node was embedded in wet mud just centimeters from the water of the river.

The news comes as Delimiter has recently published a gallery of questionable locations for NBN nodes. The NBN company is currently deploying about 30,000 of the cabinets all around Australia.

opinion/analysis
Oops.

One does rather wonder if this node is still powered and functioning. If so, the NBN company might want to turn it off, before it electrocutes people walking through the water around it? Or is that not a problem with FTTN nodes? We’ll be contacting the NBN company this morning to ask.

Image credit: Supplied by reader

89 COMMENTS

  1. These would have to have some form of safety inbuilt along with being able to be partially submerged (likely why the batteries are at the bottom. I do not see the power companies proving them with active power lines otherwise (at ground level or below).

    Question becomes I guess what happens when the level hits the top of the batteries.

    • HAHAHAHAHAHAHAHAHA that shows how little you know about the actual infrastructure.. There is a live power feed into these cabinets. They are fully supplied with power, straight from the pole.

    • The cabinet is metal and would be earthed. A circuit breaker will trip in the event of any fault, e.g. as may be caused by flooding.
      Electrical safety is a serious business; the penalties for substandard equipment are very severe – including potential jail time for senior management at nbn or the power company if someone were hurt.

      • yup this is what I was intending to suggest. Having dealt with linesmen during storm clean ups you get to appreciate just how seriously they take things so there shouldn’t be anyway those cabinets can become live from the grid (because the power companies sure as heck would take it seriously and not allow NBN to cut corners).

        The likes of batteries and solar panels though always seem to cause grief (but again properly earthed it should™ be ok in those cabinets).

        • @sm Caught in the act of questioning the Borg; abuse descends.

          Of course power is isolated when a short detected. Squealers have no idea about power, like everything else.

          • The amusing thing is, the “abuse” you claim, is coming from someone who clearly supports FTTN and is wrong.

            Nobody who is a “squealer” as you put it, has said anything even close to abusive him.

        • Another quandary is when the water goes above the vent level and the cabinet fills I’m going to assume its going to stay ‘full’ of water to said vent level until someone opens the cabinet too.

    • And there’s the fundamental problem. The only way nbn can save money compared with a full fiber network is by cutting corners and building a significantly less reliable and much lower performing network – and even then, it’s only the initial capital costs that they are saving money on.
      It’s not just nodes that can be flooded; it’s ignoring people who have the misfortune to be a long way from the node; it’s doing the bare minimum of remediation of the last mile copper; it’s telling people that if they get 25 Mbps once in a 24 hour period then they have no grounds for complaint.

      • David it’s not only that but they won’t even test the connection to see if you can get the 25Mbps once a day

        • But…but….didn’t you write the plan for them? Bloody idiots can’t even read….the countries going to hell in a handbasket!

        • They “stuffed up” because they are under pressure to do it for the minimum cost. As is often the case, it ends up costing more.
          Same story with the whole MTM mess. We’ll do it on the cheap for 30 billion. Hang on, it’s harder than it looks, make that 40 billion. Oops; we were a bit optimistic there, lets say 50 billion.

        • “Cmon, the team that installed this node stuffed up. Nothing more.”

          Now once you accept that this is just the physical manifestation of the overwhelming sillyness of the whole FTTN deployment, you’ll be on track.

  2. There are some FTTP “nodes” (forget their name) in the marsden park area in NSW that may have flooded recently, I wish I remembered to go and check. Would love to know how long they work for when submerged in muddy water!

    • Passive equipment, should pretty much last even fully submerged (Someone on Whirlpool quoted a PDF from NBNCo about it).

    • Forever.
      If not subject to freeze-thaw cycles, underground passive optical infrastructure lasts until someone cuts it.

      • Well, maybe. A severe bushfire can destroy pretty much anything – even solid metal objects. Underground cables would be ok but I wouldn’t be betting on an above-ground cabinet surviving.

        • They wouldn’t be silly enough to deploy fiber above ground in bushfire prone areas…. or would they.

          You might have read that we have massive fires in California over recent years. FTTP connectivity is still available in the aftermath /shrugs

          There have been plenty of floods and tornados as well.

          EPB Chattanooga, TN has Gigabit FTTP citywide (100/100Mbps – 1Gpbps/1Gbps – 10Gbps/10Gbps). It also drives their SmartGrid technology. Tennessee is prone to tornadoes on a regular basis which causes power outages. However the energy utility can pinpoint the exact break in the power supply (IoT) and they only need to send out one truck and repair crew, rather than ten, to repair the fault and power is restored in less than an hour, rather than days. This saves millions of dollars to the economy and maintenance costs.

          SmartGrid technology is the killer gigabit app.

          • FttP should be ‘safer’ from fire simply due to requiring less cabinets but even then the fires are capable of some pretty insane temps and things being in a pit won’t guarantee the temps won’t melt or explode various things (I guess at that point there’s not homes to service the internet into though).

          • “(I guess at that point there’s not homes to service the internet into though).”

            House might be burnt, But it’s easy enough to stick an ONT onto the existing fiber and your back on in minutes.

            Like I said about wildfires in California but you might have missed it.

            You probably remember Hurricane Sandy. In New York there wasn’t a wireless tower left standing. Verizons HQ and vaults in Manhattan were flooded and thousands of miles of copper ruined which were immediately replaced with fiber to restore services. But Verizons FiOS survived – all-fiber FTTH network.

            All of these issues might be new to Australians and they might be ignorant of the facts which have been proven time and time again over a long period of time!

            http://www.theverge.com/2012/11/17/3655442/restoring-verizon-service-manhattan-hurricane-sandy

  3. David Attenborough: the node, a land creature, is struggling to stay afloat in its new aquatic habitat. Lonely and wet, the node can be dangerous to touch and produce a sharp zap of electricity. The sexy or boom boom dances should not be preformed nearby as the node can be lethal if turned on.

  4. The benefit here is that people are used to not getting full internet or working internet during winter, so FTTN doesn’t throw any ‘unusual’ spanners in the machine, like working internet during winter..
    /s

  5. I really want to know how much of the node survives, and how much needs to be completely replaced.

    • Throw your computer into a bath with the power switched on and see what happens.

      The fiber components might survive because they don’t require power. However the electronics such as the VDSL related hardware would be fried and batteries would probably be rooted!

  6. FYI Bowral nodes are not yet RFS. They got pushed from April to July and now are scheduled for the 26th August. My parents are moving back to part of this area. We lived there for 25 years.

    However, the entire population areas of the Southern Highlands were scheduled to get FTTP in Sept. 2016 during the final 3 year rollout update just before the 2013 election. Bowral has some of the worst copper in NSW too. Self professed by at least a half dozen Telstra techs who have visited my parents house in the last 4 years because of no dial tone, crackling on the line or unusable data streams. We got royally screwed with FTTN.

    This area is a perfect example, as many others are, of exactly why this MTM rollout has destroyed any long term gain of high speed, stable internet in regional areas.

  7. Maybe a sticker on EVERY NODE with suitable caption and picture of Turnbull… reminding everyone what a politically motivated stuff-up Turnbull has made of the NBN at taxpayers expense.

  8. Node Electricity = money to run
    Wonder what the cost per year to run is…
    As long as ”Mr Turnbull” is not effected why would he care

    • I believe it’s been worked out at between $3000-4000 a year per node. Plus battery replacements every 3-5 years. It works out to around $120m a year. But that’s FTTN right across the country. For the current model, I believe it’s approx. 60% of that.

    • Leaked IOP2.0 FTTN Review shows power costs p11:

      FY20 (@ completion) $31m / 24,544 (nodes) = $1,263 / node / annum.

      • Is that code for something Ricky?

        I think it’s …something….something…squeeelers?

      • Considering the TI line driver for VDSL2 uses, on average, 4W by itself (http://www.ti.com/general/docs/lit/getliterature.tsp?genericPartNumber=ths6214&fileType=pdf)

        Not including power dissipation, other card electronics, vectoring, backhaul backplane or battery charging (let alone battery replacement), the node uses (with 300 lines) 1200w. Even using best industry bulk power rates (around 12c per kWh) it’s 15c per hour. That’s $1314 per year. So clearly, those numbers are on the extremely optimistic side.

        As I said, most other users of VDSL2 (including AT&T) estimate between $3000-4000 per year per node. (Around 12-13w per line)

          • Fully populated 7330 max peak is ~1600w, most will be half.

            VDSL2 17a approx 1.5-2w per line.

            Nodes provisioned with spare capacity for copper serving area. Add expected take-up ~80%.

            IOP2.0 costing look pretty accurate.

  9. I don’t think anyone has given the cabinet engineers any credit. A quick google of some high-res photos will show just about every FTTN cabinet has rubber seals in the doors. i guess one can infer everything is okay until water gets to those vent levels, at which point unless there is an internal mechanism to seal the vent internally and then shutdown FTTN cabinet to save itself, you’re up the proverbial creek without a paddle. My searching shows no such thing in any photos, so my guess is once it gets to vent levels, it’s stuffed. I’m guessing disaster management worked out the water level at the vents, if water got to such a level, it woul be a 1 in 150 year flood etc. so it’s a calculated risk.

    Of course, a passive GPON wouldn’t need to worry about such issues in the first place, but at least in this case, looks like no issue until another foot of water.

    • True. The seals would avert the worst of the water. I’m not sure how watertight they are though in terms of floods. And don’t forget, what used to be a 1 in 150 year flood is now more likely 1 in 50 or 75 at the most. I lived in the Highlands for 25 years. We saw 1 flood in the first 20. Then 2 in the next 5. This makes 3 in 10 years. And worse than that, the water can come up the conduit from inside during flooding. That’s how most Telstra pits are flooded. And unless all pits are replaced and sealed (most definitely not happening by default), it can happen anywhere.

      Passive infrastructure will always be better for a country of extremes like ours. But unfortunately, a 3 year budget and election cycle takes precedence over 50 years and tens of billions in maintenance and repairs. I think I might move to Japan. 100 year business plans sounds good to me….

      • “don’t forget, what used to be a 1 in 150 year flood is now more likely 1 in 50 or 75 at the most.”
        Don’t forget, the party running this shambles actively purports climate change to be a myth, so any calculations along those lines would be against the party ideology.

    • A quick google of some high-res photos will show just about every FTTN cabinet has rubber seals in the doors.

      Oh, so they could connect Tasmania with them!!? How do they vent any gas from the batteries?

      Cool story bro!

    • Don’t forget that these nodes are being installed & maintained by the same people who seal the copper cable network joints, some are sealed better than others.

      The cables entering the node (telecom’s & electrical) enter from underground pits in conduits, they are ‘sealed’ at about the height of the concrete plinth the node stands on with a mastic material, water under any pressure will seep through these seals (if ants can get in, water has no problem).

      The node may isolate from power but any water/moisture combined with Direct Current on the copper communications terminations throughout the height of the node will have the same effect as it does now on poorly sealed cable joints, high resistance faults caused by corrosion, deteriorating over time.

      So are they going to go back and re-terminate every connection on new terminal blocks? The only solution is FTTP, no copper to give us the same problems we have currently in the network.

  10. Perspective.
    Just the beginnings of one of my concerns before the election, the other is the location of NBN HQ in the dockside precinct.
    The intense rain issue is going to get far worse, and it is happening world wide.
    For every 1C increase in temp it is an extra 7% atmospheric water carrying capacity, plus the heat provides updrafts pushing the moisture and clouds higher and building them up untilll it alll collapses in intense downpours,
    Japan may not be the best idea 7T
    https://robertscribbler.com/2016/06/24/bad-rains-fall-across-globe-700000-evacuated-in-kyushu-deluge-as-worst-flood-in-100-years-inundates-west-virginia/
    Bad Rains Fall Across Globe — 700,000 Evacuated in Kyushu Deluge as Worst Flood in 100 Years Inundates West Virginia

    In Kyushu, Japan on Friday, government officials urged 700,000 residents to evacuate as record heavy rains and severe flooding inundated the city for the fifth day in a row. Half a world away in West Virginia, another unpredicted record deluge dumped 8.2 inches of rain, washed out roads, cut off shopping malls, flushed burning homes down raging rivers, and left more than 14 people dead and hundreds more stranded.

    Individually, these events would be odd. But taken together with what are now scores of other extreme flooding events happening around the world in the space of just a few months and the context begins to look a lot like what scientists expected to happen due to human-forced climate change.

    • Being fair, intense ran this time of year in Japan isn’t strange at all. It IS rainy season, which is then followed by typhoon season, if there weren’t evacuation warning sent to places around here, I’d be confused.

  11. Into the vault: the operation to rescue Manhattan’s drowned internet
    Hurricane Sandy’s storm surge flooded Verizon’s downtown office, rendering miles of copper wiring useless
    http://www.theverge.com/2012/11/17/3655442/restoring-verizon-service-manhattan-hurricane-sandy

    Did Verizon replace the destroyed copper with copper? To this very day there are still areas on the east coast without telecommunications left in Sandy’s wake.

    Do you think US telcos such as AT&T and Verizon are stupid enough to repair their copper networks?

    You can perish that thought because in the last few years (since 2009) the effort is now concentrated on retiring the out-dated telephone network with the Federal Communications Commission (on behalf of US Congress) and President Obama’s blessing.

    Verizon are replacing their copper wireline & extending their footprint with FiOS and extending existing footprint. Likewise with AT&T and dozens of others. The only objectors to this initiative are not from politicians but from their competitors, mainly cable companies.

    It is quite funny hearing Turnbull and others bleating “no one needs a gigabit”. I know where this cliche comes from and it had nothing to do with conservative politics. This falsehood has been debunked more than 10 years ago!

    At the rate the IP Transition is moving, the ancient telephone network will cease to exist by 2020 as the U.S. rapidly move toward 100% IP communications networks over fiber for the sake of economic development, national security and the ravages of climate change.

      • That depends entirely on the 2016 Election results Saturday 2nd July like it did in 2013!

        Hope its not a repeat of Brexit

  12. Richard:

    The 7330 Might have a max peak of 1600w, but that is the 7330 only. Not the heat dissipation (fan), the battery charger or power conditioning. I work with all 3. There’s easily another 400-500w in those components. 2000W per node, plus redundancy provisioning, 2200W per node. That’s ~ $2400 per year/node. NOT including battery replacements.
    (8 Deep Cycle, high operating temp batteries/5 years- UPS equivalent batteries at 48V/5000Wh (enough for 16 hours) are wholesale $950 each- I’m certain NBNCo. could get them cheaper bulk)

    And if you do the calculations, they are not mostly half full. 24 500 nodes by 192 lines is 4.7 Million premises. Approx. 3.4 Million premises are being covered by FTTN in the MTM (at 80% uptake). So the majority will be feeding close to 140 lines.

    Or $60 million in power alone. Not far off 60%-70% of the $120 million I suggested initially.

    We could go around and around. The fact is, the power these nodes draw is not insignificant. Nor is it ludicrous when amortised across the network. My point being- power is not necessary for FTTP GPON infrastructure. That’s a saving of ~$75m a year. $750m over a decade (the maximum period before FTTN would need upgrading). Plus batteries at about the same cost over a decade. And that doesn’t cover replacements if/when nodes are submerged during floods, as is the case here. That’s total node replacement at ~$60k. Not necessary in FTTP. Assume a loss of 10% of nodes over a decade (not unreasonable in QLD and NSW) due to flooding, that’s another $1.5 billion. So before we’ve even started, it’s assumed the operating cost of FTTN will be $3B more over a decade due to active components.

    Does it pay for FTTP? No. But when you start adding the differences up (passive for FTTP compared to active upgrades for FTTN, reduced maintenance callouts from removal of copper etc.) It’s not hard to see that spending another $10-15B now makes perfect sense.

    • And that’s just the node component isn’t it Seven, that doesn’t include the copper maintenance, etc?

    • “And if you do the calculations, they are not mostly half full. ”

      192-lines is half capacity.

      “The fact is, the power these nodes draw is not insignificant.”

      True.

      “Nor is it ludicrous when amortised across the network.”

      True, power cost is actually insignificant.

      “My point being- power is not necessary for FTTP GPON infrastructure. That’s a saving of ~$75m a year. $750m over a decade (the maximum period before FTTN would need upgrading).”

      Direct opex of FTTN is acknowledge as more than FTTH. My calculations put it at ~$100 / customer more a year (demolishing Ferror’s analysis), ~$0.5b a year more.

      Whilst large its nothing when compared to earlier revenue (faster deployment), capturing majority of available revenue (80% choosing 25mbps or less), and lower capex (1/2).

      Provisioned CVC (1.05mbps / customer) has far more potential revenue growth than AVC (even utilising ADSL2+).

      “It’s not hard to see that spending another $10-15B now makes perfect sense.”

      Depends what you look at, numbers not supportive. The policy is a complete disaster, all models; NBNCO will never be profitable without additional revenue (industry levy) or massive write offs. Handing over another $15b or borrowed cash? Not for me.

      • “192-lines is half capacity”

        Irrelevant for power draw. The engineering specifications require a minimum of 3 racks power to be provided for in the DC-DC converters (485w x3), even if used single shelved.

        “True, power cost is actually insignificant.”

        Really? $70m a year is insignificant for a telecommunications company? No further comment.

        “Direct opex of FTTN is acknowledge as more than FTTH. My calculations put it at ~$100 / customer more a year (demolishing Ferror’s analysis), ~$0.5b a year more.”

        I don’t know what your calculations are based on. But that seems reasonable.

        “Whilst large its nothing when compared to earlier revenue (faster deployment), capturing majority of available revenue (80% choosing 25mbps or less), and lower capex (1/2).”

        FTTN hasn’t been faster. In fact, it is currently 8 months behind predictions (500k by right now actually). FTTP was scheduled to be finished in 2021 +- 1 year. Assuming the current delay begins to shorten once up to speed, as FTTP did, FTTN would be finished in 2022. Making the rest of this paragraph bunk.

        “Provisioned CVC (1.05mbps / customer) has far more potential revenue growth than AVC (even utilising ADSL2+).”

        This is an argument based on finances only. Irrelevant for a GBE who’s purpose is to provide long term telecommunications services for a country.

        “Depends what you look at, numbers not supportive. The policy is a complete disaster, all models; NBNCO will never be profitable without additional revenue (industry levy) or massive write offs. Handing over another $15b or borrowed cash? Not for me.”

        NBNCo. is not supposed to be profitable. Not in a business sense. If NBNCo. as an GBE exists until 2050 (not unreasonable re-Telstra’s 30 years as PMG/Telecom) it has 30 years to pay off any existing debts and upgrade infrastructure. If we meet again in 2035 and it is still $60 billion in debt, then perhaps I will soften my view on its’ success as a GBE. Basic arithmetic and CVC provisioning (as you’ve already pointed out) predict that outcome to be almost inconceivable.

    • Just to let everyone know, all new designs for at least the last 6 months are for 384 port cabinets the 192’s have gone the way of the Dodo! Hopefully the same will happen to FTTN.

Comments are closed.