POWERED DOWN: Flood takes out AAPT datacentre

By
Renai LeMay
-
18

The Queensland floods have claimed their first datacentre victim, with AAPT’s facility at 167 Eagle St in Brisbane confirmed to have powered down. And a number of other facilities may be about to follow it, as the statewide disaster continues to hit technology providers where it hurts.

AAPT this morning distributed a notice to customers stating the 167 Eagle St facility had gone offline. “Due to the flooding of the Brisbane region, AAPT will power down the equipment at the 167 Eagle St, Brisbane site as a safety requirement,” the telco said in a statement. Sources added the telco’s other facilities at 344 Queen St and up the road at 123 Eagle St were also expected to go down shortly.

A spokesperson for the telco this morning was unable to be contacted on the matter, and technicians at AAPT’s network operations centre declined to comment, saying they weren’t authorised to do so. In its statement, AAPT said it was currently investigating the issue and would provide further information once it became available. It apologised for the inconvenience caused.

It is unclear how large and how critical the AAPT facilities are to the running of the state’s telecommunications systems, although the 167 Eagle St facility is believed to be a major one, in the heart of Brisbane’s financial and legal district. And AAPT may not be the only company to suffer problems with its datacentres in the wake of the floods hitting the state capital. PIPE Networks may also shortly be facing power shortages at some of its Brisbane facilities.

The company has sent out a string of notes to clients this morning. At first it noted that it did not expect any problems, but later operations manager Matt Whitlock noted in an email that emergency services agencies and energy utility Energex had advised the company that it may be necessary to disconnect its 127 Creek St facility from grid power.

“Currently the concern is regarding the substation located in the lower basement of the building,” wrote Whitlock. He noted that if power was cut to the facility, PIPE’s UPS systems and generators would carry on delivering power to its DC1 and DC2 facilities, with its DC3 and DC4 facilities continuing to run on mains power.

“We have confirmed that there is sufficient fuel to operate the building for at least 24 hours,” he added.

However, later in the morning Whitlock noted the situation had changed, and PIPE had been advised that its DC1 and DC2 facilities at 127 Creek St would be “one of the last buildings to be disconnected from the grid by Energex” as it housed a critical Qld Government department coordinating the State Emergency Service response to the flood.

“Energex have advised that if the flooding reaches 15-20cm of standing water in the lower basement car park of 127 Creek St, all power (including the generators) will be cut to the building,” wrote Whitlock. “This will result in DC1 & 2 going dark until such time as Energex advise us that power can be safely restored to the building.” DC3 and DC4 won’t be affected, however.

And this could be a problem for PIPE, given the company’s previously mentioned 24 hour time frame.

“Worst-case scenarios puts this time at 48 hours,” said Whitlock of the potential outage from the grid power system. PIPE has now evacuated the 127 Creek St office — including its network operations centre, although it continues to monitor its network remotely. “As I’m sure you can understand, we are working in a very changeable climate and trying to react to the changes in advice,” wrote Whitlock.

Not everyone’s coming so close to the wire. Melbourne IT spokesperson Tony Smith said his company’s datacentre was in Wickham St in Fortitude Valley, on the edge of the central business district. It currently had full power from mains supply, and obviously generators for backup, he said.

“We’re going to keep monitoring the situation, and trying to get in touch with Energex, to see what their plans are. We’ll essentially be ready,” Smith said.

Image credit: Tom Jankowski, Creative Commons

Related posts:

  1. AAPT: Multiple Brisbane co-lo sites “under threat”
  2. AAPT chief backs Queensland facilities
  3. COO Yuile takes AAPT’s Broad reins
  4. Active Directory, AAPT the culprit in Ray White outage
  5. NBN Co picks Global Switch for datacentre

18 COMMENTS

  2. Maybe people should get their facts straight before publishing their articles for everyone to see – there isn’t an AAPT datacentre in that building (167 Eagle Street), only equipment that services customers who are located in that building who may lose service as a result of flooding.

    Just shows how ignorant some people are and how they would like to market their businesses…

  3. You might want to check your ‘sources’. The AAPT data centre is functional and not expected to go down at all.

    • Hi Larkon and SnakeDoctor,

      I have the original email issued by AAPT to customers (which states they will be powering down equipment at the site) and have contacted AAPT to ask for comment on the matter.

      I’ll be happy to update the story if and when they feel like clarifying the matter to their customers and to the press.

      Cheers,

      Renai

  4. AAPT has a capacity to house their own/and or customer equipment at 100 Wickham St in Fortitude Valley. This is the same building that also houses WebCentral/MelbourneIT

    Wickham St in Fortitude Valley has been listed as a location facing threat of flood
    http://www.couriermail.com.au/news/queensland/list-of-brisbane-flood-suburbs/story-e6freoof-1225985228833

    Based on this, I would strongly suggest that mains power will be shut down.
    Even though this facility has UPS and backup diesel generators, they will no doubt put the facility into shutdown due to significant threats not to mention the OH&S risks.

    The Eagle St location seems to only be a in building POP servicing building tenants and would therefore have no significant impact on the operation of the AAPT network either nationally or within QLD.

    PIPE Networks has two locations in QLD and both of these locations are listed as threatened by flooding or within a very short distance to flooding
    – 127 Creek St, Brisbane (PIPE DC 1 & 2)
    – 148 Brunswick St, Fortitude Valley (PIPE DC 3 & 4)

    If PIPE was to shut down both QLD locations, then this would have a very significant impact on PIPE’s network in QLD

  5. This site is not actually listed by AAPT anywhere as a customer facility, woudl suggest a network POP. Battery power will not hold the facility indefinitely.

    UPDATE:

    ——–
    AAPT Alert NOTIFICATION: SERVICE PROBLEM:

    Window : 13:57 AEDT Wednesday 12-Jan-2011
    Event : Outage

    Affected Services : All Services in QLD in all POPs

    Message:
    AAPT Reference – 1040404

    Initial Notification

    Location Affected – 123 Eagle St, Brisbane (Riverside Colo)

    Network at fault – Transmission

    Services Affected = IP, VOICE & DATA, INTERNET, ETHERNET

    Additional Information – Due to the flooding of the Brisbane region, mains power to the facility at the Riverside Centre site has been shut down as a safety requirement. Site is currently running on battery backup.

    ETR – TBA

    AAPT apologises for any inconvenience caused.

    AAPT Network Operations Centre
    1800 802 813

    ——–

    • Yeah, well, I’ve got a follow-up story coming detailing AAPT moving servers out of co-location facilities. There are problems aplenty up there.

  6. There are three critical requirements for ANY Data Centre, this is well known, even in EU and US as a must…

    They must never be located in:

    1 – An area prone to earthquakes
    2 – An area prone to flooding
    3 – An area near any critical infrastructure that may be deemed a target by terrorists

    AAPT and PIPE both *FAIL* in #2 and although close by, I deem only AAPT to also fail in #3, those who know that area know why I say this.

    Subsequently, ISP’s colocating their equipment with PIPE, are as much to blame in #2.

    Back in ’06 127 creek st had a full power outage due to a fire condition (which mandates NO generator operations), most ISP’s customers in Qld went byebyes as all relied upon PIPE, because everyone placed their eggs in one basket, only AAPT/Powertel and Veridas customers remained online IIRC because neither soley relied upon that facility for core operations.

  8. The company has redundancy however FaktorTel VOIP provider has a interconnect running directly into that POP. With out a second interconnect into their second site the DID’s are down.

    Our VOIP and they route the national numbers (Brisbane, Sydney, Melbourne) through that location.
    They will be working with AAPT to move the interconnect to their redundant site but its still down and has been for hours.

Comments are closed.