Active Directory, AAPT the culprit in Ray White outage


Real estate giant Ray White has clarified that it was actually an inability for its Active Directory installation to communicate with the rest of the world which has taken down its email systems this week during the Queensland floods — courtesy of an AAPT outage at its facility in the Riverside building in Brisbane.

Yesterday, Delimiter reported the Microsoft Exchange portion of the company’s national email platform had been rendered unavailable by the statewide disaster — with the Google Apps portion remaining up. The organisation is partway through migrating to the Google platform.

However, this morning Ray White director of IT & property management Ben White said the real culprit was actually Active Directory, which Ray White’s thousands of staff used to authenticate their identity with both Exchange and Gmail.

An AAPT link in the company’s head office in the Riverside building in Brisbane had gone down, he said, meaning the Active Directory installation could not communicate with its main datacentre in Fortitude Valley — let alone Google’s global systems hosted overseas.

The situation had arised from a quirk of network design history at Ray White, which had seen its telecommunications systems routed through its Brisbane office, with White citing one proprietary cable — “the last metre of connection between our servers and the internet” as the problem.

It’s not surprising the AAPT link in the building is down — the telco yesterday warned customers a number of its Brisbane sites were under threat, and there was speculation on the Australian Network Operators Group mailing list yesterday that the Riverside building’s generators had run out of fuel to supply the building with power, in the absence of mains power.

“They whole node is destroyed and they have to rebuild the whole node somewhere else,” said White today. “There’s no answer on how long it will take.”

The Active Directory installation, the executive said, was a legacy system which was used not only for authenticating users with Ray White’s email platforms — but also with other property systems. “It has tentacles into all sorts of things,” he said.

To resolve the situation — which is keeping Ray White staff nationally from accessing their email — White said the company’s technicians were currently working on three potential resolutions.

The first, he said, would see AAPT resolve its network issues, while the second would see Ray White switch to a different telco. Currently, the company is negotiating with Optus. Normally getting a new connection of this type would take between two and three weeks, White said, but Optus was promising a resolution within the next day or so.

The telco already had a connection close to Ray White’s facility, he said, which could be extended. “One of the nice things about a crisis, it brings out the best in people,” he said.

But the final option might be a more radical one for Microsoft’s software. White said his company was also considered an emergency switch from using Active Directory as its identity management software and letting staff login to Google Apps directly.

“We’re working on all three,” said White, noting Ray White had a team on each option. “We just have to make a decision on who gets there first.”

Image credit: Jono Haysom, Creative Commons