Massive Flooding Damages Several NYC Data Centers - Data Center Knowledge

Flooding from Hurricane Sandy has hobbled two data center buildings in Lower Manhattan, taking out diesel fuel pumps used to refuel generators.A third building at 121 Varick is also reported to be without power. There were also reports of outages for some tenants at a major data hub at 111 8th Avenue, and many other New York area facilities were running on generator power amid widespread utility outages.

Both Internap and Peer 1 are struggling to continue operations at 75 Broad Street after basement-level flooding disabled critical diesel fuel pumps, leaving the providers now way refuel generators on mezzanine floors.

Customers of Datagram were knocked offline Monday evening as water flooded the basement of its building at 33 Whitehall, knocking out high-traffic sites including Gawker, Gizmodo, Buzzfeed and Mediate.

Both 75 Broad and 33 Whitehall were located in the “Zone A” flood zone, which was inundated by a storm surge of more than 13 feet as Sandy’s landfall in New Jersey coincided with high tide in lower Manhattan.

There were also multiple reports of downtime for some tenants at 111 8th Avenue, a major communications hub which is owned by Google. Some reports attributed the outages to network problems, while others said that an electrical panel had failed while parts of the building were switching to generator power.

Meanwhile, several data centers in northern New Jersey were reported to be running on generator power, as PSE&G and other utilities suffered widespread outages.

Datagram

Datagram hosts a number of high-profile blogs, including the Gawker network, which remains offline Tuesday morning. “Gawker sites down after power cut off at Datagram, our data center down on Whitehall St,” tweeted Gawker publisher Nick Denton. “Backup power didn’t kick in fast enough.”

In fact, the problems at 33 Whitehall went beyond the speed of its power response. Other tenants reported that the basement had been flooded, knocking out diesel fuel pumps that support generators.

Buzzfeed said it was able to work with content delivery provider Akamai to restore a static version of its web site. It posted a short story about the problems experienced by Datagram, including a text sent to Buzzfeed from a Datagram official. “Basement flooded, fuel pump off line â€" we got people working on it now. 5 feet of water now,” the official wrote.
It wasn’t clear when Datagram would be able to restore service.

75 Broad

Diesel pumps were also the focus of problems at 75 Broad, where basement levels experienced major flooding that forced Internap and Peer 1 to shut down their operations at the facility.

“Around 9 PM the storm surge, combined with high tide, caused flooding to many portions of lower Manhattan, including in and around our data center at 75 Broad,” Internap said in a  blog post. “At the peak there was three feet of flood water in the lobby at the site, causing both basement floors to be inaccessible. As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down. The system continued to run until all fuel within the secondary feeder tanks were exhausted and our facility lost power.”

The Internap facility lost power at about 11:45 a.m. this morning.

“We are working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation,” Internap said. “It is unclear how long it will take ConEd to restore utility power to the site, but we are preparing for the possibility of remaining on generator power for many days.”

UPDATE: “We have coordinated fuel deliveries and pumps, as well as engineers to fabricate pipe to bring the fuel directly to the generators on the mezzanine level,” Internap reported this afternoon. “There is currently no ETA available for the completion of these activities, but we are working through all options to bring the UPS infrastructure back online as soon as possible. We will continue to work toward resolution and
provide status updates with any new information.”

After attempting to arrange temporary diesel fuel storage and pumps, Peer 1 Hosting said early Monday that it will “implement a controlled shutdown of NY Data Center at 10:45 ET.” But as of 12:30 p.m., the site was still online, and the company said it was awaiting a diesel delivery.

“A fuel delivery is en route to the facility but due to the various street closures and general road conditions caused by Hurricane Sandy, (and) the expected ETA of the fuel truck is approximately 1 hour, Peer 1 said on its support forum. “If the fuel delivery is not able to be made before 2:00PM EST, PEER 1 will need to initiate a suspension of all services at our affected facility beginning approximately at 2:15PM EST.”

Peer 1 has not provided any updates since the 2:15 deadline.

Atlantic Metro / 121 Varick

On Tuesday morning Atlantic Metro reported that it was experiencing “wide spread network and facility power outages due to Sandy and its flooding.” Steadfast Networks reports that power has failed at 121 Varick Street, a data center managed by Atlantic Metro (via The WHIR).

“We have just been informed that due to an auxiliary electrical failure, power systems have failed at our nyc16 facility at 121 Varick St in New York City,” Steadfast reported on its support site. “All New York servers and services are now offline and we have no ETA for service restoration. We will provide updates as we have them.”

111 8th Avenue

There were multiple reports on Twitter of outages for some tenants at 111 8th Avenue, one of Manhattan’s largest buildings and a key hub for Internet traffic. Companies affected included Equinix, Voxel/Internap and XO Communications.

“NY9 experienced a failed generator that impacted service to several customers,” Equinix said on it blog, referring to its facility at 111 8th Avenue. “We made repairs and service was returned this morning. The site remains on generator power.”

111 Eighth Avenue is among the world’s most wired buildings. It was originally built as the Port Authority Commerce Building in 1932, and was redeveloped for telecom use by Taconic in the late 90s.

Telehouse reported that its facility at 25 Broadway in lower Manhattan has remained online throughout Sandy and its aftermath, as have its data center in Chelsea and at the Staten Island Teleport. The colocation provider said it switched to generator power proactively ahead of Con Edison’s utility power shutdown. “Since that time, these facilities have remained operational without any critical system (power, cooling) outages,” Telehouse reported. ” These facilities will remain fully staffed and off-grid (independent power generation mode), with periodic refueling until utility power is restored and stable.

“During this period, both Manhattan sites (25 Broadway, Chelsea) were operationally unaffected by local flooding,” the company added. “Building engineers and equipment, at both Manhattan facilities, are engaged in mitigating this threat. The Teleport facility, on Staten Island, had no flood threat during this time.”

We will continue to update as we get more news. You can email us at news@datacenterknowledge.com with news and tips.

I used to work in New York City quite a few years ago now. That’s just incredible how the water flooded so many places. Obviously people never envisioned the possibility their fuel pumps becoming submerged.

I’m sure there’s going to be a lot of learning opportunities from this storm.

Cheers!

Patriot Clean Fuel has emergency teams on standy-by for emergency fuel related issues including full reconditioning, water sepearation and removal, filtration and fuel procurement. We have mission critical partners in this area of Manhattan.