The data center migration plan derailed from the leaking roof • The Register

on call If it’s Friday it must be time for another On Call episode, The registerweekly column celebrating readers’ escapes from ugly scratches.

This week, he meets “Danny,” who started a new job with an automotive company in 2016 and was immediately commissioned to renovate the organization’s data center.

“The project included replacing four power distribution units and moving all power and data cables from under the floor to new overhead walkways,” explained Danny. This meant eliminating the raised ceiling to make room for the cable trays, as well as replacing the old fluorescent lighting with the new LED kit.

All that work required an expansion of the fire suppression system and raising the dry sprinkler heads from about 18 inches above the racks to the ceiling itself.

Just to make sure the job was particularly difficult, it had to be done while the data center kept running.

The day the racks were covered with protective foil so the ceiling could lower without crushing the servers or clogging the fans with dust.

But things quickly turned pear-shaped as demolition of the ceiling panels revealed a lattice of cinder block above the datacenter, rather than the expected pleasantly solid concrete slab.

The situation quickly worsened when the site facilities manager arrived and asked for an explanation for the mess.

The director accepted Danny’s explanation, but then insisted on restoring the recently destroyed ceiling.

“It is the only thing that protects the racks from ceiling leaks in the event of flooding of the upper floors,” Danny was told.

“Leaking from the ceiling?” Danny asked, incredibly, before being told that several times in the preceding decades the irrigation systems had been activated accidentally. The resulting streams of water found several holes and cracks in the concrete block ceiling, raining water on the equipment shelves and damaging the devices.

The site facilities director explained that water sensors had been installed to detect such floods … but the organization had come to the assumption that water would accumulate above the ceiling tiles, which would provide an enough barrier to prevent the datacenter from getting seriously drenched.

This news “caused a sudden and immediate shutdown of the project as we searched for and found contractors who specialized in sealing any crossings in a ceiling to make it watertight,” said Danny. The register.

Three months and several hundred thousand dollars later, the roof was sealed, procedures were written to deal with the leaks, and Danny was able to finish the job.

“The rest of the project was completed without significant incidents and from then on, any new visitors from the IT leadership were taken to see the refurbished data center and allowed to see the uniquely sealed cinder block ceiling. “concluded Danny.

Have your renovations ever been attacked by reality? If so, let us know with an email at On Call and your story may be featured here on a future Friday. ®