Rebuilding Broadband Infrastructure following Katrina – Lessons LearnedGreg Bicket, Vice President and Regional ManagerMichael Latino, Vice President of EngineeringCox Communications Louisiana

Agenda Pre-storm Preparations Event Monitoring & Damage

Assessment Developing the Restoration Plan Challenges to Restoration Hardening Network Infrastructure Summary

Pre-Storm Preparations Friday, August 26th, 2006

NHC guidance indicated that Katrina would make landfall Monday morning on the Panhandle of Florida.

Knowing we would feel some impact from the storm, we initiated our Business Continuity Plan.– Securing Facilities– Establishing Communication

resources including regularly scheduled conference bridges and our Employee Storm Line.

– Dispatch our Safety/Risk Manager to the Office of Emergency Preparedness.

Pre-Storm Preparations By Saturday morning, the

projected path indicated New Orleans would take a direct hit.

Senior leadership met and decided to cease normal business operations as of 1pm Saturday.– Shutdown of all offices.– Cancellation of all work orders.– Release of all personnel

Immediately we began full implementation of our BCP including evacuation of our first responder teams.

First Responders were to arrive at designated evacuation locations by noon Sunday.

Pre-storm Preparations Sunday, as Katrina

approaches, we establish our remote monitoring center in Baton Rouge.– Along with our Atlanta

Network Operations Center (NOC), we monitor the health of our network as the storm approaches.

First Responders with their families, vehicles and equipment arrive in Baton Rouge and teams begin restoration planning.

Event Monitoring & Damage Assessment

Utilizing several tools, we are able to remotely diagnose and survey the health of our network.

By constantly monitoring the sequence of events and correlation of the data, we are able to determine the most likely cause of outages.– Power outages– Fiber Breaks– Flooding

System status is monitored non-stop and reported on two hour intervals.

Event Monitoring & Damage Assessment

Immediately after landfall, initial damage assessment was conducted by a system flyover.– Information was used to

organize the first wave of on-the-ground damage assessment.

– Flyover assessment indicated that aerial plant had survived the storm fairly well with initial assessment of 10-15% of aerial plant suffering wind damage.

We also utilized satellite imagery to determine the level of flooding and extent of wind damage to network facilities.

•Add Flyover Picture

Event Monitoring & Damage Assessment

On site damage assessments began Wednesday morning.– Because of flooding, initial

assessment was limited to St. Charles and Jefferson parishes.

– Damage was noted on maps and this information was used to dispatch repair teams beginning Thursday.

Contractor resources were organized to augment Cox resources to expedite recovery. – Difficulty in securing temporary

housing created challenges in securing outside resources.

A war room was created to track all damages and restoration activity utilizing system maps.

We were unable to access the most heavily flooded areas for almost 4 weeks to perform on the ground damage assessments.– Flood depth maps were used to

estimate damage in these areas.

Event Monitoring & Damage Assessment

Results of Initial Damage Assessments:– The entire network was

virtually dark.– Two network facilities and

3 service centers flooded.– Approximately 10%-15%

of aerial plant sustained wind damage.

– Over 450 miles of underground plant in flooded areas.

– 40% of drop network damaged or downed.

– 70+ flooded vehicles

Developing the Restoration Plan

The information collected from the damage assessments was combined with parish information on repopulation.– Focus was on rebuilding

the areas repopulating first.

– Prioritization on restoring critical governmental and commercial customers.

Teams were focused on restoring main hubs and fiber distribution network.– As commercial power

outages continued, we established regular refueling intervals for generators.

To expedite service restoration, crews were established to replace/reconnect drops street by street.

Challenges to Restoration Coordination with other utilities proved to be difficult.

– The number of additional contract resources made it difficult to coordinate the activities and led to unnecessary damages in the process of restoration.

Safety Issues– Concerns for health and security issues delayed response to some areas of

town.– All of our First Responders and their families were given inoculations for

Tetanus, Hepatitis A & B– Corporate Security personnel were brought in to accompany our field teams

and secure of facilities. Debris Removal/Continuing Damages

– As storm debris was collected, pedestals were destroyed by front-end loaders.

– Over-loaded debris trucks ripped down overhead trunk, feeder and drops. FEMA

– 2500 Gallons of fuel confiscated by FEMA in route from Florida.– Cox Communications was not recognized nationally as a telecommunications

provider.– Identifying points of contacts for FEMA controlled processes (Debris

removal, Trailer placement, etc) proved futile.

Hardening Network Infrastructure

All points of failure were analyzed and additional redundancy and protection were implemented to improve network survivability– Additional SS7

connectivity to maintain connectivity to the PSTN.

– Reallocation of network infrastructure to locations proven to be less exposed to flooding.

– Utilize Cox’s national backbone to provide backup of services.

Coordinated with other utilities to synchronize priorities for outside plant restoration and minimize additional damages.

Summary Having a thorough, practiced

Business Continuity plan will protect your people and enable a quick recovery.

Expect the unexpected. Plan for multiple layers of redundancy with facilities and vendors.

The time to create the tools needed to track and manage restoration progress is now!

Know your restoration priorities. Ensure your teams know them too.

Take care of your teams. Your network restoration depends on them.