Crisis Management The New Role of the CIO/CTOs & IT staff

The City of New York

• Resident population of over 8 million; daytime population of 10 million

• Over 350,000 City employees, 300,000 retirees

• New York City Government includes 5 counties

• The 1 million student school system reports to the Mayor

• Annual budget exceeds $54 billion dollars

– Trails only the Federal Government, and the states of California & New York

– If New York City was a private sector corporation, it would be in the Top 30 of the Fortune 500 companies

• Over 120 agencies, offices, and organizations make up “The City”

New York City’s Public Safety Record

In December 2005, out of the nation's 10 and 25 largest cities, New York City ranked the safest with the lowest overall crime rate.

Among cities with 100,000 residents or more, New York City ranked 211th out of 227 cities.

New York City’s Major Incidents - the past 5 years

1. Terrorist Attack on World Trade Center – September 11th 2001

2. Crash of Flight 587 - November 12th, 2001

3. Northeastern U. S. Blackout – August 14th & 15th, 2003

4. Staten Island Ferry crash – October 15th , 2003

5. One and a half hour partial 911 outage – March 26th, 2004

6. Major Transit Strike - December 20th – 23rd 2005

7. Blizzard, largest recorded snowfall in NYC- February 12th 2006

NYC’s Major Incidents –Effect on Telecomm (1)

• Land line & cellular phone utilization peaks, exceeding network capacities.

–Phone service spotty and extremely limited.

–Challenge for critical communication for command and control.

– 911 may not be accessible.

–Government Emergency Telephone Service (GETS) may be inaccessible via cell phone.

NYC’s Major Incidents –Effect on Telecomm (2)

• A prolonged power outage, further erosion of phone service occurs.

– Cell sites lose power and fail over time 3-4 hours (100%UPS, 50%Generator Couplings, 10%Generators)

– Public Safety Radio infrastructure fails (unless supported by UPS)

– Non-copper based telephone infrastructure fails. Fiber optics premise based multiplexers

– Cable television service fails. Radio stations w/o UPS go off the air.

– VoIP service dependent upon operating non-copper based infrastructure fails

– Premise equipment w/o battery backup fails – wireless phones, cable modems, routers.

The role of the CIO/CTO during power outages and events impacting infrastructure

• What is the impact on public safety communications?

• What is the impact on public telecommunications?

• What do the city’s web site, 311, & 911 need to do?

• What IT support needs to be provided for the emergency response?

• What needs to be done to keep Municipal IT functioning?

Telecommunications - What is the impact, what is the status, what type of volume are is being handled, are there any issues/problems, any developing?

Are 911 systems operating normally?

Are the telephone systems operating?

Are cellular networks operating?

Are our public safety radio systems operating normally?

Have any critical Public Safety sites been effected?

Police/Fire/EMS sites, Command and Control – Cityhall, OEM, Hospitals, Key agency buildings and operational facilities

Have any sensitive locations/installations been effected?

Nuclear power plants, Chemical plants, refineries…..

Have any critical commercial sites been effected?

Exchanges, strategic industries, banks, alarms, surveillance, monitoring and security systems

Are cable systems up and operating?

Public Facing IT Utilities – What needs to be done?

• 911, 311, 211

How high is call volume, do we have enough call takers, is the next shift going to be able to get in to work? Supplement call takers?

What is the message, “What do we tell people?”

What are people telling us? Who needs to know?

Are there any new 311 functions that need to be brought up quickly? How quickly? What information do we need from callers, what is the process?

• Web site

How is the web site performing?

What is the message/content?

Do we post an emergency home page (OEM)

• Municipal Television

Is there information to go out on our crawls. Do we want to facilitate press coverage by providing a pool feeds,

where are the press conferences going to be held?

Municipal IT infrastructure

What is the impact, what is the effect, how big a problem is it, should DR procedures be implemented?

Are data centers effected? Is there potential for integrity to be compromised?

Are networks operational? Is there a security threat?

Are any non-data center based systems adversely effected?

Are systems critical to support emergency operations up and running properly?

Web sites, GIS, E-Teams, CRM (311), Email, BES servers

Have other technologies been adversely effected?

PBXs, Traffic Pattern Control systems, Building Control Systems

Providing IT support for the emergency - Wireless

• Wireless

Does anyone need radios, satellite phones, cell phones, blackberries, EVDO?

Do we need to deploy additional batteries, chargers, accessories?

Is the radio infrastructure stable, deploy back up systems?

Are there Interoperability requirements? Deploy ACU 1000s, system level interoperability?

Do we need to deploy Cells on Wheels (COWS), Cells on Light Trucks (COLTS)?

Do we need to deploy temporary public pay phone banks?

How is the infrastructure holding up, any systems failing, Batteries, UPSs, Generators Operating, is fuel required?

Providing IT support for the emergency – General Technology

Is there a need for new office environments to be established, an EOC, field command centers, Family Assistance Centers?

Is there a need for credential systems?

Does anyone need laptops?

Is there mapping and GIS support required for press announcements, traffic control and logistical support?

Are there new applications that need to be brought up quickly?

Are there IT support staff in EOC’s, public safety agencies that need to be relieved?

Aftermath of the Attack on the World Trade Center – September 11th 2001

• Needed to Coordinate IT equipment requests, triage, stage, inventory, distribute, manage FEMA reimbursement applications, pay vendors. (coordinate and track donations) Lesson Learned – Develop procedures, establish Emergency IT Contract

• Needed to build facilities rapidly

– OEM Emergency Operation Center & Family Assistance Center

– Backup Emergency Operation Center & Executive Offices

Lessons Learned - LAN in a box. (Network equipment and Notebooks ready to deploy), Emergency IT Contract, pre-configure

• Needed to coordinate the restoration of telephone service

Lessons Learned – Strengthened Mutual Aide Restoration Consortium Agreements

• Developed a GIS capability at OEM Emergency Operation Center

• Implemented Emergency Resource Management Software

• Developed Agency by agency business /continuity recovery plans

• Centralize - City data centers and network fared very well, agency servers and networks failed

Crash of Flight 587 - November 12th, 2001

• Developed Mobile GIS capability

• Rapid deployment of Family Assistance Center

Northeastern U. S. Blackout – August 14th & 15th, 2003• Needed to have better control over the Maintenance of UPS’s

and Motor Generators at some of our sites

• Needed to strengthen our protocols with the Telco’s, needed insight into their Motor Generator maintenance and testing

• 311 – Handled 172,000 calls in a single day. Harden 311.

– 311 provided feedback from public

– 311 helped to relieve 911, 911 experienced normal volume

• Needed additional battery capacity for Public Safety Radio repeaters – spares, portable generators

• Needed satellite phones and radios for executives

• Needed to harden city hall

• Data centers & network available, no users

Staten Island Ferry crash – October 15th , 2003

• Needed a system on hand at 311 to track missing, & casualties / next of kin.

One and a half hour partial 911 outage – March 26th, 2004

• 311 – received and routed 911 emergencies to precincts via non-emergency CRM workstations

Lessons Learned – strengthen procedure to route 911 calls thru 311, ring down phones between NYPD/FDNY/EMS

• NYPD had work around procedures for 911 outage

• Painstaking change control procedures with Verizon

• The highest levels of redundancy need to be implemented for 911

Major Transit Strike - December 20th – 23rd 2005

• Needed to develop telecommuting policies and capabilities

• Needed a plan and the capacity to transport 911 and 311 call takers to work and back.

• May have needed uniformed NYPD trained to handle 911 calls if 911 call taker attendance was too low. Needed to have NYPD officers trained.

• The benefit of Call center capacity outside of the region began to be apparent

• NYC 311 handled 243,000 calls in one day, normal 911 call volume

Blizzard, largest recorded snowfall in NYC- February 12th 2006

• The need for call center capacity outside of the region was reaffirmed.

New York City 3-1-1 - Invaluable Everyday – Essential during Emergencies

• All calls are answered by a live operator, 24 hours a day/7 days a week/365 days a year

• Provides immediate access to language translation services in over 170 languages

• Allows callers to quickly and conveniently:

– Be directed to a specific City, State or Federal agency or program

– Request detailed information about services and programs

– File a request for City services

• Professional, courteous, knowledgeable, accessible

New Yorkers only need to remember two numbers to contact City government: 911 for emergencies and 3-1-1 for everything else.

311 Providing Access to Govt.– Increasing call volume

Monthly Call Volume

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

• 3-1-1 has increased the public’s access to non-emergency government services.

– Since March 9, 2003, 3-1-1 serviced over 30 million calls, 15 Million/year

• 3-1-1 averages approximately 47,000 calls per day.

• Since its inception, the average number of calls per day has steadily increased

• 3-1-1 has set of a service level of answering over 98% of calls in less than 30 seconds, no calls on hold beyond 3 minutes.

New York City --- Why 3-1-1 is necessary