Scott McPhersonScott McPhersonCIO, Florida House of RepresentativesChairman, Florida CIO Council Pandemic Preparedness Committee

Keeping Information Technology Up If People Go Down

Challenges to Business and Government

There has never been an influenza pandemic in what we would define as the “Internet Age.”

IT is “The forgotten resource.” No one outside of IT fully realizes that no

work can be done without properly functioning mainframes, servers and PCs/Macs.

Now, with a pandemic second wave looming, everyone needs to understand how important IT really is.

Once the second or third wave begins…

A frantic agency head or senior manager will rush into your office and scream, “I need a work at home plan – NOW!”

You will calmly look back and say, “That is why I put in all those purchase

requisitions for protective equipment, additional bandwidth and a secure virtual network. Since they were all turned down, we will have to do things a different way.”

That is when you must take over the discussion within your entity.

Forrester survey, 2009 In a recent joint Forrester and Disaster

Recovery Journal survey, 285 BC/DR decision makers were asked if their company had strategies for workforce recovery in their BCPs.

68% said yes. This means that 32% have a lot of work to

do. Of the 68% that have strategies in place,

86% use remote access procedures as part of their strategy.

Forrester/ZDNet survey, April 2009

However, these questions went unasked:

What do you do if you cannot fabricate a satisfactory telework solution?

What if your distanced employees cannot access via a broadband connection?

What if the business processes are not sufficiently digitized to allow for remote processing of bits, as opposed to atoms?

What if the data center staff or networking staff become sick themselves?

Part One

What would Ike do?

“The plan is useless – it’s the planning that’s important.”

Ike's point is that events will never go according to The Plan -- but a mature planning process will help you prevail. 

Believe me, no one is smarter than Ike on this matter. No one.

Add context: What will happen all around us?

The pandemic plan for IT organizations PLAN ONE CATEGORY HIGHER

Create corporate/agency pandemic planning team Prepare succession plan Update DR and COOP plans NOW, as existing DR and

COOP plans without pandemic modifications will NOT WORK in a pandemic.

Prepare to shut down nonessential IT services Ensure “retail business/government” ops continue Acquire protective equipment Monitor employee absenteeism Cross-train your staff Design, implement and support Work at Home plans Prepare for supply chain failures Prepare Communications Plan Teach protective actions Gain an understanding of influenza and how it works Leverage this planning for similar scenarios

IT Issues to consider Do you have a succession plan in place? Which services do you turn off or allow to fail? Data Center operations (lights out operation, automated

patching) Public Website for emergency notices such as openings and

closures, if necessary posted by IT staff at home, or even by non-IT staff as needed

Remote Access (Citrix, RAS, Terminal Services) as alternatives to SSL VPN

Don’t forget field staff!! Maintaining agency cybersecurity in the midst of all this Do you enable or eliminate Help Desk operations? PC support for employee personal computers? NO Ensuring security of access and data while dealing with

employee personal computers POLICY ENFORCEMENT via SSL VPN

Videoconferencing as alternative to face-to-face meetings – how will you support it if it malfunctions?

Recovering from cascading emergencies (swine flu on top of hurricanes, terrorism, etc.)

IT Services

It is just not practicable to expect to support 100% of all your applications/services.

Now is the time to sit down with leadership and ask which IT services may be turned off in a pandemic, and which can be allowed to fail without restarting.

Doing this now sets the expectation bar, reinforces the urgency of the coming second wave, and allows you to cross-train more efficiently and effectively.

Prepare to shut down services

Grab your list of IT services (ITIL) and (re)prioritize them with governance board

Prepare to shut down ALL nonessential services or to abandon SLAs for same Do not bring these services back up if they

fail, unless you can support them later If not an ITIL shop, then work with upper

management to prioritize applications by criticality. Maintain the list and review annually as part of Dr/COOP.

Have you cross-trained your staff?

Create written instructions/ procedures for critical processes that can be carried out by others

Cross-train your staff, ideally three-deep Anticipate 30% morbidity (illness) within staff Assume absenteeism due to closure of other schools, day care

centers

Train by TASK, not by what somebody does Maintain a matrix of staff training and widely

distribute and post in disaster recovery books and agency COOP plan

Cross-train inside and outside of Data Center; in other words, cross-train non-data center people in simpler technical tasks such as tape rotation.

Don’t cross-train on services you will disconnect or allow to fail!

16

Methodology Categorize all operating tasks within one of four operating

quadrants Supporting: Break and fix oriented tasks; user issues;

application outage, etc Operating: Task needed to keep the wheels turning -- backups,

routine jobs, admin work Changing: Application updates, small code drops; hardware

renewal Optimizing: Large projects, version upgrade, performance

management

Determine in which quadrant the critical tasks are located What type of work will we focus on in the event of a crisis What type of work can we stop doing without affecting our

services

Determine amount of time spent on activities from within each quadrant Calculate how many FTEs are assigned to each quadrant Determine how many FTEs can be reassigned to other functions.

17

Geographic Skill Matrix Overview

Team FTE SAN MessagingKnowledge

ManagementCollaboration

External Website

Europe 60 2 6 0 0 0

Americas 128 8 9 2 13 6

Total FTE ‘s 188 10 15 2 13 6

Minimum FTE’s required to support skill

2 3 1 5 1

25% reduction 141 7 11 1 9 4

50% reduction 94 5 7 1 6 3

75% reduction 47 2 3 0 3 1

18

Individual Skill Matrix Overview

FTE Location Team SAN MessagingKnowledge

ManagementCollaboration

Website

Joe Glen Mills Hosting Ctr. 3 4 4

Jane LondonPlatform Architecture

1 2 2 3

Charlie ChicagoPlatform Dev.

1 2 4 3

Theresa Prague Help Desk 5 1

Blank = No experience

1 = limited experience

2 = some experience - basic knowledge

3 = good experience - can do configuration setup, some basic troubleshooting

4 = strong experience - strong troubleshooting, configuration, setup skills

5 = very strong experience - configure, troubleshoot - go to person

Is the answer in the cloud? Cloud computing may be an alternative, but remember certain public-access cloud solutions are subject to frequent outages; run on the greater Internet, not a secure private network; and, for the most part, are not very secure (think Sarah Palin and myriad Facebook/Twitter hacks). Webmail (Gmail, Yahoo mail, Hotmail) Group scheduling and calendaring features on

Google and other sites Creating a Facebook presence Using Twitter in place of emergency

notification solutions Placing middleware in front of Twitter to

enhance security

Other cloud solutions/SaaS Email failover solutions such as

MessageOne, which automatically allows failover of email to their cloud in an emergency or even due to routine maintenance.

This makes email a Tier Zero app, with 24/7/365 availability.

Chief lesson learned from 9/11 is that email must never go down.

Do you have examples of other cloud-based solutions you are using that you can share?

Monitor your employee absenteeism, even if no one else does. Do not rely solely on “actual” totals coming

from WHO and CDC. If you see a spike in employee absenteeism, it is probable that the virus has gained a foothold in your workforce.

This can be done without HIPAA violations, and should be performed agency-wide, statewide, and don’t forget field staff.

Encourage an enterprise-wide or agency-wide initiative to monitor for absenteeism which will be critical to track both the spread of the virus (reporting daily roll-ups to DoH) and to determine operational readiness statewide.

Infrastructure issues Upgrading networks to Priority Restoration Upgrading/Increasing Service Level

Agreements (SLAs) for hardware support, network support, Wide Area Network support

Maintaining appropriate levels of information security on employee personal computers

Preventing kids from hacking Mom’s corporate network while she sleeps or is away

Broadband – who has it and who doesn’t?

Castling oneself into checkmate (outsourcing to India and China) Half of India is age 25 or younger Much of India’s IT support operations

were hard-hit in the pandemic’s first wave (and continue to be hit)

Sign up for Google Alerts with keywords like “Bangalore H1N1” to get news and blog accounts of the impact of the pandemic in those areas.

Have you prepared for supply chain failures?

In a pandemic of any severity, the supply chain will falter.

In a 1918-type (or worse) pandemic, the supply chain will fail.

If possible, keep essential supplies/ parts stockpiled in advance (4-6 week supply).

Survey your suppliers. Resurrect the old Y2K adage: If they can’t articulate their plans for pandemic flu preparedness, be wary of their ability to survive.

In fact, go find your Y2K plans, turn to the tab marked “Supply Chain Workarounds,” update it and put it into your DR/COOP Pandemic Annex.

Has your agency prepared internal and external communications plans?

How will key managers communicate among themselves?

How will information be conveyed to employees?

How will employees know who to call in specific situations?

How will information be conveyed to business partners?

How will the public know which “safety net” offices are open and which are closed?

Consider emergency notification services

Companies such as Dialogic Communications, TechRadium, Dell/MessageOne and others have affordable, hosted services that allow an agency to push information to employees via any type of device

Eliminates the old “phone tree” tedium Includes voice synthesis and fax Can allow agencies to poll their workforce to

see who can work and who is too sick to report

Will be critical when trying to open offices or trying to tell people which office to report for work

Communicate with employees and teach preparedness at work and at home

Conduct an awareness campaign within your organization.

Teach employees how to prepare themselves and their families FIRST. And do it now, instead of later – when it is too late

Cover work and home issues Teach protective actions and personal

hygiene Prepare them for moving from office to office

– even from agency to agency.

Teach protective actions NOW

Hand washing without recontamination Covering cough, not using hands Avoid putting hands to face, mouth,

nose, eyes. Staying home if any signs of illness Proper use of protective equipment Cleaning hard surfaces, wearing gloves,

using hand sanitizer and wearing masks

Part Two

Telework reality check1. Are you currently using any remote-

access work solution beyond Web-based email?

2. Are your business processes adequately “digitized” that you can conduct a flow of business without being in the office and without touching paperwork?

3. Do your employees have the broadband capabilities necessary to move mountains of ones and zeroes from home?

4. Have you exercised your telework capabilities routinely?

If you answered “no” to questions one through three…

Then your organization has a very limited chance of successfully deploying telework beyond a very narrow scope.

Is telework due to social distancing even practicable in this pandemic? Yes. It is a serious option for those who must stay

home to take care of sick loved ones. Yes. It is a serious option for anyone having to stay

home due to school or day care center closures. Yes. It is a way to get important work done while

minimizing the risk of infection, especially for immuno-compromised employees.

No. It is sometimes difficult to measure performance unless the job is specifically structured in a way that managers can quickly determine productivity from homebound employees. Airline call centers are a good example of this.

No. It is almost impossible to run an organization solely based on telework unless that organization was designed from the ground up to do so.

Viral reality check There are multiple opportunities, at

home and in the community, for an employee (or you) to become infected.

Indeed, the CDC recently said it would “probably not” change guidance on people returning to school or work, even when new evidence shows contagion can exist for eight days or longer.

The CDC’s reasoning is that the virus is so pervasive, such lengthy extensions of isolation would not produce positive results.

First, a message on leadership. Leaders do not “lead from the bench,” nor

do they lead from home in a pandemic. Many businesses and most governments

will be unable to function adequately using telework alone, and so many people will eventually have to come in to the office.

Managers and executives will lose all respect from their employees if they sequester at home and rarely come in to the office.

A loss of respect will translate into a loss of employees at their first opportunity.

A good place to start:

Existing COOP and COG plans that account for employee displacement due to anthrax/ricin preliminary detection, hurricanes, tornadoes/derechos, “snow days,” floods, ice storms and other disasters that render normal office operations impractical or impossible and forces employees to work from alternate locations.

Just imagine the alternate location being the employee’s home or home office.

Candidate jobs for telework(feel free to add your own)

IT application developers IT Tier One help desk and support staff (with

caveats such as call forwarding via VoIP, PBX, along with help desk software they can access remotely)

Sales, PR/communications and marketing Procurement staff with access to Web-based

purchasing solutions HR and personnel officers and staff Legal staff with Westlaw access from home and

access to corporate/government files or PDFs Senior decision-makers (with leadership slide

caveat)

Enemies of telework

Will their Work at Home plans really work?

In response to a pandemic, business, corporate America and government are all attempting to enable “Work at Home Plans.”

But just what is meant by working at home? How will paper get home to people? Who will deliver it? How will people input data?

How will people complete their work? What infrastructure will be necessary in order to facilitate this? Who will pay for it?

How will you secure the thousands of home PCs needed to fully implement such a plan?

Be prepared to “lose the Internet” (Booz, Allen)

Let’s define “work” Government and much of the private

sector still runs largely on paper Forms have to be inputted into computer

systems The business process must be taken apart

in order to be streamlined Tremendous opportunity to further digitize

government – and we cannot afford to lose this chance to streamline government and business processes!

Inventory business processes with intent to Webify them as “eGov” operations

Deconstruct and reconstruct the entity’s business processes.

Have business analysts work with Department staff to seek to streamline/digitize processes

Remember, only those processes that are mission-critical should be candidates for conversion

Aim for both a Webified solution and a manual-to-digital solution

Concentrate on alternatives to moving paper.

Now identify which employees are candidates for telework.

It is amazing to see just how many of your employees do not have broadband connectivity.

They may choose not to pay for it. Their home may be too remote or rural

to receive it. They cannot receive broadband even

via aircards. This makes them unsuitable for

telework.

The home office is key. Does the user have broadband?

Eliminate those who do not, or prepare a plan to have the agency pay for home broadband.

Does the user have a PC? Eliminate those who do not, or prepare to supply users

with laptops. Does the user have the appropriate applications

suite, antivirus and antispyware? Prepare to have legal review your existing

licensing agreements (“Seat” may allow you to install on a home PC if the office PC is turned off).

Is the OS patched? Any other security nightmares such as children?

What is the method of logging in?

Citrix VPN Remote takeover/remote control Windows Terminal Services

Potential failures in work at home plans

How will paper get home? USPS? Irregular deliveries UPS? FedEx? DHL? They too will suffer

loss of dependable service. Will agencies put together their own

delivery routes? If gas is scarce, how will deliveries take

place? Is it realistic to expect government to set

up its own postal service?

Potential failures in work at home plans

Paper must be quarantined, lest employers inadvertently sicken otherwise healthy homes CDC and St. Jude say virus becomes inert after

12 to 24 hours on paper and porous surfaces Each stage in the paper handling process

requires a day quarantine to prevent infection. Remember the terrible lessons learned from the deaths of Inuits (Eskimos) in 1918. The mailman brought the virus to them.

Are PDFs and PDF-driven forms an answer? Possibly, if you have enough time to convert forms

to fillable PDFs. PDF-driven forms would have to immediately be

designed by IT, with accompanying metadata captured and placed into databases for movement and action.

Scanning and emailing of documents would eliminate any potential for infection.

Scanning staff would need (and should demand) protective equipment.

The agency would need to set up a scanning solution to account for tens of thousands of documents at each “retail” site, or in each regional office.

Excessively large attachments might overwhelm the agency networks.

ISPs will not accept large attachments, so emailing to private accounts is unrealistic.

SSL VPNs and you

Time for an SSL VPN solution with rigid, unforgiving policy enforcement.

Implement an SSL VPN service and be prepared to scale it radically upward

Be prepared to “lose the Internet,” as network service providers will also experience high absenteeism and be forced to scale back SLAs (Booz Allen)

That is one compelling reason to upgrade to priority restoration.

Substitutes for SSL-VPNs

Go to My PC (made by Citrix, it requires a work PC to be on and functional in order to take it over via remote control

Above-average security. Individual seats or enterprise licensing A quick substitute for organizations

who have no VPN established, or for those who do not have the ability to “scale” their VPN upwards.

Conclusion, W@H plans: Governments and businesses cannot afford to

implement “perfect” work at home plans in the current financial and pandemic climate.

W@H plans can be successful, if the process does not involve the moving of paper or constant online access to legacy systems; if applied properly; if created with enough advance planning; and if exercised frequently.

Once the second wave kicks in, be prepared to have difficulty in obtaining equipment.

Corporations are usually better-equipped to proceed with broader work at home plans. They are usually more “digital” than government.

Alternatives and workarounds to face-to-face meetings and expensive conferencing solutions Audio teleconferencing as an

alternative to videoconferencing Low-cost videoconferencing services

as an alternative to face-to-face meetings

Desktop videoconferencing point-to-point via Skype, other platforms in lieu of face-to-face meetings

Second Life as alternative to meetings GoToMeeting and WebEx (with or

without video option) as alternatives to managing your own conferencing solutions or as backups to your established services

FedEx Office conferencing services in key locations in the US as alternatives to air travel.

Low-tech alternatives to telework Staggered shifts within the office,

e.g., 7AM to 4PM, 4PM to midnight Limitations on personnel at any

given time within “cubicle cities” Heavy emphasis on cleaning of solid

surfaces Heavy use of corporate-supplied

hand sanitizer Moving to four-day weeks

Part Three

BIG rule of thumb:

If your agency or organization has not formulated a pandemic planning team, that organization is NOT PREPARED FOR A PANDEMIC.

If your agency or organization has not exercised its pandemic plan via a tabletop exercise or stronger simulation, that organization is NOT PREPARED FOR A PANDEMIC.

What would Ike do?

“The plan is useless – it’s the planning that’s important.”

Ike's point is that events will never go according to The Plan -- but a mature planning process will help you prevail. 

Believe me, no one is smarter than Ike on this matter. No one.

Add context: What will happen all around us?

There should be already in place, a pandemic planning team

Legal Human Resources Training Development Information Technology Procurement/Purchasing Communications/Press Secretary Facilities/Maintenance/janitorial And at the top….. The CEO, agency head,

or second in command. If there isn’t one… they’re not ready for a

pandemic.

Have they created a realistic pandemic annex to their DR AND COOP plans NOW?

Armed with your information and in conjunction with all agency peers, urge creation of a Pandemic Annex to your organization’s COOP and Disaster recovery plans now.now.

Additional questions to answer:1. Does agency COOP Plan have an event horizon

beyond 30 days?2. Within a 30 day – to – 120 day context, does it have

a new definition of essential and nonessential personnel?

3. Does it contain a well-defined succession plan for agency leadership?

4. Just for grins, imagine if you had to support agency operations AND a hurricane or terrorist event came during a wave of the pandemic. Recall that 75% of all pandemics occur during hurricane season.

5. What plans would you engage?

Audit to ensure “retail” ops continue

Is an organization prepared to engage in core activities even if the following is true: Loss of raw materials/finished goods due to

failures in JIT supply chain. Everyday drop in workforce from 10% to

40% during a pandemic wave Loss of customers due to pandemic Forced closure of certain industries

(restaurants, public gatherings such as football games, dry cleaners, boutiques).

Ensure “retail government” ops continue

Retail government service delivery offices in the social, medical and law enforcement “safety net” domains will have to remain open, and certain employees will have to remain exposed to citizens while at work.

Keeping offices open will be essential to preserve the integrity of government and to care for its customers.

Can you imagine what will happen if government fails its citizens when they need it the most?

Have they cross-trained their staff?

Help them understand and mentor them on the correct ways to go about this.

Acquired protective equipment?

N-95 masks or surgical masks 3 per employee per workday for 6-12

weeks (you do the math) Alcohol-based hand sanitizer

Enough for 6 – 12 weeks Gloves (vinyl – some are allergic to

latex) Same ratio as masks – 3 pairs per

workday for the duration of a wave of a pandemic

Have you/they decided who gets protective equipment?

Front-line workers with constant exposure to the general public

The social and public safety “safety net” workers (AWI, DCF, HSMV, local equivalents, etc.)

Data center employees “The checks gotta roll.”

Working with Procurement

Do they know what to buy? Do they know the quantities? Do they know the context?

Everyone and their brother will want the same items and be willing to pay more for them.

Does your agency have the money to increase expenditures on staples?

Leverage this planning for similar scenarios

Anthrax (loss of building for 3-26 weeks) Ricin (loss of building for 3-26 weeks) Bioterrorism or chemical weapons Natural disasters striking your facilities

Hurricanes Tornadoes

Civil Disturbances Common themes:

What if my headquarters building were heavily damaged or destroyed, or people could not occupy it for an extended period of time?

Where would I put all the employees, and how could my agency recover and resume its core mission more quickly?

Plan ,plan, and plan some more.

Gain Global Context

Gain an understanding of influenza and its history

Read John Barry’s book The Great Influenza

Subscribe to Google, Yahoo and news services’ RSS feeds/email alerts on “bird flu,” “H1N1,” “swine flu” and “pandemic”

Learn how to protect yourself against all influenzas

Learn how to recognize the global “warning signs” when the pandemic intensity status is about to be raised.

FOLLOW THE TAMIFLU.

Ten tips on pandemic flu planning (from UK)

Lessons from the past are not always learned - we must do better than with Foot and Mouth and SARS

It won't go according to plan - biological, political and economic impact will be complex

Keep a sense of proportion - most people will survive Not just a health issue - businesses will suffer Lots of heroes out there - people will outperform expectations Communication will falter - we need tried and tested systems

for contacting staff The media loves disasters - can be good as well as bad Good enough is good enough - less than perfect care

and rationing is inevitable Low tech is reliable - hygiene as important as

oseltamivir Involve the public - be honest and open Source: Hilary Pickles, Hospital Doctor Pandemic Flu conference, London

References and Suggested Reading Materials “The Great Influenza,” John M. Barry (Viking, 2004 hardcover).

“America’s Forgotten Pandemic: The Influenza of 1918,” Alfred W. Crosby (Cambridge University Press, 1976/2003 softcover).

“Flu: The Story of the Great Influenza Pandemic of 1918 and the Search for the Virus that Caused It,” Gina Kolata (Diane Pub. Co., 2001 hardcover).

“The Monster at Our Door: The Global Threat of Avian Flu,” Mike Davis (New Press, 2005 hardcover).

“Influenza,” Dr. Edwin D. Kilbourne (Plenum Publishing, NY, 1987 hardcover).

“The Devil's Flu: The World's Deadliest Influenza Epidemic and the Scientific Hunt for the Virus That Caused It,” Pete Davies (Owl Books, 2000 softcover)

“The Swine Flu Affair: Decision-making on a Slippery Disease,” Richard E. Neustadt, Harvey V. Fineberg (University Press of the Pacific, 2005 softcover)

“Bird Flu: A Virus of Our Own Hatching,” Dr. Michael Greger (Lantern Press, 2006, hardcover)

Centers for Disease Control and Prevention, www.cdc.gov and www.pandemicflu.gov

World Health Organization Avian Influenza Website http://www.who.int/csr/disease/avian_influenza/en/index.html

National Academy of Sciences, Institute of Medicine http://www.iom.edu/

Recombinomics, Dr. Henry Niman’s Website, http://www.recombinomics.com/

University at Albany (NY) Public Health Pandemic Course, http://www.ualbanycphp.org/learning/registration/detail_Pandemics.cfm

commentary available commentary available at at www.scottmcpherson.newww.scottmcpherson.nettFollow me on Twitter:Follow me on Twitter:@scottwmcpherson@scottwmcpherson