Case: CareGroupSteven AdamsTravis DonselmanRuwani Palihapitiya

9 November 2009

Case: GroupCare/Adams et al. 1

2

Waltham

Deaconess Hospital Needham

Case: GroupCare/Adams et al.

SWOT Analysis

Case: GroupCare/Adams et al. 3

Strengths Weaknesses• Halamka’s leadership • Y2K Disaster Recovery Plan• Info. Week’s #1 IT organization• 90% cut in capital expenditures• 2nd largest healthcare system in

region

• Outdated network components• Weak financial statements• Inadequate security controls• Inadequate change controls• Lack of networking knowledge• Outdated Disaster Recovery

Plan

Opportunities Threats• Partnership with Cisco• Recovering economy (Q4, 2002)• Vendor to Waltham and Glover• Synergies from merger• Further cost cuts due to

leveraging IT• Many hospitals closing, leading to

less market saturations

• Volatile time for industry• Stagnant industry growth • Mass. Gen.-Brigham and

Women’s merger

Symptoms and Problems

Case: GroupCare/Adams et al. 4

Symptoms Problems Evaluation CriteriaConsolidation in industry Mass. Gen.-B&W Hosp.

Price Pressure HMOs want cost cuts

Hospitals losing money MAH, BIDH, NEBH, WH

Excess Supply in health care industry < 60% bed occupancy

Fierce Competition Many hospitals closing

Efficiency increase of 20% Debt-equity ratio under 1 Cost cuts limited to 5% All CG hospitals earn profit for Q4, 2003 10% ROE for FY 2005

Uncertainty about future New CEO, CFO, CIO

Lack of centralized IT system Legacy IT systems

New business strategy Merger of different hospitals MAH, BIDH, NEBH, DNH,

DGH, WH

95% of personnel know CG mission and departmental goals and objectives All hospitals on Meditechsoftware by end of 2001

Collapse of IT network Out of spec network Layer 2 based

Lack of IT knowledge No network specialist

Lack of Controls No record of changes Napster-like attack

“Five nines” uptime for a period of two years No unscheduled downtime for two years Network is “in-spec” at all times for two years Meet Gartner benchmark

Theory and Concepts

Case: GroupCare/Adams et al. 5

HIGH PATH TO BUSINESS TRANSFORMATION

IT Impact on Core Capabilities

Business Process Design/Reengineering Centralized IT System Improved efficiency

Business Transformation Merger of five hospitals Greater contracting power against HMOs

Incremental Improvement Use of synergies: strong image (Beth Israel), lower debt costs

Emerging Opportunities Vendor to divested Waltham and Nashoba branches Reorganization of Beth Israel and Deaconess-Glover

LOW IT IMPACT ON CORE STRATEGY HIGH

Timeline of Network Collapse

Case: GroupCare/Adams et al. 6

Napster-like internal attack

Wed., afternoon

Service restoredThurs., 4:00 am

Intermittent failures

Thursday

CAP declared Thurs., 4:00pm

Network blocked; backup procedures

implemented Fri.,

morning

Redundant core builtSaturday

Network fully

operational Sunday,

night

The network had grown haphazardly due to the Caregroup merger, the standardization of Meditech in all hospitals, and the lack of change controls and networking knowledge. Significantly, legacy applications required a flat topology.

Additionally, the use of aged outdated networks components contributed to the failure and prolonged the recovery process. This is a result of not meeting the Gartner benchmark for IT spending as a percentage of revenue (1.9% vs. 2.7% target).

Reboot of core and Layer 3

Friday

Case: GroupCare/Adams et al. 7

RenaissanceParkswitch-rca switch-rcb

switch-rcc

5500 5500

5500

EastCampus

switch-ccell118

switch-rob05

switch-ly030

SiSi

switch-br203

5500

5509

5500

5500 ATM7/1

FEC 9/1-2 FEC10/5-6

FEC 9/1-2

WestCampus

switch-spg06b

switch-spg06a

switch-ccw00m4

5500

5500

5500

SiSi

ATM 5/1

FEC 9/5-6

SiSi

FEC 8/1-3FEC 10/1-4

FEC 8/3-4

FEC 9/1-4FEC 8/1-2

FEC 8/1-2

FEC 8/1-2

SiSi

FEC 10/1-2

SiSi

FEC 11/5-6

FEC 8/1-2

FEC 9/1-2FEC 10/5-6FEC10/1-4

FEC 11/1-2

FEC 9/1-2

FEC 11/3-4

ATM10/1ATM10/1

FEC 6/23-24 FEC 6/23-24

ATM

7/1

ATM

10/

1

FEC 3/21-24FEC 4/21-24

FEC 3/21-24FEC 4/21-24

ATM 10/1

ATM 5/1

148 12

12

12

12

1212

8 12

14

14 14

12

8 8

8

(8) 5505 Switches

(1) 5505 Switch

(15) 5505 Switches(1) 5500 Switch

(37) 5505 Switches

(1) 5500 Switch(21) 5505 Switches(2) 5509 Switches

(1) 3500 XL Switch (PACS)

(18) 5505 Switches

(21) 5505 Switches(1) 3500XL

(4) Ren Ctr (rc5, rc6, rc7, rcc)(1) Mount Auburn (Remote)

(1) 5500 Switch(1) 5505 Switch

(3) 5509 Switches

SiSi

(12) CC East Campus(4) HIM

(3) 109 Brookline Ave(2) 2127 Burlington(3) Research North

SiSi

(24) CC West Campus - - 1 is Dual Homed w/spg06b(1) PACS

(1) Research East

(1) 5500 Switch(1) 5505 Switch(1) 5509 Switch

(4) Dana(3) East(7) Feldberg(2) Finard(3) Kirstein(4) Reisman(5) Rose(1) Service(5) Stoneham(2) Yamens

(1) Baker(4) Deaconess(7) Lowry Medical(1) Maintenance(3) Palmer(1) CC West (Dual-homed w/ccw00m4)

(13) Farr(6) Kennedy(2) Lowry Medical(1) Masco

(3) Ren Ctr (rc7, rc8, rcc)

SiSi

ATM OC-3 (155Mbps) over SonetATM OC-3 (155Mbps) dark fiberFast Etherchannel (400 Mbps)Fast Etherchannel (800 Mbps)Not Active

SiSi

Technical Explanation Layers 2 and 3 were out of specification with 802.1d

standards:◦ The standard calls for a maximum of 7 hops. ◦ Multiple bridges were 10 hops from the bridge root ◦ Bridge loops present◦ Insufficient use of routers

Case: GroupCare/Adams et al. 8

802.1d Standards CG Network One VLAN per subnet One VLAN for multiple subnets

Limited or no bridging Extensive use of bridging Switching limited to Layer 2 Switching used for Layers 2 and 3

Alternatives and EvaluationAlternatives Pros Cons

Mitigation• Disaster Recovery Plan• Emergency Access System

• Limited access available during outages• Outages run more smoothly

• Moderate capital expenditures• Moderate time requirements• Limited risk involved

Avoidance• Contract with Cisco• Upgrade Components• Sec./Change Controls• SETA

• Minimizes risk of outages• Expertise and knowledge of Cisco• Changes are documented• Employees are less likely to harm the network

• High expenditures $300,000/yr. for Cisco

• High level of network knowledge required• Significant time investment• Some uncertainty involved

Acceptance• Do nothing

• No project planning• No capital expenditures• No time required

• System subject to future failures• Threat to company’s reputation and profitability

Transference• Outsource Network Management

• Skilled administrators• Can focus on core competency (knowledge management)

• Need to trust another company• Very expensive• Loss of control over process

Case: GroupCare/Adams et al. 9

Implementation

Case: GroupCare/Adams et al. 10

0 5 10 15 20 25 30

Install Emergency Access System

Plan and Conduct SETA

Revise Disaster Recovery Plan

Institute Stricter Change Control

Revise Employee Use Policy

Upgrade Network Components

Contract with Cisco

Recommendations Use multiple spanning tree protocol Implement monitoring and diagnostic tools to

maintain the health and hygiene of the network:◦ NATkit◦ CiscoWorks◦ HP Network Node Manager

Implement exclusively routers in Layer 3 Replace all legacy applications Review and, if necessary, replace Meditech PAC

software Use dense multicast instead of broadcast Hire network VP, network managers, network

administrators, and network techniciansCase: GroupCare/Adams et al. 11

Case: GroupCare/Adams et al. 12

Review of Case How did CareGroup respond to the disaster?◦ Implemented backup procedures◦ Brought in Cisco’s Emergency Response Team◦ Reverted to paper system◦ Installed routers, restored critical systems, reviewed crisis

Your thoughts on how CareGrop handled the crisis.◦ Well organized, decisive action◦ Cisco Emergency Response Team was the right decision◦ Should NOT have changed system before ERT arrived

Your thoughts on network management in light of the case.◦ Vigilant system maintenance and upgrades are critical◦ Control access to the main system at all times◦ Make sure backup plans are in place and well-practiced

Case: GroupCare/Adams et al. 13