jump-starting the command center centralizing ... · protiviti • jump-starting the command center...
TRANSCRIPT
Jump-Starting the Command Center – Centralizing Communication for High-Risk Information Technology Events
1Protiviti • Jump-Starting the Command Center – Centralizing Communication for High-risk it Events
introduCtion
Communication among key internal teams and quick, organized incident resolution for customers are essential to preserving business continuity when organizations undertake high-risk and high-impact information technology (IT) events. A command center, with a well-trained team devoted to documenting, prioritizing, escalating and helping to resolve incidents, can help ensure the success of such a major IT event and reduce the risk of business interruption.
The command center is not a project management organization, but a temporary cross-functional team composed of the appropriately skilled personnel organized in advance of planned events and designed to serve potentially as a 24/7 main line of support during pre-production and post-go-live IT events that are medium- or high-risk and/or high-impact to internal and external operations and customer transactions. A large-scale data migration or conversion event where major applications are being implemented or modified would be a likely candidate for a command center, but an organization may implement the function to support other IT events such as a pilot, acceptance testing, performance/scalability testing, mock testing or a production release.
2Protiviti • Jump-Starting the Command Center – Centralizing Communication for High-risk it Events
EStabliSHing tHE Command CEntEr: a FlExiblE StruCturE
A command center’s staffing level and duration will vary based on the level of risk and need for dedicated support for a specific high-risk IT event. The organizational chart above outlines a suggested structure for a command center and details the roles of personnel typically required.
The command center’s senior leader is the key decision-maker who assigns severity levels to reported incidents and escalates issues when necessary. The severity level, which is assigned before an incident is passed on to a subject-matter expert (SME) for resolution, dictates how closely and quickly the issue must be monitored or resolved. This role is typically filled by a senior leader from the client’s existing structure who can act as a liaison between the command center team and other executive leaders within the organization.
The command center coordinator, who reports to the senior leader, ensures the staffing plan, secures the necessary resources (e.g., rooms, computers, bridge lines), and establishes documentation and processes related to incident reporting around the IT event.
The incident manager interacts directly with customers who contact the command center, gathering all pertinent information about an issue and then passing it on to the relevant support groups. The application SMEs (resources with the knowledge and insight to resolve key issues); the infrastructure representative (who provides expertise into infrastructure components, such as server, network and database components); and the resolution manager (who facilitates resolution of reported issues) round out the team.
Smaller IT events (such as minor code deployments, testing events or other lower-risk activities) may require only one or two staff members to perform the responsibilities of each role in the basic structure. Additionally, the incident manager and resolution manager roles may be combined. The command center organization is meant to be flexible and assign roles as necessary to support the type and significance of the event as needed.
Command Center Basic Structure
• Administrative role to record all issues and decisions
• Analyzes incident reports for trending issues
• Provides roll-up reporting
Incident Manager
• Facilitates resolution of issues that are reported to the command center
• Coordinates the paging of necessary on-call and support resources
Resolution Manager
• Provides insight into detailed operations
• Resource to resolve key issues
• Provides general oversight of release
Application SMEs
• Provides expertise in infrastructure components
Infrastructure Representative
• Makes key decisions
• Assigns severity levels to incidents
• Escalates issues when necessary
Command Center
Senior Leader
• Ensures staffing plan
• Secures necessary resources
• Establishes documentation and processes
Command Center
Coordinator
Figure 1: basic structure for a command center (smaller it events may require fewer staff members)
3Protiviti • Jump-Starting the Command Center – Centralizing Communication for High-risk it Events
The War Room
During high-risk IT events that typically require elevated staffing levels and rapid response, it may be necessary for an organization to separate staff between a command center and a “war room” function. A war room is a specialized team of SMEs drawn from across the business who are tasked with resolving high-priority incidents as they occur, freeing the command center to triage and route less critical tasks to other support groups. The command center staff (see blue boxes in Figure 2) handles incoming issues and assigns incidents to the war room staff (see gray boxes in Figure 2) for investigation and resolution.
• Administrative role to record all issues and decisions
• Analyzes incident reports for trending issues
• Assists coordinator in providing roll-up reporting
• Provides expertise in infrastructure components
• Facilitates resolution of issues that are reported to the command center
• Contacts enterprise support groups when necessary
• Coordinates the paging of necessary on-call and support resources
• Provides insight into detailed operations
• Resource to resolve key issues
• Provides general oversight of release
Incident Manager
War Room Lead
Application SMEs
Infrastructure Representative
• Makes key decisions
• Reviews severity levels assigned to incidents
• Escalates issues when necessary
Command Center
Senior Leader
• Ensures staffing plan
• Manages command center operations during the event
• Responsible for communications updates to other command centers when necessary
• Sends status communications (deployment/incident) as agreed upon with each team
Command Center
Coordinator
Figure 2: Some high-risk it events may require a separate war room staff
4Protiviti • Jump-Starting the Command Center – Centralizing Communication for High-risk it Events
SEnding tHE rigHt inFormation to tHE rigHt PEoPlE
The command center helps promote effective communication about incident resolution among support personnel, project teams, other support teams and command centers, and IT/application groups. It receives initial calls for help from internal customers (i.e., the application, test or pilot teams) when issues, such as a service outage, arise during a major IT event. Depending on the size and duration of the event, customers may be given several ways to contact the command center – for example, a toll-free number, a conference bridge line, an instant messaging application, or a shared email box. They also may be able to submit their issues electronically through a designated support portal.
The command center team documents and then relays information to the appropriate SMEs for resolution. The team logs all calls and emails it receives from customers, whether the issue reported is an incident (e.g., degradation of service) or a defect (i.e., a bug or issue discovered as a result of testing), and provides level one support for the caller. Additionally, the command center will interface with other IT Service Management groups throughout the enterprise and ensure that appropriate resolution teams are engaged. While the command center does not function as a centralized defect reporting center, or fix defects, it will log such reports into the tracking system and communicate that information to the testing team, which establishes the process for handling reported defects.
At the close of each supported IT event, statistics related to the reported issues are calculated and inserted into a report that is presented to project managers, program leads and work stream leads. The information the team documents about reported errors and defects and how they were resolved is useful knowledge the organization can apply in the future when preparing to undertake and provide appropriate support for other significant IT events.
ConCluSion
When establishing a command center, organizations should strive to create a structure that will be easy to replicate quickly – and lessons learned after each event is completed should be applied as soon as possible. And when undertaking high-risk IT events, they also should identify a set of metrics for measuring results that are both meaningful to management and usable by support staff. Primary metrics should be in alignment with an organization’s “business as usual” status, and be actionable. If failures, missed targets, or data outliers are discovered, appropriate and timely action can be taken to investigate and remediate such failures.
Technology is always changing, and so too are business needs, which means the likelihood of a large organiza-tion undertaking more than one high-risk IT event in the near future is likely. The support that a designated command center function can provide is necessary for the success of any high-risk IT project; there is simply too much at stake for an organization not to have a well-trained team in place to manage centrally the incident reporting and resolution process throughout all critical stages of a major event.
5Protiviti • Jump-Starting the Command Center – Centralizing Communication for High-risk it Events
about Protiviti
Protiviti (www.protiviti.com) is a global consulting firm that helps companies solve problems in finance, technology, operations, governance, risk and internal audit. Through our network of more than 70 offices in over 20 countries, we have served more than 35 percent of FORTUNE® 1000 and Global 500 companies. We also work with smaller, growing companies, including those looking to go public, as well as with government agencies.
Protiviti is a wholly owned subsidiary of Robert Half International Inc. (NYSE: RHI). Founded in 1948, Robert Half International is a member of the S&P 500 index.
The Command Center in ActionWhile assisting a major financial institution with merger activity, Protiviti’s on-site project teams identified a need to mitigate a number of production support-related risks. A key middleware team implementing a new platform to eliminate a significant number of partner-to-partner interfaces was concerned about how to control the inherent risks associated with upcoming deployment events. Making these changes was critical to allow legacy and go-forward systems from both organizations to communicate effectively without extensive code changes.
Protiviti worked with the client to develop and implement a command center and ongoing production support model. In its final form, the command center was a joint effort between our team and the client’s staff. Protiviti created much of the organizational structure and the leadership team, and was responsible for back-office preparations both before and during the supported events.
The core team was augmented with SMEs from application teams during high-risk events, who provided the necessary details as production incidents occurred. This model allowed the command center to successfully integrate into the overall corporate command center structure.
About Our IT Consulting Protiviti’s IT Consulting practice helps IT leaders design and implement IT governance, security, data management, application and compliance solutions. Our professionals partner with CIOs and other IT and business leaders to ensure their organizations maximize the value of information systems investments while mitigating key risks. Our IT Consulting practice has brought technology and business solutions to more than 1,200 clients worldwide. We group our IT consulting services into three areas of focus for CIOs:
Michael SchultzManaging [email protected]
Jonathan WyattManaging Director+44.207.024.7522 [email protected]
Jason RiddleSenior [email protected]
Kyle [email protected]
• Managing the Business of IT • Managing IT Security & Privacy • Managing Applications & Data
We invite you to explore at www.protiviti.com/ITconsulting how we create value by helping clients identify and solve critical business and technology issues.
For additional information about the issues reviewed in this white paper or Protiviti’s services, please contact:
© 2012 Protiviti inc. an Equal opportunity Employer. Pro-0712-103046 Protiviti is not licensed or registered as a public accounting firm and does not issue opinions on financial statements or offer attestation services.
aSia-PaCiFiC
australia Brisbane Canberra Melbourne Perth Sydney
China Beijing Hong Kong Shanghai Shenzhen
india Bangalore Mumbai New Delhi
indonesia Jakarta**
Japan Osaka Tokyo
Singapore Singapore
South Korea Seoul
* Protiviti member Firm ** Protiviti alliance member
tHE amEriCaS
united States
Alexandria Atlanta Baltimore Boston Charlotte Chicago Cincinnati Cleveland Dallas Denver Fort Lauderdale Houston
Kansas City Los Angeles Milwaukee Minneapolis New York Orlando Philadelphia Phoenix Pittsburgh Portland Richmond Sacramento
Salt Lake City San Francisco San Jose Seattle Stamford St. Louis Tampa Washington, D.C. Woodbridge
argentinaBuenos Aires*
brazilRio de Janeiro* São Paulo*
CanadaKitchener-Waterloo Toronto
mexico Mexico City* Monterrey*
PeruLima*
venezuelaCaracas*
middlE EaSt
bahrain Manama*
Kuwait Kuwait City*
oman Muscat*
united arab Emirates Abu Dhabi* Dubai*
EuroPE
France Paris
germanyFrankfurt Munich
italy Milan Rome Turin
the netherlandsAmsterdam
united KingdomLondon