wp_196 essential elements of datacenter facility operations-140902125904-phpapp02

8/10/2019 WP_196 Essential Elements of Datacenter Facility Operations-140902125904-Phpapp02

http://slidepdf.com/reader/full/wp196-essential-elements-of-datacenter-facility-operations-140902125904-phpapp02 1/27

Essential Elements of DataCenter Facility Operations

Schneider ElectricData Center Science Center

Schneider Electric – Data Center Science Center WP 196 Presentation – February 2014

White Paper 196



70% of data center outages are directly attributable to humanerror according to the Uptime Institute’s analysis of their“abnormal incident” reporting (AIR) database 1. This figurehighlights the critical importance of having an effective operations


and maintenance (O&M) program. This presentation describesunique management principles and provides a comprehensive,high-level overview of the necessary program elements foroperating a mission critical facility efficiently and reliablythroughout its life cycle. Practical management tips and adviceare also given.



IntroductionImportance of operations and maintenance (O&M) program

• Most facility outages attributable to human (operator) error• Majority of data center facility TCO is in OPEX, not CAPEX, where greatest

potential cost savings reside• Largest portion of OPEX are energy costs, which are rising


• Drive for energy efficiency reducing capacity safety margins and systemredundancy, increasing importance of proactivemaintenance and data center infrastructuremanagement (DCIM)

• High levels of facility automation and equipment

performance data have created new opportunitiesfor enhancing reliability while reducing costs,when properly managed



Mission Critical Mentality

● Focuses on risk mitigation● Grasps interconnectedness of facility

and IT systems

Failure is not an option


● Highly complex, fast-paced changesin mission critical facility

● Challenging to manage● Unique outside pressures

● Government regulations● Customer audits

NOTE: In this paper, only system planning is covered. System planning refers to the power, cooling, racks,

and other support infrastructure systems. Planning related to the IT equipment is not discussed here.



Mission Critical MentalityCode of Conduct

“Mission Critical Mindset” principles Impact

Focused on risk mitigation in all operational andmaintenance activities, work processes, andprocedures

Proactively deals with all potential threats tosystem availability and worker/occupant safety


Acting with confidence and patience that is anoutgrowth of careful planning and preparation

Prevents risks from becoming problems;enables faster response times and fewer errorsif problems do arise

Analytical, process-driven approach to riskavoidance and problem solving

Helps identify and mitigate risk in complexenvironments; ensures predictable and safeoperation

Comprehensive understanding of the function andinterconnectedness of facility systems andcomponents

Quickly identify and resolve potential threatsor actual problems; avoid or reduce systemdowntime

Commitment to continuous learning and processimprovement

Increases skills and operational efficiency tomaintain an edge in a constantly changingenvironment



12 Essential Elements of an O&M ProgramEnvironmental Health and Safety

● Key components include● Injury, illness prevention● Electrical safety● Hazard anal sis


● Hazard communication



12 Essential Elements of an O&M Program

Environmental Health and Safety

Key Program Attributes Description

Safety plans and trainingWritten safety plans must be established that describe the safe work practices andprocedures to be observed by all workers. Regular training on the programelements must also be conducted.

Hazard analysisAll operational procedures shall start with an analysis of the possible hazards


. .

Lockout/tagout proceduresProper procedures to prevent the unexpected energizing or startup of machines orequipment (or which causes a release of stored energy) shall be used whenservicing or maintaining equipment.

Personal protective equipment(PPE)

Appropriate protective equipment should be provided, properly sized, stored,maintained, and utilized as required to mitigate identified safety hazards.

Hazardous material handling

Hazardous materials must be properly identified, labeled, stored, maintained, and

used in conformance with manufacturer’s requirements, local laws, andordinances.

Hazard communications programIncludes a list of hazardous chemicals, use of material safety data sheets (MSDS),proper labeling of all hazardous materials containers, and employee training on useof and protection from hazardous materials.

Compliance with all applicablehealth and safety laws andregulations

Requirements will likely vary by region and by level of government (e.g., local,state, federal).




Personnel Management

● Hiring and training● Competent, team-oriented people with

mission critical mentality-


● Develop staffing model● Clearly defined roles and responsibilities




Emergency Preparedness and Response

● Develop emergency operatingprocedures – EOPs – for all high-riskfailure scenarios

● Develo , rehearse escalation


procedures● Conduct regular scenario drills● Formal failure analysis for significant

facility events

See White Paper 199, “ Data Center Emergency Preparedness and Response ”, for

more information.




Maintenance Management

● Key tasks● Asset management● Work order management● Spare parts management


● Ensure power and cooling continual performance

● Improved reliability with● Good asset intelligence● Proactive and preventative predictive

maintenance plan● Results in

● More accurate maintenance budgetforecasts

● Minimized TCO and downtime




Maintenance Management > Asset Management

● Accurate, consistent tracking of critical facility assets● Computerized maintenance management system (CMMS)

● Record, track, and manage asset data and maintenance history● Sco e of service SOS


● Defines maintenance frequency, specific activities, # of man hours● Establishes standard for procurement of

● Service agreements● Maintenance scheduling● Procedure development● Continuous program improvement




Maintenance Management > Asset Management

● Recommended asset management information● Type - top level classification (e.g. electrical,

mechanical, fire system)● Sub-type (e.g. PDU, UPS, CRAH)


●

Text description of asset● Make - asset manufacturer name● Model - manufacturer model #● Size or rating● Location ID (room/area)● Trade responsible for maintenance● Manufacturer serial #● Install date● Warranty expiration date● Date asset to be replaced




Maintenance Management > Work Order Management

● Tool for service process management● Allows work to be


● orrec y pr or ze● Assigned the right resources● Complete d on schedule

● Standalone ticketing system OR● Integrated work order module in a

CMS or DCIM system● Provide valuable information to facility personnel




Maintenance Management > Spare Parts Management

● Shortens mean time to recovery MTTR● Inventory should include parts with lead times longer than acceptable

downtime● Maintain s are arts list


● Stock frequently used items● Re-evaluate annually




Change Management

● Method of Procedure - MOP- process

● Detailed checklist ofspecified tasks


●

MOP helps control workactivity along with● Operational procedure

development and review● Risk analysis and

communication● Structured work practices● Vendor/contractor

supervision




Documentation Management

● Facilitates development of● Accurate procedures● Proper training● Workplace safety


●

Process improvement● Document management software application

● System to keep critical infrastructure recordsorganized, up-to-date

● Detailed checklist of specified tasks● Manual process can also work




Training

● Establish training program that organizes operational and maintenancetasks into categories● Mapped to capability levels – basic, intermediate, advanced

● Train and evaluate personnel to certify them


●

Require annual recertification exams● Ongoing education keeps personnel current




Infrastructure Management

● System to match facility resources with changing IT requirements● Prevent downtime● Improve resiliency

and res onse


● Reduce operatingexpenses

● Provide a soundbasis for capacity

planning decisions● Three key tasks

● Facility monitoring● Capacity management●

IT/Facilities integration




Quality Management

● Key components● Quality Assurance (QA): Typified by process and procedure

standardization● Qualit Control QC : Qualit checks, ins ections, and audits


● Continuous Quality Improvement




Energy Management

● Energy typically the singlelargest data center expense

● 3 core tasks of an effective


energy management program● Performance benchmarking● Efficiency analysis● Strategic energy sourcing

● Optimized energy sourcing● Reduce exposure to price volatility● Secure pricing that fits budget and business objectives




Financial Management

● Financial-related issues can impact facility’sday-to-day availability and resiliency


●

Processes should focus on● Purchasing● Invoice matching● Financial reporting/analysis

● Facility managers and purchasing departmentshould maintain close relationship




Performance Monitoring and Review

● Regularly monitor and review facilityperformance● Determines health and effectiveness

of O&M program


●

Shows where it is trending● Quality process should incorporate

facility KPIs● Benefits

● Aligns operational activities withbusiness goals● Positive reinforcement for innovation

and process improvement



Common MistakesCommon Mistakes Description

Maintenance program is not drivenby metrics

Often the result of poor asset managementNo linkage made between break/fix maintenance

activities and preventative maintenance

Poor trainingTraining is not formalized and/or is not taken seriouslyOver-reliance on technician “shadowing”No linkage between certification level and tasking

Inadequate risk analysis

-


No defined process for performing critical work tasksFailure to consistently test &evaluate skills

Existing skills/training level not formally evaluatedScenario drills are not employedIncident and drill results are not evaluated

Poor documentationNo coherent sequence of operationsDrawings and schedules are outdated

Lack of revision control and/or lack of digitizationFailure to develop and implement aquality control system

Lack of governance or resources to measure, monitor,and review performance

Stuck in manual mode Failure to implement CMMS, EDMS, DCIM, etc

Overconfidence Assumption that future performance can be predicted

by past experience



Facility Operations Services

Using Outside Vendors for O&M Programs

● Offer services for both existing and new data centers● Advise on● Develop


● Operate

See White Paper 198, “ How to Write an Effective RFP for Data Center FacilityOperations Services ”, for more information.




Performance Monitoring and Review > Recommended Facility KPIs

●

Critical load uptime● Load redundancy

maintained● Su ort s stem u time

●

Safety policy and procedureadherence● Procedure development,

management and use


● Maintenance completion● Staffing coverage● Security policy

conformance● Emergency preparedness

drills● Emergency response

procedure adherence

●

Quality control/improvement● Training compliance● Process improvement● Operational reporting● Proper event notification and

escalation● Timely and accurate cost reporting



Conclusion

● Efficient Operations & Maintenance program

● Mitigates threats, effects of human error● Focus on 12 essential elements of O&M program● Must have facilities operation team with “mission critical” mindset


● risk mitigation● Preparedness● standardized processes● continuous improvement



ResourcesFacility Operations Maturity Model for Data CentersWhite Paper 197

How To Write an Effective RFP For Data Center Facility Operations ServicesWhite Paper 198

Data Center Emergency Preparedness and ResponseWhite Paper 199

Classification of Data Center Infrastructure Management (DCIM) Tools


Browse all APC white paperswhitepapers.apc.com

Browse all APC TradeOff Tools™

tools.apc.com

How Data Center Infrastructure Management (DCIM) Software Improves Planning and CutsOperational CostsWhite Paper 107

Avoiding Common Pitfalls of Evaluating and Implementing DCIM Software

White Paper 170

wp_196 essential elements of datacenter facility operations-140902125904-phpapp02

Documents