data centre energy

8/9/2019 Data Centre Energy

1/54

Data centre energy efficiency metricsExisting and proposed metrics to provide effective understanding and

reporting of data centre energy

Liam Newcombe Data Centre Specialist Group


2/54

Data centre energy efficiency metrics http://dcsg.bcs.org

Data centre energy efficiency metrics 1

1 Introduction ........................................................................................................................ 2

2 Summary ........................................................................................................................... 6

3 Overview of metrics ......................................................................................................... 12

4 Data centre energy efficiency .......................................................................................... 155 Estimation of fixed and proportional overheads .............................................................. 31

6 Metrics when selecting data centres and equipment ...................................................... 37

7 Monitoring and measurement.......................................................................................... 40

8 Impact of external temperature on data centre efficiency ............................................... 44

9 Glossary........................................................................................................................... 50

10 Acknowledgements ......................................................................................................... 52

11 References ...................................................................................................................... 53


3/54



1 Introduction

Whilst there is considerable coverage of IT energy use, despite the work of the European andother government agencies there is still little hard information on the total size, powerconsumption or efficiency of the data centre market. In the absence of this information it isdifficult to defend the data centre industry, predict growth or set effective metrics and targets.To deal with these issues, properly understand the scale of the problem and deliverimprovements it is essential that an initial set of measurements and metrics is agreed uponand data collection commenced on a large scale.

1.1 Demands and constraints upon data centre operators

In recent years the commercial, organisational and political landscape has changedfundamentally for data centre operators due to a confluence of apparently incompatibledemands and constraints.

The energy use and environmental impact of data centres has recently become asignificant issue for both operators and policy makers. Public perception of climate changeand environmental impact has changed substantially, delivering real commercial impactsfor corporate environmental policy and social responsibility. Unfortunately, data centres

represent a relatively easy target due to the very high density of energy consumption andease of measurement in comparison to other, possibly more significant areas of IT energyuse. Policy makers have identified IT and specifically data centre energy use as one of thefastest rising sectors. At the same time the commodity price of energy has risen faster thanmany expectations. This rapid rise in energy cost has substantially impacted the businessmodels for many data centre operators and has already driven changes in the way datacentre capacity is charged for commercially. Energy security and availability is also fastbecoming an issue for data centre operators as the combined pressures of fossil fuelavailability, generation and distribution infrastructure capacity and environmental energypolicy make prediction of energy availability and cost difficult.

Figure 0-1 Demand and constraints on data centre operators

Opposing these constraints are demands from the business consumers of the data centre

services. The underlying growth in demand for IT services to the business is continuing andnow in addition many businesses are looking toward IT systems to reduce their


4/54



environmental impact in other areas, e.g. logistics systems for a road transport or tele-commuting. As businesses have become more dependent upon IT services therequirements for availability and continuity of services have increased, multiplying theequipment requirements. This is a particular issue in sectors where regulators cover ITsystems. A failure to understand the relationship between the falling capital cost of ITequipment and the rising costs of housing and powering it in the data centre is also creatingcapacity and financial problems for many operators1.

1.2 Metrics

In this context of rising energy cost, energy security concerns, environmental pressure andbusiness demand data centre operators will soon be targeted, measured, grouped orlabelled by the efficiency of their facility. Many streams are currently underway both in theEuropean Union and worldwide to develop and apply efficiency metrics; specifically thispaper investigates the metrics suggested for the EU Data Centre Code of Conduct.

The scope of the metrics discussed in this paper is restricted to the data centre mechanicaland electrical infrastructure. These metrics do not reflect the efficiency with which ITservices, the end product, are delivered to users. This is a clear end goal for a metricsdevelopment work streams and the capability to form part of a holistic set of system metricsis a core consideration.

1.2.1 Methodology

The DCSG believes that for the industry to make real progress any data centre efficiencymetric will need to be part of a measurement methodology designed to calculate areasonable and fair approximation of the total environmental and financial cost of theservice provision from the data centre.

1.2.2 Reporting measures and metrics

Many parties have identified the need for measures or metrics to describe how efficientlya data centre transfers power from the source to the IT equipment and define whatconstitutes an IT load versus what is overhead. For example, the Green Grid have

defined the PUE (Power Usage Effectiveness) and DCiE (Data Centre infrastructureEfficiency) metrics which have been useful in promoting both the understanding thatthere is an issue and enabling further discussion of what effective metrics would need todescribe.

1.2.3 Analysis and diagnostic measures and metrics

Whilst the DCiE metric approach is effective in providing initial recognition of a problemand helps justify the need to implement energy saving changes this is only half of thesolution. Once the issues are recognised there is a requirement for analysis metrics andtools to determine why the efficiency is poor and to assist operators in selecting andmaking effective financial and environmental improvements. This relationship betweenreporting and analysis metrics is shown in Figure 0-2.

The BCS Data Centre Specialist Group has investigated these issues, specifically in aEuropean context and from the perspective of how the IT hardware interacts with thedata centre (building) infrastructure. In this paper the DCSG will discuss more detailedproposed analysis metrics for operators that support detailed analysis and prediction ofthe impacts of changes, specifically we will present a breakdown analysis of DCiE as;

Facility fixed overhead multiplier

Facility proportional overhead multiplier

These fixed and proportional metrics for the data centre are directly analogous to thefinance concepts of fixed and variable cost and we will use them in a similar way tounderstand the real energy and cost behaviour of the data centre and how that impactsthe cost and energy use of operating IT equipment within the data centre.


5/54



1.2.4 Stabilit y of metrics

The reporting measures and metrics are measures of performance at a point in time oraveraged performance over the measurement time frame. To show changes inperformance these measures are inherently sensitive to changes and the values arevolatile.

Analysis measures and metrics have the opposite requirement and should to be asstable as possible for each data centre and as independent as possible of the varying ITworkloads and IT equipment contained within the facility to support effective decisionmaking and planning.

Figure 0-2 Roles of reporting and analysis metrics

1.3 Phases of metrics development

Many bodies within the industry see IT energy efficiency reporting metrics development forthe data centre in three distinct phases;

1. Data centre infrastructure, specifically how efficiently energy is delivered from thepower source to the IT equipment in the facility

2. IT equipment, how many units of computing (or storage and networking) work caneach IT device deliver per unit of energy consumed

3. Useful work, how many units of useful, end user work can each IT service deliverper benchmark unit of computing work consumed

There are several development streams underway in phases 2 and 3, to describe the ITequipment efficiencies (Green Grid DCP, SPECPower

a etc) and IT useful work (DEST

b)

and in additional metrics to deal with the stages beyond the data centre infrastructure asshown in Figure 0-2. This paper concentrates on the first of these phases of metricsalthough the analysis is informed by the DCSG work on holistic modelling of the datacentre.

ahttp://www.spec.org/power_ssj2008/

bhttp://www.elsparefonden.org


6/54



1.4 Driving behavioural change through per service accounting

To significantly reduce the environmental impact and inefficiency of current data centresand housed IT services the BCS DCSG believes that it is necessary to enable and drivechange at the demand as well as the supply side of IT. The data centre only exists to house

the IT equipment whose purpose is to deliver IT services to support the business processesof users or customers; it is not an end in itself. The BCS has been promoting the use of perservice accounting since November 2007.

1.4.1 Relating IT or useful work to business value

Even once metrics have been developed and the information required to calculate themcaptured there will still be an issue relating these to business value or using them foreffective comparison of operators. How would we compare the value of a supercomputermodelling protein synthesis with an online auction site or sales of insurance policies?

1.4.2 Targeting demand

Often facilities and IT departments have little control over the nature of the IT estate and

facility that they manage. This is because most of the services delivered and supportedand frequently the associated IT equipment are dictated by the business units theysupport. Finance systems for finance department, HR systems for HR department, etc.

There have been a number of calls recently from pressure groups to make the ITdepartment pay the power bill for the data centre with the view that this would solve theIT energy use problem. Unfortunately, applying targets, carbon taxes, incentives orregulations to the IT departments increases the pressure on the IT department but thiswill not necessarily translate effectively to changes in business policy as the ITdepartment will have to individually sell green solutions and change to each businessunit. To effectively reduce the carbon footprint of IT we should apply the cost andtherefore incentives to those who control or are responsible for the business processesand creating the IT demand. If we are to transfer the power or carbon bill it needs to goall the way to the consumer of the service to apply the behavioural change pressure at

the most effective point.

1.4.3 Per service financial and environmental costs

Many of the current approaches and metrics under development are intended to answerthe question what do I get out of my data centre for each unit of energy I put in?. TheDCSG proposes that the industry works instead towards metrics and models that cananswer the question what is the financial and environmental cost of each IT service thatmy data centre delivers to support a business process?. Approaching the issue in thisway allows businesses to account effectively for the rising overall cost of IT services(financial and environmental), rather than the coarse IT budget cost allocation formulasfrequently used. The per-service cost accounting approach allows direct comparison ofeach IT service cost with its business benefit.

A common example of why the business unit or owner should be targeted is the issue ofretention of legacy systems, when an IT department requests permission todecommission an existing service the business owner frequently objects and the serviceis maintained. Where an external service provider is used the business unit will frequentlybear the cost of this service directly, thus it is more likely to be decommissioned,virtualised, or archived to be made available if and when required.

This issue will become more significant with the implementation of business carbonaccounting where IT departments will have to justify their carbon budget alongside theirfinancial budget. If internal IT departments can be assisted to achieve per-service costand energy accounting the carbon and financial costs of each service can be effectivelyallocated to the business units responsible for them using an internal market model. Thistrend is already underway in the increased proportion of IT departments receiving theirpower bill and working to pass this on through internal accounting processes

2.


7/54



2 Summary

There has been considerable progress in the market in a relatively short time in identifyingdata centre efficiency reporting metrics. There is a general consensus developing that thedevices directly involved in delivering the useful work of the facility, that is the IT equipment,servers, storage, networking and appliances are the energy targets and that any otherconsumption of energy such as power conditioning and distribution or cooling is overhead.

2.1 Recommendation to operators

The DCSG recommends to facility operators that energy measurements be taken for theirfacility and a combination of the Green Grid DCiE and DCSG fixed and proportionaloverheads be used to report on and understand the efficiency of the facility. Operators whowish to achieve significant cost and carbon reduction should consider extending theircapability to full simulation modelling using an open, independent tool such as that beingdeveloped by the BCS in partnership with the Carbon Trust

3as soon as possible.

2.1.1 DCiE reporting

A time averaged (energy reading) DCiE can be used to report to senior management thecurrent efficiency of a data centre in a form that can easily be understood Our DCiE is0.5, which means that only 50 percent of the power we paid for actually went to ITequipment. This is the primary utility of the DCiE measure.

The DCiE metric is not a fixed value for each data centre; it varies depending upon the ITelectrical load, which is a variable and site specific function of the IT software,architecture, hardware, load and efficiency. Due to this variability the DCiE is not bestsuited to predicting the impact of changes to the data centre or IT and DCiE should bereported in the context of this caveat. DCiE is not a measure on which decisions or plansshould be based.

2.1.2 Fixed and proportional overhead

Whilst the DCiE provides a useful representation of the achieved efficiency of a datacentre, operators also need a set of analysis metrics that can be used to understand thebehaviour of the facility, its response to changes in M&E or IT equipment and the utilitypower used to operate the IT equipment it supports.

To provide operators with an intuitive understanding of energy use the DCSG has foundit useful to represent the energy use of a data centre in two parts, the fixed andproportional energy use as described in Section 4.6. The fixed energy use of a facility isthe power that would still be drawn from the utility feed if all of the IT equipment wereturned off (without reconfiguring the M&E equipment) whilst the proportional energy useis the power drawn in response to IT equipment electrical load. These metrics can beeffectively used to determine the impact and ROI of changes such as consolidation andvirtualisation programs as well as select or compare data centre facilities.

The DCSG fixed and proportional overheads are reversible metrics and as stated in theintroduction, part of a measurement methodology that extends beyond the data centreinfrastructure which is designed to deliver per service cost and environmental accounting.

2.1.3 Estimating fixed and propor tional losses

The measurements required to determine fixed and proportional losses are the same asthose to determine the time averaged DCiE, the utility power input and the IT power ofthe facility at a number of time points. The simple spreadsheet supplied with this papercan then estimate the fixed and proportional loss values for the facility as well asanalysing and forecasting the DCiE.

2.1.4 Implement energy measurement too ls now

Due to the combined pressures of rising IT energy use, rising energy costs and risingimpact of environmental considerations all operators should develop a program of energy


8/54



measurement for their facilities. This should, at the very minimum be independentmetering of the utility power to the facility.

There are many sources of variability to the energy use and efficiency of a data centreincluding the IT workload and external temperature. These can create quite significantdifferences in the measured efficiency of the facility at different times of day or year. For

this reason it is important to measure the total energy used by the data centre and thetotal energy delivered to the IT equipment as a long term average as well as recordingthe individual data points to obtain useful information

4.

2.1.5 Measurement frequency

The two parameters of DCiE, utility electrical energy use and IT electrical energy useshould be measured and logged on an hourly basis, even if this is only with temporaryequipment for a few days each quarter. This provides the basic information necessary tounderstand the characteristics of the facility, both to determine the fixed overheads asdescribed in Section 5 Estimation of fixed and proportional overheads.

2.1.6 IT electrical load and workload

Information from the IT monitoring system should be used to compare hourly readings ofIT electrical load with IT workload as described in Section 7.3. How effectively the ITelectrical load tracks the applied workload provides useful information on theeffectiveness of any power management capabilities, technologies or processes in use inthe data centre. Many operators find that although equipment has power managementcapability this is not activated or working correctly and that reconfiguration can yieldsubstantial benefits. This comparison is particularly relevant when old equipment isreplaced to determine how well the new equipment delivers on power managementpromises.

2.1.7 Develop an energy measurement plan and approach

A phased approach to energy measurement, leading to integrated IT and energymonitoring is described in section 7 Monitoring and .

2.1.8 Provision to the peak power of IT devices

Operators should avoid the legacy approach of provisioning power and cooling to thenameplate power of IT devices. At a minimum, operators should move to provisioning forthe as configured peak power of each IT device if not statistical or dynamic provisioningapproaches.

2.2 Using metrics to understand marginal energy and cost

With the increasing commercial focus on the energy cost component of IT services there isdemand from IT management and business units to forecast the marginal energy cost of achange to IT systems. This may be a cost reduction in the case of a decommissioning orvirtualisation program or a cost increase in the case of a new service.

2.2.1 DCiE

As DCiE does not contain any data to separate fixed and variable costs it cannot giveany useful information about the marginal energy or cost of new or reduced IT electricalload. As a combined, point measure the DCiE of the data centre will change under anysignificant change in load.

2.2.2 Fixed and proportional

The proportional overhead measure gives a direct understanding of the marginal energyor cost of new or reduced IT electrical load.


9/54



2.3 Are the proposed metr ics effective?

To determine whether the proposed metrics are effective we will assess them both againsttheir goals and under a range of common use cases to determine where they are effectivein terms of reporting, targets, education, analysis and decision support.

2.3.1 Goals of metrics

To determine whether a metric is effective and meaningful it is necessary to describe thegoal(s) for data centre energy efficiency metrics;

Goal DCiEDCSGF&P

Provide a clear, preferably intuitive understanding of the measure Y

Provide a clear, preferably intuitive direction of improvement Y

Describe a clearly defined part of the energy to useful work function ofthe IT services

Y Y

Be persistent, i.e. the metrics should be designed to be stable andextensible as the scope of efficiency measurement increases, ratherthan confusing the market with rapid replacement

Y Y

Demonstrate the improvements available in a modern design of facility Y Y

Demonstrate the improvements available through upgrade of existingfacilities using more efficient M&E systems

Y Y

Provide a clear, preferably intuitive understanding of the impacts ofchanges

Y

Be reversible, i.e. it should be possible to determine the energy use atthe electrical input to the data centre for any specified device or groupof devices within the data centre

Y

Be capable of supporting what if analysis for IT and data centreoperators in determining the energy improvement and ROI forimprovements and changes to either the facility or the IT equipment ithouses

Y

Table 2-1 Goals of metrics

As shown in the table neither the DCiE nor DCSG fixed and proportional metrics addressthe full set of goals individually but complementary use of the two methods of analysismeets all of the identified goals.


10/54



2.3.2 Comparison of metrics under common use cases

The following table provides an overview comparison of the DCiE/PUE, the DCSG fixed& proportional metrics and full simulation modelling in a number of common use casesfor data centre and IT operators.

Task Averaged DCiEFixed and

proportionalSimulationmodelling

Managementreporting

Simple value, easy tocommunicate

Slightly more complexto explain

Effective businessreporting, morecomplex to perform

Assess anexisting datacentre

Reporting only as anaveraged energyreading.

More complex tounderstand butprovides betterunderstanding, DCiEcan be calculated

More complex tounderstand butprovides detailedunderstanding, DCiEcan be calculated

Benchmarkagainst otherdata centres

Basic level, prone tosignificant errors

More complex tounderstand butprovides betterunderstanding

More complex tounderstand butprovides detailedunderstanding.

Cost and energyforecast a datacentre

Weak, prone tosignificant errors ofunknown scale

Good indicator of costand energy use

Strong indicator ofcost and energy use

Marginal cost orenergy forecastan IT service

within a datacentre

Weak, prone tosignificant errors of

unknown scale

Reasonable indicatorof cost and energyuse in conjunction

with effective ITmodels

Strong indicator ofcost and energy useincluding effective IT

models

Select a newdata centre

Single point indicator,no information aboutIT load, equipment orutilisation changes

Good prediction ofenergy use andefficiency undervarying IT load,equipment andutilisation scenarios

Strong prediction ofenergy use andefficiency undervarying IT load,equipment andutilisation scenarios

Assess changesto IT equipment

Very little information,difficult to predict theimpact of changes

Good indicator of theimpact of changes

Strong indicator of theimpact of changes

Assess changesto IT powerprovisioningprocesses

Very little information,difficult to predict theimpact of changes

Good indicator of theimpact of changes


Assess cost orenergy benefitof Virtualisation

Weak indicator, ROIlikely to beoverestimated byunknown margin

Good indicator,effective ROIprediction

Strong indicator,effective ROIprediction

Assess M&Echanges toexisting datacentre

Little information,difficult to predict theimpact of changes

Poor indicator of theimpact of changes


Table 2-2 Use case comparison of metrics


11/54



As before this table shows that a combination of metrics and methods is required toeffectively support operators and improve the efficiency with which IT services aredelivered.

2.4 Efficiency targets for data centresThere have been a number of groups who have suggested that data centres or operatorsshould be ranked, grouped or targeted using a metric of data centre efficiency. The DCSGhas raised concerns regarding the use of DCiE targets for data centre operators as thispresents the risk of mixed incentives, specifically, the installation of more efficient IThardware is likely to reduce the measured DCiE efficiency.

2.4.1 Green Grid proposed DCiE ranking system

The Green Grid recently proposed a ranking system for data centres based on their DCiEmetric

5, this ranged from recognised at a DCiE of 0.4 (PUE of 2.5) up to platinum for data

centres achieving a DCiE of between 0.8 and 0.9 (PUE 1.25 1.11). This is areasonable set of targets, not only differentiating between facilities on the basis of

achieved efficiency but also providing some high targets to differentiate the bestperforming facilities.

2.4.2 Mixed incentives and weighting systems

One issue with the DCiE is that, as the metric is not stable, it presents the risk of mixedincentives. Specifically, an operator carrying out an energy efficiency program andreducing their total IT electrical load will probably find their DCiE getting worse due to theimpact of the fixed overheads of their facility.

Some proposals contain a mechanism of weighted bonuses against the DCiE typemeasurement for operators who are impacted by this when reducing their IT electricalload. The DCSG argues that these mixed incentives are simply an aspect of the rankingsystem and that ranges of weightings will reduce the value of the rankings.

2.4.3 Resilience and compensation factors

Operators running higher resilience facilities of older design can suffer from higher fixedoverheads and therefore lower efficiency. There have been suggestions that higherresilience facilities should have some fudge factor applied to improve their efficiencyscore.

There are many issues with this proposal;

This weighting is unfair to operators who have taken the decision to build atlower resilience levels to improve their energy efficiency.

We would not subsidise the capital cost of an operator choosing higherresilience, this is a cost versus requirements decision. Equally the efficiencypenalty should remain as an incentive to only use effective and appropriate

designs justified by the business requirements The combination of lower fixed losses in modern M&E equipment and modular

deployment substantially reduces the efficiency penalty of 2N+ resilience

The DCSG rejects the proposal of weightings based on resilience level as it wouldsubstantially devalue any ranking system at the same time as creating greater mixedincentives than those described above.


12/54



2.4.4 Age of the facil ity

The major differentiator in efficiency between facilities is the age of the facility and theM&E equipment installed. As there is a substantial embedded manufacturing anddisposal energy cost in both the physical building and the infrastructure, ranking systems

that reward building a new facility when the overall environmental impact is negativewould be counterproductive.

To mitigate these issues, if targets are to be set for operators then separate targets forexisting and new build facilities should be set. These targets should then beprogressively tightened based on reporting data from the market. This also creates anincentive for M&E component vendors and data centre builders to determine theembedded energy costs of their products to justify the replacement of old facilities oneffective environmental grounds.

2.4.5 Geographic weighting

During the review period for this paper the DCSG received a significant number ofcomments on the matter of Geographic weighting.

The external ambient temperature and humidity can substantially affect the availableefficiency of a data centre. Some operators who are already monitoring their energyefficiency have been able to show significant seasonal variations in their efficiency due toexternal temperature. See Figure 1 in Enabling the Energy-Efficient Data Center

6for an

example of this. The fresh air and economised cooling technologies are particularlyimpacted as they require the external temperature to be lower than the set cooling planttemperature to operate in their high efficiency modes.

Whilst some operators have flexibility in the location of their data centres this is not ageneral case, a Telecom operator for example has very little choice. Due to this issuethere is considerable discussion over whether to apply external climate weighting factorsto data centre infrastructure efficiency targets. The concern is that un weighted targetswould unfairly penalise data centres in warmer climates

This is essentially a policy rather than a technical matter, therefore the DCSG makes norecommendation.

2.4.6 DCSG recommendation

The DCSG recommends that the DCiE metric be understood as a partial metricrepresenting only the data centre infrastructure and that any ranking or targeting programbe provisional until effective, holistic systems of metrics or models have beenimplemented. Weighting factors for level of resilience or improvement in overall ITelectrical load are unnecessary and will weaken any target or ranking system.

Operators should also separately report their total IT and Utility energy use, thecomponents of the DCiE to show any changes in their IT and overall energy use.


13/54



3 Overview of metrics

There is considerable opportunity for improvements in ICT and specifically data centre energyefficiency, in order to realise these potential improvements it is important to provide not onlyreporting measurements or targets but also analysis metrics and tools that assist operators inunderstanding their facility and the impacts of their choices.

In this section we discuss the scope of current metrics, the development path to holisticmetrics and some of the issues with the DCiE metric, specifically why it is a reporting metricand should not be used to make decisions about a data centre or to build a business case forany data centre or IT changes.

3.1 Limited scope of metrics in this paper

The metrics described in this document focus on the data centre building and themechanical and electrical equipment housed within it. These efficiency measures andmetrics are therefore not intended to be an effective description of the efficiency with whichIT services are delivered from the facility and should not be mistaken for holistic metrics.

3.1.1 Comparing operators on data centre infrastructure metrics

It is not possible to compare the delivery efficiency of two operators based solely ontheir data centre infrastructure measurement. This is only one component of the deliverychain and does not reflect the efficiency achieved by the IT equipment, software orsystems architecture.

3.1.2 Metri cs considered

In this paper we consider the current Green Grid metric, the data centre infrastructureEfficiency (DCiE) along with the proposed DCSG analysis metrics, the data centre fixed

and proportional overheads.

3.2 Holist ic IT efficiency metrics

There is a clear goal for IT efficiency metrics to be able to report and predict the energy andfinancial costs of delivering IT services to the user base as described in the introduction.Both business cost modelling and internal or external carbon markets will require this levelof capability to deliver effective management information about IT services. This goalrequires data centre infrastructure analysis metrics that are reversible and independent ofthe IT equipment to determine the total energy use of an IT device or devices within themeasured data centre.

3.2.1 Chained component level metrics

There have been a number of chains of individual, single value, measures and metricsproposed to indicate the efficiencies of the various layers of the IT delivery, softwareefficiency, IT hardware utilisation, IT hardware efficiency and data centre efficiency.Some of these have proposed that the product of the component metrics is descriptive ofthe overall efficiency.

The DCSG argues that this is a flawed approach and should be avoided by operators asthe results are at best misleading. These metrics are particularly weak in any form ofeconomic analysis and should not be used to try and determine marginal cost. The BCSDCSG has demonstrated by holistic system modelling that these layers interact in amore complex way than is captured in these single value layered metrics.

3.3 Why we need analysis metrics in addition to reporting metrics

Effective sets of metrics should educate and inform the data centre operator and provideenough information to predict the cost and energy impacts of equipment, load or process


14/54



changes. Once reporting metrics such as DCiE have identified an efficiency issue werequire metrics with analysis capabilities to enable operators to make the practicaldecisions required in selecting a new, or improving the efficiency of an existing, data centre.

3.3.1 Why fixed and propor tional overhead metrics?

Fixed overhead drives an understanding of the committed power to a facility. High fixedoverhead facilities should be upgraded, partially decommissioned or filled as close tocapacity as possible to minimise the impact of the fixed losses.

Proportional overhead provides the understanding of how the utility energy use will varywith the IT equipment energy use, specifically what reductions can be expected if the ITelectrical load is reduced.

Fixed and proportional overhead scalars provide a more intuitive understanding of thefacility efficiency to operators, particularly where modular provisioning is used and thesescalars change incrementally as infrastructure is installed or enabled.

3.3.2 Vehicle metrics comparison

Whilst the DCiE and PUE metrics are attractive for their simplicity this also presents a

significant issue if we wish to take decisions based upon them. The metric is amanagement report where three dimensions of data {time, IT load, utility load} have beenreduced to one and decision making information has, necessarily, been destroyed in theprocess. If we take the analogy of the fuel efficiency and load carrying capacity of threedifferent vehicles we can demonstrate this issue. If we take three vehicles, a 38 tonarticulated lorry, a four ton Luton van and a family hatchback and multiply their miles pergallon rating by their load capacity in tons we get their ton miles per gallon load transportefficiency.

38 tonarticulated

4 ton Lutonvan

Familyhatchback

Fuel economy 8 30 60 MPG

Load weight 30 3 0.5 Tons

Load economy 240 90 30Ton miles pergallon

Table 3-1 Comparison of vehicle fuel economies and load capacities

In the above table, if we only consider the ton miles per gallon metric it is clear that weshould all buy 38 Ton articulated lorries to do our shopping. The issue is that we havediscarded the information that tells us which is appropriate for our use and the metricsare now likely to lead us to the wrong conclusion.

3.3.3 IT equipment consol idation comparison

The DCiE metric can produce mixed incentives as we can show with an IT equipmentconsolidation example. In this example an operator with a data centre with a design DCiEof 0.6 expends significant capital, operational expense and tolerates migration risk toreduce the power use of their primary service platform. This platform is several years oldand the IT equipment draws 200kW at the PSU, the new platform the service is migratedto is significantly more efficient, drawing only 50kW at the PSU.


15/54



Before After

Design DCiE of facili ty 0.6 0.6

Fixed utilit y load of facility 100 100 kW

Rated IT load of facility 500 500 kW

Fixed loss mult iplier 0.2 0.2 W/W

Proportional loss multiplier 1.5 1.5 W/W

Rated utility load of facility 850 850 kW

IT electrical load 200 50 kW

Proportional electrical load 300 75 kW

Total utili ty load 400 175 kW

Achieved DCiE 0.5 0.29

Table 3-2 Results of data centre consolidation program

As shown in the table and graphs thefacility has a fixed overhead of 100kW,a rated IT electrical load of 500kW andwould draw 850kW from the utility feedat full rated IT load.

The operator had measured their DCiEat 0.5 before the program andtherefore predicted a 4:1 reduction inutility power to match the reduction inIT power, the power reductionmeasured after the consolidation isonly 2.3:1 which impacts the businesscase and ROI point.

This is due to the combination of thefixed overhead of the data centre notchanging with the IT load and the ITload multiplier being only 1.5 not 2 assuggested by the DCiE. The finalproblem for this energy consciousoperator is that when the expectedreductions in utility power are notrealised and the DCiE of the facility ismeasured again it has fallen to 0.29even though the utility power draw hasbeen reduced by more than half.

The fixed and proportional analysis technique provides a far more effective understandingand forecast of the impacts of the response of the data centre to the consolidation exercise.This both allows a more effective ROI prediction and clearly demonstrates the need to tunethe M&E infrastructure to unlock the full benefits of the IT equipment consolidation.


16/54



4 Data centre energy efficiency

A key part of national Market Transformation Programs and the EU Code of Conduct for DataCentres is to create effective incentives and reporting measures for the efficient use of energyin IT. It has been identified that a measure of data centre to IT equipment energy transferefficiency is key to driving understanding of the energy and cost inefficiencies in current datacentres and thus key to changing the behaviour of IT operators.

4.1 Power use in the data centre

The data centre is a complex environment that is designed to house IT equipment. Utilitypower entering the data centre has to pass through a number of stages of voltagetransformation, distribution and cleaning before finally being delivered to the IT equipment.Most of the power used within the facility is converted to heat, requiring significant coolingsystem capacity which draws an additional load in a traditional, recirculating air data centre.There are also a number of ancillary support systems in the data centre such as lighting,generator pre-heaters, fire suppression systems as well as human occupied areas whichalso require electrical power.

Figure 4-1 IT power delivery path and losses in the data centre

Figure 4-1 shows a simplified representation of the power delivery and loss path in a datacentre. Utility power enters the building on the left and passes through the power delivery

chain to the IT equipment on the right. Each stage in the delivery chain has inherent losses,shown by the red arrows as well as the specific overheads shown as their own paths.

The actual implementation of a data centre is considerably more complex than in thisdiagram and detail such as whether the CRAC units are fed from the UPS may vary. Thisdiagram is provided to provide a general understanding of how power flows through thefacility for the reader.


17/54



4.2 Green Grid data center inf rastructure efficiency metr ic

The metric that is currently used in the working draft of the EU CoC is a version of the datacenter infrastructure efficiency (DCiE) metric originally recommended by the Green Grid.This metric has been selected as it does not consider the IT equipment or software

efficiency for which we do not yet have metrics or units and as such offers the possibility ofcomparing the data centre buildings. This metric concentrates on the proportion of the utilityelectrical load presented by the data centre that is delivered to the IT equipment housed bythat facility. In the EU Code of Conduct the measurements are of energy consumed permonth rather than instantaneous power readings.

4.2.1 DCiE definition

The Data Center infrastructure efficiency metric is defined as the fraction of the ITequipment power divided by the total facility power;

PowerFacilityTotal

PowerEquipmentIT=DCiE

The total facility power is defined as the power measured at the incoming utility meter.The IT equipment power is defined as the power consumed by the IT equipmentsupported by the data centre as opposed to the power delivery and cooling componentsand other miscellaneous loads. For a full description of DCiE see the Green Grid paperon DCiE and PUE7.

There is a view that devices such as KVM switches and monitors represent overheadand not devices directly involved in delivering the useful work output of the facility. TheDCSG supports this view in principle and agrees that such devices should, whenreporting granularity allows, be excluded from the IT equipment power part of thecalculation.

4.2.2 Power usage effecti veness

The PUE metric is simply the reciprocal of the DCE metric;

PowerEquipmentIT

PowerFacilityTotal1==

DCEPUE


18/54



4.3 Data centre IT power to uti lity power relationship

As described above, in order to understand, measure or model the overall energy efficiencyof a data centre it is necessary to understand the relationship between electrical load of thehoused equipment and the utility power the facility draws to power and cool the housed

equipment.The power used by the IT equipment in a data centre is delivered through a series of powerconditioning and distribution devices, each of these exhibits inefficiency and thus a certainamount of power is lost. As this equipment is typically housed within the data centre theselosses, converted to heat, add to the heat output from the IT equipment. This total thermaloutput must then be handled by the HVAC systems.

The graph in Figure 4-2 shows a power transfer function for a 2N+1 resilient (Tier 4) datacentre with a rated 1MW IT load. The power drawn by the data centre from the Utility feedis determined by taking the IT power draw and then applying sequence the losses of eachcomponent within the mechanical and electrical infrastructure. In this relatively simplemodel the inefficiency loss of each component is composed of three factors;

Fixed losses, devices such as the CRACs and UPS have a fixed load component

as soon as they are turned on, before any IT equipment is supported. For examplein a UPS there is the battery charge maintenance power.

Proportional losses, these losses are proportional to the load drawn through thedevice. In a chiller this would be the compressor pumps that switch on and offdependent upon the cooling load.

Square law losses, these are frequently RI2

electrical losses which areproportional to the square of the current carried. This type of loss occurs, forinstance, in transformers and cabling.

The example 2N+1 resilient data centre used in this paper is derived from the following datacentre equipment loss parameters;

Device Rated power(Watts)

Fixed losses Proportionallosses

Square lawlosses

Cabling and switchgear 1,000,000 0.0% 0.0% 1.5%

Power distribution units 1,600,000 0.5% 0.0% 0.5%

Uninterruptible powersupply

1,100,000 2.0% 2.5% 5.0%

Computer room airconditioners

c

1,200,000 10.0% 1.5% 0.0%

Chiller plant 1,500,000 5.0% 30.0% 0.0%

Transformer 2,100,000 0.5% 0.0% 2.5%

Table 4-1 Data centre equipment loss parameters

The fixed losses are particularly significant in this type of design as the 2N+1 resiliencedoubles the impact of the fixed losses of each component, although this is partially offset bythe reductions in square law losses achieved by running the equipment below rated load

d.

c Note that this is a simplistic model for CRAC and chiller plant but provides a useful

approximation at this point. A more detailed model is used later in this paper

d

Note that as the fixed losses of data centre M&E equipment improve the reductions insquare law and other losses in 2N and 2N+1 facilities start to offset the increased fixedlosses, reducing the overheads inherent in 2N type infrastructure resilience.


19/54



Figure 4-2 Data centre overall power transfer function

The graph shows the increasing power requirements at each stage of the delivery chain,note that due to the fixed loads in the delivery chain such as UPS battery maintenance

power and CRAC fan power the data centre would draw a significant proportion of its peakpower even if all of the IT equipment were turned off. For instance, the figure above showsthat at zero IT electrical load the data centre would draw around 670kW.

4.3.1 Fixed overheads drive propor tional overheads

It is not only the fixed load overheads that impact the fixed load power draw at zero ITelectrical load. The fixed load overheads of devices in the power and cooling chains,such as the UPS create an electrical load upon their parent devices as well as heat andtherefore drive further proportional and square law losses in the power and coolingsystems, increasing the zero load fixed overhead beyond that of the individual fixedlosses.

4.3.2 Chiller effic iency and external temperature

Most data centre HVAC systems are impacted by both the internal and external airtemperatures. The efficiency of the chiller pumps typically improves as the externaltemperature falls. This is particularly significant where air or water side economisers orfresh air cooling systems are used to reduce cooling energy use. The analysis presentedin this section uses an averaged external temperature for simplicity. Section 8 examinesthe impacts of external temperature in more detail.

4.3.3 Reductions in fixed overhead through operator actions

The graph in Figure 4-2 does not represent the reductions in fixed overhead that can beachieved in a data centre that is only partially filled where CRAC units or other M&Einfrastructure components may be turned off in unoccupied areas. However, it doesprovide a reasonable representation for the purposes of evaluating the impact of lower

power or variable power equipment replacement programs within an existing facility.There are very substantial efficiency gains possible in a partially occupied data centrefrom such measures as well as modular UPS and chiller systems that are scaled as ITequipment is installed. These measures are investigated later in this paper. SeeElectrical Efficiency Modelling of Data Centers

8 for a more detailed analysis and

description of these loss functions.


20/54



4.4 DCiE & PUE metrics

The data centre modelled in 4.3 would achieve a DCiE of 0.5 or a PUE of 2.0 at the full1MW rated IT load. This, of course is unlikely to ever be achieved as this would require thatthe facility IT electrical load be 100 percent of the rated IT electrical load. This would

require that the facility be fully provisioned at the IT equipment peak power rating, notnameplate power ratings and that all IT equipment be at full power draw simultaneouslywhich would normally be recognised as a failure in capacity planning.

IT electrical load DCiE PUE

0% 0.00 5996226.8

5% 0.07 13.4

10% 0.14 7.4

15% 0.19 5.4

20% 0.23 4.4

25% 0.27 3.8

30% 0.30 3.435% 0.32 3.1

40% 0.35 2.9

45% 0.37 2.7

50% 0.39 2.6

55% 0.40 2.5

60% 0.42 2.4

65% 0.43 2.3

70% 0.44 2.3

75% 0.45 2.2

80% 0.46 2.2

85% 0.47 2.1

90% 0.48 2.1

95% 0.49 2.0

100% 0.50 2.0

Table 4-2 Data centre DCiE and PUE by IT electrical load

As shown in the table above and the graphs below, both the DCiE (Figure 4-3) and thePUE (Figure 4-4) are non linear functions and are significantly influenced by the ITelectrical load in the data centre.

Figure 4-3 Data centre infrastructure efficiency by IT electrical load


21/54



Figure 4-4 Data centre power usage effect iveness by IT electrical load

This presents an issue with the use of DCiE and PUE metrics as the IT electrical load atwhich they should they be measured for each facility is not defined.

If the DCiE or PUE is to be measured and reported by data centre operators there is anincentive to measure this at maximum IT electrical load to optimise the result whilst if IT orutility electrical load is to be reported as has also been suggested, the incentive would be tomeasure at the lowest power point, where the DCiE is at its worst. This presents a conflictin measurement objectives and approaches and would require the definition of some sort ofstandard load profile which is unlikely to effectively represent the range of data centresand their utilisation.

Point measurements of a non-linear function

The basic issue with the PUE and DCiE metrics is that they are point measurements of anon linear function, a single data point is not sufficient to describe this function oreffectively compare between facilities. This issue is compounded by the varyingutilisation of the facilities and IT electrical load within each facility.

This can be addressed by measuring the total energy use of the data centre and ITequipment as a long term average in addition to the point measurements. The level ofvariation in the point measurements is useful and informative data in itself.

4.5 IT equipment power draw

Load to power linear IT equipment

Many component and system vendors are currently expending considerable effort in thedevelopment of new IT devices whose power draw is far more linear with their ITworkload than currently installed equipment. This extends beyond hardware and intosoftware solutions such as VMWares VMotion which provides the capability to moveworking virtual servers under load to optimise the IT load on servers or server blades andshut down unused blades when aggregate workloads fall.

The use of virtualisation technologies can also provide significant benefits in reducing theoverall IT equipment power draw as well as the provisioned power to IT equipment.

Whilst these approaches reduce the IT equipment electrical load and the overall utilityload at the data centre they also, unfortunately, tend to reduce the measured DCiE of thefacility


22/54



Variable IT electrical load

As these new generation devices are installed by data centre operators the IT electricalload will become more variable, spanning a larger minimum and maximum powerconsumption range and exacerbating the problem of measuring overall data centre

performance with a single point measure such as DCiE.

4.6 Fixed and proportional overhead metrics

The key factor that influences the DCiE is the fixed load overhead of the facility. Oneapproach to clearly representing this issue in an easily understandable form that the DCSGhas used is to represent the facility power draw in two components, the fixed and variablepower draw. This is represented by the fixed and proportional overheads. Whilst there issome non-linearity from the square law losses these are dominated by the fixed andproportional losses, allowing this representation to be an effective indicator for operators asshown in Figure 4-5.

ZeroPowerFacility The power drawn at the Utility feed at zero IT electrical load

FullPowerFacility The power drawn at the Utility feed at full IT electrical load

LoadITRated The rated IT electrical load of the facility

LoadITRated

PowerFacilityOverheadFixed Zero=

Fixed overhead has no units as the component units are Watts / Watts.

LoadITRated

PowerFacilityPowerFacilityOverheadalProportion ZeroFull

=

Again, proportional overhead has no units as the component units are Watts / Watts.

Once these two values are determined for the facility the two loss components can beplotted together, in the case of the data centre example from 4.3 these are;

Fixed Overhead = 0.65

Proportional Overhead = 1.41

Note that these two values sum to the PUE at full IT electrical load of 2.06.

Figure 4-5 Data centre power transfer as fixed and proportional losses


23/54



Slope and intercept measurement of the power transfer function

By considering the data centre overheads to be formed of a fixed and proportionalcomponent we are able to approximate the power draw of the facility in terms of the ITpower draw with the two values of fixed overhead (intercept) and proportional overhead(slope).

Whilst there are non-linear elements in the proportional power function these aredominated by the linear elements. The linear proportional overhead is a useful estimatorto the true variable overhead as it provides an effective first-cut analysis and prediction ofthe data centre behaviour. If additional accuracy is required then a full simulation modelsuch as that being developed by the BCS

3should be used.

4.7 IT equipment provisioned power and facil ity f ixed overhead

When IT equipment is installed into a data centre power and cooling are provisioned or setaside for that equipment. This is typically for the nameplate, or power supply maximum ofthe equipment although this practice is being superseded by provisioning to the peak powerdraw of the device in its installed configuration.

As each facility has a maximum total IT electrical load capacity, the power and cooling

provisioned to each device represents a proportion of this facility capacity and therefore canbe considered to represent a proportion of the facility fixed overhead. Our analysis showsthat in many existing facilities the fixed power overhead allocated to a server exceeds itspeak PSU power draw and can become the dominant factor in the server energy use andcost.

Fixed overhead as a motivator to change provisioning processes

Representing this allocation of the fixed overheads in terms of the provisioned power tothe device is also an effective motivator to operators who are still nameplate powerprovisioning to change their processes as there is a direct cost and energy benefit fromdoing so as well as being able to fully utilise the rated IT electrical capacity of the facility.

4.8 IT device power under data centre overheads

The second significant use for data centre efficiency metrics is to allow operators tounderstand the utility electrical load that an IT device such as a server is responsible for.This will be of particular significance in the EU once carbon accounting and carbon cap andtrade mechanisms come into force as both IT departments and service providers will bepushed toward carbon and cost accounting of the IT services they deliver.

IT Device power characteristics

For this example we will choose a commodity x86 1U rack server representative of thosefrom major manufacturers. This is a typical corporate server for a single applicationdeployment with two x86 processors and 2 local hard disks, although the powercharacteristics are unsurprisingly similar to a dual processor blade.

Server power data

Server provisioned power (nameplate) 700 Watts

Server zero IT workload power 200 Watts

Server peak power 350 Watts

Table 4-3 Server power data

This server exhibits a fairly high minimum power though there are many significant anduseful efforts already underway within the industry to reduce this and produce devicesthat exhibit a far more linear relationship between IT workload and PSU power draw asdescribed in 4.5.


24/54



4.8.1 IT device uti lity electrical load

The PSU power draw value is not sufficient to determine the overall energy used by an ITdevice in delivering a service, for either device comparison or carbon accountingpurposes. What is necessary is to be able to determine the utility electrical load thedevice is responsible for within the data centre housing it.

This can be approximated by breaking the device power into two components, the fixedand proportional power draws at the utility feed.

PowerDrawnPSUOverheadalProportionDrawUtilityalProportion

PowerdProvisioneOverheadFixedDrawUtilityFixed

=

=

These two values can then be summed for any workload value to determine the utilityelectrical load required to power and cool the server in the data centre.

Figure 4-6 Server uti lity power draw by fixed and propor tional overheads

Figure 4-6 above shows the power drawn by the server defined in Table 4-3 across therange of applied IT workload, for the example data centre with a fixed overhead of 0.6and proportional overhead of 1.4. The mauve area shows the servers power drawranging from 200W to 350W at the plug whilst the yellow area shows the proportionallosses of the data centre applied to the servers power draw curve, including theproportional losses the server power now varies from 284W at idle to 497W at full load.

We can determine the fixed utility load of the server by allocating a proportion of thefacility fixed power draw based upon the proportion of the facility rated IT electrical loadthat is provisioned to this server

e. This server uses a standard, hot swap power supply

module rated at 700W, even though its peak draw, as configured is only 350W. With this

nameplate power of 700W the purple blue area shows that even when turned off, theprovisioned power multiplied by the fixed overhead gives a draw of 441W at the utilityfeed, this is substantially more than the peak draw of the server.

To determine the overall utility power for this server we add the fixed and proportionalloads together. The servers total utility draw at the idle PSU draw of 200W is 737Wwhilst at the full load PSU input of 350W the utility draw is 949W as shown by the blueline.

e

Note that in the DCSG data centre model the overall utilisation of the facility is also takeninto account and that the fixed overheads of IT devices in a partially utilised data centre canbe much larger.


25/54



4.8.2 Prediction of IT device power savings

Another key use for any data centre power transfer efficiency metric is to assist inforecasting the energy use and cost of operating an IT device as well as the savingsavailable from new, lower power or power linear equipment.

Figure 4-7 Server power draw by fixed and proportional overheads vs. PUE

Unfortunately, as shown in Figure 4-7 the fixed and proportional overhead approachproduces a significantly different result to simply multiplying the servers power draw bythe power usage effectiveness (PUE) of the data centre. This is shown for both thedesign PUE by the red line and the achieved PUE at the operating IT electrical load bythe orange line. The PUE and DCiE are not able to account for the difference betweenthe provisioned power and the actual power drawn, or the fixed power floor of the datacentre and significantly underestimate the power drawn. Further the PUE and DCiE canbe misleading and may seriously overestimate the power savings at utility feed. This willlead to overestimated ROI for low power IT equipment, which could damage operator

confidence in these technologies as described in 3.3.3.

4.8.3 Failure to achieve design PUE

It will be in the interests of any data centre designer to quote the best efficiency for a newfacility design, due to the presence of fixed overheads this is likely to be at 100% rated ITelectrical load

f. Unfortunately the data centre is unlikely to ever operate at full IT electrical

load as this would require that it was fully provisioned and that all equipmentsimultaneously drew its full provisioned power which would, in most instances constitutea failure in capacity planning. This is particularly unlikely in a nameplate provisioningscenario where the IT devices will never reach their full provisioned power. The second,

orange line in Figure 4-7 shows the more likely achieved PUE of 3 (DCiE of 3.0 ) for thisfacility once fully provisioned. As shown the server utility electrical load does intersect the

PUE calculated value at the mean IT workload but diverges significantly on either side.This error will increase as vendors improve the power to load linearity of new ITequipment.

f For traditional chiller technologies, facilities with variable speed pumps, air or water sideeconomisers may achieve optimum performance below 100% rated IT electrical load.


26/54



4.9 Data centres run below capacity

When a new data centre is provisioned one of the defining measures is the total rated IT

equipment load, although this is also frequently expressed as 2m

Watts . It is not

uncommon to build a data centre with sufficient capacity to meet several years worth of

expansion, resulting in the facility operating at a fraction of its rated IT electrical load forconsiderable part of its operating life.

Much of the large infrastructure of a data centre such as the air conditioning and UPS onlyachieves optimum efficiency at close to 100 percent load. If the load presented reducesthen the facility overhead increases. This issue is exacerbated by the high levels ofredundancy in a typical data centre with 2N or 2N+1 resilience, components such as UPSare frequently operating at less than 30 percent of their rated load which substantiallyincreases the impact of the fixed loss overheads, see section 6.1.1 for an example of thisissue.

Al locat ion of f ixed load overheads

The fixed load overhead of the data centre does not change with the IT equipment load.

To understand the energy use and cost for IT equipment in the data centre both the fixedand proportional overheads must be considered in the same way that fixed and variablecosts would be treated elsewhere in business finance. There are two basic approachesto the allocation of fixed load, one is the proportion of the rated IT electrical capacity thatis provisioned to the device and separate accounting for the overheads of the unusedcapacity, the second is the proportion of the total provisioned capacity that is allocated tothe device

g. Either of these approaches drives an understanding of the energy and cost

implications of a partially utilised facility with high fixed load overheads, thus drivingbehavioural changes in operators.

Installed estate vs. efficient new M&E equipment

Whilst many M&E equipment vendors are now producing equipment with verysubstantially improved fixed and proportional losses, these still have overheads and will

not replace the installed estate of equipment in working data centres for several years.

Failure to realise the benefits of load to power linear IT equipment

Another issue for both vendors and operators is to develop the operator understandingthat it is necessary to address the data centre fixed overheads in order to realise the fullbenefits of the power savings offered by the new IT equipment they are installing.

4.9.1 Modular data centre infrastructure

Many M&E equipment vendors and data centre design and build specialists are nowproducing modern, modular designs for data centres. These designs allow the operatorto scale their M&E infrastructure in a more linear fashion to meet the IT load andutilisation of their facility. This represents a significant improvement in both the cost

profile and efficiency of these facilities as the equipment will be running closer to ratedload, thus minimizing the fixed loss overheads. An important secondary benefit of thisapproach is that the M&E infrastructure is more likely to be of the same technologygeneration as the IT equipment it supports and therefore more likely to deliver efficientand effective service.

The graphs below, in Figure 4-8 and Figure 4-9 both show the same data centre,measured with both DCiE and fixed & proportional overheads.

gThe DCSG data centre simulator provides both methods of cost allocation.


27/54



Figure 4-8 Data centre fixed and proport ional losses under modular provis ioning

The graph in Figure 4-8 shows the data centre from section 4.3 with the same inefficient

M&E equipment but this time the M&E equipment provider uses modular provisioning ofthe PDU, UPS, CRAC and chiller systems in 200kW

h steps for rated IT electrical load.

This provides substantial efficiency improvements in the early years of the facilityoperation where the facility is at low utilisation as well as reducing initial capital costs andimproving flexibility. Whilst more complex than just the two values of fixed andproportional, this graph is easy to visualise from an understanding that the fixedoverheads will increase in steps as the M&E infrastructure is provisioned.

Figure 4-9 DCiE under modular provisioning

The graph in Figure 4-9 shows the same modular provisioning approach in terms of thefacility New DCiE against the Base DCiE under monolithic provisioning as shown inFigure 4-3. The DCiE function now varies significantly through the life of the facility withdistinct saw-tooth steps. Although the DCiE is significantly improved across much of theIT load range, if this facility were to be measured purely on DCiE the results may beconfusing to the operator as well as difficult to explain to business management. This isnot a set of results that a facility operator would intuitively expect to see.

h200kW IT load steps, the actual increments are larger for most devices due to the losses

further down the chain


28/54



Figure 4-10 Data centre fixed overhead under modular provisioning

To further illustrate the fixed and proportional overhead analysis for the modular

provisioning data centre Figure 4-10 shows the fixed overhead (Watts drawn / Wattprovisioned) and the proportional overhead (utility Watts drawn / IT Watts drawn) for thefacility at each of the 5 provisioning steps.

Note that in this facility with modular provisioning of the same M&E equipment, theproportional overhead is very nearly constant whilst the fixed overhead is slightly higherat the lower provisioned capacities. This is what one would intuitively expect to happenwithin such a facility as there is no change in the nature of the equipment creating theproportional losses and as the capacity increases the overheads of the 2N+1 resiliencereduce in proportion to the overall load.

4.9.2 Powering down existing CRAC units

In an existing data centre environment where consolidation or virtualisation is undertaken

or where there is still unused capacity there may be the opportunity to clear areas of thedata centre and turn off existing, fixed speed fan CRAC units where they are no longerrequired. This would reduce not only the fixed load of the CRAC units but also theproportional losses they drive.

The graph in Figure 4-11 shows the same inefficient facility as in 4.3 with all of the M&Eequipment provisioned from zero IT electrical load but with the CRAC units turned on in200kW

isteps.

iAgain, 200kW IT load steps


29/54



Figure 4-11 Fixed and proportional losses powering down unused CRACs

Whilst this does not produce as effective a reduction as modular provisioning of thewhole infrastructure, note that each 48kW step in CRAC fixed overhead produces morethan 65kW reduction in overall utility electrical load due to the reduction in proportionaloverheads elsewhere in the infrastructure.

Figure 4-12 DCiE powering down unused CRACs

The graph in Figure 4-12 shows the same data, represented as the DCiE again againstthe Base DCiE under monolithic provisioning as shown in Figure 4-3. The improvement is

smaller here than in the modular provisioning approach but this is action is available in anexisting facility at no capital cost. Note that whilst the curve is smoother and morepredictable the DCiE still varies significantly through the IT load range.


30/54



Figure 4-13 Fixed and proportional overheads powering down CRACs

Figure 4-13 shows the facility overheads for this scenario, as expected the proportional

overheads are nearly constant whilst the Fixed Overhead falls substantially as more ofthe installed M&E equipment is utilised by IT equipment load and the fixed power drawbecomes a smaller fraction of the overall power draw.

4.10 Dynamic modular provisioning

As described in 4.5 many new IT devices are being developed whose PSU power draw ismore variable and related to the IT workload, from energy efficient servers to storage arraysthat power down when idle. This will result in the IT electrical and heat loads in workingdata centres varying significantly through the day and week cycles as the IT workloadchanges. This adds an additional efficiency issue to the data centre design as the M&Eequipment will be provisioned to meet the peak demands of the IT services delivered. Withthe ability for devices to go into suspend and sleep modes IT electrical and heat loadvariations in excess of 50 percent are not unlikely. This will, again, result in unnecessarily

high fixed overhead losses from the facility at low IT workloads, even under modularprovisioning, reducing the energy and cost benefits of this equipment.

Figure 4-14 DCiE curve family under modular provisioning

Many vendors, such as APC and Chloride, already produce modular M&E equipment whichcan be installed in small increments to meet demands. Unfortunately once a unit of capacityis provisioned the fixed overhead associated with that capacity is applied and instead of thesawtooth efficiency curve of Figure 4-9 the facility is better described by the family of DCiE

curves shown in Figure 4-14.


31/54



We suggest that a natural extension of modular M&E equipment be equipment that selftunes to the current electrical and thermal loads by turning off modular components whenthe IT load drops. This would also allow for more flexible installation and more effectivemanagement of modular provisioned solutions as additional modules could be installedahead of requirement and automatically provision themselves when the load breaks the setthreshold.

As for the previous examples fixed and proportional overheads will highlight these issuesmore effectively and in a more intuitive manner that drives understanding of the issues andthe available mitigation strategies.

4.11 Ratio of fixed to proportional overhead

The curved shape of the DCiE graphs is due to the mixture of fixed and proportionaloverhead. If the overheads of the data centre were purely proportional it would exhibitconstant efficiency with IT electrical load and the DCiE graph would be a flat, straight line.The inclusion of a fixed overhead causes the DCiE to be zero at zero IT electrical load. Thisrequires that the DCiE increase from zero to the design DCiE of the facility at full ITelectrical load. The curvature of this line and how quickly the DCiE approaches the designDCiE depends upon the ratio of fixed to proportional overhead.

Fixed 0 Fixed 0.1 Fixed 0.5 Fixed 0.9

Design PUE 2 2 2 2

Design DCiE 0.5 0.5 0.5 0.5

Fixed 0 0.1 0.4 0.9

Proportional 2 1.9 1.6 1.1

Table 4-4 Fixed, proport ional and DCiE

This is illustrated in Figure 4-15 for the set of facilities, all with the same design DCiE of 0.5described in Table 4-4.

Figure 4-15 DCiE curves by f ixed overhead


32/54



5 Estimation of fixed and proportional overheads

It is important that facility operators be able to determine their fixed and proportionalOverheads to make these metrics useful. Whilst the most effective approach is to takedetailed measurements and develop an operating model of the data centre, the samemeasurements used to determine the averaged DCiE can be used to estimate the fixed andproportional overheads.

5.1 New facili ties

It is expected that the data centre design and build contractor should be able to predict thefixed and proportional overheads of the facility at design stage as we have done in thispaper. These parameters are key decision criteria in vendor and design selection. Tomaintain control of the facility capital and operational costs the data centre operator mustspecify efficiency targets whether the facility is built under their direction or via a contractor.Apparent capital cost savings on M&E infrastructure can be outweighed by increasedoperational costs in a matter of months.

More advanced data centre operators using simulation models such as that being

developed by the BCS should request the M&E device performance data and layout toallow simulation of the facility to provide ongoing optimisation and management.

5.2 Existing facilities

In an existing facility there is a multi stage approach to initial estimation and subsequentimprovement of understanding of the fixed and proportional overheads leading towardeffective simulation of the facility.

5.3 Constraints on application

The estimation approach presented here for fixed and proportional overheads provides agood estimate of facility performance for traditionally cooled data centres but is lesseffective for facilities with step variability in load such as those using fresh air cooling orsignificant amounts of air / water side economisation where temperature averaged valuesare required. These can be handled in simulation models of the facility as in section 8 butnot from simple measurements.

It is also important to understand that the proportional overhead is a linear approximation toa compound function of linear and non linear components. Whilst the overall power transfercharacteristic of the data centre is sufficiently linear for these approximations to be useful tooperators only full simulation will provide full detail for facilities with more dynamic powertransfer characteristics.

The primary target for the fixed and proportional estimations are operators who do not haveenough information to enter into a simulation tool, those with new build facilities shouldhave component and overall power loss functions for the data centre from the designer.

This analysis is intended as an approximation tool for operators to predict the response of

their data centre to changing IT electrical loads. Some facilities will experience moresubstantial variation than shown in the example in their efficiency based on externaltemperature, particularly those in more variable climates. For these facilities a multipleregression technique including the external temperature is required.

5.3.1 Required measurements

The first stage is to take a range of measurements of the IT and utility electrical loads asfor the DCiE measure. The table of sample values below, representing 24 hourlymeasurements across one day, is generated from a more detailed model than used insection 4.3 including simulation of the variable cooling loads due to external temperature


33/54



variationj with the addition of 20 percent noise to simulate significant measurement

error.

MeasurementExternal

temperature

Data floor set

temperature

IT electrical

load

Utility

electrical load1 8 21 348,087 1,163,344

2 8 21 352,853 1,191,991

3 8 21 355,194 1,150,685

4 9 21 357,512 1,226,698

5 11 21 375,301 1,078,595

6 13 21 396,073 1,268,379

7 16 21 419,348 1,303,868

8 17 21 471,575 1,388,577

9 18 21 493,588 1,347,376

10 19 21 496,626 1,379,983

11 20 21 481,173 1,340,160

12 22 21 471,575 1,235,116

13 22 21 490,524 1,467,842

14 22 21 499,638 1,464,687

15 21 21 493,588 1,267,441

16 20 21 468,318 1,263,546

17 18 21 451,582 1,215,70818 17 21 415,578 1,329,254

19 16 21 400,067 1,110,475

20 14 21 379,572 1,079,568

21 12 21 357,512 1,058,440

22 11 21 352,853 1,030,307

23 10 21 352,853 1,202,706

24 9 21 348,087 1,132,554

Table 5-1 Set of IT and util ity electrical load measurements

The graph, Figure 5-1 of the measured load data in Table 5-1 shows noticeable scatteraway from a straight line due to the random error used to simulate measurement errorsand other variations.

jSee section 8 for further analysis of the impacts of external temperature


34/54



Figure 5-1 Scatter plot of IT and utility load measurements

5.3.2 Regression analysis of measurements

The next step is to perform a regression analysis of these measurements; this is a simpletask in a spreadsheet. The graph in Figure 5-2 shows the scatter plot with the regressionline overlaid. The important aspect of this graph is that the line indicates a significantutility electrical load at zero IT electrical load at the intersection with the vertical axis. Thisis the approximation to the fixed overhead of the facility, dividing the utility electrical loadat the intercept by the rated IT electrical load of the facility gives the estimated fixedoverhead Watts per Watt of provisioned power for the data centre.

Figure 5-2 Scatter plot of IT and ut ility load measurements with regression line

In the Microsoft Excel spreadsheet provided to DCSG members9this regression analysis

is achieved with the LINEST() function providing the following estimated values for thefixed and proportional overhead as well as the 95

thpercentile upper and lower confidence

boundsk;

kThe upper and lower confidence bounds are determined by using the standard error and

degrees of freedom outputs of the LINEST function as inputs to a two tailed t-test


35/54



Proport ional overhead Fixed overhead

Lower bound 1.102 0.304

Estimated value 1.665 0.541

Upper bound 2.228 0.779

Table 5-2 Estimated fixed and proportional overheads

These values are shown in the graph in Figure 5-3, the error bars show the 95th

percentile upper and lower confidence bounds for the estimation.

Figure 5-3 Estimation of fixed and proportional overheads

From these regression analysis values we can estimate the fixed and proportionaloverheads for the facility to be;

54.0000,000,1

578,558

LoadITRated

PowerFacilityOverheadFixed Zero ===

67.1LoadITRated

PowerFacilityPowerFacilityOverheadalProportion ZeroFull =

=

These values are a reasonable first approximation of the real values for this facility;

Fixedoverhead Proportionaloverhead PUE(100% IT load)DCiE

(100% IT load)

Measured 0.54 1.67 2.2 0.45

Actual 0.65 1.41 2.1 0.47

Table 5-3 Measured vs . actual values for facil ity


36/54



5.3.3 Impact of external temperature on estimation

There is a tendency in a corporate data centre for the IT workload to be low whenexternal temperature is low overnight and higher when the external temperature risesduring the day. This can cause the regression analysis approach to underestimate the

fixed overhead of the facility as the utility power is rising through the influence of bothrising IT electrical load and external temperature. This effect is particularly visible whenthe variance in IT electrical load is small.

This is shown in the sample simulation of the impact of workload and temperature acrossa single day in Figure 5-4, the output power area shows the total thermal load presentedto the chillers which varies through the day, the proportional losses increase through thewarmer parts of the day where the IT workload is also higher. See section 8 Impact ofexternal temperature on data centre for a more detailed examination of these effects.

It is recommended that the external temperature and data floor set temperature are alsorecorded in the spreadsheet supplied to BCS members

9 to allow for later multiple

regression analysis including the impact of external temperature on the facility. This hasnot been included in this initial, simplified version as temperature compensation requiresfurther information from the operator on the type of cooling system in use in their facilityand its performance characteristics.

Figure 5-4 Chiller plant power transfer by IT load and external temperature

As shown in Figure 5-4 the overhead of the data centre is influenced by the external

temperature. In many facilities this is a major influence on the overall utility load andtherefore the efficiency. In the context of this variability the fixed and proportionaloverheads are considered to be instantaneously fixed or proportional. The fixed part ofthe load is still that which would remain if all IT electrical load was removed and theproportional still the remainder. This allows us to average the fixed and proportionaloverheads of the facility over the operating temperature ranges.


37/54



5.3.4 Further analysis and modelling

To improve understanding of the data centre overheads there are two sets of additionalanalysis that would yield useful data. Both of these approaches seek to break down theoverall loss values into the component losses of the data centre infrastructure. This data

would be input into a simulation model such as that being developed by the BCS3

of thedata centre M&E infrastructure to further understand the fixed and variable overheads,where their primary sources are and what mitigation steps are available in that facility.

Equipment audit

An audit of the M&E equipment in the facility should be conducted, gathering the powerspecifications of the components from the manufacturer, maintainer or specificationplates. This is particularly relevant in the case of fixed load items such as chilled waterpumps or

data centre energy

Documents