data centers system studies best practices

8/3/2019 Data Centers System Studies Best Practices

1/5

January-February 2011

Conducting vital power system studies in data centers

Photo: istockPhoto/Petrovich9

By Karl A. Homburg

Electrical equipment in data centers is in constant heavy use, and sometimes abuse, on a daily basis. One useful method tomaintain optimal performance of the various utility sources and generators present in a data center environment is to conductroutine power system studies.

The unique challenges in electrical equipment applications, such as multiple operating modes, must be carefully implemented toachieve desirable system performance under both normal and abnormal conditions. Short Circuit Studies, Coordination Studies,and Arc Flash Hazard Analysis are important elements to the best practice of data center operations to avoid costly damage andsystem downtime, and should be supplemented with regular maintenance for the best performance.

Short Circuit StudiesCollecting and Assessing Data and Common Pitfalls to Avoid. The first step in short circuit studies is finding the maximumavailable fault currents at all locations in the power system. Those results are then compared with power system componentratings to determine equipment ratings needed to safely withstand or interrupt calculated fault currents. The results from shortcircuit studies are also used when conducting the subsequent coordination and arc-flash

studies.

Results are usually presented in a table listing each bus along with voltage and equipment connections, calculated available faultcurrents, and equipment short circuit ratings. The table indicates that the equipment is either adequate or inadequate based onthe available short circuit current.

Inadequate ratings must be addressed, if found, to comply with the National Electrical Code, and other regulations. If direct actionis not taken, a data center risks serious safety hazards such as melted or bent bus bars, enclosure rupture, or even failure of faultinterrupting devices, any of which can cause extensive damage and loss of service continuity.

There are several pitfalls to be wary of when conducting short circuit studies in a data center. The operating scenario that will

http://distributedenergy.com/forms/print-29196.

11/23/2011 10


2/5

Photo: Schneider ElectricDC offsets are important factors when calculating theinterrupting duty of fault interrupting devices like circuitbreakers.

supply the most fault current should be referenced and all sources of fault current considered. In most cases, the operatingscenarios and motor information are not readily apparent upon examining the one-line diagram of the power system. The systemstudy engineer must obtain this information from the system designer during the study.

Parallel Operation. Even in data centers that do not intentionally operate with parallel sources, many of them experience sometime in which sources are paralleled. This is the case if switching is performed as a closed transition. In this scenario, sources arebriefly paralleled to allow the switching of sources without interruption to the load. If done automatically, the time the sources areparalleled is brief, up to only a few seconds. If switching manually, the time can be significantly longer based on the time neededfor the worker operating the equipment. During the time that the sources are paralleled, the power system equipment is exposed tothe much larger fault current of the paralleled sources.

Single Line-to-Ground Faults. Single line-to-ground faults when a delta-wye-connected 3-phase, 4-wire system orwye-grounded generators are employed pose yet another pitfall, especially when substation transformers or generators are close-coupled to distribution switchgear. The smaller the distance, the smaller the zero sequence impedance, compared to positive andnegative sequence impedance. This may cause single line-to-ground fault magnitude up to a theoretical maximum of 150% of thethree-phase fault magnitude.

X/R Ratio. The X/R ratio is always larger than zero since inductive impedance is found in every power system. An inductor storesenergy in a magnetic field, which requires current flow. Since that energy cannot be instantaneously dissipated, the inductoropposes changes to current flow, thereby creating a DC offset after a fault is initiated. During fault conditions, the higher the X/Rratio, the higher the DC offset.

DC Offsets. DC offsets are important factors when calculating the interrupting duty of fault interrupting devices like circuitbreakers. For LV interrupting devices, the result used to compare to the device interrupting rating will be higher than the actualcalculated fault current if the X/R ratio calculation shows the X/R ratio of fault is greater than that for which the device has beentested. This becomes an issue if the interrupting rating of the device is only marginally higher than the available fault current, aconcern especially in data center applications containing low-voltage generators operating in parallel.

Coordination StudiesThe purpose of a coordination study is to determine the proper settings forovercurrent protective devices in the data center power system. Ideally, theselection of the proper settings will both protect the power system equipmentas well as remove the smallest portion of the electrical system as necessaryfrom service in order to isolate a fault. In most instances, compromises must bemade in order to provide the best overall system reliability.

To establish the proper setting for the overcurrent protective devices, they areplotted on time-current curve graphs (TCCs). The time-current characteristicsof properly coordinated devices will not overlap on a TCC graph.

Interpreting the TCC GraphThe TCC graph relies on the information determined in the short circuitanalysis to determine where the device time-current curve is cut off. Eachdevice has its characteristic cut off at the maximum fault current available forthe connected bus.

Many data centers utilize ANSI draw-out style switchgear, or similar hybridswitchboard construction, at their service entrances. This type of switchgearhas the benefit of the ability to conduct rated fault currents for time periods upto 30 cycles. This capability to withstand allows the instantaneous function ofthe main circuit breaker to be turned off in many cases avoiding overlap andproviding for better coordination.

There are some exceptions to the no-overlap concept, for example, if twodevices are operating in a current-limiting mode at the available fault currentthe curves will fall below 0.01s on the TCC graph. Even though they do notshow overlap on the graph, additional techniques are required to judge


11/23/2011 10


3/5

Photo: Schneider ElectricA system study engineer analyzes a variety of operatingscenarios.

selectivity. Generally, this occurs with fuses and requires the use of fuse ratio tables to determine selectivity.

With circuit breakers, there are some instances where the overlap of device characteristics does not necessarily mean that thedevices do not coordinate. Most often this occurs with thermal magnetic circuit breakers. Several circuit breaker manufacturerspublish what are called selectivity tables for their circuit breakers. T hese are similar to the fuse ratio tables that have existed forfuses for many years.

Coordinating Parallel GeneratorsThe coordination of parallel generators can pose a great challenge in a data center. It is not uncommon for the feeder breakers in

generator paralleling switchgear to have a similar long time trip rating as the generator breakers. In this scenario, it may beimpossible to adjust the devices to have good selective coordination, while providing protection that matches the generatorsdecrement curve.

If two identically sized generators with identical impedance characteristics are operating in parallel, the fault current will be splitapproximately equally between them both. Meaning, the generator breakers will only see half of the fault current that a feederbreaker would see. Using this relationship, the generator breaker curves can be shifted to the right by a factor of two. This shiftingof the curves allows a visual representation of the actual conditions that exist during the parallel operation. The same methodologyapplies to numbers of generators greater than twothe curves would be shifted by a factor of three for three generators, four forfour generators, and so on. This shifting of curves applies only to faulted conditions where the fault is downstream of bothgenerators with shifted curves, and not to overloads. If only one of the sources is in operation, however, fault overcurrentcoordination would still be based on the non-shifted curves.

This technique can additionally be applied to a generator paralleled with a transformer, or two generators of different sizes. Thescaling factors would simply need to be adjusted based on the ratio of the impedances of the sources.

Arc Flash Hazard AnalysisAn arc-flash hazard analysis has become a standard and incredibly importantpart of many system studies. The analysis determines the available arc-flashincident energy at all of the buses in the power system so that electricalworkers are aware of the potential hazard and can make informed choicesabout personal protective equipment.

The foundations of arc-flash analysis are the IEEE-1584-2002 [4] and NFPA

70E-2009 [5] standards. The calculation methodologies for arc-flash analysisrequire the results from the data centers short circuit and coordination studies.The results of these studies have to be applied carefully to the arc-flashanalysis so that realistic estimates of the available incident energy are made.

Multiple Operating ScenariosMultiple operating scenarios need to be considered by the system study

engineer in order to properly calculate the incident energy levels in the system for the arc-flash analysis. Unlike short circuitstudies where the main objective revolves around the maximum available fault current, when it comes to arc-flash studies, allpossible available fault currents are a concern. In many cases, the highest incident energy levels are produced by the scenariothat produces the lowest available fault current. This is a result of fault currents falling just below the instantaneous or, short timepickup of circuit breakers that will last for a much longer time period, exposing an electrical worker to more energy.

PDUs and TransformersAccording to the IEEE 1584 standard, buses operating at 240 V or less, fed by transformers less than 125 kVA in size, do not needto be considered in an arc-flash hazard analysis and can be assumed to be Category 0 (1.2 cal/cm 2). Many data centers,however, use transformers larger than 125 kVA in their power distribution units (PDUs). This can pose a problem if trying to keepthe incident energy levels in the power system low.

There are two problems associated with 480- to 208-V transformers in relation to arc-flash within data centers, the first concerningthe physical construction of the PDUs and the second is related to the low level of arcing faults on the 208-V terminals of thetransformer.

PDUs are typically constructed with 208-V secondary breakers and a branch circuit panelboard within the same enclosure. Thisspecific construction poses two risks. First, the primary of the transformer represents a 480-V hazard while a worker is performing


11/23/2011 10


4/5

work on the 208-V components. The second risk is exposure to the line side of the 208-V secondary breakers. This location isquite often an arc-flash Hazard Category 3. The arcing fault current on the secondary of a 480- to 208-V transformer is usuallyrelatively low when compared to the instantaneous pickup of the 480-V breaker feeding the PDU. For that reason, the primarybreaker usually takes several seconds to clear a fault on the 208-V side. When the arc-flash calculations are stopped at twoseconds, per IEEE 1584, a Category 3 arc-flash hazard/risk rating is very common. Something to always keep in mind is that forlocations where clear egress is not possible, it may be prudent to extend the calculations past two seconds. Since all thecomponents are in the same enclosure, this rating must be applied to the entire PDU. Yet there is a possible solution to this PDUproblem.

To work around this, consider the equipment construction and layout. A solution is to move the branch circuit panelboards to aseparate enclosure. Most of the time the 208-V breaker feeding the branch circuit panelboard is capable of interrupting the arcingfault fast enough to allow for a Category 0 rating at the panel. By moving it to a separate enclosure the worker is no longerexposed to the unprotected transformer secondary when working on the branch circuit panelboard. The hazard still exists in thePDU, but workers need access to the PDU less often than the panelboards.

Arcing Faults on the Output of a UPSWhen performing arc-flash studies the assumption that an UPS system will switch into bypass during an arcing fault is the norm. Ifthe arcing fault current is too low, however, this may not be the case.

Information about the behavior of a UPS system for low level arcing faults is difficult to obtain from most of the major UPSmanufacturers and any information that is available shows that different UPS models have different behaviors, even if from thesame manufacturer.

Generally, if the arcing fault current is less than the overload rating of the UPS, the unit will not switch into bypass and the onlycurrent flow will be from the output of the UPS itself. For slightly higher faults the unit may operate in pulsed parallel mode in whichboth the UPS output and the bypass are paralleled for approximately 40 milliseconds, at which time the UPS output breaker willopen, leaving only the bypass to supply the fault.

The third scenario is one in which the fault current significantly exceeds the UPS overload rating. In this case, the UPS willimmediately switch to bypass mode.

The scenario that produces the worst-case incident energy level could vary significantly depending on the UPS size, availablefault current and overcurrent protection types. Each system should be carefully considered in consultation with the UPSmanufacturers published data to make a final decision.

Power System Study Data AccuracyAccurate data about the power system in a data center is absolutely essential for any system study to accurately predict itsbehavior. Sometimes even seemingly insignificant errors in the system data can produce significant errors in results. This isespecially the case with an arc-flash hazard analysis. Even small errors in the power system data can have a very significanteffect on arc-flash calculations. Underestimating cable lengths, for example, can cause arc-flash incident energy levels to beunderreported. Overestimating lengths could also have an effect when comparing the withstand ratings of equipment whenperforming a short circuit analysis if the fault current falls close to the actual equipment rating.

Another area that could cause significant errors is neglecting to consider the type of conduit used for feeders. Assuming amagnetic conduit when PVC is actually used or vice-versa can have a significant effect on the fault current calculations.

Finally, a third common area that can introduce errors is the available utility fault current.

Incorrect Cable LengthsIncorrectly estimating the lengths of the power cables in the system can have a negative impact on data center study results.There are two concerns that are in opposition with each other, the first concern would be overestimating the length. This wouldcause the study results to underestimate the available fault current at a bus and determine that devices are adequately rated,when in actuality they are not. Generally, this would only be a problem for a gross overestimation of the lengths since most of thetime equipment ratings are not chosen to be too close to the available fault current. It is also relatively easy to guard against thissituation by simply scanning through the short circuit comparison table for devices that are close to their interrupting rating. If thereis little confidence in the accuracy of the cable lengths some time could be spent on only these cables to obtain more accurateinformation.

The second concern would be underestimating the cable lengths. It is more difficult to spot check cable data for an arc-flash


11/23/2011 10


5/5

analysis that for a short circuit analysis. Both cables that are estimated too long and too short can produce erroneous results. Ageneral rule of thumb would be to keep any estimation error to less than 10% of the actual length.

Utility Fault Current DataA final, common area of concern, when considering the data accuracy is the available fault current as given by the electric utility.Although it has improved in recent years there are still many utilities that only give the maximum available fault current at acustomers service entrance. This is most often the case with 480-V service entrances. For medium voltage service entrances it iscommon to get the maximum value along with an actual present value.

The maximum value is needed to run the short circuit analysis however lower values should always be considered whenperforming an arc-flash hazard analysis since lower fault currents quite often produce higher incident energy levels.

In ConclusionData center power systems present unique challenges to system study engineers and design engineers. The selection andarrangement of the components in a given power system significantly affect the arc flash study outcome. Design practices thatused to be common such as infinite bus short circuit calculation, large PDU transformers, etc., are not conducive to controlling arcflash energies. Arc flash mitigation must be designed into the data center power system and incorporated into the initial conceptstage.

Furthermore, due to the various operating scenarios, data center power system studies require a diligent study engineer. Selectinga study team that has sufficient resources to evaluate the system parameters completely and accurately, and investing thenecessary efforts to acquire a reasonable compromise between selectivity/reliability and arc flash energies is the true key to asuccessfully running data center.

Topics: Data Centers, Facilities, Power Systems


data centers system studies best practices

Documents