energy efficient datacenters: electrical design · 6 traditional datacenter electrical design sun...

ENERGY EFFICIENT DATACENTERSELECTRICAL DESIGN Michael Ryan, Brett Rucker, Dean Nelson, Petr Vlasaty, Ramesh KV, Serena DeVito, and Brian DaySun Global Lab & Datacenter Design Services

Sun BluePrints™ Online

Part No 820-6213-10Revision 1.0, 3/11/09

Sun Microsystems, Inc.

Table of Contents

Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1

About This Article . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Traditional Datacenter Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Utility Power and Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Switchgear . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Backup Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Rack Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Datacenter Tier Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Sun’s Approach to Electrical Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Increasing Electrical Distribution Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Component Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Equipment Power Supplies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Power Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Appropriate Levels of Redundancy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Flexibility Through Modularity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Datacenter Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Extending the Datacenter Lifecycle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Electrical Yard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

Building Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Pod Power Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Sun’s Electrical Design At Work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Santa Clara, California . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Broomfield, Colorado . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Sun Modular Datacenter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

About The Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Ordering Sun Documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Accessing Sun Documentation Online . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

1 Introduction Sun Microsystems, Inc.

Chapter 1

Introduction

Good datacenter electrical design supports increased energy efficiency, stability, long-

term growth, and business agility. As the following examples indicate, electrical design

has become paramount in this equation:

• Electrical design that supports growth gives datacenters a longer life. Many

datacenters built even as recently as 5–10 years ago have reached a premature end of

life because their infrastructure was overtaken by density, power, and cooling

demands that were not planned for.

• Modular designs, from the electrical yard to server racks themselves, can support

both rapid growth and rapid change, giving organizations the agility they need to

adapt to changing business conditions.

• System efficiency can be increased while not comprising availability by careful system

design and component selection, such as choosing high-efficiency transformers and

uninterruptible power supplies (UPS).

• Appropriate use and implementation of redundancy can raise efficiency and reduce

construction and operational cost over the life cycle of the datacenter. Everyone

wants a Tier 4 datacenter, but there is inherent capital and operational inefficiency in

having two complete distribution systems. Instead, a flexible and modular electrical

design can support multiple Tier levels in the same datacenter, allowing the cost of

the higher service level to be spent only on the systems requiring it.

• Metering at the right places in the electrical distribution system is a key feature

that’s needed in order to assess operating conditions of the system and components.

Good metering design practices are imperative in the datacenter to continuously

assess system heath datacenter efficiency. Metering should be implemented starting

at the utility feeds and cascading through the system down to the rack level.

About This ArticleThis Sun BluePrints™ article discusses the electrical design principles that Sun’s Global

Lab & Datacenter Design Services (GDS) organization incorporated into its energy-

efficient, pod-based datacenters. Sun’s GDS organization is responsible for Sun’s

technical infrastructure (datacenters, labs, server and communication rooms and

wiring closets) worldwide, and has ushered in a new generation of energy-efficient

datacenters at Sun.

• Chapter 2, “Traditional Datacenter Electrical Design” on page 3, discusses how

datacenter electrical systems are typically designed.

• Chapter 3, “Increasing Electrical Distribution Efficiency” on page 8, highlights the

opportunities that we have found to improve modularity and efficiency in datacenter

electrical design.

2 Introduction Sun Microsystems, Inc.

• Chapter 4, “Flexibility Through Modularity” on page 17, shows how our design

choices create flexible datacenters that contribute to business agility.

• Chapter 5, “Extending the Datacenter Lifecycle” on page 21, discusses the

importance of coordinating expected datacenter lifecycles with equipment refresh

cycles. The chapter shows the many ways in which datacenters can be built from day

one to support future expansion

• Chapter 6, “Sun’s Electrical Design At Work” on page 24, provides a tour of our

datacenter designs from the perspective of the one-line diagrams that describe their

electrical systems. We focus on Sun’s Santa Clara, California datacenters, including

our most recent addition of Sun™ Modular Datacenters at the site.

3 Traditional Datacenter Electrical Design Sun Microsystems, Inc.

Chapter 2

Traditional Datacenter Electrical Design

From an electrical design standpoint, a datacenter is a hierarchy of electrical devices

that transmit power from a utility feed to server racks (Figure 1). One or more feeds

arrive from the electrical utility before or after their voltages are transformed to usable

levels. Switchgear provides, among other things, a disconnect point for the utility feed.

Where needed, UPS and generators provide transient and longer-term backup power.

Conduit and wireways feed distribution systems in the datacenters themselves. Power

distribution mechanisms deliver electricity to racks and standalone systems. Within

racks, smaller distribution systems provide power to individual servers. This chapter

describes the various steps in the distribution chain; the next chapter describes the

ways in which efficiency, modularity, and flexibility can be introduced into each step.

480VSWITCHGEAR

UPSSWITCHGEAR PDUs RACKSFACILITYPOWER

TRANSFORMER

GENERATOR

MORE THAN ONE OF EACH DEVICE INSTALLED AT EACH LAYER

Figure 1. A typical datacenter power distribution system transforms and delivers utility power to equipment racks.

Utility Power and TransformerPower from the electric utility is usually delivered at voltages anywhere from 480V to

21 kV and higher. The first step in the distribution process is to transform this voltage to

a level that is usable by the building and datacenter equipment. The transformer may

be owned by the utility but installed on a pad provided by the customer or it may be

owned by the customer. Both models are used at Sun, and Figure 2 illustrates

transformers owned by Sun at Sun’s Santa Clara, California campus.

In many parts of the world, a single transformer reduces medium voltage from the

utility to 400/230V, with 230 the final voltage supplied to datacenter equipment. (The

400/230V notation indicates three-phase power where the first number indicates phase

to phase and the second number indicates phase to neutral voltage). In the United

States, voltage is reduced in two steps. A first step to 480/277V is used to supply devices


including HVAC equipment and lighting throughout the building. A second step to

208/120V brings the voltage down to that used by compute equipment. Each time

voltage is reduced, some energy losses are introduced into the power distribution

system.

Transformers

Generators

Figure 2. This view of the main switch yard in Sun’s Santa Clara, California facility

shows four transformers connected to three separate utility feeds and switchgear,

Backup generators are in the background.

SwitchgearIn industrial electrical design terminology,

“medium voltage” ranges from 1-33kV.

Datacenters use switchgear and distribution boards to safely distribute power from the

utility to the datacenter floor. The switchgear includes circuit breakers and switches for

managing medium and low voltages and they are typically used to distribute large

amounts of power to various locations within datacenters and buildings.

The circuit breakers installed in switchgear tend to be more sophisticated than breakers

in distribution panels. Switchgear breakers typically have intelligence for monitoring

and control, have configurable trip points, and can include ground fault interruption

circuitry (depending on the design). The purpose of the breakers is to interrupt and

disconnect the load from the power source in the event of abnormally high current

conditions that could damage the electrical system or create unsafe operating

conditions such as a phase-to-phase or phase-to-ground faults. Switchgear may also

include tie breakers and cross-connection schemes that allow the power path to be

reconfigured through the same set of switchgear, or around it in case of a fault. For

example, in a dual-path power system design, it is important to be able to re-route


power around a fault in the system. Tie breaker schemes allow switchgear that would

normally be fed from one source to be switched over to a secondary source. Automatic

transfer switches can also be used to automate this process.

At Sun, a tie breaker scheme allows power supplied to buildings to be doubled. By

installing double-ended switchgear with internal tie breakers, an additional power feed

can be incorporated by installing a new transformer at the opposite end of the

switchgear. The tie breakers then separate the switchgear from one bus into two

distinct buses.

Backup PowerBackup generators are used to power critical equipment in the event of a utility power

failure. Backup power is sometimes supplied to the main switchgear, sometimes it is

provided further down in the distribution system.

Backup power generators cannot start instantly, so another form of backup power

needs to continue to power datacenter equipment during the interval of eight seconds

or less that is needed to start the generators and ready them to carry the datacenter

and support loads. The traditional approach is to use battery backup to support the

datacenter load during this time. Rotational UPS that draw power from a flywheel are

starting to prove themselves superior to battery backup. Both approaches introduce

losses that can be mitigated as discussed in the next chapter.

Power DistributionConduit and wireway distribute the conductors that run from the switchgear to the

datacenter itself, where typically one of three techniques are used to distribute power

to racks. The trade-offs between these approaches are discussed in detail in “Datacenter

Power Distribution” on page 17.

• Distribution panel boards use standard wall-mounted distribution panels at the

datacenter’s perimeter with any combination of conduit, wire mold, raceways,

and armored cable whips running from the breaker panels to individual racks.

Power is typically delivered underneath a raised floor in the datacenter, but also

can be delivered overhead.

• Power distribution units (PDUs) typically consist of multiple distribution panel

boards and can also include internal transformers. (PDUs without transformers are

typically called remote power panels (RPPs). PDUs tend to have a significant

amount of built-in intelligence and monitoring capabilities. PDUs are typically

installed on the datacenter floor throughout the space. PDUs can be used in both

slab and raised-floor datacenters.


• Busway provides a modular track above or below rows of racks. ‘Cans’ combine

breakers and short whips that can be plugged into the busway to provide power

directly to rack-mounted power strips. Busways are capable of distributing three-

phase power across the datacenter floor. The cans are designed to use any

combination of the three phases, allowing different types of equipment to be

powered by plugging in different cans, and for power consumption to be balanced

across phases. Sun has found that busway power distribution provides the most

efficient and modular way to distribute power to the racks and has standardized

on technology for new datacenters.

Rack Power DistributionInside each rack is a power strip or rack-mounted power distribution unit (rPDU) that

distributes power to the IT equipment installed in the rack. Sun has standardized on

rPDUs that monitor power and allow each plug to be powered on and off over the

network. These rPDUs provide data for power monitoring as well as allowing remote

users the ability to power cycle equipment. As the datacenter continues to evolve,

power monitoring and control of individual outlets will increase in importance.

Knowing actual power loads at the server and rack level enables the datacenter

manager to see the power utilization and distribution across the datacenter space. This

information can then be used in the IT equipment provisioning process. It can also be

used to dynamically adjust loads based on server utilization as compute load changes.

Datacenter Tier LevelsAs defined by Uptime Institute's white paper Tier Classifications Define Site

Infrastructure Performance, Tier levels range from 1–4, with Tier 4 being the most

fault tolerant.

Datacenters typically are classified into four Tier levels that are defined by the levels of

redundancy in their power and cooling systems. From the perspective of electrical

design, the Tier levels are defined as follows:

• Tier 1 uses a single path for power distribution, without redundant components.

• Tier 2 uses a single path for power distribution. Redundant components include

multiple UPS and generator power.

• Tier 3 uses multiple power distribution paths along with redundant components.

This allows one distribution path to be taken out of service for maintenance or

repairs without having to take the datacenter offline.

• Tier 4 uses fault tolerant multiple power system and system distribution paths,

redundant components, and uses only equipment with dual power supplies so

that any distribution path or complete system can fail completely without

affecting datacenter uptime.


Sun’s Approach to Electrical DesignThe Green Grid’s definition of PUE and DCiE is

available in their white paper Data Center

Power Efficiency Metrics at:

http://www.thegreengrid.org

Sun’s approach is to use the opportunities that exist within a datacenter’s electrical

design to increase efficiency, flexibility, and provide room for growth. Efficiency

increases help to reduce power consumption and increase the datacenter’s Power

Usage Effectiveness (PUE) or its reciprocal DCiE. Building in flexibility allows a

datacenter to change rack power configurations in minutes, rather than days or

months, contributing to the company’s ability to quickly adapt to changing business

conditions and product development efforts. Preparing for future growth extends

datacenter life by anticipating rapid hardware refresh cycles and the increased power

requirements that accompany them.


8 Increasing Electrical Distribution Efficiency Sun Microsystems, Inc.

Chapter 3

Increasing Electrical Distribution Efficiency

A typical datacenter can easily lose 10 percent of its power in the power distribution

system. Transformers, UPS, and distribution contribute to power loss. Using a number

of the techniques discussed in this section, we have reduced the power distribution

system loss in our Santa Clara, California software organization datacenter from ten

percent to approximately four percent of the total datacenter facility load.

The calculation is as follows: 800 kW x .06 = 48 kW 48 kW x 8760 hours x $0.08 = $33,638

A savings of six percent may not seem particularly significant. Consider one of our

datacenters that has an IT load of 800 kilowatts (kW). Reducing overhead by six percent

is a savings of 48 kW, or $33,638 per year. Any loss due to inefficiency is converted to

heat, so the savings is even greater when one considers that greater efficiency means

less cooling is needed to remove that heat, another cost reduction.

Component SelectionThe low-hanging fruit for increasing electrical system efficiency is choosing high-

efficiency components from transformers and UPS to server power supplies. This

straightforward choice is often overlooked due to higher initial costs. However, if the

total cost of ownership is evaluated, the payback period proves to be very short. With

the current green initiative trend, these component costs are starting to come down,

making the decision easier to justify.

High-Efficiency Transformers

In the United States, most datacenters have at least two power conversion steps that

require transformers. Typical transformers impose about a 3 percent loss, making them

around 97 percent efficient. Using a high-efficiency NEMA TP1-rated transformer can

increase performance to the mid-98 percent efficient range. This not only saves energy

in the electrical system but since all transformer loss are converted to heat it also

reduces electrical room cooling loads.

High-Efficiency UPSThere are many factors that contribute to the efficiency of UPS systems. However, a

common factor across most UPS systems is that the higher the utilization, the more

efficiently they operate. For example, if we look at a typical UPS efficiency curve, the

rated efficiency of the unit is usually stated for loads greater 70% of capacity. Because

of UPS redundancy requirements, UPS system are rarely operated in such a high region.

Performance of the UPS begins to drop off when loads are 50 percent or less of rated

loads. At around 30 percent utilization, UPS efficiency drops very quickly as utilization is

reduced. Since UPS systems are a major contributor to electrical system losses it is


important to make sure the UPS system is designed so that it can be operated at a

higher utilization level and that the efficiency of the unit is maximized for the expected

design operations.

APC white paper #1 describes UPS topologies.

Please refer to: http://www.apc.com

There are many different UPS topologies that can be used in datacenter applications,

including double on-line conversion, delta conversion, and line interactive. In our Santa

Clara datacenter, we chose to use a delta-conversion UPS that uses a modular approach

to support capacity increases. This allows us to right size the UPS for day one needs and

then grow the UPS using 200 kW power modules as the load increases, all without

major construction. This choice provides us with UPS efficiency into the 97 percent

range which is typically about 3–4 percent better than the traditional double-

conversion UPS.

Alternatives to Battery-Based UPS

Battery-based UPS are costly to maintain. Batteries must be replaced over time,

sometimes as early as when they are five years old; rooms must be vented to prevent

hydrogen gas build up; catch pans must be in place to protect against acid spills; and

battery-based UPS take a significant amount of space. Batteries are also not very

environmentally friendly and require controls on the hazardous materials both when

handling and disposing of them.

A low-maintenance, more environmentally friendly approach uses the rotational inertia

of a flywheel to generate electricity during the gap between a power failure and

generator startup (Figure 3). There are two common approaches to rotational UPS:

Figure 3. A line-interaction UPS is

in use at Sun’s Broomfield,

Colorado datacenter.

• The traditional approach uses a typical UPS topology, such as a double-conversion or

line-interactive model, replacing the batteries with a flywheel that stores energy

through rotational inertia. When utility power is lost, the energy stored in the

flywheel powers the datacenter. It can substitute directly for a battery-based UPS

within a building or datacenter.

• Another approach couples a diesel generator with an induction clutch. When power

fails, the flywheel turns the generator until the diesel engine starts and comes up to

speed.

Flywheel-based UPS are finding increasing acceptance throughout the industry because

they greatly reduce maintenance costs and require less space in the UPS room than

traditional battery-based UPS systems. The biggest concern of datacenter operators is

the relatively short ride-through time provided by the flywheel. Battery UPS system are

typically sized to support loads for 10–15 minutes while the generators come on line. A

typical generator, however, is ready to accept full load in 8–10 seconds.

If a generator does not start up in 8–10 seconds, there is typically very little that can be

done to repair it during the 15 minutes the batteries can deliver power. While rotational

UPS do require maintenance on mechanical parts such as bearings, they take far less

http://www.apcmedia.com

http://www.apc.com/

http://www.apc.com/

http://www.apc.com


space and maintenance than battery backup systems. The current drawback is that

flywheel UPS systems tend to have a higher initial cost as compared to a traditional

battery UPS. Reduced maintenance costs usually offset the difference in a short time.

Equipment Power SuppliesThe power supplies on server, storage, and networking equipment also contribute to

power distribution losses. Power supplies are always sized to handle the worst-case

scenario: a server with the maximum memory, internal disk, and peripheral cards

installed and with the CPUs and fans running at full power. Many servers, however,

actually operate at 50 percent or less of their name plate power (Figure 4), a range

where historically, average power supplies can be relatively inefficient.

Percent of Name Plate Power Output

Efficient

45100

100

Average

Perc

ent

Effi

cien

cy

0

Figure 4. Power supplies such as those used by Sun are more efficient across a

broader output range.

Information on the 80 Plus program is

available at http://www.80plus.org.

With today’s focus on energy efficiency, server vendors including Sun are striving to

make their power supplies more efficient at low utilization levels. In the U.S., an electric

utility-funded program called “80 Plus” is promoting the use of power supplies that are

at least 80 percent efficient at 20, 50, and 100 percent of their rated load. Most of the

power supplies on servers from Sun exceed this specification, with some of them 93

percent efficient at an 80 percent load.

Another issue to consider is the fact that systems with redundant power supplies

operate at far less than 50 percent load, often putting them into an inefficient range of

operation. If the application supported by the systems is designed to be resilient to

single points of failure, then servers with single power supplies can be used. This allows

their power supplies to run in a much more efficient portion of their operating range.

Regardless of whether single or dual power supplies are used, care needs to be taken to

ensure that each power supply is plugged into a dedicated circuit.

http://www.80plus.org

http://www.80plus.org


Power CharacteristicsAn organization’s choice of power characteristics — voltages and AC vs. DC —has an

impact on datacenter power distribution efficiency.

208V Vs. 230VThis topic is discussed in detail in the Green

Grid’s white paper 16 titled: Quantitative

Efficiency Analysis of Power Distribution

Systems for Data Centers The paper is

available at: http://www.thegreengrid.org

In the U.S., we typically reduce the utility’s medium-voltage supply in two steps, using

two stages of transformers, down to 208/120V. Other parts of the world use higher

voltages, such as single-phase 230V (line to neutral), yielding some efficiency benefits.

The higher voltage can eliminate the inefficiency of a second transformer step, and it

allows the use of less copper, as distribution paths internal to the datacenter can carry

more power at lower amperage.

Virtually all datacenter equipment is built to accept a range of input power from 100–

240V, so the choice of which voltage to use is one that is dependent on comfort levels

and local electrical codes. Had we had more time to evaluate, test and educate the

project team when we built our datacenters in California, we may have been able to

increase efficiency even further by deploying 230V throughout.

Alternating Vs. Direct Current

There is a long-running controversy regarding the relative merits of using high-voltage

DC versus AC power in the datacenter. The argument is this: high-voltage (380–575V) DC

eliminates the requirement for large amounts of copper in the distribution system,

making this aspect competitive with AC power. Using DC throughout the datacenter

eliminates three AC-DC power conversions that add to inefficiency (Figure 5):

• 480 VAC is converted to direct current at the UPS to charge the batteries.

• Battery power is inverted to 480 VAC to supply the datacenter.

• The first stage of datacenter equipment power supplies converts incoming AC power

to high-voltage DC, which is then regulated down to the low DC voltages used within

the server, storage, or networking equipment.

UPS PDU

AC/DC DC/AC AC/DC DC/DC

Power Supply Unit (PSU)

Inte

rnal

Volt

ages

480 VACSupplyPower

Power Inefficiencies 380 VDC (typical)

ServerLead-Acid Batteries

Figure 5. The argument for DC power in the datacenter is that using DC power

throughout eliminates multiple inefficient AC-DC power conversions.


http://www.thegreengrid.org/~/media/WhitePapers/White_Paper_16_-_Quantitative_Efficiency_Analysis_30DEC08.ashx?lang=en

http://www.thegreengrid.org/~/media/WhitePapers/White_Paper_16_-_Quantitative_Efficiency_Analysis_30DEC08.ashx?lang=en


The final report resulting from this project is

available at:

http://hightech.lbl.gov/documents/ DATA_CENTERS/DCDemoFinalReport.pdf

In conjunction with a California Energy Commission-sponsored project, Sun

implemented a proof-of-concept that demonstrated the power savings of using DC in

the datacenter environment. Sun created two identical datacenters, one with AC and

one with DC power distribution. Off-the-shelf equipment from Sun, Cisco, and Intel was

used, with the power supplies modified by hand to eliminate the power conversion

circuitry that converts alternating to direct current. Incoming power was rectified,

distributed at 380 VDC, and fed directly into the modified power supplies. This was a

proof-of-concept demonstration only, as 380 VDC power supplies are not available as

products from Sun. (Sun does offer 48 VDC-powered equipment through its Netra™

product line designed for communication-carrier environments). The results showed a

2–5 percent improvement in power supply efficiency, and up to a 10–20 percent

improvement when the use of a flywheel vs. a battery-based UPS was factored into the

equation.

We believe that the benefits demonstrated by this experiment are superseded by

improvements in power supply efficiency and that similar improvements are possible

without the use of DC power. The equipment used in this experiment used older

equipment equipped with power supplies where a greater improvement could be

shown compared to today’s more efficient power supplies.

Another approach that is being discussed is using the telco standard of 48 VDC.

Regardless of which DC power approach is used, we see some key disadvantages:

• Equipment that accepts high-voltage DC directly is not available on the market today.

There is a limited selection of equipment that can use 48 VDC.

• There is a relative lack of experience and comfort using DC power in the datacenter.

Distribution systems, system power connections, and breakers are all different, and

everyone from electrical contractors to system administrators would need training.

• Maintenance and arc-flash suppression has not been uniformly addressed.

• For most companies, the ability to leverage the newest and best server, storage, and

network technology is key to maintaining a competitive edge. Today, the choice of DC

power restricts equipment choices from almost all major manufacturers.

Appropriate Levels of RedundancyDatacenter power efficiency can be increased by using redundancy in appropriate ways.

First, choosing a level of redundancy that is matched to the organization’s mission, and

their level of accepted risk, increases efficiency by reducing the amount of redundant

equipment that is necessary, and also by reducing layers of components (such as UPS)

that add to inefficiency. Second, redundancy can be implemented so that equipment is

used in a more efficient part of its operating range.

http://hightech.lbl.gov/documents/DATA_CENTERS/DCDemoFinalReport.pdf

http://hightech.lbl.gov/documents/DATA_CENTERS/DCDemoFinalReport.pdf


Choosing Tier Levels

Everyone wants to have a Tier 4 datacenter, with two complete power distribution

systems so that either can be taken out of service without affecting datacenter uptime.

In fact, most organizations only require a small portion of their datacenter to be built to

Tier 4 specifications — if any at all. The cost of implementing a Tier 4 design and

continuous operation at that level can be more than five times greater than a Tier 1

facility.

Although our electrical design approach can support any Tier level, our datacenters at

Sun are used for internal product development and typically require Tier 1 or Tier 2

support. Our IT specific datacenters provide more critical services, and therefore require

a Tier 3 level of service.

A pod, as defined in the Sun BluePrints Series

article The Role of Modularity in Datacenter

Design, is a small, self-contained group of

racks and/or benches that optimize power,

cooling, and cabling efficiencies. A typical

pod is a collection of 20–24 racks with a

common hot or cold aisle along with a

modular set of infrastructure for it to handle

its own power, cooling, and cabling.

Our datacenters can support multiple Tier levels in the same room. Different

configurations of our modular busway can feed multiple parallel power distribution

paths down to individual racks. Since most of our equipment does not actually require

UPS power, we can run two busways, one with UPS power and one without, to each of

our pods. Racks that require redundancy and UPS power can connect to both busways.

Those that don’t require UPS power do not. In the end, only approximately 20 percent

of our systems actually require UPS power, helping to increase efficiency and

significantly reduce the capital and operating costs associated with uninterruptible

power.

Our pods are self-sufficient from a power, cooling, and cabling perspective, so a Tier 3

pod can be placed in the same room with Tier 1 and Tier 2 equipment. The same would

be true if the corporation required Tier 4 as long as a dual-path system with or without

redundancy was carried back to the central plant.

Using Distributed Redundancy

Using an appropriate level of redundancy in the electrical distribution system is one

means to increase efficiency; the way in which you implement redundancy also has an

impact on efficiency.

2N Redundancy

A typical UPS configuration uses 2N redundancy, where one of two UPS systems can

support the entire load should one UPS fail. This model operates each UPS at less than

50 percent utilization. Typical maximum utilization levels are 40–50 percent so that

when the UPS has to support 100 percent of the datacenter load it does not exceed its

full capacity rating and trip offline. It should also be noted that UPS systems operating

in this range are relatively inefficient especially if operated at less than expected full

design loads. A hypothetical configuration illustrated in Figure 6 shows two UPS

powering a datacenter consisting of three racks.

http://wikis.sun.com/display/BluePrints/Energy+Efficient+Datacenters+-+The+Role+of+Modularity+in+Datacenter+Design



Figure 6. In a 2N redundant model, each UPS is sized to support 100 percent of the

workload but operates at less than a 50 percent utilization level.

N+1 Redundancy (Parallel Redundant)

This model uses N+1 UPS to support a load where one UPS can fail and the remaining N

are sufficient. The example in Figure 7 shows a three UPS N+1 UPS system, each UPS

supports 33 percent of the power draw, but is sized to support 50 percent in the event

that one fails. This puts each UPS at a more favorable 66 utilization level. Because each

UPS is sized too small to power the entire server workload, the outputs must be

synchronized to a common bus, and it is a single point of failure for this model.

Distributed Redundancy

A distributed redundancy model uses three or more UPS connected with multiple paths

to servers. A distributed redundancy UPS system model can be used to achieve multiple

distribution path redundancy where required without the need for a 2N system.

(Figure 8). For example, a three-UPS distributed system can achieve a utilization level of

66 percent. This is the same utilization level as the N+1 model, but the configuration

eliminates the need to tie outputs together and eliminates the bus as a single point of

failure. The benefits can include lower capital cost as compared to a 2N system,

increased efficiency, and dual power paths to servers.


Figure 7. In an N+1 redundant model, all UPS are connected to all servers, but the

bus is a single point of failure.

Figure 8. In a distributed redundant model, each UPS supports one third of the

servers and operates at a 66 percent utilization level. There is no single point of

failure.


As with all approaches to UPS redundancy, distributed redundancy requires adjusting

the balance of servers to UPS, and this requires a flexible, modular power distribution

system that can adapt as a datacenter grows. Modularity and Flexibility are the topics

of the next chapter.

17 Flexibility Through Modularity Sun Microsystems, Inc.

Chapter 4

Flexibility Through Modularity

Modularity is a key datacenter design principle that enables flexibility and the ability to

grow. At Sun, our modular, pod-based design creates energy-efficient building blocks

that can be duplicated easily in any size or Tier-level datacenter worldwide. The pod

design allows different modules to be selected and configured to best meet client

requirements as a datacenter changes over time. Following our modular, pod-based

approach, different pods in the same room can to be built to different Tier-level

specifications because our electrical design can support it. The modular approach

allows pods, and racks within pods, to have radically different densities, with a rack

using 4 kW next to one that requires 30 kW. The modular approach is flexible so that a

4 kW rack can be replaced with one using 30 kW — or vice versa — in a matter of hours. The same modular design that supports brick-and-mortar datacenters also can

be used to support Sun Modular Datacenters as discussed in “Sun Modular Datacenter”

on page 34

Datacenter Power DistributionFor a more detailed discussion of the trade-

offs summarized here, see the Sun BluePrints

article: Energy Efficient Datacenters: The

Benefits of Modular Design.

The feature that most supports the day-to-day flexibility of our datacenters is our choice

of power distribution on the datacenter floor itself. The challenge here is two-fold: the

power distribution system must be able to support growth so that the datacenter can

handle increased density at every equipment refresh cycle. The power distribution

system also must be able to handle day-to-day equipment churn that significantly

changes the power draw at the rack level. At Sun, this is extremely important, as spaces

used for new hardware bring-up must allow for the next generation of products and

even higher densities to be rolled in and powered up without long delays. Our power

distribution system helps future Sun products to get to market faster because all outlet

types and loads are supported. Our datacenters have products in them today that our

customers won't see for two years.

Hard-Wired Power Distribution

The oldest, and the least-flexible power distribution system is one that uses circuit

breaker panels at the datacenter perimeter with conduit, wire moulding, or armored-

cable whips running from the breaker panels to individual racks or groups of racks.

This approach limits flexibility, as a significant amount of electrical work is

necessary in order to add or change service to a rack. Every time a panel is opened,

all of the equipment attached to it is at risk.




Power Distribution Units

PDUs combine transformers and breaker panels in units that are placed on the

datacenter floor. PDUs are typically installed throughout the datacenter floor, with

individual flexible whips powering individual racks. PDUs occupy valuable

datacenter floor space and add to the datacenter’s heat load. Whips are difficult to

manage, and every time that a PDU is opened to add or change a configuration, it

introduces the possibility of human error, putting everything the PDU powers at risk.

Although we have used PDUs in some of our earlier designs, we now see it as a

limiting factor in those datacenters’ flexibility. This is because all changes require an

electrician, the PDU must be powered down, much more time is required for changes,

and the cost can be up to ten times more over the life of the electrical system when

compared to busway systems.

The Benefits of Modular Busway

Our pod design removes all transformers and power distribution units from the

datacenter floor. Instead, we use a hot-pluggable busway overhead or under a raised

access floor to distribute power to each rack (Figure 9).

Figure 9. Modular busway is installed overhead or under a raised access floor that

is not used for air distribution.

Each busway includes three phases, one neutral, and one ground conductors that are

accessed by ‘cans’ that tap into the right combinations for the power to be delivered.

The busway supports an intermixing of 120V, single-phase power up to 208V, three-

phase power (or the equivalent geographical power configuration for datacenters


located outside of the U.S.). Despite its capacity to handle up to 100A through a single

can, an electrician can install a new circuit in a matter of minutes. This flexibility is key

in our datacenters where equipment churn occurs on a regular basis.

This modular approach can be used to support different Tier levels of service on a pod-

by-pod basis. A Tier 1 pod can be supported with two busways, one over each row of

racks in a pod. Tiers 2–4 can be supported with four busways, with the second pair of

busways providing access to a redundant power source. This allows us to provide the

required level of service to our customers while saving the cost and inefficiency of

providing redundant power to every single rack. The pod illustrated in Figure 10 shows

four busways delivering redundant power to the pod.

Two busways aboveeach rack

Busway ‘cans’

Figure 10. A pod using four busways provides two redundant power paths that

support Tier 2-4 levels of availability.

We have found busway to work equally well whether used with the in-row cooling

approach illustrated in Figure 10, or with overhead cooling units as illustrated in

Figure 11.


Two busways aboveeach rack

Busway ‘cans’

Figure 11. Busway can be used with overhead cooling units as easily as with in-

row cooling.

21 Extending the Datacenter Lifecycle Sun Microsystems, Inc.

Chapter 5

Extending the Datacenter Lifecycle

The industry is seeing a new wave of datacenter construction that is driven by

premature obsolescence of existing datacenters that has resulted from the high

demand driven by the global consumption of technology and services. Facilities built

during the late 1990s were expected to last 15–20 years, and most have reached a

premature end of life. While these facilities may have sufficient space, their power and

cooling systems no longer can support the IT equipment in them.

The well-known reason for this problem is a mismatch between the datacenter’s

projected life span and the frequency and impact of equipment refresh cycles. If a

datacenter is built to last 15 years, then it must be built to accommodate equipment

refresh cycles that range from 3–5 years. For example, a high-density datacenter in the

year 2000 consumed around 3–4 kW per rack. Now 3–4 kW per rack is below the

industry average, and today’s high-density racks are exceeding 20–30 kW per rack!

In our datacenters at Sun, we have incorporated expansion capacity to support three-

year equipment refresh cycles. From day one, we have built a modular system that can

accommodate growth through the addition of new modules, and we have built extra

capacity in terms of additional equipment locations and power paths. This “future

proofing” of infrastructure required an additional investment of 10–15 percent in pipe

work for the mechanical/electrical systems. But this scalability enabled Sun's agility

and paid for itself within the first year.

This chapter follows the path of electricity from the utility to servers. It illustrates how

datacenters can be built to support the rapid change and increases in density that

occur over time without requiring major construction inside or outside of in the

datacenter itself.

Electrical YardThe electrical yard is where the utility power is provided to the datacenter. When

building a datacenter, or retrofitting an existing building, it’s important to plan for the

future with the electrical utility: be sure that long-term power needs can be satisfied by

the utility and under what terms. In other words, design backwards from the amount of

power you will require or can receive to reach your end state. Build your electrical yard

with expansion in mind:

• Plan for additional transformers that connect to new utility feeds by preparing

concrete pads for them from the beginning.

• Design switchgear so that new feeds can be installed and cut over seamlessly.


• Configure switchgear so that there is room for growth. One technique is to use cross-

connect (cross tie) schemes to allow a bus to be split in half and powered by two

separate utility feeds in the future.

• Leave room in the switchgear for additional breakers, and run empty conduits —

sized to accommodate the largest possible wire — from the electrical yard to future

equipment locations in buildings. Trench and run conduit underground only once. At

Sun’s Santa Clara facility, we have installed conduit from the electrical yard to

datacenter spaces to accommodate growth within datacenters. We also have run

conduit from the electrical yard to existing office spaces that may become

datacenters in the future.

• The electrical yard is typically home to one or more backup generators; plan space for

growth by allowing space for additional generator capacity to support your end state.

Building InfrastructureEach datacenter has a significant amount of electrical distribution infrastructure that

needs to be designed and sized with room for expansion:

• Prepare to expand UPS systems as electrical loads grow. This requires not only a UPS

design that is modular and can be expanded, but also floor space for additional UPS

modules and battery racks or flywheel cabinets.

• Use transformers, not PDUs. Our modular, pod-based design uses transformers in

separate rooms rather than PDUs on the datacenter floor. Reserve space for

additional transformers that growth in electrical loads will require.

• Design power distribution with room for expansion and preinstall conduit to support

the future growth. For example, if the goal is to support 100 percent growth over the

datacenter lifecycle, then design distribution boards to be 50 percent full on day one

to allow for easy expansion over time. Empty conduits to support future expansion

should also be run at the time of original construction to reduce the impact of the

expansion on the operating datacenter.

• Use industrial busway to provide power from the distribution panels to the pods. This

allows power feeds to be changed as power consumption within pods grows.

Pod Power DistributionEach of our pods use a modular overhead busway to deliver power from the distribution

system to the equipment racks. Modular busway contributes to a datacenter’s agility by

making it possible to reconfigure equipment racks for completely different power

requirements — voltage, phase, and amperage — in a matter of minutes rather than

hours, days, weeks, or months. Busway also enables a datacenter’s expansion capacity

by supporting increasing power draws as equipment is replaced gradually over time.

Rather than sizing the overhead busway to the maximum expected power draw, we

have designed a strategy that can double the power provided to a pod in two phases —

allowing the power draw to increase by a factor of four over time.


For example, consider a pod with two busways, one above each row of racks (Figure 12).

• Our initial configuration is to jumper the two busways together through a conduit

bridge and provide a single power feed for each pair of busways.

Phase One:One feed for twobusways using

Phase Two:One feed foreach busway

Phase Three:One feed foreach split busway

jumper

Figure 12. As a pod’s power draw increases, its supply can be increased in two

phases, each phase doubling the previous power capacity.

• The pod’s power feed can be doubled by removing the jumper and providing a power

feed to each busway.

• The pod’s power feed can be doubled again by splitting each busway in half and

providing a separate feed for each half.

This expansion strategy avoids the up-front cost of purchasing high-amperage busway,

while providing a way to support a four-fold increase in power consumption over time

with minimal disruption or construction costs. This level of expansion in a traditional

datacenter electrical distribution system would require a complete repowering of the

datacenter space along with significant downtime and expense. Keep in mind that this

strategy requires your conduit to be larger and pathways to be pre-defined to support

the end state quantity and size of conductors. Choosing one size busway that supports

the final load will allow the re-use of cans over the functional life of the datacenter.

These busways also support higher voltages as discussed in the previous chapter.

24 Sun’s Electrical Design At Work Sun Microsystems, Inc.

Chapter 6

Sun’s Electrical Design At Work

The design principles that we have discussed to this point are embodied in our

datacenter designs worldwide. This chapter provides a top-to-bottom overview of our

Santa Clara, California datacenters followed by a discussion of Sun’s newest facility in

Broomfield, Colorado, and an overview of Sun’s use of Sun Modular Datacenters at the

Santa Clara site.

Santa Clara, CaliforniaOur electrical design tour of our Santa Clara datacenter is through a series of one-line

diagrams. These diagrams illustrate the various components and how they are

connected without the detail of showing each wire for each phase — hence the term

“one line” diagram.

Electrical Yard

Our Santa Clara project, one of the largest datacenter consolidation efforts in Sun’s

history, allowed us to construct an entirely new electrical service yard to supply power

to new datacenters and buildings.

This electrical yard receives two 12 kV power feeds from Silicon Valley Power, each of

which allows us to draw 4.5 MVA of continuous power. To ensure that we have power to

grow over time, we have negotiated with the local utility to provide up to 24 MW of

additional power to the electrical yard through additional 4.5 MVA feeders.

A set of medium-voltage switchgear allows the additional feeds to be incorporated into

the electrical yard without disrupting existing service. The medium-voltage switchgear

feeds three similar sets of 480V switchgear through step-down transformers, one of

which is illustrated in Figure 13.

The switchgear supplies 480V power to office buildings and datacenters on the Santa

Clara campus. The switchgear includes breakers for existing spaces and for future

expansion, with conduit runs already installed to the future spaces.

We accommodate expansion at this level of the distribution hierarchy by supporting

each of the three sets of switchgear with a single transformer. Each set of switchgear is

configured with two halves that are connected with a tie breaker that is currently

closed. When the time comes to double the power supply, a second transformer will be

connected to a new utility power feed, and the tie breaker will be opened. Now each

half of the switchgear has a 3 MW continuous operating capacity.


Existing transformer suppliesswitchgear with 480/277V

Future transformer allowsa doubling of capacity

Tie breaker remains closeduntil future transformer is installed

Breakers, future breaker locations, and power feeds to buildings and datacenters

Figure 13. Each of three sets of 480V switchgear can support a doubling of power by adding a second transformer and

opening a tie breaker.

A photograph of the Santa Clara switch yard gives a visual perspective (Figure 14).

• In the foreground are three rows of similar switchgear. From right to left is the 12 kV

switching upstream of the transformer. Next is the transformer, followed by the 480V

switchgear. Note the empty space at the end of two of the rows to accommodate an

additional transformer and switchgear. One already has been installed.

• Behind the 480V switchgear is the medium-voltage, 12 kV switchgear that

accommodates existing and future utility feeds.

• Also visible in the photograph are two diesel backup generators.

12kV Switchgearand Transformers

Generators

480VSwitchgear

ExpansionLocations

12 kV Utility Feedsand Switchgear

LoadBank

Figure 14. A view of the Santa Clara electrical yard shows the 12 kV to 480V

transformers and switchgear in the foreground.


Backup Generators

Because the majority of our datacenters are for research and development efforts, the

pods in them are designed to Tier 1–2 specifications. For 20 percent of the load, we

deploy UPS to support more critical operations. As operational risks increase, we deploy

generators behind the UPS to provide continuous power.

Figure 15 shows a portion of the one-line diagram showing our backup generator

configuration. Two backup generators are currently installed, with space, conduit, and

breaker positions for a third. This provides 2 MVA of power today, with 3 MVA when the

third generator is installed. The generators provide power to a set of paralleling

switchgear that combines the generator outputs. A 1000 kW load bank is installed to

provide a full load when testing the generators. Regular 100 percent load testing of

generators is important to make sure the generators are operating correctly and always

ready to accept load.

The paralleling switchgear currently feeds one 800 kVA UPS with provisions for future

growth. In the event of a failure, the feed is switched through automatic transfer

switches (ATS). Two automatic transfer switches (ATS) switch the power feed for the UPS

from utility power to generator power in the event of a power failure. A third ATS

supplies power to keep the cooling system operating.

Existing twogenerators

Futuregenerator

Feed from480Vswitchgear

Supply to UPS

Automatictransferswitch

Figure 15. Our Santa Clara electrical yard includes three generators that provide

backup power to UPS.


Battery-Based UPS

Our Santa Clara site is equipped to accommodate a fourth UPS to bring the total to

3.2 MVA. The site uses three 800 kVA UPS capable of providing up to 2.4 MVA of

uninterruptible power. We chose APC Symmetra MW units for the Santa Clara site for

their modularity and efficiency. The APC UPS is unique in that it enables the UPS system

to grow in 200 kW increments. At the Santa Clara site, two of the three UPS are sized

with 800 kW frames with only 400 kW of power modules installed, right sizing the UPS

to our day one needs. As our loads grow, we will add additional power modules that

grow the UPS to its full 800 kW capacity with only minimal construction in the UPS

room to match battery capacity to UPS capacity.

Power supply fromautomatic transferswitch

To transformers anddistribution panels

Overhead busways providing uninterruptiblepower to pods

Figure 16. An 800 kVA UPS supplies power to three transformers, each of which

supplies 208/120V uninterruptible power to a set of busways.

The UPS illustrated in Figure 16 takes its input power from the automatic transfer

switch illustrated in Figure 15. Its output feeds a distribution panel that connects to

three transformers that step down the 480/277V power to the 208/120V power that


feeds the busways. One transformer is illustrated in Figure 16. It supplies power to a

distribution panel that feeds a set of 225A and 400A busways inside the datacenter

spaces.

Pod Power Distribution

For the majority of our pods that only require Tier 1 service, a single 480V feed from the

switchgear supports multiple transformers that bring the voltage down to the 208/120V

supplied to pods. The example shown in Figure 17 illustrates pairs of busway jumpered

together as described in “Pod Power Distribution” on page 22.

Supply from480V switchgear

480V transformer

Figure 17. Power is distributed to non-critical loads through transformers that

feed 208/120V power to overhead busways.

Metering and Power Management

One of the most important aspects of datacenter electrical design is an integrated

metering and energy management system. These systems are critical because they

provide the ability to evaluate how power is being used in the datacenter, power

quality or disturbance events, how efficiently the datacenter is operating, and what

impact any energy-saving measures have on overall efficiency.


As we discussed in the Sun BluePrints Series article Energy Efficient Datacenters: The

Role of Modularity in Datacenter Design, power usage effectiveness (PUE) is the ratio of

total facility power to the IT equipment load. In order to obtain this ratio, and to be

able to observe energy consumption of the various datacenter subsystems, appropriate

metering capabilities must be placed at strategic locations cascaded through the

electrical system. In our Santa Clara datacenters, we have metering and power

management in place at the following locations:

• The utility switchgear

• All major switchgear

• All UPS, ATS, and generators

• Each overhead busway

• Each rack is equipped with a rack-mounted power distribution unit (rPDU) that

monitors overall power consumption and also provides remote on/off per-outlet

control. These rPDUs can be monitored over the network, providing fine-grained data

on IT equipment loads.

Broomfield, ColoradoOur Broomfield, Colorado location is the site of our most recent datacenter

construction project. Spurred by the need to consolidate the former StorageTek site into

our Broomfield campus, there are two significant differences between this datacenter’s

electrical design and the Santa Clara design described above:

• No electrical yard. In Broomfield, the utility provides 480V power through a set of

primary voltage transformers located at the campus perimeter. For redundancy

purposes, these transformers are loop fed through utility company-owned switch

cabinets. This is an example of where the utility owns the primary voltage feeders,

transformers, and switch cabinets.

• Flywheel UPS. The Broomfield campus is the largest installation of rotational UPS

at Sun. The double-conversion UPS used in Broomfield supplies 900 kVA of backup

power, sufficient to power critical loads between the time of a power failure and

diesel generator startup. The UPS pictured in Figure 18 provides 15 seconds of

backup power to critical loads, sufficient to cover the 8 seconds between a power

failure and diesel generator startup.

Main Power Distribution

Power to datacenter equipment located on floors 1–4 of the Broomfield building

originates from utility transformers located at one end of the building. One

transformer feeds power to a 4000A, 480V main distribution board that serves

equipment loads on the first and fourth floors. The second transformer feeds power to

a 5000A, 480V main distribution board that serves equipment loads on the second and

third floors. The two electrical services combined are capable of delivering 7.1 MW to


datacenter equipment on floors 1–4. A transient voltage surge suppression device is

installed at each service entry point to protect the building electrical system from

voltage surges generated outside the building.

Figure 18. The rotational UPS used in Broomfield supplies 960 KW kVA of backup

power, sufficient to power critical loads between the time of a power failure and

diesel generator startup.

The main switchboards are equipped with spare circuit breakers and spaces with

conduit runs to the secondary electrical rooms located on the second and fourth floors

for future expansion. A one-line diagram illustrating this design is illustrated in

Figure 19.

The central plant transformers are fed from a new utility transformer located northeast

of the building. It feeds power to a 2500A, 480V main distribution board which serves

mechanical loads supporting the datacenter. The electrical service is capable of

delivering 2.2 MW to mechanical load in the central plant.

Pod Power Distribution

The two main distribution switchboards distribute 480V power to a series of 225 kVA

and 500 kVA high-efficiency dry-type transformers located in secondary electrical rooms

on each floor. We sized these transformers so the would have a much greater load than

the industry standard of 30 percent. High-efficiency transformers were selected because

at a higher load factor, efficiency tends to remain constant and at a higher level. In

contrast, the efficiency of standard dry-type transformers decline sharply beyond the 30

percent load factor.


Figure 19. Broomfield datacenter main power distribution

Each transformer feeds a large 120/208V distribution panelboard also located in the

secondary electrical rooms. A transient voltage suppression system (TVSS) device is

installed at each distribution panelboard which protects sensitive datacenter

equipment downstream from voltage surges internally generated within the building.

The distribution panelboards are equipped with spare circuit breakers, spaces, and

conduits routed to locations of future datacenter panelboards. The distribution

panelboards feed 120/208V panelboards that distribute power to all 225A busways in

the datacenter. The diagram shown in Figure 20 illustrates how pod power is distributed

from a typical secondary electrical room to datacenter busway.


Figure 20. Pod power is supplied from a 500 kVA transformer located in a

secondary electrical room.

Rotational UPS

A 960 kW/1200 kVA, 480V centralized rotational UPS provides approximately 15 seconds

of back-up power for critical datacenter equipment located in the mainframe, server

room and one pod in the CTC lab located on the second floor. The UPS has four

modular 240 kW/300 kVA flywheels in which three of the four flywheels are needed to

support the critical loads. This N+1 design approach provides redundancy where one

flywheel can be completely taken offline for maintenance purposes. The UPS is

equipped with a full-maintenance wrap-around bypass so that it can be taken out of

service without interrupting power to critical loads. At 97 percent, the efficiency of the

rotational UPS is higher than a traditional battery-based system due to the fact that the

two electrical conversions do not occur under normal operating conditions — only

when the flywheel is carrying the critical load.

The UPS illustrated in Figure 21 takes its input and maintenance by-pass power from a

1600A, 480V emergency distribution panel. This distribution panel is equipped with a

TVSS device which protects sensitive datacenter equipment downstream from voltage

surges internally generated within the building. The UPS output feeds a 1600A, 480V

main distribution panel that distributes power to a large 225 kVA and 500 kVA high-

efficiency dry-type step-down transformers. The transformers feed 120/208V

panelboards that distribute power to all 225A busway in the datacenter.

Back-up Generator

Emergency back-up power to the datacenter and central plan is served from an existing

1250 kW/1562.5 kVA diesel generator located outside the building. The back-up

generator feeds three emergency main distribution boards located in main electrical

room which supports life safety loads and critical secondary mechanical equipment


that supports cooling to critical datacenter and the rotational UPS. It also feeds an

emergency main distribution board that supports critical primary mechanical

equipment in the new central chiller plant.

Metering and Power Management

The Broomfield datacenter has metering and power management throughout. Three-

phase energy and power measurement with data logging and power quality analysis

can be obtained locally from each meter location or remotely from the campus central

control room or off campus from a Web browser. Metering has been installed at the

following locations:

• The two 480V main distribution switchboards are monitored to determine total power

consumption for the four floors of the datacenter.

• Each branch distribution feeder in the 480V main distribution switchboards is

monitored.

Figure 21. The Broomfield rotational UPS are integrated into the power design so

that any one of them can be taken out of service without disrupting loads.


• Input and output distribution panelboards monitor the flywheel UPS power and

quality.

• Each overhead busway incorporates power monitoring, allowing a per-pod evaluation

of power consumption of equipment.

• Each rack is equipped with an rPDU that monitors overall rack power consumption.

These in-rack PDUs can be monitored over the network, providing fine-grained data

on IT equipment loads.

Sun Modular DatacenterFor more information on the Sun Modular

Datacenter, please visit

http://www.sun.com/sunmd

Recent expansion at our Santa Clara site has used two Sun Modular Datacenters (Sun

MDs). This unit is a self-contained containerized datacenter configured in an enhanced

20-foot shipping container with integrated high-efficiency cooling and power

infrastructure. Sun MD is available in 2 configurations: the original eight-rack S20

model, and a new seven-rack D20 edition. Both support a 200 kW capacity in only 160

square feet of space. The Sun MD can typically be deployed much more quickly than a

traditional datacenter; all that is required is to provide appropriate chilled water,

networking and power. This section describes the product power features and an

overview of the external power requirements for deploying a Sun MD.

Power Overview

The Sun MD is available in two different electrical configurations. The low-voltage

version operates on 190 to 220 VAC, three-phase, 50 or 60 Hz electrical service. The high-

voltage version operates on 380 to 415 VAC, three-phase, 50 or 60 Hz electrical service

Power Distribution in the Sun MDPower is routed into the Sun MD through two electrical feeder entrances, one on each

side of the container, and can be provided by one or two external power sources.

The Sun MD has a primary and a secondary internal power distribution panel

corresponding to the two electrical feeder entrances (Figure 22). The internal power

panels provide redundant power to each of the racks. Each panel supplies power

through a set of circuit breakers to each of the racks and to each of the HX fan control

units in the Sun MD.

Supplying Power to the Sun MD

Power can be supplied to a Sun MD through a commercial utility, an on-site utility

power generator, or a standalone power generator in any combination. There are

several factors to consider in designing and deploying external power infrastructure for

the Sun MD, including requirements for redundancy, uptime, mobility, and existing site

power infrastructure.




Figure 22. Sun Modular Datacenter power distribution panels. This view of a low-

voltage unit is from the back of the Sun MD, with A panel on the left and the B

panel on the right.

Uptime is typically a primary concern in the deployment of a datacenter. In order to

provide the highest levels of service, a means of maintaining power in the event that an

external power source fails is highly recommended. This involves a design using an

automatic transfer switch to switch from the primary source of power to a secondary or

backup source of power such as a generator. A UPS is used to supply power to the Sun

MD during the time interval when the primary power fails and when the secondary

source comes online.

While the actual configuration of the electrical power distribution system will vary, a

typical external power infrastructure will have many of the following components:

• Commercial electric utility service

• Site distribution panel

• Automatic transfer switch (if required)

• Chiller/heater sub-panel (if required by building code)

• Transformer (if required to match line voltage)

• Generator (optional)

• Uninterruptible power supply (optional)

• UPS power distribution panel (if required by building code)

When we deployed Sun Modular Datacenters at our Santa Clara site, we utilized the

flexibility of the electrical infrastructure described in previous sections and literally

snapped in the modular datacenters. We installed high-efficiency transformers outside

next to each container to convert 480V to 208V. Installing the transformers outside


eliminated the need to provide cooling for the transformers. The containers in Santa

Clara are installed downstream of our UPS system which is currently providing power to

both the Sun MDs as well as our building-based datacenters.

37 Summary Sun Microsystems, Inc.

Chapter 7

Summary

A datacenter’s electrical design plays a critical role whose influence ranges from

protecting real estate investments to reducing day-to-day power consumption.

Traditional electrical designs can limit growth, reduce business agility, and add, rather

than subtract, from skyrocketing utility bills. In contrast, the datacenter designs that

we have used at Sun facilitate growth, increase agility, and provide significant savings

in energy consumption. Rather than being a static aspect of datacenter design that

hampers business operations, our datacenter electrical design flexibility helps to better

align our datacenters and business operations.

Our modular electrical design provides greater flexibility that helps our datacenters

better support our business operations. Our modular design allows our engineering

organization to roll in and power a next-generation server in a matter of minutes. The

same modularity that allows us to support different Tier levels within the same

datacenter also saves costs by delivering only the required amount of redundancy.

Our philosophy of preparing for future growth from day one has a dramatic impact on

the datacenter lifecycle. By building a datacenter electrical system that can support

future generations of IT equipment, we have better matched equipment refresh cycles

with our planned datacenter life cycles. This not only helps prepare our business for the

future, it helps to protect the immense investment in real estate by increasing the

longevity of our datacenters. This agility has become a competitive advantage for Sun.

Ultimately, a datacenter electrical system must deliver power from the electric utility to

each piece of IT equipment. Taking advantages of the opportunities to increase

efficiency along the way, making sure not to overprovision for day-one capacity, and

through careful monitoring, we have been able to achieve about a six percent greater

efficiency that amounts to an annual savings of $33,638 in one datacenter alone. From

energy-efficient transformers to appropriate implementations of redundancy,

datacenter electrical designs offer abundant opportunity to reduce power consumption

and overall operating costs.

One of the best aspects of our datacenter electrical design principles is that they are

straightforward and easy to implement, so organizations outside of Sun can reap

similar benefits. While our designs initially seemed radical compared to traditional

designs, the choices that we have made are now embodied in mainstream practices at Sun. These practices are also starting to become prevalent in the industry.


About The Authors

Michael Ryan

Mike Ryan is a Senior Staff Engineer in Sun’s Global Lab & Datacenter Design Services

organization. Mike graduated from San Jose State University with a degree in

mechanical engineering. He is a licensed professional mechanical engineer and has

been involved in the design, construction, and operation of mission-critical facilities for

the past 18 years. His experience spans numerous industries such as gas turbine co-

generation, semi-conductor manufacturing, and mission-critical datacenters. For the

last seven years he has focused on the design and operation of mechanical and

electrical infrastructure systems supporting high-availability datacenters. Mike joined

Sun as a Staff Engineer for the GDS organization in March of 2006. Mike is the primary

author of the GDS physical standards and was the lead design engineer for the Santa

Clara, California datacenter project, one of the largest and most complex in Sun's

history. He also provided supporting engineering services for the Broomfield, Colorado

datacenter project. Mike's design work supports the Act portion of Sun's Eco strategy,

and was showcased at Sun's Eco Launch in August 2007.

Mike is a member of ASHRAE (American Society of Heating, Refrigeration, and Air-

conditioning Engineers). He also participates in the Data Center Pulse and the Critical

Facilities Round Table group. Mike was the recipient of Sun's prestigious Innovation

Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of

2008.

Dean Nelson

Dean is the Senior Director of Global Lab & Datacenter Design Services (GDS) in the

Work Environments (WE) business unit of Sun Microsystems. The GDS organization

bridges the gap between Facilities, IT, and Engineering, and is responsible for managing

Sun’s multi-billion dollar global technical infrastructure portfolio including datacenter

design, standards and strategy. The GDS work resides in the “Act” portion of Sun’s Eco

strategy, and was showcased at Sun’s Eco Launch in August, 2007, and the Colorado

Data Center grand opening in January, 2009.

Dean has been in the technology industry for 19 years, of which 16 have been with Sun.

He spent four years in Sun manufacturing in roles ranging from component-level debug

to managing quality. This included helping the drive to achieve ISO 9002 certification.

Dean joined the Sun Engineering community in 1993. He led systems and network

administration support for some of Sun’s largest and most complex R&D lab

environments.

Dean left Sun in 2000 to join a networking startup company called Allegro Networks. At

Allegro, he built a world-class QA team, state of the art global R&D lab environments

and fully integrated automation system.


In 2003, Dean returned to Sun joining the newly formed N1™ Software organization. He

orchestrated the integration of Terraspring and Center Run R&D labs and the merger of

Sun™ Cluster software into N1 technology. In mid-2003, Dean took over management all

of the N1 organization’s R&D labs, build engineering, automation, and capital budget

responsibilities world-wide. In 2004, he became a leading member of the Global Lab &

Datacenter Design Services (GDS) team tasked with creating a strategy to standardize

Sun’s multi-billion dollar technical infrastructure portfolio. He was the architect of the

GDS operating model and lead design engineer for lab and datacenter projects world-

wide. In 2006, he became the GDS Director. Since taking over the group, Dean has

delivered the GDS strategy, executing the largest technical infrastructure consolidation

in Sun’s history compressing the 1.4 million square foot portfolio by 48 percent, a $250

million dollar investment.

Dean holds numerous industry technical and business board positions including

Founder of Data Center Pulse (http://datacenterpulse.org) - an exclusive datacenter

owner community. Dean was featured in Contrarian Minds, a focus on the engineers,

scientists and dreamers of Sun Microsystems. Dean was also the recipient of Sun’s

prestigious Innovation Award presented by Sun’s CEO, Jonathan Schwartz and CTO,

Greg Papadopoulos in July of 2008. Dean lives in Northern California with his wife and

daughter.

Serena Devito

Serena DeVito is a Datacenter Design Engineer in Sun’s Global Lab & Datacenter Design

Services organization. Serena joined Sun over ten years ago and has spent the bulk of

her career as a system administrator within the IT organization where she designed,

deployed, and supported critical infrastructure for the company. She then moved to the

Software organization and managed the beta testing project for the Sun Cluster

product group. Serena acted in a customer capacity running early releases of Sun

Cluster software on production home directory clusters with over one hundred active

users as well as clustered web services. Serena identified, documented, and filed bugs

while supporting one of Sun's largest and complex engineering lab environments.

Serena has been a core member of the GDS team since 2004. In December 2005 she

joined the GDS organization to drive lab and datacenter consolidation projects world-

wide. Along with building numerous labs and datacenters in the Americas, Serena was

the technical lead and customer engagement engineer on the Santa Clara, California

datacenter consolidation project, one of the largest and most complex datacenter

consolidations in Sun's history. For this work, Serena was the recipient of Sun's

prestigious Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg

Papadopoulos in July of 2008.

Serena is a graduate of the University of Adelaide (Australia) with a Bachelors of

Architectural Studies and a focus in Computer-Aided Design. She is a Sun Certified

http://datacenterpulse.org


Solaris Administrator for the Solaris™ 7, 8 and 9 Operating Systems. She also loves

traveling and riding horses.

Ramesh KV

Ramesh has been in the technology industry for over 13 years and has held a variety of

technical & managerial roles since joining Sun in 2001.

Ramesh began his career as a customer support engineer at Kumdev computers, where

his clients included companies in the manufacturing, pharmaceuticals, telecom and IT

industries. He later joined SAS (now know as SASKEN) as a Systems Engineer, where he

supported the NORTEL infrastructure. He then moved on to TATA Elxsi as a Design and

Development Specialist, where he was involved in various key infrastructure projects.

Ramesh spent the initial years at Sun working for the RPE organization as Global Lab

Manager, where he drove the global lab model and designed, deployed, and supported

critical infrastructure across USA, EMEA & India. In early 2006, Ramesh moved into the

role of IT/ Lab Council head for the India Engineering Centre (IEC) reporting to the site

VP as well as the Global Lab & Datacenter Services (GDS) lead for the APAC region. In

this role Ramesh led the physical and technical design for the Bangalore, India

Datacenter consolidation project and supported all APAC lab and datacenter projects. In

2007 Ramesh became a Regional Work Place Manager (RWPM) covering Asia South and

India in addition to his GDS responsibilities. In this role he has been responsible for all

workplace resources activities across Asia South & India, including portfolio

management, real estate services & facilities management. Ramesh has been a critical

member of the GDS technology team since 2004 and has designed multiple labs and

datacenters across Asia Pacific. Ramesh is the recipient of a research disclosure on

“Web Based Multi OS Installation,” PC Quest IT Implementation project of the year

award in 2007 and has received 12 internal awards globally including Sun's leadership

award in 2005 and 2007. Ramesh was the recipient of Sun's prestigious Innovation

Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg Papadopoulos in July of

2008. Helives in Bangalore, India with his wife and daughter.

Petr Vlasaty

Petr has been working in the IT industry for more than 12 years. His experience that

started as an implementation engineer and project coordinator in 1996 has grown into

a variety of people, project, process and service management areas of datacenter

operations and technical support services. Petr worked for ANECT to define and develop

the System Support and Maintenance department responsible for systems

implementations and support for key customers including the Czech Ministry of

Finance, Commercial Bank, and others. Petr formed a 12-person team that defined

internal and external processes, established contracts, and represented support

services to customers. In November 2004, Petr joined the Sun Software organization as


R&D Datacenter Manager and Team Lead for the Prague Datacenter Management

Team. Petr designed and implemented two internal datacenter projects in Prague in

partnership with the GDS team. He applied the GDS processes and standards for the

new Prague datacenter expansion that was completed in June 2006. Due to Petr's

outstanding performance in Prague, he was assigned as the technical lead for the

Louisville, Colorado consolidation project in August of 2006. He spent four months in

Colorado developing the very detailed and complex plan. Petr joined the GDS

organization full time in January of 2007. He is responsible for the Europe, Middle-East

and Africa (EMEA) region and continues to act as technical lead for the Colorado

datacenter consolidation project, the largest and most complex in Sun's history.

Petr is a Staff Engineer, Datacenter Architect, and the GDS Technical Infrastructure Lead

for the EMEA region. He loves sports like mountain hiking (year round, especially

winter) and rock climbing. Petr lives with his wife Vera and 10 month-old son Erik in

Prague, Czech Republic.

Petr is a CCSE (Check Point Certified Security Engineer), CCSA (Check Point Certified

Security Administrator) and SCSA (Sun Certified Solaris Administrator). Petr was the

recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan

Schwartz ad CTO, Greg Papadopoulos in July of 2008.

Brett RuckerBrett Rucker is a Staff Engineer in Sun’s Global Lab & Datacenter Design Services

organization where he focuses on physical environment issues. Brett has been in the

Corporate Engineering, Construction and Management business for 25 Years. A

Graduate of the University of Colorado, Boulder, and a licensed Professional Engineer,

Brett began his career with Ball Aerospace Systems Division, a rich opportunity that

gave him complete design and management responsibility for all construction projects.

He supervised products including building additions, clean rooms, X-Ray rooms,

datacenters, laboratories, anechoic chambers, secure areas, gallium arsenide facilities,

high elevation satellite dish structural mounts, RF shielded rooms, and office,

conference, and customer show areas. Highlights of Brett’s time at Ball include design

and construction of large secured program area in 7-days to support critical Corp.

project, stick build design and construction of MAMA Lab Clean Room facility for 1/3

market cost (still in operation today with one of Ball's premier products), metal finish

Area with a $30,000 per year energy savings in 1992 base, and creation of the Facility

Engineering Program at Ball Aerospace which grew to take over Ball corporate facilities.

In 1992, Brett joined a newly forming engineering group at StorageTek to address

facility backlog and project issues. Brett and his team created corporate building

specifications, drawing standards, and quickly eliminated all backlog. Brett drove the

high tech areas, and HVAC infrastructure as well as general projects. He represented

StorageTek on refrigerant conversion and reported on this project at an ASHRAE


function. Brett became manager of the New Project Management Group in 2000

directing projects at headquarters and in the field. His responsibilities soon included

StorageTek’s energy management program, which deployed processes and delivered

high-quality, low-cost projects with consistent high metrics with limited staff support.

This group lead all project designs, construction and project budgets. The energy

management group delivered ISO Certification, installed monitoring systems, metrics,

monthly log and issue reviews, best payback energy projects and commodity contracts

that beat the market consistently. Brett joined Sun’s GDS team in 2006 as a design

engineer and is the physical design lead for the largest and most complex datacenter

consolidation in Sun's history, the StorageTek move from Louisville, to Broomfield,

Colorado.

Brett is a member of ASHRAE, and AEE, and he was the recipient of Sun's prestigious

Innovation Award presented by Sun's CEO, Jonathan Schwartz ad CTO, Greg

Papadopoulos in July of 2008. Brett is active in developing youth programs and

coaching his daughter’s competitive fast-pitch team with great support from his wife

and two daughters.

Brian Day

Brian Day is a Senior Program Manager in Sun’s Global Lab & Datacenter Design

Services organization. Brian has been working in the technology industry for over 10

years, of which more than five have been with Sun. He spent over two years in Sun’s

Systems group, developing automation programs for server, storage, and software QA

functions. This work included testing, configuring, and administering some of Sun's

largest systems at the time, including the Sun Enterprise™ 10000 and Sun Enterprise

6500 servers, and Sun StorEdge™ D1000 storage device. Brian then served in a similar

role for iPlanet™ software, developing server side Java™ technology-based tools that

automated core team processes (ticketing system, equipment tracking, on-call

notification).

Brian left Sun in 2000 to pursue his MBA full time. He then joined Adobe Systems in

2001 as a Program Manager, leading the development of several Digital Imaging plug-

ins for Adobe Acrobat. In 2003 he took over full time program management of the

Photoshop Elements product, releasing an all new version (3.0) in the fall of 2004. This

included the coordination of all key product teams (engineering, QA, marketing,

localization, documentation) and ensuring a successful release for the critical holiday

shopping season. Brian returned to Sun in early 2005 to program manage the GDS

initiative. As Senior Program Manager & Chief of Staff for GDS, Brian is responsible for

first engagement with new customers, ensuring world-wide GDS projects are resourced,

deliverable expectations are set, and commitments honored. Brian's work in GDS has

significantly contributed to Sun's modular design and the teams success. Brian was the

recipient of Sun's prestigious Innovation Award presented by Sun's CEO, Jonathan


Schwartz ad CTO, Greg Papadopoulos in July of 2008. Brian holds a Bachelor of Science

in Computer Science and a Masters of Business Administration.

AcknowledgmentsThe authors would like to thank Steve Gaede, an independent technical writer and

engineer, for his probing questions, deep insights, and ability to create a coherent story.

Steve is a frequent contributor to Sun Microsystems technical documents, including

many Sun BluePrints articles. He is a member of ACM and USENIX, and is active in the

Boulder Colorado professional community, having been a coordinator of the Front

Range UNIX Users Group since 1984.

Thanks to Mark Monroe for providing input on power supply efficiency trends, and to

James Monahan for his detailed review comments. Thanks also to the Sun MD team

members who spent time contributing to this paper, including Maurice Cloutier, Liz

From, Brian Kowalski, and Bob Schilmoeller.

ReferencesMore information on Sun’s energy-efficient datacenters can be found at: http://www.sun.com/datacenterdesign.

Ordering Sun DocumentsThe SunDocsSM program provides more than 250 manuals from Sun Microsystems, Inc.

If you live in the United States, Canada, Europe, or Japan, you can purchase

documentation sets or individual manuals through this program.

Accessing Sun Documentation OnlineThe docs.sun.com web site enables you to access Sun technical documentation

online. You can browse the docs.sun.com archive or search for a specific book title or

subject. The URL is http://docs.sun.com/

To reference Sun BluePrints OnLine articles, visit the Sun BluePrints OnLine Web site at: http://www.sun.com/blueprints/online.html

http://www.sun.com/datacenterdesign

http://docs.sun.com

http://www.sun.com/blueprints/online.html

On the Web sun.com

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com

© 2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, the Sun logo, BluePrints, iPlanet, Java, N1, Solaris, StorEdge, Sun Enterprise, Sun Fire, and SunDocs are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the U.S. and other countries.Information subject to change without notice. Printed in USA 3/2009

Energy Efficient Datacenters: Electrical Design