building private clouds for hpc with opennebula: reference deployments & lessons learned

Ruben S. Montero

ISC Cloud’11 – Modeling HPC CloudsSeptember 26th Mannheim Germany

© OpenNebula Project. Creative Commons Attribution-NonCommercial-ShareAlike License

Acknowledgments

The research leading to these results has received funding from the European Union's Seventh Framework Programme ([FP7/2007-2013] ) under grant agreement n° 261552 (StratusLab Project)

Chief Architect Technical Coordinator

Building Private Clouds for HPC with OpenNebula

Ruben S.Montero


Building Private Clouds for HPC with OpenNebula: Reference Deployments & Lessons Learned

• Cloud Computing in a Nutshell• HPC & IaaS Clouds: Two+ Basic Approaches• Sample Deployments: CERN & Fermilab• Lessons Learned• A Third Approach: Grid & Cloud Computing


What Who

On-demand access to any application

End-user(does not care about hw or sw)

Platform for building and delivering web applications

Developer (no managing of the underlying hw & swlayers)

Delivery of a raw computer infrastructure

System Administrator(complete management of the computer infrastructure)

Cloud Computing Disciplines

An Introduction to Cloud Computing

Platform as a Service

Infrastructure as a Service

Physical Infrastructure

Software as a Service


Infrastructure as a Service (IaaS)


• Simple Web Interface

• Raw Infrastructure Resources

• Pay-as-you-go (On-demand access)

• Elastic & “infinite” Capacity

Public Cloud








Public Cloud

A “Public Cloud behind the firewall”

• Simplify internal operations

• Dynamic allocation of resources

• Higher utilization & operational savings

• Security concerns

Private Cloud








Public Cloud

A “Public Cloud behind the firewall”

• Simplify internal operations

• Dynamic allocation of resources

• Higher utilization & operational savings

• Security concerns

Private Cloud

• Suplement the capacity of the Private Cloud

• Utility Computing dream made a reality!

Hybrid Cloud


High Performance & IaaS Cloud Computing

The pre-cloud time

LRMS (LSF, PBS, SGE…)

Grid MiddlewareAc

ces

sP

rov

isio

n

The pre-virtualization/cloudtimes



Two Basic Approaches: An Infrastructure Tool

Virtual WorkerNodes

LRMS (LSF, PBS, SGE…)

Grid MiddlewareAc

ces

sP

rov

isio

nS

erv

ice

• Consolidation of WNs• Simplified management • Physical – Virtual WNs• Dynamic capacity partitioning

• Traditional interfaces• Grid Integration• Batch Processing• Custom Execution Environments

Service-ProvisioningDecoupling



Two Basic Approaches: A Provisioning Tool

Pilot Jobs, SSH…

IaaS InterfaceAc

ces

sP

rov

isio

nS

erv

ice

• Simple Provisioning Interface• Raw/Appliance VMs

• Custom access to capacity• Not only batch workloads• Not only scientific workloads

• Improve utilization• Reduced service management• Cost efficiency


High Performance & Cloud Computing

Using hybrid cloud computingP

rov

isio

n

External ProvidersPhysical Resources

• Extend provisioning layer• IT managed as an Utility• Interoperability

• Networking Management• Appliance Management

• Cost• Peaks• Fault-Tolerance


Examples of HPC Cloud Deployments

The CERN’s lxcloud

Goal & Characteristics• Virtualization of Batch Nodes: limted life-time, dynamic capacity• Golden Nodes + CERNVM• EC2 Query – OpenNebula API

Some Technical Notes• Custom Network Integration MAC/IP pinning• Disk I/O Tunning• Fast boot of VMs: Bittorrent + LVM snapshots• OpenNebula tests up to 16000 VMs (great feedback!)


Examples of HPC Cloud Deployments

Fermicloud

Goal & Characteristics• Scientific stakeholders get access to VMs (no sysadmin needed)• Developers & Integrators of new Grid applications • Improve utilization of power, cooling and humans • Legacy applications

Some Technical Notes• VMs access Fermilab Networking and Storage Services• Using KVM• OpenNebula + X509 support (contributed back!)• Other Areas: HA, Batch queues look-ahead, cluster on-demand…


Lessons Learned

HPC & Cloud Computing

Do not hide the Complexity, Architect the Cloud!• Let IT staff adapt/desgin/develop their cloud deployments

• Design to Automate

• Scale your deployments: Storage

Be Interoperable, Have a Rich set of API’s & Interfaces • Be prepared for completely different uses cases (don’t look at the service)

• Be prepared to deal with different solutions (storage, hypervisors, networks)

• Be prepared to interoperate at different levels (API’s, Cloud services)

And for HPC Cloud builders…• Scientists want it their way (VM images, do not put access restrictions)

• Deal with tunning : CPU & I/O (image formats, hypervisor I/O policies)


A Third Approach: Grid and Clouds

The StratusLab Project

Grids

Customize Environments

Uniform Security

Resource Management

Scientific Applications

Resource Sharing

Flexibility & Simplicity

Usage• Job Processing

• Big Batch System

• File Sharing Services

Achievements • Federation of Resources

• VO Concept

But… • User experience

• Complexity

Usage• Raw infrastructure

• Elasticity & Pay-per-use


Achievements • Agile Infrastructures

• IT another Utility

But… • Interoperability

• Federation

Clouds


A Third Approach: Grid & Clouds

The StratusLab Project

Grid Services

Grid Services Cloud API Grid Services Cloud API

MarketPlace

Cloud/Grid Sites

• Sharing existing VM images• Federation facilities• Security

External Clouds

•Needs to consider Image & VLANs•Needs to consider placement

Multi Cloud

Cloud Bursting

Ruben S.Montero


Building Private Clouds for HPC with OpenNebula: Reference Deployments & Lessons Learned


The research leading to these results has received funding from the European Union's Seventh Framework Programme ([FP7/2007-2013] ) under grant agreement n° 261552 (StratusLab Project)

@ruben_s_monteroblog.opennebula.org

building private clouds for hpc with opennebula: reference deployments & lessons learned

Technology

cloud computingsoftware

private clouds

grid cloud computing3

nutshellhpc iaas clouds

service iaasan introduction

raw infrastructure resources

cern fermilablessons

elastic infinite capacity