opennebulaconf2015 2.14 cloud service experience in teidehpc infrastructure - carlos ignacio...

OpenNebula implementation in TeideHPC infrastructure

Carlos I. González Vila

OpenNebulaConf 2015 - Barcelona

ITERFounded by the Cabildo Insular de Tenerife in 1990, the island's administrative authority. Its objectives are to promote research activities and technological development, especially those related with the use of the renewable energies.

ITER

ITER

MA

NA

GE

ME

NT

RENEWABLE ENERGIES

Departamento de Fotovoltaica Departamento de Eólica

DIS

EM

INA

TIO

N A

RE

A

MA

INT

EN

AN

CE

Photovoltaics

Sustainable architecture

Wind Energy

INFORMATION AND NEW TECHNOLOGIES

Electronics

Informatics

Groundwater resources

Environmental quality

Volcanology

ENVIRONMENT SCIENCE

TeideHPC

TeideHPC

● Weather forecasting– Wind power generation

– Volcanic smoke dispersion prediction

– Solar power generation

● Bioinformatics● Computational Fluid Dynamics

– Aerodynamics, aeroacoustics

– Industrial design

TeideHPC

● Astrophysics– Solar composition

– Cosmological Parameters Estimation

– Analysis of solar spectro-polarimetric observations

● Computer sciences● Streaming services● Render ● Cloud computing

OpenNebula

● Implementation● Study cases

– Infiniband virtualization

– Interoperability

● Use cases– Multidatastore (lbox)

– Remote visualization

– Render farm

OpenNebula Implementation

● API● Cobbler● Chef● NetApp Storage● Monitoring ● Repository

OpenNebula Implementation

High Performance Computing Cloud computing

● Ethernet network configuration● Infiniband configuration● User auth & permissions

API

OpenNebula Implementation

● API● Cobbler● Chef● NetApp Storage● Monitoring● Repository

OpenNebula Implementation

● API● Cobbler● Chef● NetApp Storage● Monitoring● Repository

OpenNebula Implementation

● API● Cobbler● Chef● NetApp Storage● Monitoring● Repository

OpenNebula Implementation

● API● Cobbler● Chef● NetApp Storage● Monitoring● Repository

Study case – Infiniband virtualization

Study case – Infiniband virtualization

● OpenNebula Ecosystem● KVM SR-IOV Driver

– Single root I/O Virtualization

Virtual functions

PCIe device

Study case – Infiniband virtualization

● OpenNebula Ecosystem● KVM SR-IOV Driver

– Single root I/O Virtualization

Virtual functions

PCIe device

# lspcib0:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]b0:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]b0:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

● Enable option in BIOS● Enable IO virtualization in kernel boot line● Define number of Virtual Functions in kernel module load● Burn SR-IOV capable firmware in the HCA (OEM)● Upgrade OFED version (2.3-2.0.5-rhel6.6)

Study case – Infiniband virtualization

● OpenNebula Ecosystem● KVM SR-IOV Driver● Great A'Tuin

– GPU devices support was funded by SURFsaraOpenNebula 4.14 Release notes... Support for GPU consumables, giving the ability to give exlcusive PCI passthrough access to VMs to GPU cards, for HPC computing.

Study case – Interoperability limitations

Study case – Interoperability limitations

● 2 research centers– Share resources

● Conectivity● Federation● Cloud bursting

Study case – Interoperability limitations

● 2 research centers– Share resources

● Conectivity● Federation● Cloud bursting

● End-to-end connection

– L2-L3

– Jumbo frames

● 1 lambda - 10 Gbps

● Ping < 40 ms

● VPN IPsec

● +600Mbps between 1Gbps hosts

Study case – Interoperability limitations

● 2 research centers– Share resources

● Conectivity● Federation● Cloud bursting

Tightly coupled

Study case – Interoperability limitations

● 2 research centers– Share resources

● Conectivity● Federation● Cloud bursting

Study case – Interoperability limitations

● 2 research centers– Share resources

● Conectivity● Federation● Cloud bursting

OpenNebula 4.8 docs... “The remote provider could be a commercial Cloud service, such as Amazon EC2, IBM SoftLayer or Microsoft Azure, or a partner infrastructure running a different OpenNebula instance”

Use case – Multiple datastore

Use case – Multiple datastore

● Host load balancing required

● Migrate / live migrate● Time of migration (live)

– VM RAM size

– Host bandwith (IB)

– Network

Use case – Multiple datastore

● Storage overloaded● Delete-recreate & Deploy to new

datastore● Take image snapshot (same DS)

& create template & deploy– VM HDD size

● Manual migration: – Stop VM + copy image to new DS

+ delete-recreate + deploy + replace image

Use case – Remote visualization

● Lack of rendering hardware● Low performance with ssh -X (even with

compression)● Spice

– Audio+video

– Low BW requirements

● QXL driver (variable desktop resolution)– 1024 x 768 … 2560 x 1600

Use case – Remote visualization

Use case – Remote visualization

Capture the Flag

● 404 VMs● Connected to client private

infrastructure through IPsec tunnel

● 20,8 TB HDD● 12,8 TB RAM● 12800 cores● 100 KW

Thank you

teidehpc@iter.es @teidehpc teideHPC

References

● http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/

● http://docs.opennebula.org/4.12/advanced_administration/data_center_federation/introf.html

● http://docs.opennebula.org/4.14/release_notes/release_notes/whats_new.html#opennebula-core