OpenStack for HPC: Meeting Your Varying HPC Requirements with a Flexible Private CloudIt is no secret that more and more organizations are moving to cloud-based architectures. Many are using OpenStack, the open source cloud computing platform for this transition. And increasingly, OpenStack ecosystems being considered and used to execute High-Performance Computing (HPC) workloads.

siliconmechanics.com | 866-352-11732

Why OpenStack for HPC?OpenStack use is becoming quite common. More than one-third of cloud users use private clouds, and more than half of those are OpenStack clouds.[i]

Uses Cases of OpenStack for HPCA suitably configured solution using OpenStack can deliver HPC capabilities to organizations in a number of ways.

In organizations that have not traditionally had HPC capabilities, an OpenStack-powered cloud can aggregate compute, storage and networking resources into a powerful platform to run demanding workloads. As a result, low-cost commodity hardware can be leveraged at a fraction of the cost of installing a new cluster to run HPC workloads.

For those organizations with HPC facilities, an OpenStack-based system can complement those existing systems and offload some demands from constrained resources. The OpenStack Foundation offers a selection of User Stories demonstrating successful HPC deployments. One case study features Argonne National Laboratory, a multidisciplinary science and engineering research center. The lab has an extensive portfolio of HPC resources including two systems ranked in the Top 500[iii] of the world’s most powerful supercomputers.

Argonne was an early adopter of OpenStack, deploying its first system in 2011.[iv] Administrators had considered a number of cloud options and selected OpenStack because it was very straightforward to build and deploy, and offered the ability to scale much better than the previous platform.

OpenStack for HPC

OpenStack is enthusiastically embraced because the software platform lets organizations set up private or hybrid clouds by controlling large pools of compute, storage and networking resources throughout a datacenter. Users provision the resources through a web-based dashboard, through command-line tools or through the OpenStack API. In a typical deployment scenario, organizations use OpenStack to create an Infrastructure-as-a-Service (IaaS) cloud where administrators use a dashboard to manage the environment and users provision resources through a web interface. Because of this flexibility, OpenStack’s characteristics are well suited for many types of workloads and are aligned with the needs of many HPC environments.

HPC environments must accommodate a large amount testing and development work, as well as support a wide variety of workloads. OpenStack helps accomplish these goals by allowing organizations to establish virtual clusters that can easily scale up or down, making it ideal for varying HPC workloads. With an OpenStack solution, users can take advantage of self-service capabilities to test various workloads without interfering with production environments. Once a workload is tested, the job can then be run using the required compute resources to complete the job in a timely manner.

ONE HALFOF PRIVATE CLOUDS ARE

OPENSTACKCLOUDS

OPENSTACK

OThEr ClOud

OvEr 50%

Rather than providing a dedicated and fixed node cluster where users are locked into a set number of nodes, an OpenStack-based HPC system is not restricted to one distribution and allows the end-user to choose and in some cases create their own environment. In fact, by using OpenStack, organizations can offer its researchers, data analysts, business units and any other group needing HPC resources, a Cluster-as-a-Service (CaaS) solution. In this way, users that need HPC resources can get a right-sized private cloud to run their jobs. Essentially, users get an Amazon-like experience, on-premise.

[i]

siliconmechanics.com | 866-352-1173 3

Early on, the lab built systems consisting of 700 to 800 nodes,[v] that included a combination of compute, storage, memory and management nodes. Now, an average large system begins with about 1,000 unique hardware elements with many scaling to over 10,000 nodes.[vi]

An OpenStack deployment can also help an HPC facility improve delivery of its services, as was the case with the Van Andel Institute. Van Andel Institute (VAI) is an independent biomedical research and science education organization that dedicated to determining the epigenetic, genetic, molecular and cellular origins of cancer, Parkinson’s and other diseases and translating those findings into effective therapies. Its researchers’ compute requirements were widely disparate with huge variations from job to job, and at times were required to utilize shared NSF resources to complete their work. To continue to grow and support their on-site efforts, VAI looked to an OpenStack solution to reinvent how HPC resources were delivered to researchers.

Examples of HPC ApplicationsOpenStack offers a unique way to deliver HPC services to users and it has other things in its favor, too, that should be factored in when selecting a cloud platform for HPC.

OpenStack is a mature offering, and like other popular open source software solutions, including Linux, Apache and open source firewalls, OpenStack is backed by a vibrant developer community. That backing ensures that common problems will be quickly fixed, and continuous improvements and feature enhancements will be introduced over time.

As is the case with popular open source software, the developer community is very active with creating OpenStack-aware applications. There are many management, performance monitoring and utilities oriented applications available. Additionally, there are a variety of HPC applications for OpenStack[vii] including:

•database: Trove is Database-as-a-Service for OpenStack. It’s designed to run entirely on OpenStack, with the goal of allowing users to quickly and easily utilize the features of a relational or nonrelational database without the burden of handling complex administrative tasks. Cloud users and database administrators can provision and manage multiple database instances as needed. Initially, the service will focus on providing resource isolation at high performance while automating complex administrative tasks including deployment, configuration, patching, backups, restores and monitoring.

•hadoop: The Sahara project provides a simple means to provision a data-intensive application Hadoop cluster on top of OpenStack. OpenStack Sahara offers elastic Hadoop on demand. Sahara lets organizations rapidly configure, auto-deploy and scale Hadoop clusters on OpenStack.

•Bare Metal Provisioning: Ironic is an integrated OpenStack program which aims to provision bare metal machines instead of virtual machines. It is best thought of as a bare metal hypervisor API and a set of plugins that interact with the bare metal hypervisors.

•devOps: Zaqar is a multi-tenant cloud messaging service for web developers. It combines the ideas pioneered by Amazon’s SQS product with additional semantics to support event broadcasting. The service features a fully RESTful API, which developers can use to send messages between various components of their applications, by using a variety of communication patterns.

•Service Orchestration: Heat is the main project in the OpenStack Orchestration program. It implements an orchestration engine to launch multiple composite cloud applications based on templates.

OpenStack for HPC

BIGGEST BArrIEr TO OPENSTACK AdOPTION

The combination of applications and OpenStack core services lowers the barrier for organizations to offer HPC services to its users. However, many organizations need help to take advantage of the potential benefits OpenStack can provide for their HPC efforts. A major obstacle to deployment is that many organizations do not have the in-house expertise to deploy OpenStack. In fact, 45 percent of respondents to the annual OpenStack enterprise adoption survey cite lack of internal IT skills as the biggest barrier to OpenStack adoption.[viii]

lACK OF INTErNAl IT SKIllS [vii]

4

As a result, many organizations are turning to technology partners that bring the expertise and deep industry knowledge to provide complete, custom or turn-key HPC solutions. One solution to help with today’s HPC compute requirements would combine the benefits of open source with enhanced features and services to reduce TCO (Total Cost of Ownership), and assure production workloads have enterprise-class capabilities.

Silicon Mechanics, Inc., is a leading provider of servers, storage and high-performance computing technologies to the world’s most innovative organizations. Since 2001 Silicon Mechanics has supported customers with its “Expert included.” approach, reflecting the company’s passion for providing complete customer satisfaction and customer confidence in the return on their technology investments. Recognized as one of the fastest growing companies in the greater Seattle technology corridor, Silicon Mechanics is changing the way systems providers engage with customers. To learn more, please visit http://www.siliconmechanics.com.

© 2015 Silicon Mechanics, Inc.

OpenStack for HPC

TOP 3 rEASONSEnterprises Deploy

OPENSTACK1) Reduced time of deployment of services2) Reduced IT operational costs3) More responsive infrastructure

Silicon Mechanics as your partnerSilicon Mechanics can help an organization take the headache out of deploying OpenStack supported HPC. Using its “Expert included.” approach, Silicon Mechanics collaborates with customers to architect, build and deploy a custom, turn-key system, followed by ongoing support and warranty services.

These tailored solutions give users the full benefits of a flexible private cloud to carry out their work and run their HPC workloads. Self-service capabilities let users access the computing resources they need without being slowed down waiting for IT to provision and deploy additional resources.

Silicon Mechanics offers both turn-key and fully customized HPC solutions. It is a sole-source provider for deployments of all sizes. It delivers a cost effective and right-sized customized solution for your HPC needs. Knowing that every organization has unique needs and requirements, Silicon Mechanics offers an extensive, flexible and customizable product portfolio.

[ii]

[i] http://www.datacenterknowledge.com/archives/2015/01/30/survey-half-of-private-clouds-are-openstack-clouds/

[ii] https://www.redhat.com/en/about/press-releases/new-openstack-survey-highlights-strong-adoption-plans-among-global-enterprises

[iii] http://www.top500.org/list/2015/06/

[iv] http://openstackconferencespring2012.sched.org/event/2b976f6b3ac0727d82680d6f83595a1d#.Vij7En6rRII

[v] https://www.openstack.org/user-stories/argonne-national-laboratory-us-department-of-energy/

[vi] http://www.slideshare.net/raydelott/openstack-hpc-talk

[vii] http://www.techvalidate.com/product-research/red-hat-enterprise-linux/facts