https://portal.futuregrid.org

Virtual Appliances

CTS Conference 2011PhiladelphiaMay23 2011

Geoffrey Foxgcf@indiana.edu

http://www.infomall.org https://portal.futuregrid.orgDirector, Digital Science Center, Pervasive Technology Institute

Associate Dean for Research and Graduate Studies,  School of Informatics and Computing

Indiana University Bloomington

Exploit electronic infrastructure to enhance learning

• Several quite old approaches are critical and dominant– “Just a bunch of web pages” aka digital library– Video conferencing– Shared material as in Webex, Adobe Connect

• Note asynchronous interaction via Twitter, Blackboard, Google docs etc. much easier (and successful) than synchronous (Polycom, access grid, Webex) approaches

• Interactive web learning environments such as www.whyville.net

• Virtual worlds such as Second Life have not taken off but some think this will change as performance of clients and networks are improving dramatically (VRML failed ~1999)

• Must move to an environment consistent with world view of current students aka the “Twitter University”

C4

ContinuousCollaborative

ComputationalCloud

C4I N

T EL

IG

L

EN

CE

MotivatingIssues job / education mismatch Higher Ed rigidity Interdisciplinary work Engineering v Science, Little v. Big science

Modeling& Simulation

C(DE)SEC4 Intelligent Economy

C4 Intelligent People

C4 Intelligent Society

NSFEducate “Net Generation”Re-educate pre “Net Generation”in Science and EngineeringExploiting and developing C4

C4 Curricula, programsC4 Experiences (delivery mechanism)C4 REUs, Internships, Fellowships

Computational Thinking

Internet &Cyberinfrastructure

Higher Education 2020

CDESE is Computational and Data-enabled Science and Engineering

Educational appliancesEducational appliances

• One component of C4

• A flexible, extensible platform for hands-on, lab-oriented education (on FutureGrid)

• Need to support appliances representing clusters of resources

• Virtual machines + social/virtual networking to create sandboxed modules– Virtual “Grid” appliances: self-contained, pre-packaged

execution environments– Group VPNs: simple management of virtual clusters by

students and educators

Why use Virtualization?

• Traditional ways of delivering hands-on training and education in parallel/distributed computing have non-trivial dependences on the environment

• Difficult to replicate same environment on different resources (e.g. HPC clusters, desktops)

• Difficult to cope with changes in the environment (e.g. software upgrades)

• Virtualization technologies remove key software dependences through a layer of indirection

Appliance Infrastructure - guiding principles

• Fidelity: activities should use full-fledged, executable software: education/training modules– Learn using the proper tools

• Reproducibility: Creators of content should be able to install, configure, and test their modules once, and be assured of the same functional behavior regardless of where the module is deployed– Incentive to invest effort in developing, testing and

documenting new modules

Appliance Infrastructure - guiding principles

• Deployability: Students and users should be able to deploy modules in a simple manner, and in a variety of resources– Reduce barriers to entry; avoid dependences upon

a particular infrastructure

• Community-oriented: Modules should be simple to share, discover, reuse, and expand– Create conditions for “viral” growth

Towards this vision in FutureGrid

• Executable modules – virtual appliances– Deployable on FutureGrid resources– Deployable on other cloud platforms, as well as

virtualized desktops

• Community sharing – Web 2.0 portal, appliance image repositories– An aggregation hub for executable modules and

documentation

https://portal.futuregrid.org

What is a virtual appliance?

• An appliance that packages software and configuration needed for a particular purpose into a virtual machine “image”

• The virtual appliance has no hardware – just software and configuration

• The image is a (big) file• It can be instantiated on hardware

9

Virtual appliance example• Linux, Java, Hadoop, configuration scripts

copy

instantiate

Hadoopimage

A Hadoop workerAnother Hadoop worker

Repeat…

VirtualizationLayer

https://portal.futuregrid.org

What about the network?

• Multiple Web servers might be completely independent from each other

• Parallel processing: workers are not– Need to communicate and coordinate with each other– Each worker needs an IP address, uses TCP/IP sockets

• Cluster middleware stacks assume a collection of machines, typically on a LAN (Local Area Network)

11

Virtual cluster appliances• Virtual appliance + virtual network

copy

instantiate

Hadoop+

VirtualNetwork A Hadoop worker Another Hadoop worker

Repeat…

Virtual machine

Virtual network

https://portal.futuregrid.org

Virtual cluster appliances• Virtual appliance + virtual network

13

copy

instantiate

MPI+

VirtualNetwork An MPI node Another MPI node

Repeat…

Virtual machine

Virtual network

https://portal.futuregrid.org 14

BackgroundBackground

• Virtual appliances– Encapsulate software environment in image

• Virtual disk file(s) and virtual hardware configuration

• The Grid appliance – Encapsulates cluster software environments

• Current examples: Condor, MPI, Hadoop

– Homogeneous images at each node– Virtual LAN connecting nodes to form a cluster– Deploy within or across domains

https://portal.futuregrid.org

Grid appliance in a nutshell

• Plug-and-play clusters with a pre-configured software environment– Linux + (Hadoop, Condor, MPI, …)– Scripts for zero-configuration– “Virtual machine” appliance; open-source software

runs on Linux, Windows, Mac

• Hands-on examples, bootstrap infrastructure, and zero-configuration software – you’re off to a quick start

15

https://portal.futuregrid.org

Grid appliance in a nutshell

• Creating an equivalent Grid on your own resources, or on cloud providers, is also easy

• Deploy image on FutureGrid, Amazon EC2• Copy the same appliance to clusters, PC labs• Simple deployment and management of ad-hoc

clusters– Opportunistic computing– Testing, evaluation– Education, training

16

https://portal.futuregrid.org

Virtual Clusters in FutureGrid

17

Nimbus

Eucalyptus

Appliance

imageEducationTraining

https://portal.futuregrid.org

Social virtual private networks• Education/training: deploy your own cluster!

18

copy

instantiate

MPI+

VirtualNetwork An MPI worker Another MPI worker

Repeat…

Virtual machine

GroupVPN

GroupVPNCredentials

(fromWeb site)

Virtual IP - DHCP10.10.1.1

Virtual IP - DHCP10.10.1.2

https://portal.futuregrid.org

Where to go from here?• You can download Grid appliances and run on your own

resources• You can create private virtual clusters and manage groups

of users• You can customize appliances with other middleware,

create images, and share with other users• Tutorials available at FutureGrid.org• More information on Grid appliances also available at Grid-

appliance.org• Contact Renato Figueiredo renato@acis.ufl.edu for more

information about appliances

19