session e: prs - cis.temple.edugiorgio/old/cis307s09/readings/cloud... · 6 open cirrus stack prs...

Session E:PRS

PRSRichard Gass

AgendaSessions:(A) Introduction 8.30-9.00 (B) Hadoop 9.00-10.00

Break 10.00-10.15Hadoop 10.15-11:30Lunch 11.30-12.30

(C) Pig 12.30-1.30Break 1.30-1.45

(D) Tashi 1.45-3.30Break 3.30-3.45

(E) PRS 3.45-5.00

I. OverviewII. Plans/StatusIII. User ViewIV. AdminsitrationV. InstallationVI. Summary

Overview

Open Cirrus Stack

Compute + network + storage resources

Power + cooling

Management andcontrol subsystem

Physical Resource set (PRS) service

Credit: John Wilkes (HP)

Open Cirrus Stack

PRS service

Eucalyptus Tashi/HDFS NFS storage service

Experiment

PRS clients, each with theirown “physical data center”

Open Cirrus Stack

PRS service

Experiment

Virtual cluster Virtual cluster

Virtual clusters

Open Cirrus Stack

PRS service

Experiment

Virtual cluster Virtual cluster

BigData App

Hadoop

1. Application running2. On Hadoop3. On Tashi virtual cluster4. On a PRS5. On real hardware

Web Service

Open Cirrus stack - PRS• PRS service goals

– Provide mini-datacenters to users– Isolate mini-datacenters from each other

• PRS service approach– Allocate sets of physical co-located nodes, isolated

inside VLANs.

• Initial PRS implementation from HP• Re-write from Intel (in collaboration with HP)

contributed to Apache Software Foundation

PRS service

Further Motivation

• Enable innovation in virtualization

• Allow running without virtualization overhead– Necessary for predictable QoS

• e.g. cache interference

• Reduce complexity in allocating physical resources• Gain User Confidence

– Show users that we can efficiently allocate/deallocate resources

• Stop the squatting– Incentives

• HP’s tycoon (economic model)• Simple points scheme for good behavior• Early return

Responsibilities of PRS

• Isolate domains• Provision system software• Provide platform control

– On/Off• Provide boot debug

VLANPXEIPMI

VLAN• Virtual LAN technology allows a single physical

network to appear as several isolated networks– Ethernet packets are tagged with a VLAN id– Switches and NICs enforce the policies associated

with each VLAN

• By associating PRS domains with different VLANs, they can be isolated from each other

• The PRS system provides the interfaces necessary to abstract switch configuration programming across multiple switch vendors

• Enables provisioning of OS image over the network

• On machine boot, the NIC firmware contacts a PXE server via the DHCP process for the appropriate kernel and initrd to load

• Once loaded, the init scripts in the initrd can pull the filesystem to the machine

• In our environment, we download the desired filesystem to a ramdisk from a NFS server– enabling a very rapid provisioning (30 seconds or less) while leaving the host filesystem undisturbed

eXecution

Environment

• Defines a standardized, abstracted, message-based interface to intelligent platform management hardware

• Defines standardized records for describing platform management devices and their characteristics

• Enables cross-platform management software

Intelligent

Platform

Management

Interface

Status/Plans

PRS Roadmap• Stage 1

• Manages all cluster hardware• Handles resource provisioning• Provides interfaces for VLAN definition/programming• Administrator is still in the allocation decision-making loop

• Stage 2• Introduces a request queue and primitive scheduler• Admin may still be in loop, definitely for special cases• Enables provisioning of OS to local disk• Enables virtual disk conversion to physical

• Stage 3• Incentives module added (Tycoon)• Tashi integration

Some History

• Previous prototype developed at HP Labs• Focus on economic model

• Nice web interface which will be available upon reconvergence of code

User View

PRS Roles

• Admin: root of all authority– Controls the physical resources

• User: requests domains– Controls the domain, once allocated

Domains

• A Domain is the unit of PRS isolation• A simple domain is a set of compute

nodes gathered into a single VLAN

• Nodes are allocated from pools of available resources

The PRS Interface• Users and Admins currently interact with the

PRS system through a command line interface

• This interface both:– Queries and updates records in the PRS database– Wraps the various commands that must be issued to

effect changes in the cluster

• PRS is currently a centralized system; users log into the PRS manager to issue commands– An RPC interface is planned for the near future

PRS Usage

Usage: prs <options>Standard options:--help [show this help message and exit]--version [show program's version number and exit]--verbose [be verbose]

Common options:--nodeName <name> [Specify node]--switchPort <port> [Specify switchport switchname:portnum]

Common admin options:--userName <name> [Specify user name]--uid <UID> [Specify user id]

Image Management Interface

--addImage <img> [Add image to PRS]--delImage <img> [Delete image]

User Allocation Interface--createDomain <name>

– May fail if name already exists --submitDomainRequest <name>--destroyDomain –domain <name>

--requestNodes --domain <name> [--count <N>] [--nodeName <name>] [--cores <n> …]

– Add the requested nodes to the domain--assignImage <kernel> <image>

– Assign image to resource

--associateNewVlan –domain <name>– Allocate an unused VLAN number to domain

--createReservation <YYYYMMDD> <YYYYMMDD>– Specify duration of node reservation where start time may be “ASAP”

--reservationNotes “notes”--updateReservation

Admin Allocation Interface

--allocateNode [Assign node to a user]--releaseNode [Release node

allocation]

--vlanIsolate <vlanid> [Specify vlan for isolation]

Hardware Control

--hardware [Make hardware call]--powerStatus [Get power status]--rebootNode [Reboot node (Soft)]--powerCycle [Power Cycle (Hard)]--powerOff [Power off node]--powerOn [Power on node]

Query Interface--showReservations [Show current node reservations]--showResources [Show available resources to choose from]

--procs <N> [Filter by number of processors]--clock <N> [Filter by processor clock]--memory <N> [Filter by amount of memory (Bytes)]--cpuflags “flags” [Filter by CPU flags]--cores <N> [Filter by number of cores]

--showPxeImages [Show available PXE images to choose from]

--showPxeImageMap [Show PXE images host mapping]

Administration Interface--admin Enter Admin mode

--addPxeImage [Add PXE image to database]

--enableHostPort [Enable a switch port]--disableHostPort [Disable a switch port]

--removeVlan <vlanId> [Remove vlan from all switches]--createVlan <vlanId> [Create a vlan on all switches]--addNodeToVlan <vlanId> [Add node to a vlan]--removeNodeFromVlan <vlanId> [Remove node from a vlan]--setNativeVlan <vlanId> [Configure native vlan]--restoreNativeVlan [Restore native vlan]--removeAllVlans [Removes all vlans from a switchport]

--sendSwitchCommand “<command>” [Send Raw Switch Command, BE CAREFUL]--interactiveSwitchConfig “<switchname>” [Interactively configure a switch]

--showSwitchConfig <nodename> [Show switch config for node]

Administration

Typical Workflow

1. Admin queries available systems2. Admin requests systems with desired user configuration

1. i.e., cores, memory, image, duration, etc

3. Request goes in queue4. PRS locates resources and provides a list to admin/Tashi.5. Admin/Tashi moves VMs to free resources

1. Add node to blacklist and tell hadoop to reload

6. PRS allocates resources1. Provides estimated time to get resources2. User can query 3. PRS sends notification when allocated

7. PRS reclaims resources and adds them back into respective pools1. User may extend time period before expiration

After allocation

• A returned PRS node is typically untrusted– update the system to default settings

• Clean physical node by PXE booting a reset image• Restore all setting to defaults (address, IPMI passwords)• Repartition and format disks

• (Option) Trust images from some users– No re-format needed

• Clean network configuration (VLAN)

Example: Minicluster./prs –addimage hardy-rgass-testing:hardy:8.03./prs –assignimage hardy-rgass-testing –nodename

r1r1u25./prs –allocatenode –nodename r1r1u25 –username rgass

–reservationDuration 30 –vlanisolate 300 –notes “Practice allocation”

./prs –addnodetovlan 300 –nodename r1r1u25

./prs –hardware –rebootnode –nodename r1r1u25

Example: CloudConnect 1

• Network isolate a rack of machines and PXE boot them with a user’s kernel and initrd

• Create a VM that acts as a SSH gateway and a NAT for the private cluster

• Dynamically configure switches to support the networking experiment

1Gb/s Switch

100Mb/s Switch

Rack A region

Rack B region

Rack C region

Rack D region

4x1Gb trunk link

1 Gb/sSwitch

MRack A

4Gb/s Switch

- server

VLAN #2: Optical

VLAN #1: Electrical

100Mb/s Switch

Rack B Rack C Rack D

- switch

M - manager

Example: CloudConnect 2for i in r1r1u12 r1r1u13 r1r1u14 r1r1u15;do

./prs --admin --setnativevlan 300 -n ${i}

./prs --admin --addnodetovlan 800 -n ${i}

./prs --admin --addnodetovlan 801 -n ${i}

./prs --admin --addnodetovlan 802 -n ${i}done./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface range ethernet g(25-28); spanning-tree

disable"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g25;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g26;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g27;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g28;switchport mode trunk;exit"./prs --admin --switchport sw0-r1r1 --sendswitchcommand "config;interface ethernet g28;switchport mode trunk;exit"

./prs --admin --switchport sw0-r1r1:25 --setnativevlan 802 -v

for i in $(seq 12 16);do./prs --hardware --rebootnode -n r1r1u${i}

Example cluster: Networking Testbed

• Network isolate a rack of machines and PXE boot them with a user’s kernel and initrd

• Create a VM that acts as a SSH gateway and a NAT for the private cluster

• Dynamically configure switches to support the networking experiment

Future Work

• Integration with Tashi…– Would enable free exchange of resources

between the Tashi pool and the free pool

PRS client queries PRS server

for available resources

System Servers

VMVM VM

PRS server

Tashi Cluster Manager

VMVM VMManagement Servers

PXE server

PRS client

Administratoror

Cluster ManagerVM

PRS queries DB to locate available

resources

Node 1 : 8 Core, 16G memory, 6TB disk,30dayNode 2 : 8 Core, 16G memory, 6TB disk,30 dayNode 3 : 8 Core, 16G memory, 6TB disk,90 dayNode 4 : 8 Core, 16G memory, 6TB disk,1 dayNode 5 : 8 Core, 8G memory, 2TB disk, 90 dayNode 6 : 8 Core, 8G memory, 2TB disk,90 dayNode 7 : 8 Core, 8G memory, 2TB disk,90 dayNode 8 : 8 Core, 8G memory, 2TB disk,90 dayNode 9 : 8 Core, 8G memory, 2TB disk,90 dayNode 10: 8 Core, 8G memory, 2TB disk,30 day…

Results are sent back to

the client

User choosesmachine attributes

and submits a request for the resources

for some time period

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

Request Queue

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

PRS processes request and

identifies physical machines that satify

the user request

VM VM VM

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

PRS sends requestto Tashi to free selected nodes

Tashi moves virtual machines off of selected nodes

VM VM VM

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

PRS allocated the physical machines to the requested user and isolates them from the network

using VLANsTashi notifies PRS that migration of

virutal machines has completed

PRS reboots the physical machine and

sets PXE image to users VM

Virtual disk image is converted to PXE

Physical machines boot up with PXE

PXEPXEPXE

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

VMVM VM VM

PRS updates reservation database

PRS client queries server for allocation

User connects to the machines and starts running experiments

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

System Servers

VMVM VM

PRS server

PXE server

PRS client

Administratoror

Cluster ManagerVM

Installation

Necessary Components• DHCP Server• PXE Server• NFS Server• DNS Server (optional)• Configurable switches

– New switch types may require new PRS modules• Hardware access method

– E.g. IPMI– IP-addressable PDUs enable rescue if IPMI becomes

compromised

Internals

Notes on Current Software

• PRS client code is Python 2.5• PRS database implemented in MySQL

– Reachable through python-MySQLdb interface

• pExpect used for switch configuration• User information currently obtained

through LDAP

Summary

PRS• PRS lays the foundation of the Open Cirrus

software stack– easing management of multiple projects in a single cluster

• PRS enables partitioning clusters into isolated domains of physical resources

• Current implementation allows rapid provisioning of system software

• PRS code base is open source software available through Tashi project in Apache Incubator– Contributions welcome

session e: prs - cis.temple.edugiorgio/old/cis307s09/readings/cloud... · 6 open cirrus stack prs...

Documents

プロジェクタースクリーン（prs-wb6090,prs …...

hdfs internals

tashi teacher's notes 2010 - bmec.com.au · 6! preparing...

yoritomo tashi

buddhist summer school 2016 - nyima tashi

be k tashi surrender

de unos fantasmas tashi y los fantasmas que acechan...

tashi homework

tashi delek

tashi and pets xmas 2010

tashi - aadl

tashi - math.muni.cz

tashi norbu. boeddha in de polder

influence – how to exert it by yorimoto tashi ·...

yoritomo-tashi - el espíritu de organización

hadoop and hdfs overview -...

tashi the bird lama

tibet 11, tashi lhumpo

human development & family science...

hdfs documentation