utilizing public and private clouds with bright cluster manager
TRANSCRIPT
Utilizing Public AND Private Clouds
with Bright Cluster Manager
Ian Lumb
Product Marketing Manager
Bright Topics Webinar RECORDING
March 25, 2015
2
How Bright Cluster Manager allows you to seamlessly
make use public clouds like Amazon Web Services
(AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
WEBINAR RECORDING
3
Public cloud
• Off-premise IT capabilities or applications, provided by others
Private cloud
• On-premise enablement of cloud capabilities with existing IT
Hybrid cloud
• Some combination of public and private clouds
Definitions
http://www.wired.com/2014/10/public-vs-private-cloud/
4
How Bright Cluster Manager allows you to
seamlessly make use public clouds like Amazon
Web Services (AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
“Both public and private clouds have management implications. However, by choosing a public cloud solution, an organization can offload much of the management responsibility to its cloud vendor. In a private cloud scenario, there is significant demand on resources to specify, purchase, house, update, maintain, and safeguard the physical infrastructure. Financially, deploying a private cloud can also create a large initial capital expense, with subsequent investment required as new equipment and capacity is added.”
http://www.akamai.com/html/solutions/public_private_cloud.html
Scenario I — “Cluster on Demand”
node001
head nodenode002
node003
Cloud Utilization
Scenario II — “Cluster Extension”
head node
node001 node002 node003
node004
node005
node006
node007
Cloud Utilization
node004
node005
node006
node007
head node
node001 node002 node003
Cloud Utilization
data
1. User submits job to queue
2. Bright creates “data-transfer” job
3. Bright runs compute job when data-transfer job is complete
4. Bright transfers output data back after completion
job
Data-Aware Scheduling to the Cloud
Case Study: Life Sciences
Customer: Major pharmaceutical company
Existing clusters at four locations
Regulatory and compliance concerns
Challenge
Integrate AWS into existing IT infrastructure • … with minimal administrative overhead
Solution
Bright Cluster Manager + PBS Professional + AWS• Secure AWS instance added as an on-demand location
Access to licensed software
Persistent resources/services– Bright Cloud Director
– AWS S3 and Glacier
Results
AWS seamlessly integrated into existing IT infrastructure with minimal administrative overhead
Case Study
10
How Bright Cluster Manager allows you to seamlessly
make use public clouds like Amazon Web Services
(AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
11
1. Cost effectiveness
• Trade CAPEX for OPEX
Open source is OPEX neutral
2. Data security
• IT security better aligned with business needs
3. Better control
4. True flexibility
• Customize as needed, not limited to a generic offering
5. Sync speed
• Internal networks don’t need to hide latency
10 reasons you should build your own cloud
Wallen, TechRepublic, 3/2013http://www.techrepublic.com/blog/10-things/10-reasons-you-should-build-your-own-cloud/
12
6. Company integration
• Leverage existing in-house services – e.g., storage, ID
7. Unlimited size
• Trade CAPEX for OPEX
8. User management
• Customize as needed, not limited to a generic offering
9. Backup control
• IT backups better aligned with business needs
10.HIPPA compliance
• Compliance challenges simplified
10 reasons you should build your own cloud
Wallen, TechRepublic, 3/2013http://www.techrepublic.com/blog/10-things/10-reasons-you-should-build-your-own-cloud/
http://docs.openstack.org/admin-guide-cloud/content/figures/2/figures/openstack-arch-havana-logical-v1.jpg
OpenStack Architecture (Havana)
Role-Based Service Management
• OpenStack consists of services
• Services accomplish specific tasks
• Dependencies, interactions, …
• Bright role
• A task that can be assigned to a node
• Assigning/un-assigning role will
• Write out config file based on role parameters
• Start/stop/monitor relevant services
• Add/remove entries to Keystone service registry
• Bright roles can be composed
• Addresses dependencies and interactions
• e.g., introduction of HA capabilities
OpenStack Configuration
OpenStack configuration through roles
• OpenStack consists of several services to accomplish specific tasks
• Tasks can be assigned by assigning roles to nodes
• Example roles:
Compute (Nova) Image (Glance)
Block Storage (Cinder) Dashboard (Horizon)
Auth (Keystone) Networking (Neutron)
• Assigning/unassigning role will:
• Write out config file based on role parameters
• Start/stop/monitor relevant services
• Add/remove entries to Keystone service registry
• OpenStack configuration is dynamically updated with cluster changes (e.g. hostname, network settings)
• Instance migration works out of the box
Managing OpenStack Clusters
• Managing OpenStack Clusters just as difficult as other types of clusters
• Without proper infrastructure, OpenStack will not be able to run
• Setting up OpenStack manually is often extremely complex for production setups
• Bright Cluster Manager provides single-pane-of-glass to manage and monitor all aspects of OpenStack cluster
• Includes:
• Hardware (set up, configuration, monitoring)
• Operating system (provisioning, updates)
• OpenStack installation
• OpenStack configuration
• Users
• Bright Cluster Manager provides perfect environment for OpenStack to run on
17
OpenStack Metrics & Healthchecks
OpenStack related metrics. For example:
• Per-tenant quotas
• Swift: Object size, number of containers, API request rate, data transfer rates
• Cinder: Volume size
• Nova: CPU time used, disk I/O, network I/O
• Neutron: remaining number of floating IPs
OpenStack related healthchecks
• Verify that OpenStack related services are running properly
Storage & Networking Management
Ceph is used by default for all storage related to OpenStack
• Bright integrates with Ceph to make it easy to set up, manage and monitor
• Ceph used for S3 & Swift compatible object storage
• Ceph used for block storage of VMs and storing OpenStackimages, volumes and snapshots
Bright uses VLAN pools or VxLAN to allow private networking to be set up between VMs.
• Several types of network setups supported
• No headaches
Provision VMs using Bright
Allow VMs to be provisioned using Bright software images
• In plain OpenStack VMs are instantiated using static image
• After initial deployment managing software on VMs will become challenging
• Bright adds value by booting VMs into Bright node-installer and provisioning Bright software image onto VM
• VMs can then be managed the same way as physical nodes
• VMs will show up in Bright management tools
• VMs can be updated by updating the Bright software image and using Bright’s provisioning system to propagate updates
Virtualized Sub-clusters
• Virtualized Bright clusters can be spun up with a few clicks within minutes using OpenStack Heat orchestration
• Allows tenants to have full control over their own Bright clusters
• Clusters can be easily be resized to meet tenant demands
• OpenStack Ironic also allows non-virtualized sub-clusters to be created
• Single CMGUI instance can manage both physical cluster as well as sub-clusters in single view
• SR-IOV allows HPC resources (e.g. InfiniBand and GPU) to be made available directly to VMs at near-native performance (work in progress)
Product Differentiation
• Installs on bare metal through to the OpenStack distro
• Single-pane-of-glass management interface for HPC and Big Data Analytics clusters/clouds
Addresses the physical cluster as well as OpenStack
• Fully manages OpenStack services
• Customized monitoring and health checks
• Multiple instances of OpenStack
Architected specifically for OpenStack
• Simultaneous, independent instances on dedicated hardware
• Time-sliced instances on shared hardware
Supports various OpenStack usage models
Slide courtesy Adaptive Computinghttps://vimeo.com/115659153
24
How Bright Cluster Manager allows you to seamlessly
make use public clouds like Amazon Web Services
(AWS)
How Bright Cluster Manager allows you to rapidly
deploy a private cloud based on OpenStack
Key Takeaways
Additional Slides
server
farmsbig data
clusters
HPC
clusters
private
clouds
storage
clusters
database
clusters
Trends
- Centralization
- Consolidation
- Standardization
- Cloud
- General purpose
clusters
- Private clouds
- Across on-premise
and cloud
28
The Problem — “clusters are difficult”
All clusters may need
• deployment
• provisioning
• updating
• managing
• monitoring
• security
• health checking
• alerting/alarming
• virtualization
• cloud
• workload management
What makes it hard
• everything affects everything
else
• everything changes all the
time
• hardware breaks all the time
• cloud, virtualization increase
complexity
IT Pain
• Downtime
• Delays
• Personnel cost
• Inability to react
29
The Problem According to IDC
Clusters are still hard to
setup, operate, use and
manage
New buyers require “ease-of-
everything”
The shortage of skilled people is
a major hindrance
Software is still the #1 roadblock
Better management software is
needed
“
”
30
The problem with “Toolkits”
Toolkits — “A patchwork of disparate tools”• Tools typically used: Ganglia, Nagios, Cfengine, System Imager,
Puppet, Chef, Cobbler, Hobbit, Big Brother, Zabbix, etc.
• Scripts
Issues with the “toolkit” approach:• Scripts poorly documented and hard to maintain
• Tools not designed to work together
• Each tool has its own user interface (CLI/GUI)
• Each tool has its own agent and database
• Tools rarely designed for scale & high performance
• Accelerators (GPUs, Phi) often not supported
Making a collection of unrelated tools work together• Requires a lot of expertise and scripting
• Rarely leads to a really easy-to-use and scalable solution
31
A Better Solution
Bright Cluster Manager takes a much more fundamental & integrated approach• Designed and written from the ground up
• Single cluster management agent provides all functionality
• Single, central database for configuration and monitoring data
• Single UI for ALL cluster management functionality
Which makes Bright Cluster Manager …• Extremely easy to use
• Extremely scalable
• Secure & reliable
• Complete
• Flexible
• Maintainable
32
CMDaemon
Bright Architecture
SOAP/JSO
NAPI+SSL
Cluster
Management
GUI
Cluster
Management
Shell
Web-Based
User Portal
Third-Party
Applications
SQL database
head node
node002
node004
node003
node001
node005
GPUs in the Cloud? The Top Four Reasons
1. You can realize possibilities using the cloud
• You can scale up and scale out
2. You still realize the promise of GPU programmability
• … via HPC in the cloud
3. Your use of the cloud is transparent
• You’ve found ways to `hide’ latency
• Constraints apply for MPI apps
4. Your go-to apps still work in the cloud
http://info.brightcomputing.com/Blog/bid/196290/The-Top-4-Reasons-You-Should-Try-Cloud-Based-GPUs-for-HPC