Building a Fully Automated Cloud with System Center and Hyper-VPhilip DuffToby Alcock

MDC332A

What the marketing stuff says

What you really need to doHardware management processStorage DesignsNetwork designs Application installationApplication ElasticityBackend integration (Legacy)Application modelsSecurity

And stuff . . .

AgendaWhat we builtOutline the overall solution and some of the concepts in building large multi-tenant cloud

What worked well from MicrosoftOf the many pieces of software, what saved us

What did not workWhen we hoped Microsoft would solve our problem, and they didn’t

What we wished we had usedThe bits of technology that looked like they solved our issues

What we had to build ourselvesIn building something like this, what are the bits that nobody will give you

What this session is• This is a shopping list, please take away what

you need

• This was a service provider cloud, not all concepts will relate directly to the Enterprise

• We assume you know most of the products we are discussing

• We are going to discuss our experiences, mileage may vary

Marketing slide free zone

What we built

What we built• Enterprise Class Business Productivity• SharePoint, Exchange and Lync

• Supporting services • Active Directory, FIM, UAG, SQL, ADLDS, Hyper-V, VMM, OpsMgr, Orchestrator, etc etc

• Provides a dedicated instance• All VM’s are dedicated to one customer• Management and Infrastructure are shared• Ensures security and allows for basic customization

• Cloud based service• Low cost to manage• Elastic resources based on customer demand

Datacenters and Stamps

Virtual Fabric

OperationsManager

VMM Orchestrator ConfigurationManager

ManagementStamp

Customer StampsManagementStamp

Customer Stamp

Customer Stamp

The big missing partManagementStamp

OperationsManager

VMM Orchestrator ConfigurationManager

Customer Stamp

Customer Stamp

Auto PilotSystem

Telemetry

Actions

External

Input

What needs to be in the Auto Pilot• How to provision new services• How to keep services within Desired

Configuration• What to do with specific alerts• Attempt Auto repair• Change the desired configuration

• Maintenance Tasks and Scheduling• Elasticity The goal of the Auto

Pilot isto reduce human

interaction

What worked well from Microsoft

What worked well• Lots of stuff

• Server Placement• Microsoft virtual switch and VMM

networking• Bare Metal (with some help) • Orchestrator (powershell)

Server Placement

Virtual Machine Manager Networks

Management

Domain iSCSI Public Cluster

Management

Domain iSCSI Public Cluster

Management

Domain iSCSI Public Cluster

Management

Domain iSCSI Public Cluster

Virtual Machine Manager Networks

Virtual Machine Manager NetworksLogical networksHost AdaptersPort ProfilesHost TrunksVirtual SwitchesVM NetworksPort ClassifcationsEtc etc etc

VMM Bare Metal Deployment

Orchestrator (with powershell)• Stay as modular as you can• Need to start with a well defined structure• Need a configuration repository store

What we FULLY automatedDeployment, installation and configuration of• Windows Server• Exchange • Lync• SharePoint• ADLDS• System Center OpsMgr Gateway and Orchestrator Runbook

Server• SQL 2012, mirrored, shipped and AlwaysOn• Forefront Identity Manager• Certificate requests• Active Directory Domain services• Create domain and load GPO

Orchestrator Runbooks

Business Process Runbooks

Domain Runbooks

Domain Runbooks

Domain Runbooks

Worker Runbook

s

Some quick help points• Use the VMM powershell command to run

3.0• Create a standard of error handling• Well documented for process for new

Orchestration• Automate validation steps• Unit Test, Functional Test, and then Test

again• Runbook tester is ok for debugging, it is not a test tool

• Codeplex IP SCORCH Dev Orchestrator• And more codeplex packs, SQL etc

What did not work

What did not work wellUAGService TemplatesMicrosoft best practice for sizingNIC TeamingFibre attachVersion control in OrchestratorThird party virtual switches

What does UAG stand for?

U

A

G

What does UAG stand for?

Unified

Access

Gateway

What does UAG stand for?

Un

Automatable

Garbage

What we could not automateUAG configuration

Needed to be manually configured every deployment

Lync topology builderNeeded to be run by hand for support from

MSFT

VMM Service Templates

Service Template — Multi-Tier .NET Application

Web

Scale-Out Rules

IIS

HW Profile OS Profile App Profile

App

Scale-Out Rules

App Server

Data

Scale-Out Rules

Microsoft SQL Server

MSDeploy package

and configuration settings

App-V package and configuration

settings

SQL Server data-tier application (DAC)

package and configuration

settingsW2K8.VHD OS Settings

HW Profile OS Profile App Profile HW Profile OS Profile App Profile

Networking Load Balancer

What we wished we had used

What we wish we had usedPVLANSMBData Protection ManagerSoftware load balancersHyper-v ReplicaWindows Azure Pack

Attaching shared storage to guest clusters• Need to keep the size of the VM portable• Large amounts of storage on Exchange and SQL• Large numbers of LUNS connected

• Need to be able rebalance guests across Hosts and Clusters

• Need to dynamically add and remove storage

• Want to take advantage of SAN based features

• De-Dupp, Thin Provisioning, Hierarchical Storage Management (Tiering)

Live migration maintaining Fibre Channel connectivity

Virtual Fibre Channel in Hyper‑V

Hyper‑V host 1 Hyper‑V host 2

Worldwide Name Set B

Worldwide Name Set A

Worldwide Name Set B

Virtual machineVirtual machineLIVE MIGRATION• Unmediated access to a storage area

network (SAN)

• Hardware-based I/O path to virtual hard disk stack

• N_Port ID Virtualization (NPIV) support

• Single Hyper‑V host connected to different SANs

• Up to four Virtual Fibre Channel adapters on a virtual machine

• Multipath I/O (MPIO) functionality

• Live migration

Access Fibre Channel SAN data from a virtual machine

Worldwide Name Set A

SMB file share

35

Improvements• Faster and simultaneous migration

• Live migration outside a clustered environment

• Store virtual machines on a File Share

VM VM

Target host

Live migration setup

SMB network storage

IP connection

Configuration data

Memory pages transferred

Memory content

MEM

ORY

MEM

ORY

Modified pages transferred

Modified memory pages

Storage handle movedLive migration based on server message block (SMB) share

VM

Shared Virtual Disk on Scale-Out File Server

• Example:• Two VMs• Two Hyper-V nodes• Separate disks for OS• Shared disk for data

• VHDX files sit on a Scale-Out File Share

Attaching shared storage for guest clusters

Virtual

Fibre

adapters

Shared

CSV on

SMB s

hareV

S

Benjamin ArmstrongPrincipal Program Manager LeadVirtual PC

Attaching shared storage for guest clusters

Virtual

Fibre

adapters

Shared

CSV on

SMB s

hareV

S

Couple of other quick questions

Hyper-threading

Time Sync

R2 and Drivers

What we had to build ourselves

What we had to build ourselvesThe cloud auto pilot

elasticity and deploymentjob scheduling

SQL deploymentOrchestration resilience (error and restart tech)

Related contentBreakout Sessions (session codes and titles)

Hands-on Labs (session codes and titles)

Product Demo Stations (demo station title and location)Related Certification Exam

Find Me Later At...

Developer Network

Resources for Developers

http://msdn.microsoft.com/en-au/

Learning

Virtual Academy

http://www.microsoftvirtualacademy.com/

TechNet

Resources

Sessions on Demand

http://channel9.msdn.com/Events/TechEd/Australia/2013

Resources for IT Professionals

http://technet.microsoft.com/en-au/

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.