hpe reference architecture for vmware cloud foundation on …€¦ · this reference architecture...
TRANSCRIPT
HPE Reference Architecture for VMware Cloud Foundation on HPE Synergy
Reference Architecture
Reference Architecture
Contents Executive summary ................................................................................................................................................................................................................................................................................................................................ 3 Solution overview ..................................................................................................................................................................................................................................................................................................................................... 4 Solution components ............................................................................................................................................................................................................................................................................................................................ 6
Hardware ................................................................................................................................................................................................................................................................................................................................................... 6 Software ................................................................................................................................................................................................................................................................................................................................................. 12
Design and configuration guidance .................................................................................................................................................................................................................................................................................... 16 HPE Synergy solution design and configuration ............................................................................................................................................................................................................................................. 16
Demonstration of HPE storage solutions for VMware Cloud Foundation workload domain ......................................................................................................................................... 23 Demonstrate VMware Enterprise PKS as a cloud-native platform on HPE Synergy ............................................................................................................................................................ 34 Demonstrate integrated composability with VCF and HPE Synergy using HPE OneView connector for VCF ............................................................................................ 38 Demonstrate patching and upgrading in VMware Cloud Foundation ................................................................................................................................................................................................ 39 Demonstrate stretching a vSAN cluster of a VMware Cloud Foundation management domain ............................................................................................................................... 40 Demonstrate ease of monitoring and reporting of VCF infrastructure using HPE OneView for vRealize Operations ........................................................................... 41 Demonstrate HPE Synergy firmware upgrade using HPE OneView for VMware vRealize Orchestrator ........................................................................................................... 46 Consolidated architecture deployment of VMware Cloud Foundation .............................................................................................................................................................................................. 52 Summary ...................................................................................................................................................................................................................................................................................................................................................... 53 Appendix A: Bill of materials ...................................................................................................................................................................................................................................................................................................... 54 Resources and additional links ................................................................................................................................................................................................................................................................................................ 56
Reference Architecture Page 3
Executive summary Businesses face challenges to turn ideas into services faster, respond quicker to new customer demands, and innovate better by building new services with technology to stay competitive. To meet these business demands, IT is increasingly adopting new cloud technologies, to replace expensive hardware with a software-defined model. Enterprises need an ideal Hybrid IT model that supports both traditional and cloud-native applications. Therefore, businesses are moving to digital transformation and software-defined data center (SDDC) solutions to support this shift.
Hewlett Packard Enterprise and VMware® collaborate to help customers accelerate the journey to hybrid cloud and bring the promise of the software-defined data center to life. The combination of HPE Synergy Composable Infrastructure and VMware SDDC solution dramatically improves business outcomes as well as overall value for our customers. HPE Synergy combined with VMware Cloud Foundation™ (VCF) delivers a simplified and more secure private cloud - that is flexible, easy to deploy, seamless to manage, and simple to operate. For enterprise customers looking to accelerate their journey to hybrid cloud, HPE Synergy with VMware Cloud Foundation is the right solution to support and run all your enterprise apps—both traditional and containerized—in cloud environments.
This Reference Architecture provides guidance for deploying and managing VMware Cloud Foundation on HPE Synergy for virtual infrastructure (VI) in a single VCF workload domain. It demonstrates the integrated composability with VCF and HPE Synergy using HPE OneView connector for VCF. Also, the lifecycle management for HPE Synergy firmware using HPE OneView for VMware vRealize® Orchestrator™ based firmware update workflow is discussed in further sections.
This Reference Architecture also covers using the HPE Primera, HPE 3PAR StoreServ and HPE Nimble as a principal storage as well as an ancillary storage for VMware Cloud Foundation workload domain. The following are the highlighted benefits of this solution:
• The value of combining HPE Synergy with VCF, from a deployment and lifecycle perspective, in a cost-effective and simplified management for faster time to value
• An easy-to-operate VI traditional data center in a VCF workload domain
• Agility to expand and contract physical and virtual infrastructure, on-demand, to quickly meet changing business requirements with HPE OneView connector - a unique Integration between VMware SDDC Manager and HPE Synergy Composer that provides a single console to dynamically compose software-defined and physical infrastructure
• An efficient environment for monitoring VCF using HPE OneView for VMware vRealize® Operations™ and custom dashboard
• An easy and automated way to upgrade HPE Synergy firmware using VMware vRealize Orchestrator workflow
• Guidance to attach and configure HPE Primera, HPE 3PAR StoreServ, and HPE Nimble Storage to be used as a principal storage and ancillary storage for VCF VI workload domain
• Consolidated architecture deployment of VMware Cloud Foundation
Target audience: This document is intended for IT decision makers as well as architects and implementation personnel who want to understand enterprise ready private cloud solutions using the HPE Synergy Composable Infrastructure capabilities offered by the HPE Synergy platform and VMware Cloud Foundation. The reader should have a solid understanding and familiarity with VMware Cloud Foundation, VMware vRealize Orchestrator and HPE Synergy.
Document purpose: The purpose of this document is to demonstrate enterprise-ready private cloud solutions by combining the value of VCF on HPE Synergy Composable Infrastructure that is flexible and, easy to deploy.
This Reference Architecture describes solution testing performed in March 2020.
Reference Architecture Page 4
Solution overview This Reference Architecture demonstrates best practices for customers building a cloud solution in an enterprise data center and deploying business critical applications in an automated manner. The solution design is based on VMware Cloud Foundation on HPE Synergy. VMware Cloud Foundation provides a unified software-defined data center (SDDC) platform comprising of VMware vSphere® Hypervisor, VMware Virtual SAN™ Storage (vSAN) and VMware NSX® networking.
HPE Synergy is a single infrastructure of pools of compute, storage, and fabric resources, along with a single management interface, HPE OneView, that allows IT to rapidly assemble, disassemble and re-assemble resources in any configuration. HPE Synergy architecture eliminates hardware and operational complexity, so that IT can deliver applications faster and with greater precision and flexibility. HPE OneView is your infrastructure automation engine built with software intelligence. It streamlines provisioning and lifecycle management across compute, storage, and fabric resources in the HPE Synergy system.
This Reference Architecture demonstrates the following solutions for VMware Cloud Foundation on HPE Synergy:
• HPE Storage for VMware Cloud Foundation – HPE Primera, HPE 3PAR StoreServ, and HPE Nimble Storage as a principal storage and also as an ancillary storage to a virtual infrastructure (VI) workload domain on HPE Synergy
• VMware Enterprise PKS - VMware Enterprise PKS as a cloud-native platform on a VMware Cloud Foundation NSX-T workload domain on HPE Synergy
• HPE OneView connector for VCF - Integrated composability with VCF and HPE Synergy using HPE OneView connector for VCF
• HPE OneView for vRealize Orchestrator – HPE OneView for vRealize Orchestrator workflow to update the HPE Synergy compute firmware
• VMware Cloud Foundation lifecycle management - Offline VMware Cloud Foundation update for HPE Synergy
• High Availability of VMware Cloud Foundation management domain using VMware vSAN Stretched cluster
This solution demonstrates the concept using two (2) HPE Synergy 12000 Frames each equipped with eight (8) HPE Synergy 480 Gen10 servers, and an HPE Synergy D3940 Storage Module. Each HPE Synergy 12000 Frame uses HPE Virtual Connect SE 100Gb F8 Module to provide uplink to the data center network. The sixteen (16) HPE Synergy 480 Gen10 servers are used to configure one VCF management domain and three (3) VCF workload domains. The HPE Synergy Storage Module D3940 hosts software-defined VMware vSAN storage. HPE Primera, HPE 3PAR StoreServ or HPE Nimble Storage provides the ancillary storage for the VI workload domains.
HPE Primera, HPE 3PAR StoreServ or HPE Nimble Storage in FC mode is also used to provide principal storage for the VI workload domains, when VMware vSAN is not used as a principal storage.
Reference Architecture Page 5
Figure 1 shows the physical rack layout showcasing the solution components involved.
Figure 1. Physical rack layout showcasing the solution components involved
Reference Architecture Page 6
Solution components Hardware HPE Synergy Composable Infrastructure HPE Synergy is a composable infrastructure platform that empowers IT to create and deploy resources instantly and continuously, gain control of IT resources efficiently, and simplifies IT operation using a single software-defined infrastructure for physical, virtual, and containerized workload. Developers and independent software vendor (ISVs) can programmatically control an HPE Synergy Composable Infrastructure through a single, open API that is native in HPE Synergy powered by HPE OneView. This Reference Architecture is built upon the following composability concepts and capabilities of the HPE Synergy platform.
Fluid resource pools HPE Synergy allows the transformation of traditionally rigid physical systems into flexible virtual resource pools. HPE Synergy creates resource pools of “stateless” compute, storage, and fabric capacity that can be configured almost instantly to rapidly provision infrastructure for a broad range of applications.
Software-defined intelligence The software-defined intelligence in HPE Synergy reduces operational complexity and enables IT organizations to make needed programmatic changes quickly and confidently, with minimal human intervention. HPE Synergy abstracts operational details and replaces them with high-level, automated operations. HPE Synergy uses templates to automatically implement change operations such as updating firmware, adding additional storage to a service, or modifying a network.
Unified API HPE Synergy delivers automation through a unified API that provides a single interface to discover, inventory, configure, provision, update, and diagnose the composable infrastructure in a heterogeneous environment. This fully programmable interface integrates with dozens of popular management tools such as Microsoft® System Centre, VMware vCenter® and open source automation and DevOps tools such as Chef, Docker, and OpenStack.
Figure 2 describes the three architectural principles of HPE Synergy Composable Infrastructure.
Figure 2: Three architectural principles of HPE Synergy Composable Infrastructure
HPE Synergy Composer 2 HPE Synergy Composer 2 provides enterprise-level management to compose and deploy system resources, for all your application needs. This management appliance uses software-defined intelligence with embedded HPE OneView to aggregate compute, storage and fabric resources in a manner that scales to your application needs, instead of being restricted to the fixed ratios of traditional resource offerings. HPE OneView Server Profiles and profile templates capture the entire server configuration in one place, enabling administrators to replicate new Server Profiles and to modify them as needed to reflect changes in the data center. With HPE OneView REST API and automation tools, the entire process of
Reference Architecture Page 7
server personality definition and configuration can be automated. For this Reference Architecture the HPE OneView REST API and PowerShell library were used to automate the Server Profile application to “stateless” servers.
HPE Synergy 12000 Frames The HPE Synergy 12000 Frame is a base infrastructure of compute, storage, network fabric, and power into a scalable solution that addresses and scales with various customer workloads and infrastructures. The HPE Synergy 12000 Frames reduces complexity in the IT infrastructure by unifying all these resources into a common bus, and with the myriad of available network and storage interconnects which allows the frame to interoperate with any other IT environment. At a high level, the HPE Synergy Frame supports the following:
• Twelve half-height or six full-height compute modules. The HPE Synergy design additionally allows for the inclusion of double-wide modules as well, such as the HPE Synergy D3940 Storage Module.
• Ten fans and a single frame link module for in-band and out-of-band management.
• Up to six 2650 Watt power supplies.
• Up to six interconnect modules for full redundancy of three fabrics.
The HPE Synergy 12000 Frame features a fully automated and managed composer module using HPE OneView, contained within the HPE Synergy Composer module. HPE OneView handles all the setup, provisioning, and management both at the physical and logical level.
Figure 3 shows front and rear view of HPE Synergy 12000 Frame.
FrontPanel
Bay12
Bay6
Bay7
Bay1
ApplianceBay 2
ApplianceBay 1
Synergy12000Frame
UID
SynergyComposer2
Active
Power
UID
96%2650W 200-240VAC
96%
96%2650W 200-240VAC
96%
96%2650W 200-240VAC
96%
96%2650W 200-240VAC
96%
96%2650W 200-240VAC
96%
96%2650W 200-240VAC
96%
UID
MGMT
EM
LINK
UID
MGMT
EM
LINK
SynergyComposer2
Active
Power
UID
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
UID
Synergy D3940Storage Module
L/APID UID
Mode
L1 L2Q1 Q2 Q3 Q4
L3 L4X1 X2 Q5 Q6 Q7 Q8
HPE VC SE 100GbF32 Module
Reset
7 85 63 41 2
1 2
3 4
Q 1 Q 2
1 2
3 4HPE VC SE32Gb FC Module
UID
Reset
7 85 63 41 2
1 2
3 4
Q 1 Q 2
1 2
3 4HPE VC SE32Gb FC Module
UID
UID
HPE Synergy 50GbInterconnect Link Module
ResetL1 L2
HPE Synergy12 Port 12G SAS Switch
UID
HPE Synergy12 Port 12G SAS Switch
UID
HPE Synergy 50Gb InterConnect Link
Module
FC Module
Figure 3. HPE Synergy Frame showcasing SY480 servers, D3940 storage module, SAS interconnects, VC SE 100Gb F32 and VC SE 32Gb FC modules.
HPE Synergy 480 Gen 10 Compute Module The HPE Synergy 480 Gen10 Compute Module delivers superior capacity, efficiency, and flexibility in a two-socket, half-height, single-wide form factor to support demanding workloads. Powered by the latest 2nd Generation Intel® Xeon® Scalable Processors and featuring support for up to 4.5TB of HPE DDR4 SmartMemory, flexible storage controller options, three I/O connectors, and designed to create a pool of flexible compute capacity within a composable infrastructure, the HPE Synergy 480 Gen10 Compute Module is an ideal platform for general-purpose enterprise workload performance now and in the future.
The solution as presented in this Reference Architecture contains two HPE Synergy 12000 Frames. Each HPE Synergy 12000 Frames in a rack consist of four (4) HPE Synergy 480 Gen10 Servers.
Reference Architecture Page 8
Figure 4 shows the HPE Synergy 480 Gen10 Compute Module.
Figure 4. HPE Synergy 480 Gen 10 Compute Module
HPE Synergy D3940 Storage Module The HPE Synergy D3940 Storage Module is a direct attached storage module with 40 Small Form Factor (SFF) drive bays designed for use in HPE Synergy 12000 Frames. Through the HPE Synergy 12Gb SAS Connection module, it provides composable storage for up to 10 compute modules in a single frame. HPE Synergy Storage is optimized to use as either a direct attached storage array or as software-defined storage. HPE Synergy D3940 Storage Modules support a family of 12G SAS or 6G SATA HDD and SSD Smart Drives.
Figure 5 shows the HPE Synergy D3940 Synergy Module.
Figure 5. HPE Synergy Storage Module
HPE Virtual Connect SE 100Gb F32 Module for Synergy The HPE Virtual Connect SE 100Gb F32 Module, master module based on synergy composable fabric, is designed for composable infrastructure. The disaggregated, rack-scale design uses a master/satellite architecture to consolidate data center network connections, reduce hardware, and scale network bandwidth across multiple HPE Synergy 12000 Frames. The master module contains intelligent networking capabilities that extends connectivity to satellite frames through Interconnect Link Modules. This decreases top of rack switch needs and substantially reduces costs. The components reduction simplifies fabric management at scale while consuming fewer ports at the data center aggregation layer.
The HPE Virtual Connect SE 100Gb F32 Module for Synergy eliminates network sprawl at the edge with one device that converges traffic inside the HPE Synergy 12000 Frames and directly connects to external LANs.
Reference Architecture Page 9
HPE Synergy 50Gb Interconnect Link Module The HPE Synergy 50Gb Interconnect Link Module (satellite module) is designed for composable infrastructure. Based on a disaggregated, rack-scale design, it uses a master/satellite architecture to consolidate data center network connections, reduce hardware, and scale network bandwidth across multiple HPE Synergy 12000 Frames.
HPE Virtual Connect SE 40Gb F8 Module The HPE Virtual Connect SE 40Gb F8 Module is designed for composable infrastructure. Its disaggregated, rack-scale design uses a master/satellite architecture to consolidate data center network connections, reduce hardware, and scales network bandwidth across multiple HPE Synergy Frames. The master module contains intelligent networking capabilities that extend connectivity to satellite frames through Interconnect Link Modules. This eliminates top of rack switch need and substantially reduces cost. The reduction in components also simplifies fabric management at scale while consuming fewer ports at the data center aggregation layer.
HPE Synergy 20Gb Interconnect Link Module The HPE Synergy 20Gb Interconnect Link Module (satellite module) is designed for composable infrastructure. Based on a disaggregated, rack-scale design, it uses a master/satellite architecture to consolidate data center network connections, reduce hardware and scale network bandwidth across multiple HPE Synergy 12000 Frames.
HPE Virtual Connect SE 16Gb FC Module The HPE Virtual Connect SE 16Gb FC Module for HPE Synergy is a Storage Area Network (SAN) interconnect with a wire-once change-ready technology. The Fibre Channel (FC) module in conjunction with Synergy composer powered by HPE OneView allows workloads to be moved or managed without modifying the network. HPE OneView provides an intuitive management console for seamless connectivity between virtualized Synergy Compute Modules 16Gb FC SAN fabrics.
HPE Virtual Connect SE 32Gb FC Module The HPE Virtual Connect SE 32Gb FC Module for HPE Synergy is a Storage Area Network (SAN) interconnect with a wire-once change-ready technology. The Fibre Channel (FC) module in conjunction with HPE Synergy Composer powered by HPE OneView allows workloads to be moved or managed without modifying the network. HPE OneView provides an intuitive management console for seamless connectivity between virtualized Synergy Compute Modules and 32Gb FC SAN fabrics.
HPE Synergy 3830C 16Gb Fibre Channel Host Bus Adapter HPE Synergy 3830C 16Gb Fibre Channel Host Bus Adapter connects Synergy compute resource pools to SANs over 16Gb native Fibre Channel (FC) fabrics. The Synergy 3830C supports advanced virtualization, security, dual-port isolation, dynamic power management, and low CPU utilization features.
HPE Synergy 5830C 32Gb Fibre Channel Host Bus Adapter The HPE Synergy 5830C 32 GB FC Host Bus Adapter (HBA) provides low latency and flexible connectivity to HPE Synergy Virtual Connect FC modules. The HPE Synergy 5830C 32 GB FC HBA helps minimize CPU utilization for storage I/O and is ideal for environments with greater virtual machine density and bandwidth requirements.
HPE Synergy 6820C 25/50Gb Converged Network Adapter The HPE Synergy 6820C 25/50Gb Converged Network Adapter is a key element in synergy composable fabric connecting pools of compute resources to networks with reliable, high-performing converged Ethernet connectivity up to 50Gb. The HPE Synergy 6820C 25/50Gb Ethernet Adapter provides more than two times the bandwidth compared to 10/20Gb adapters. Therefore, it simplifies hardware implementation by reducing the number of adapters, associated switches and compute modules required to achieve higher bandwidth. This Synergy 6820C adapter converges Ethernet and FCoE onto a single connection, simplifying hardware and reducing costs.
HPE Synergy 3820C 10/20Gb Converged Network Adapter The HPE Synergy 3820C 10/20Gb Converged Network Adapter is a key element in synergy composable fabric connecting pools of compute resources to networks with reliable, high-performing converged 10 or 20 Gb Ethernet connectivity. With Flex-20 technology, the Synergy 3820C converges Ethernet and FCoE onto a single connection simplifying hardware and reducing costs. Each adapter replaces a 10Gb NIC and 8Gb Fibre Channel HBA simplifying I/O hardware by 50% and reducing costs up to 60%. The Synergy 3820C is an ideal choice for any virtualized or converged data center.
Reference Architecture Page 10
HPE FlexFabric 5940 Switch The HPE FlexFabric 5940 Switch Series is a family of high performance and low-latency 10GbE, 40GbE top-of-rack (ToR) data center switches. This switch series includes 100G uplink technology and a 2-slot and 4-slot modular form factor providing ultimate flexibility for ever-changing data center requirements.
Figure 6 shows the front view of the HPE FlexFabric 5940 switch.
Figure 6. HPE FlexFabric 5940 JH396A Switch
HPE StoreFabric SN6600B Fibre Channel Switch The HPE SN66x0B Fibre Channel Switch meets the demands of hyper-scale virtualization, larger cloud infrastructures, and growing flash-based storage environments by delivering market-leading 32Gb Fibre Channel technology and capabilities. It provides a high-density building block for increased scalability, designed to support growth, demanding workloads, and data center consolidation in small to large-scale enterprise infrastructures. It is built for maximum flexibility, scalability, and ease of use. Organizations can scale from 24 to 128 ports, all in an efficient 1U or 2U package. It also provides a simplified deployment process and a point-and-click user interface, making it both powerful and easy to use. With the SN66x0B Switch, organizations gain the best of both worlds: high-performance access to industry-leading storage technology and "pay-as-you-grow" scalability to support an evolving storage environment. Figure 7 shows HPE StoreFabric SN6600B FC switch.
Figure 7. HPE StoreFabric SN6600B Fibre Channel Switch
If using a fabric attached Fibre Channel topology, a minimum of two (2) SAN switches are required. If more than two switches are needed in the solution, switches of the same vendor/type must be added in pairs, so there is always an even number of switches. It is important to ensure that you select the correct model switch for your future growth plans. Select a bigger switch with only the ports requires license to start, if you plan on expanding your HPE Synergy solution in the future.
HPE 3PAR StoreServ Hewlett Packard Enterprise leveraged HPE 3PAR StoreServ to meet the architectural guidelines of an HPE Synergy 480 solution to eliminate any single point of failure (hardware or software) in the system. To mitigate single points of failure at the hardware layer, the system is designed with redundant components, including redundant power domains. At a minimum, there are two controller nodes and two copies of the HPE 3PAR Operating System, even in the smallest system configuration.
To further the resiliency of HPE 3PAR StoreServ, nodes are powered by two (1+1) redundant power supplies and backed up by a string of two batteries. Each battery has sufficient capacity to power the controller nodes and have enough space to save all necessary data in memory into the local physical drive. Although many architectures use “cache batteries,” they are not suitable for extended downtimes that are usually associated with natural disasters and unforeseen catastrophes.
To further mitigate downtime due to component failure, HPE 3PAR Persistent Cache is a resiliency feature built into the HPE 3PAR Operating System that enables graceful handling of an unplanned controller failure or planned maintenance of a controller node. This feature eliminates the substantial performance penalties that are associated with traditional modular arrays and the cache “write-through” mode they enter under
Reference Architecture Page 11
certain conditions. For the HPE 3PAR StoreServ 8440 4-node array, any node that loses its adjacent node can dynamically form a mirrored cache relationship with another storage controller node. Figure 8 shows the HPE 3PAR 8440 4-Node Storage.
Figure 8. HPE 3PAR 8440 4-Node Storage
HPE Nimble Storage HF20 Adaptive Flash Array The HPE Nimble Storage HF20 Adaptive Flash Array is designed for both principal and secondary flash workloads. It is a hybrid flash array for mixed, principal workloads, where cost-efficient flash performance is important. It is a secondary flash array for backup and DR while allowing you to put your backup data to work. Figure 9 shows the HPE Nimble Storage.
Figure 9. HPE Nimble Storage
HPE Primera 600 Storage HPE Primera storage system features a high-speed, full-mesh passive interconnect that joins multiple controller nodes (the high-performance data movement engines of the HPE Primera architecture) to form an all-active cluster. This low-latency interconnect allows for tight coordination among the controller nodes and a simplified software model.
HPE Primera storage redefines mission-critical storage for tier-0 applications. Designed for NVMe and Storage Class Memory, HPE Primera delivers remarkable simplicity, app-aware resiliency for mission-critical workloads, and intelligent storage that anticipates and prevents issues across the infrastructure stack.
HPE Primera delivers on the promise of intelligent storage, advanced data services, and simplicity for your mission-critical applications. With a services-centric OS that sets up in minutes, HPE Primera also upgrades seamlessly to minimize risk and be transparent to applications. Figure 10 shows the HPE Primera Storage.
Figure 10. HPE Primera A670 Storage
Reference Architecture Page 12
Software The table below lists the software components used in this Reference Architecture. Further below mentions the layers of the full solution stack.
Table 1 defines the HPE Synergy 480 Gen10 software components and versions used in this Reference Architecture.
Table 1. HPE software and firmware components used in this Reference Architecture
Component Version
HPE Synergy Composer 5.00.02
HPE Synergy Composer 2 5.00.02
Custom Synergy SPP 2019.12.20200326
(HPE_Synergy_Custom_SPP_2019.12.20200326_Z7550-96866.iso)
HPE Nimble OS 5.0.8
HPE 3PAR OS 3.3.1 MU2
HPE 3PAR SSMC 3.6.1
HPE Primera OS 4.0.3
B-series SAN Fabric OS (FOS) 8.2.1c
VMware Cloud Foundation 3.9.1
Table 2 defines the VMware Cloud Foundation software components.
Table 2. VMware software and firmware components used in this Reference Architecture
Component Version
Cloud Builder VM 2.2.1.0
SDDC Manager 3.9.1
VMware vCenter Server® 6.7 (Appliance) 6.7 Update 3d(Build 15679281)
VMware vSphere 6.7 ESXi 6.7 U3 (Build: 15160138)
HPE OEM Customized vSphere Distribution 6.7 VMware-ESXi-6.7.0-Update3-14320388-HPE-Gen9plus-670.U3.10.4.5.19-Aug2019.iso (Build: 14320388)
VMware vSphere 6.7 patch bundle ESXi670-201912001.zip
(Build: 15160138)
VMware vSAN™ 6.7 Update 3b
VMware NSX®-V Data Center for vSphere 6.4.6
VMware NSX-T Data Center 2.5
Pivotal Container Service 1.5
VMware vRealize® Suite Lifecycle Manager 2.1 Patch 2
VMware vRealize® Log insight™ 4.8
vRealize Log Insight Content Pack for NSX for vSphere 3.9
vRealize Log Insight Content Pack for Linux 2.0.1
vRealize Log Insight Content Pack for vRealize Automation 7.5+ 1.0
vRealize Log Insight Content Pack for vRealize Orchestrator 7.0.1+ 2.1
vRealize Log insight Content Pack for NSX-T 3.8.2
vRealize Operations Manager™ 7.5
Reference Architecture Page 13
Component Version
vRealize Automation™ 7.6
VMware Horizon™ 7 7.10.0
VMware Cloud Foundation VMware Cloud Foundation is the industry’s most advanced enterprise-ready hybrid cloud platform providing a complete set of software-defined services for compute, storage, networking, security and cloud management to run enterprise apps whether it is traditional or containerized. VCF drastically simplifies data center operations by deploying a standardized and validated architecture with built-in lifecycle automation of the cloud stack. It orchestrates, provisions, and deploys a software-defined data center (SDDC) platform by integrating VMware vSphere, vSAN, and NSX into a full stack HCI solution that delivers enterprise-ready cloud infrastructure.
Figure 11 shows VMware Cloud Foundation components.
Figure 11. VMware Cloud Foundation
VMware Cloud Foundation components The core components for VMware Cloud Foundation are explained below.
Cloud Foundation Builder VM The Cloud Foundation Builder VM is a one-time use VM which deploys and configures the management domain and transfers inventory and control to SDDC Manager. During the deployment process, the Cloud Foundation Builder VM validates network information provided in the deployment parameter spreadsheet such as DNS, network (VLANS, IPs, MTUs), and credentials. After the management domain is up and the SDDC Manager is running, the Cloud Foundation Builder VM must be powered off and archived. Table 3 shows Cloud Foundation Builder VM resource requirements.
Table 3. Cloud Foundation Builder VM resource requirement
Components Requirements
CPU 4 vCPUs
Memory 4GB
Storage 350GB
Reference Architecture Page 14
SDDC Manager SDDC Manager manages the bring-up of the Cloud Foundation system, creates and manages workload domains, and performs lifecycle management to ensure the software components remain up to date. SDDC Manager also monitors the logical and physical resources of VCF. It allows data center administrators to configure the additional hosts and racks into a logical pool of resources and thus multiple racks can be managed as a single VCF System. SDDC Manager controls these processes by using workflows. Each workflow comprises of a series of tasks, which are executed by SDDC Manager. There are two VMs installed by VCF 3.0 for SDDC Manager, each VM is performing its own functions.
VMware vCenter Server VMware vCenter Server provides management of a VMware virtualized environment with one or more ESXi hosts. SDDC Manager deploys one VMware vCenter Server per workload domain. By default, all vCenter Servers are configured in enhanced linked mode.
VMware Platform Services Controller During bring-up, SDDC Manager deploys two platform services controllers in the management domain. These instantiate a Single Sign On domain. All vCenter Servers (management domain and compute workload domains) are registered with the SSO domain and configured in enhanced link mode.
VMware vSphere (ESXi) ESXi is a type 1 hypervisor used to implement virtualization on bare metal systems. ESXi provides compute virtualization within the software-defined data center and it is foundational building block for implementing a private cloud.
VMware vSAN VMware vSAN aggregates local or direct-attached data storage devices to create a single storage pool shared across all hosts in the vSAN cluster. vSAN eliminates the need for external shared storage, simplifies storage configuration and virtual machine provisioning.
VMware NSX data center for vSphere NSX is the network virtualization platform for the SDDC, delivering the operational model of a virtual machine for entire networks. With NSX network functions including switching, routing, and firewall are embedded in the hypervisor and distributed across the environment.
VMware NSX for vSphere is a virtualized networking component in the software-defined data center (SDDC) architecture, which programmatically creates, snapshots, deletes, and restores software-based virtual networks. With network virtualization, the functional equivalent of a network hypervisor, NSX reproduces the complete set of Layer 2 to Layer 7 networking services (e.g., switching, routing, firewall, and load balancing) in software. It allows these services to be programmatically assembled in any arbitrary combination to produce unique, isolated virtual networks in a matter of seconds. NSX also provides a platform for various security services both network and endpoint based. NSX provides various built-in services, including L2-L4 firewall and activity monitoring. Additionally, security vendors can leverage its guest introspection and network introspection frameworks to deliver service chained next-generation firewall, IDS/IPS, agentless AV, file integrity monitoring, and vulnerability management capabilities.
VMware NSX-T VMware NSX-T is designed to address application frameworks and architectures that have heterogeneous endpoints and technology stacks. In addition to vSphere, these environments may include other hypervisors, containers, bare metal, and public clouds. NSX-T allows IT and development teams to choose the technologies best suited for their applications. NSX-T is also designed for management, operations, and consumption by development organizations in addition use by IT.
VMware vRealize Log insight vRealize Log Insight delivers heterogeneous and highly scalable log management with intuitive, actionable dashboards, sophisticated analytics and broad third-party extensibility, providing deep operational visibility and faster troubleshooting.
VMware Cloud Foundation also has the following optional components for which separate licenses are needed:
• VMware vRealize Operations Manager: vRealize Operations Manager delivers intelligent operations management with application-to-storage visibility across physical, virtual, and cloud infrastructures. Using policy-based automation, operation teams automate key processes and improve IT efficiency. This is an optional component.
• VMware vRealize Automation: vRealize Automation is a cloud automation tool that accelerates the delivery of IT services through automation and pre-defined policies, providing high-level of agility and flexibility for developers, while enabling IT teams to maintain frictionless governance and control. This is an optional component.
Reference Architecture Page 15
• vRealize Suite Orchestrator: vRealize Orchestrator is a development- and process-automation platform that provides an extensive library of workflows and a workflow engine. It simplifies the automation of complex IT tasks.
HPE OneView for VMware vRealize Operations HPE OneView for VMware vRealize Operations provides integrated and highly automated performance, capacity, configuration compliance, and cost management tools to the vRealize Operations custom GUI. The plugin seamlessly integrates the manageability features of HPE Synergy with VMware analytics engine that analyzes what is normal and then applies that baseline to a dynamic server environment.
When the HPE OneView for VMware vRealize Operations is installed, the custom HPE OneView dashboards are added to the vRealize Operation custom GUI. The HPE OneView dashboards allow you to monitor resources in a vRealize environment. The attributes that can be monitored include, resource health, power, temperature (server and enclosure), and system alerts. The analytics engine allows for proactive monitoring of the HPE OneView resource environment and indicates the state of the resources. If a problem occurs, an alert is triggered and displayed. The analytics engine also provides proactive prediction, which can determine the point in the future when a resource will reach a predefined critical level.
HPE OneView for VMware vRealize Orchestrator HPE OneView for VMware vRealize Orchestrator helps customers automate complex IT tasks in an extensible and repeatable manner. It provides a predefined collection of HPE tasks and workflows that can be used in vRealize Orchestrator (VRO) with easy-to-use, drag and drop access to automation of HPE OneView managed hardware deployment, firmware updates, and other life-cycle tasks. HPE OneView for VMware vRealize Orchestrator allows the advanced management features of HPE OneView to be incorporated into larger IT workflows. HPE OneView workflows and actions can also be integrated in VMware vRealize Automation via vRealize Orchestrator.
HPE OneView connector for VCF HPE OneView connector for VCF provides a REST service to answer DMTF Redfish® compliant requests by querying HPE OneView. HPE OneView is a fresh approach to converged infrastructure management, inspired by the way you expect to work, with a single integrated view of your IT infrastructure.
DMTF Redfish is an open industry standard specification and schema that specifies a RESTful interface and utilizes JSON and OData to help customers integrate solutions within their existing tool chains.
Reference Architecture Page 16
Design and configuration guidance HPE Synergy solution design and configuration The solution involves two (2) HPE Synergy 12000 frames each equipped with eight (8) HPE Synergy 480 Gen10 servers, and an HPE Synergy D3940 Storage Module. Each HPE Synergy D3940 storage module consists of 40 drive enclosures comprising a mix of 1.92TB SATA SSD drives for capacity tier and 800GB SAS SSD for cache tier, as per VMware vSAN requirement. The HPE Synergy 12000 Frames have a redundant pair of HPE Synergy 12Gb SAS Connection Modules, to provide powerful and redundant connectivity to the HPE Synergy D3940 Storage Module, and a redundant pair of HPE Synergy Virtual Connect SE 40Gb F8 Modules, for high-speed uplink connectivity to multiple networks.
Figure 12 shows the network layout of this Reference Architecture.
Figure 12. Network layout of this Reference Architecture
HPE VC SE 40GbF8 Module
L 2L 1 Q 4Q 3Q 2Q 1 Q 8Q 7Q 6Q 5Eth / X-Link
L 4L 3 Reset
UID
PIDL/A
HPE VC SE 40GbF8 Module
L 2L 1 Q 4Q 3Q 2Q 1 Q 8Q 7Q 6Q 5Eth / X-Link
L 4L 3 Reset
UID
PIDL/A
SYS
32313029282726252423222120191817161514131211654321 Green=40Gbps,Yellow=10GbpsQSFP+
HPE FlexFabric5940 SeriesSwitchJH396A
SYS
32313029282726252423222120191817161514131211654321 Green=40Gbps,Yellow=10GbpsQSFP+
HPE FlexFabric5940 SeriesSwitchJH396A
HPE 5940 FlexFabric A HPE 5940 FlexFabric B
80G LAG-A 80G LAG-B
UID
iLO
Synergy480
Gen 10
HPE Synergy 20GbInterconnect Link Module
L 2L 1
UID
HPE Synergy 20GbInterconnect Link Module
L 2L 1
UID
HPE 3820c CNA (FlexNIC's)
P1 P2
UID
iLO
Synergy480
Gen 10
HPE 3820c CNA (FlexNIC's)
P1 P2
FrontPanel
Bay12
Bay6
Bay7
Bay1
ApplianceBay 2
ApplianceBay 1
Synergy12000Frame
UID
UID
Active
Power
Synergy Composer
FrontPanel
Bay12
Bay6
Bay7
Bay1
ApplianceBay 2
ApplianceBay 1
Synergy12000Frame
UID
UID
Active
Power
Synergy Composer
ICM 3
ICM 6
ICM 6
ICM 3
IRF link
HPE Synergy Frame 1 HPE Synergy Frame 2
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
UID
Synergy D3940Storage Module
UID
Synergy D3940Storage Module
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
1
2
UID
iLO
Synergy480
Gen10
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10 K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
SAS900 GB
10KSAS
900 GB
10K
Reference Architecture Page 17
Table 4 defines the hardware configuration used in this Reference Architecture.
Table 4. Hardware configuration
Hardware Details
Number of racks 1
HPE Synergy 12000 Frames in rack 2
HPE Synergy 480 Gen 10 Compute Module per Synergy 12000 Frame 8
HPE Synergy D3940 Storage Module per HPE Synergy 12000 Frame 1
HPE Virtual Connect SE 40Gb F8 Module for Synergy per Synergy 12000 Frame 1
HPE Synergy 20GB Interconnect Link Module per Synergy 12000 Frame 1
HPE Virtual Connect SE 100Gb F32 Module for Synergy per Synergy 12000 Frame 1
HPE Synergy 50GB Interconnect Link Module per Synergy 12000 Frame 1
HPE Virtual Connect SE 16Gb FC Modules per Synergy 12000 Frame 2
HPE Virtual Connect SE 32Gb FC Modules per Synergy 12000 Frame 2
HPE Synergy Composer Module per Synergy 12000 Frames 1
HPE Synergy 3830C 16Gb Fibre Channel Host Bus Adapter 1
HPE Synergy 5830C 32Gb Fibre Channel Host Bus Adapter 1
HPE Synergy 6820C 25/50Gb Converged Network Adapter 1
Note Since HPE Nimble Storage supports bandwidth only up to 16Gb, HPE Nimble Storage configurations in this solution are tested with HPE VC SE 16Gb FC Module and HPE VC SE 40Gb F8 Module/HPE 20Gb Interconnect Link Module.
Table 5 defines the HPE Synergy 480 Gen10 hardware components used in this Reference Architecture.
Table 5. HPE Synergy 480 Gen10 hardware components (quantities are per node)
Hardware Quantity Description
CPU 2 Intel(R) Xeon(R) Gold 6254 CPU (3.1 GHz / 18-core)
Memory 24 HPE 16GB Single Rank x4 DDR4-2933 CAS-21-21-21 Registered Smart Memory Kit
Total 384GB memory on each node
FC HBA 1 HPE Synergy 5830C 32Gb Fibre Channel Host Bus Adapter
25/50Gb CNA 1 HPE Synergy 6820C 25/50Gb Converged Network Adapter
Table 6 defines the ancillary storage hardware configuration used in this Reference Architecture.
Table 6. HPE Synergy 480 Gen10 hardware components (quantities are per node)
Hardware Details
Number of racks 1
HPE Nimble Storage HF20 Adaptive Dual Controller Array 1
HPE Primera 600 Storage 1
HPE 3PAR StoreServ 8440 Storage 1
Reference Architecture Page 18
The VMware Cloud Foundation infrastructure needs different external services for initial deployment and deployment of other optional components like vRealize Operations or vRealize Automation. Those services as Active Directory, Dynamic Host Configuration Protocol (DHCP), Domain Name Service (DNS) and Network Time Protocol (NTP) are part of customer’s data center environment.
The Cloud Foundation Builder virtual machine was installed on one of the ESXi host that will be the part of the HPE Synergy VCF environment. It is configured to have network connectivity to the management network of all ESXi hosts to be added to the VMware Cloud Foundation solution as well network connectivity to Virtual Machines providing external services as Active Directory, Dynamic Host Configuration Protocol (DHCP), Domain Name Service (DNS), and Network Time Protocol (NTP).
HPE FlexFabric 5940 configuration for VMware Cloud Foundation HPE FlexFabric 5940 Switches should be connected and configured for IRF. All the VLANs required for VCF deployment should be created on the top-of-rack (ToR) switches or allowed from customer network data center uplinks. The Ethernet downlink ports on the HPE FlexFabric 5940 should be trunk enabled allowing all VLANs required for the VMware Cloud Foundation stack. Create a LAG by selecting single port from each switch and ensure the LAG connects to the HPE Synergy Virtual Connect ports as shown in the Figure 12.
VMware introduced ‘Application Virtual Networks’ in VCF 3.9.1 where BGP configuration becomes necessary to deploy VCF 3.9.1 via Cloud Builder VM. For VCF 3.9.1, Figure 13 and 14 highlights the fields in VCF deployment parameter sheet. This is required for setting up network and routing configuration required for BGP on HPE FlexFabric 5940 switches.
Figure 13. BGP VLANs shown in VCF deployment parameter sheet
Reference Architecture Page 19
As per the above VCF 3.9.1 parameter sheet, two additional VLANs needs to be configured as ‘Uplinks’ for VMware NSX in VCF Management Domain. These uplinks will represent the NSX Edge VM uplinks for North-South communication.
Figure 14. Application Virtual Network details shown in VCF deployment parameter sheet
Before deploying the Cloud Builder VM for VCF 3.9.1 deployment, ensure to configure the HPE 5940 FlexFabric switches with BGP as shown in the following command list.
Configure VLANs required for VMware Cloud Foundation deployment with the following commands.
[snet01]display current-configuration # vlan 1 # vlan 1611 to 1618 # vlan 2711 description ESGUplink-1 # vlan 2712 description ESGUplink-2 # irf-port 1/1 port group interface FortyGigE1/0/1 port group interface FortyGigE1/0/2 # irf-port 2/2 port group interface FortyGigE2/0/1 port group interface FortyGigE2/0/2 # stp global enable #
Reference Architecture Page 20
interface NULL0 # interface LoopBack0 ip address 1.1.1.1 255.255.255.255 # interface Vlan-interface1 # interface Vlan-interface1611 description Mgmt-MGMT mtu 9000 ip address 172.16.11.1 255.255.255.0 # interface Vlan-interface1612 description vMotion-MGMT mtu 9000 ip address 172.16.12.1 255.255.255.0 # interface Vlan-interface1613 description vSAN-MGMT mtu 9000 ip address 172.16.13.1 255.255.255.0 # interface Vlan-interface1614 description VXLAN-MGMT mtu 9000 ip address 172.16.14.1 255.255.255.0 # interface Vlan-interface1615 description Mgmt-WLD ip address 172.16.15.1 255.255.255.0 # interface Vlan-interface1616 description vMotion-WLD ip address 172.16.16.1 255.255.255.0 # interface Vlan-interface1617 description vSAN-WLD ip address 172.16.17.1 255.255.255.0 # interface Vlan-interface1618 description VXLAN-WLD ip address 172.16.18.1 255.255.255.0 # interface Vlan-interface2711 mtu 9000 ip address 172.27.11.1 255.255.255.0 # interface Vlan-interface2712 mtu 9000 ip address 172.27.12.1 255.255.255.0 #
For application virtual networks in VCF 3.9.1, you need to configure BGP on the HPE FlexFabric 5940 switches. In the VCF parameter sheet, the BGP Autonomous System ID is mentioned as 65001. The universal DLR deployed as part of NSX deployment in the VCF management domain has the BGP Autonomous System ID as 65004. Hence both the BGP Autonomous Systems need to establish peering for North-South communication.
Reference Architecture Page 21
Configure the NSX Edge uplinks to BGP peer with the UDLR ESG Autonomous System ID 65004 and enable the RegionA01 and xRegion01 VXLAN logical switch networks in the BGP configuration on HPE 5940 switches as shown in the following BGP configuration.
# bgp 65001 router-id 1.1.1.1 peer 172.27.11.12 as-number 65004 peer 172.27.11.12 connect-interface Vlan-interface2711 peer 172.27.11.12 password simple <Password> peer 172.27.11.13 as-number 65004 peer 172.27.11.13 connect-interface Vlan-interface2711 peer 172.27.11.13 password simple <Password> peer 172.27.12.12 as-number 65004 peer 172.27.12.12 connect-interface Vlan-interface2712 peer 172.27.12.12 password simple <Password> peer 172.27.12.13 as-number 65004 peer 172.27.12.13 connect-interface Vlan-interface2712 peer 172.27.12.13 password simple <Password> # address-family ipv4 unicast network 172.16.11.0 255.255.255.0 network 172.27.11.0 255.255.255.0 network 172.27.12.0 255.255.255.0 network 192.168.20.0 255.255.255.0 network 192.168.21.0 255.255.255.0 network 192.168.41.0 255.255.255.0 peer 172.27.11.12 enable peer 172.27.11.13 enable peer 172.27.12.12 enable peer 172.27.12.13 enable #
HPE OneView configuration through HPE Synergy Composer The following steps summarize the HPE Synergy configuration.
Create HPE OneView networks with respect to the VLANs created on the ToR switches as shown in the Table 7.
Table 7. HPE OneView Networks
HPE OneView Networks VLAN IDs
VCF Management Domain Management VLAN 1611
VCF Management Domain vMotion VLAN 1612
VCF Management Domain vSAN VLAN 1613
VCF Management Domain VTEP (VXLAN) 1614
VCF Workload Domain Management VLAN 1615
VCF Workload Domain vMotion VLAN 1616
VCF Workload Domain vSAN VLAN 1617
VCF Workload Domain VTEP (VXLAN) 1618
Reference Architecture Page 22
Create HPE Oneview network sets to be used in the respective HPE Oneview Server Profile template of VMware Cloud Foundation workload domain nodes, as shown in Table 8 and Table 9.
Table 8. HPE OneView VCF Management Network Sets
VCF Management Domain Network Set A VLAN VCF Management Domain Network Set B VLAN
Management VLAN 1611 Management VLAN 1611
vMotion VLAN 1612 vMotion VLAN 1612
vSAN VLAN 1613 vSAN VLAN 1613
VTEP (VXLAN) 1614 VTEP (VXLAN) 1614
Table 9. HPE OneView VCF Workload Network Sets
VCF Workload Domain Network Set A VLAN VCF Workload Domain Network Set B VLAN
Management VLAN 1615 Management VLAN 1615
vMotion VLAN 1616 vMotion VLAN 1616
vSAN VLAN 1617 vSAN VLAN 1617
VTEP (VXLAN) 1618 VTEP (VXLAN) 1618
Create one Logical Interconnect Group for each physical type of interconnect used in the solution and assign the proper uplink sets as shown
Table 10.
Table 10. HPE OneView Logical Interconnect Group
Logical Interconnect Group A Values Logical Interconnect Group B Values
Name of Logical Interconnect Group LIG-A Name of Logical Interconnect Group LIG-B
Add HPE OneView Networks 1611-1618 Add HPE OneView Networks 1611-1618
Add Uplink Ports from Virtual Connect ICM3 Add Uplink Ports from Virtual Connect ICM6
Note The sample VLAN IDs mentioned above has been used in the solution. However, the VLAN IDs need to be replaced as per the customer’s requirements.
Create an Enclosure Group.
Create a Logical Enclosure and apply the appropriate firmware baseline.
Create the HPE OneView Server Profile templates, selecting the appropriate hardware type and the Enclosure Group created in the previous steps.
Create a Server Profile template for the VMware Cloud Foundation management domain and workload domain with the following settings, as listed in Table 11 and Table 12.
Table 11. Sample Values for HPE OneView Server Profile for VCF management domain
HPE OneView Server Profile template for VCF Management Domain Node Sample Values
Server Profile Template Name VCF Management Domain Template
Firmware
Connection 1 VCF Management Domain Network Set A
Connection 2 VCF Management Domain Network Set B
Reference Architecture Page 23
HPE OneView Server Profile template for VCF Management Domain Node Sample Values
Local Storage (2 SSDs) via Integrated Storage Controller RAID-1 for VMware ESXi 6.7 U3
HPE D3940 (SSDs/HDD) via SAS Mezz 1 Storage Controller Cache_Tier (SSD), Capacity Tier (SSD/HDD)
*Configure capacity and cache Tiers in Logical JBODs in HPE OneView in the Storage tab
Table 12. Sample Values for HPE OneView Server Profile for VCF workload domain
HPE OneView Server Profile template for NSX-T Workload Domain Node Sample Values
Server Profile Template Name VCF Workload Domain Template
Firmware
Connection 1 VCF Workload Domain Network Set A
Connection 2 VCF Workload Domain Network Set B
Local Storage (2 SSDs) via Integrated Storage Controller RAID-1 for VMware ESXi 6.7 U3
HPE D3940 (SSDs/HDDs) via SAS Mezz 1 Storage Controller Cache_Tier (SSD), Capacity Tier (SSD/HDD)
*Configure Capacity and Cache Tiers in Logical JBODs in HPE OneView in the Storage tab
VMware Cloud Foundation installation on HPE Synergy Initiate VMware Cloud Foundation deployment across the four management nodes via VMware Cloud Builder VM and the VCF deployment parameter sheet. For more details, refer to the VMware Cloud Foundation deployment guide at https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/com.vmware.vcf.ovdeploy.doc_39/GUID-F2DCF1B2-4EF6-444E-80BA-8F529A6D0725.html.
After the VMware Cloud Foundation is deployed successfully, log in to the SDDC Manager via browser https://<SDDC_Manager_IP>.
Demonstration of HPE storage solutions for VMware Cloud Foundation workload domain VMware Cloud Foundation supports block storage though IP based (iSCSI) external storage or FC based external storage enabling addition of principal and ancillary storage options for virtual infrastructure (VI) workload domain. The datastores provided through external storage system is connected either through iSCSI or FC protocol and managed independently through the workload domain vCenter instance.
This Reference Architecture describes the deployment of HPE Primera, HPE 3PAR StoreServ, and HPE Nimble Storage as a principal storage and as an ancillary storage to a VMware Cloud Foundation v3.9.1 VI workload domain on the HPE Synergy hardware platform.
HPE storage choices for virtual infrastructure (VI) workload domain without vSAN Starting with VCF 3.9.x, workload domain can deployed on block storages. Workload domain deployed on block storage (VMFS on FC) is termed as principal storage. The term ancillary storage refers to the next storage tier that is managed out of band either through storage specific GUI or vCenter plugin. Table 13 represents the choice of options tested as part of this Reference Architecture.
Note If HPE D9340 (vSAN) storage module is not part of the VCF workload domain deployment, HPE OneView connector cannot be used to compose servers for workload domain. Instead use HPE OneView to compose the servers with the desired HPE storage.
Reference Architecture Page 24
Table 13. List of HPE storages validated for VCF workload domain without vSAN
HPE Storages for VCF Workload Domain Principal Storage Ancillary Storage
3PAR FC
3PAR Direct Attach
Nimble FC
Nimble Direct Attach
Nimble iSCSI NA
Primera FC
HPE storage choices for virtual infrastructure (VI) workload domain with vSAN (HPE D3940) If you choose to deploy workload domain on vSAN, following options are available as ancillary storages to the workload domain and are managed out of band either through storage specific GUI or vCenter plugin.
Table 14. List of HPE storages validated for VCF workload domain with vSAN
HPE Storages for VCF Workload Domain with vSAN (HPE D3940)
3PAR FC
3PAR Direct Attach
Nimble FC
Nimble Direct Attach
Nimble iSCSI
Primera FC
Note For management domains, vSAN is the only storage option that may be assigned.
HPE Primera storage for virtual infrastructure (VI) workload domain HPE Primera is an intelligent storage technology in the form of all-flash or flash / spinning media converged architecture. It provides principal storage as well ancillary storage options for the virtual infrastructure (VI) workload domains. The volumes created from HPE Primera storage, connected to an HPE Synergy chassis through fibre channel (FC) connections, can be used to create ‘VMFS on FC’ datastores for virtual infrastructure (VI) workload domain installations. Other volumes created from HPE Primera storage can be presented as ancillary storage options for the workload domain.
Reference Architecture Page 25
Figure 13 shows the block diagrams for HPE Primera storage options for VI workload domains.
HPE Primera 4 Node Storage
HPE Synergy Frame 2HPE Synergy Frame 1
Workload domain 1Management domain
vSAN storage
NSX Controll
ers
vCenter server/
PSC
vCenter server/
PSC
vCenter server/
PSC
NSX Control
lersVM VM VM
HPE Primera Storage ArrayHPE Primera Storage Array
vSAN storage HPE Primera Storage Array
Workload domain 2
NSX Control
lersVM VM VM
Figure 13. HPE Primera Storage as principal storage for VMware Cloud Foundation v3.9.1 workload domains
Note When HPE Primera is the only choice of storage for workload domain, compose servers using HPE OneView and not via HPE OneView connector for VCF. However, if you choose to set up your workload domain with D3940 (vSAN) along with Primera, compose/decompose of servers could be performed using HPE OneView connector for VCF via SDDC Manager.
Reference Architecture Page 26
HPE Primera Fabric-Attach Fibre Channel (FC) topology Figure 14 shows a fabric-attached SAN networking using SN6600B 32Gb fibre channel switches, HPE Synergy with 32Gb uplinks, and an HPE Primera A630. For downlink connectivity, HPE Synergy 5830C 32 Gb fibre channel host bus adapters are used. Hewlett Packard Enterprise recommends cabling odd numbered Primera ports to one switch and even numbered ports to the partner switch. At the minimum, two ports per component must be used to enable continuous server operation during the failure of a switch, fibre cable, or server fibre port and to enable continuous operation during a firmware updates to SAN switches.
Figure 14. SAN network topology with HPE Primera storage example
For more information on the deployment of HPE Primera as a principal storage to VMware Cloud Foundation workload domain, see http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001409enw.
HPE 3PAR StoreServ for virtual infrastructure (VI) workload domain VMware Cloud Foundation VI workload domains can be integrated with HPE 3PAR StoreServ for various workloads and applications. VMware Cloud Foundation workload domain on HPE Synergy can have HPE 3PAR StoreServ integrated as a principal storage and as an ancillary storage through Fibre Channel (FC) protocol in either of the following ways:
• HPE 3PAR Fabric-Attach Fibre Channel (FC) topology
• HPE 3PAR Direct-Attached topology
In this solution, HPE 3PAR 8450 4-Node external storage was configured to provide FC datastores as principal storage to VMware Cloud Foundation VI workload domains through Fabric-Attach Fibre Channel topology and Direct-Attach topology. HPE Virtual Connect SE 32Gb FC
Reference Architecture Page 27
Module for HPE Synergy was mounted on the Interconnect Module Bay Set 2 (Interconnect Bay 2 and Interconnect Bay 5). HPE Virtual Connect SE 32Gb FC Module was connected to two HPE StoreFabric SN6600B Fibre Channel Switch as Storage Area Network (SAN) Switch. The same configuration can be used to provide ancillary storage to VCF workload domain servers.
Figure 15 shows HPE 3PAR StoreServ as both principal and as an ancillary storage for virtual infrastructure (VI) workload domain.
12
110 23
12
110 23
3PARStoreServ
8450
3PARStoreServ
8450
Workload domain 1Management domain
vSAN storage
NSX Controll
ers
vCenter server/
PSC
vCenter server/
PSC
vCenter server/
PSC
NSX Control
lersVM VM VM
HPE 3PAR Storage ArrayHPE 3PAR Storage Array
vSAN storage HPE 3PAR Storage Array
Workload domain 2
NSX Control
lersVM VM VM
Principal Storage
Ancillary Storage
Figure 15. HPE 3PAR StoreServ as both principal and as an ancillary storage for VMware Cloud Foundation v3.9.1 workload domain
Note When HPE 3PAR is the only choice of storage for workload domain, compose servers using HPE OneView and not via HPE OneView connector for VCF. However, if you choose to set up your workload domain with D3940 (vSAN) along with HPE 3PAR, compose/decompose of servers could be performed using HPE OneView connector for VCF via SDDC Manager.
Reference Architecture Page 28
HPE 3PAR Fabric-Attach Fibre Channel (FC) topology HPE 3PAR storage with a dedicated 32Gb Fibre Channel topology provides better storage and networking performance for HPE Synergy compute modules. HPE OneView is used to provision, manage, and administer the HPE 3PAR StoreServ thus providing a single unified management pane for storage operations for the virtual infrastructure (VI) workload domain.
Figure 16 shows the cabling of HPE 3PAR StoreServ to SAN fabrics. It is recommended to connect all the odd numbered ports in the storage to one SAN fabric and the even numbered ports to another. At the minimum, you must have two ports per component to enable continued server operation during the failure of a switch, fibre cable, or server fibre port and to enable continued operation during a firmware update to the SAN fabrics in the solution.
Figure 16. SAN network topology with HPE 3PAR storage example
Reference Architecture Page 29
HPE 3PAR Direct-Attach topology In the HPE 3PAR Direct-Attach configuration, the storage for VMware Cloud Foundation workload domain is provided by direct cabling of HPE Virtual Connect 40Gb SE for HPE Synergy module to HPE 3PAR. Each HPE Virtual Connect for HPE Synergy module must be connected to at least two separate controllers in the same array. A minimum of two ports are required per component to allow the continued server operation during controller failure and during firmware updates.
Figure 17 describes HPE 3PAR StoreServ cabling in the Direct-Attach topology.
Figure 17. HPE 3PAR StoreServ as direct-attach topology example
For more information on the deployment of HPE 3PAR StoreServ as a principal and as an ancillary storage to VMware Cloud Foundation workload domain, refer to http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001408enw.
Reference Architecture Page 30
HPE Nimble Storage for VMware Cloud Foundation v3.9.1 workload domain HPE Nimble Storage is an intelligent, self-managing storage technology in form of all flash and hybrid flash. It provides principal storage as well ancillary storage options for the VMware Cloud Foundation (VCF) 3.9.1 workload domain. The volumes created from HPE Nimble Storage, connected to HPE Synergy through Fabric-Attached Fibre Channel (FC) connection or Direct-Attached Fibre Channel (FC) connection, can be used for VMFS on FC datastore to install VCF workload domain. Volumes created from HPE Nimble Storage can be presented as principal or ancillary storage options for VCF workload load domain.
Figure 18 shows the block diagram for HPE Nimble Storage options for VMware Cloud Foundation.
• HPE Nimble Fabric-Attach Fibre Channel (FC) topology
• HPE Nimble Direct-Attached topology
• HPE Nimble Storage iSCSI topology
HPE Nimble Storage
HPE Synergy Frame 2HPE Synergy Frame 1
Workload domain 1Management domain
vSAN storage
NSX Controll
ers
vCenter server/
PSC
vCenter server/
PSC
vCenter server/
PSC
NSX Control
lersVM VM VM
HPE Nimble Storage ArrayHPE Nimble Storage Array
vSAN storage HPE Nimble Storage Array
Workload domain 2
NSX Control
lersVM VM VM
Principal Storage
Ancillary Storage
Nimble Storage
Figure 18. HPE Nimble Storage HF20 as ancillary storage for VMware Cloud Foundation v3.8.1 workload domain
Note When HPE Nimble is the only choice of storage for workload domain, compose the servers using HPE OneView and not via HPE OneView connector for VCF. However, if you choose to set up your workload domain with D3940 (vSAN) along with HPE Nimble, compose/decompose of servers could be performed using HPE OneView connector for VCF via SDDC Manager.
Reference Architecture Page 31
HPE Nimble Storage Fabric-Attach Fibre Channel (FC) topology The HPE Nimble Storage 2x 16Gb Fibre Channel 2-port adapter kit was added to the HPE Nimble Storage HF20 Base Array for FC connectivity. The HPE Virtual Connect SE 16Gb FC module for Synergy was mounted on the Interconnect Module Bay Set 2 (Interconnect Bay 2 and Interconnect Bay 5) of the HPE Synergy 12000 Frame. The HPE Virtual Connect SE 16Gb FC module was connected to two HPE StoreFabric SN6600B Fibre Channel Switches as a Storage Area Network (SAN) Switch.
Figure 19 shows the cabling of HPE Nimble Storage to SAN fabrics. All the odd numbered Fibre Channel ports in 2x16Gb Fibre Channel 2-port adapter of the HPE Nimble Storage is connected to one SAN fabric and the even numbered ports to second SAN fabric. This configuration ensures continued server operation during the failure of a switch, fibre cable, or server fibre port and to enable continued operation during a firmware update to the SAN fabrics in the solution.
Figure 19. Cabling Details for HPE Nimble Storage Fabric Attach FC topology storage solution
IN
OUT
IN
OUT
HPE VC SE 16GbFC Module
7 85 63 41 2 Q 3 Q 4
1 2
3 4
1 2
3 4
Q 1 Q 2
1 2
3 4
1 2
3 4 Reset
UID
UID
iLO
Synergy480
Gen10
UID
iLO
Synergy480
Gen10
40 51 62 73 128 139 1410 1511 3632 3733 3834 3935 4440 4541 4642 4743
48 49 50 51 56 57 58 59
52 53 54 55 60 61 62 632016 2117 2218 2319 2824 2925 3026 3127
1 3
0 2
HPE SN6600BFC Switch
HPE VC SE 16GbFC Module
7 85 63 41 2 Q 3 Q 4
1 2
3 4
1 2
3 4
Q 1 Q 2
1 2
3 4
1 2
3 4 Reset
UID
40 51 62 73 128 139 1410 1511 3632 3733 3834 3935 4440 4541 4642 4743
48 49 50 51 56 57 58 59
52 53 54 55 60 61 62 632016 2117 2218 2319 2824 2925 3026 3127
1 3
0 2
HPE SN6600BFC Switch
PORT
0
PORT
1
T X R X T X R X
FC16
PORT
0
PORT
1
T X R X T X R X
FC16
HPE Synergy 480 Gen10 Compute Modules
HPE Synergy Virtual Connect SE 16GB FC Module InterConnect
Module Bay 5
Nimble Storage HF20 Adaptive Array
HPE Synergy Virtual Connect SE 16GB FC Module InterConnect
Module Bay 2
HPE StoreFabric SN6600B Fibre Channel Switch
HPE StoreFabric SN6600B Fibre Channel Switch
2*16Gb Fibre Channel 2 Port Adapter
Reference Architecture Page 32
HPE Nimble Storage Direct-Attach topology Figure 20 shows the cabling of HPE Nimble Storage connected directly to HPE Synergy Frames. In a two frame HPE Synergy solution, the HPE Nimble Storage is connected directly to port 3 of Interconnect Module (ICM) located in Bay 3 for HPE Synergy Frame 1 and Port 3 of Interconnect Module located in Bay 6 for HPE Synergy Frame 2. In a single frame HPE Synergy solution, the Interconnect Module would be in Bay 3 and Bay 6. The Interconnect Module (ICM) used is HPE Synergy Virtual Connect SE 40 Gb F8 Module.
Figure 20. HPE Nimble Storage Direct-Attached Fibre Channel (FC) topology
Reference Architecture Page 33
HPE Nimble Storage iSCSI topology Figure 21 shows the cabling diagram of HPE Nimble Storage (iSCSI) with HPE Synergy. The cabling shows that the HPE Synergy 12000 Frame is connected to VCF management network on HPE Synergy Frame through eth0a and eth0b on each HPE Nimble Storage controller via the HPE Synergy Virtual Connect SE 40GB Interconnect Module. HPE Nimble Storage is configured to use two different networks, management network of speed 1Gbps and data network of speed 10Gbps.
Figure 21. Cabling diagram of HPE Nimble Storage (iSCSI) with HPE Synergy
HPE OneView is used to provision, manage, and administer the HPE Nimble Storage.
For more information on the deployment of HPE Nimble Storage as an ancillary storage to VMware Cloud Foundation workload domain, refer to http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001414enw.
Refer the HPE Nimble Storage documentation to understand the best practices of configuring HPE Nimble Storage at https://infosight.nimblestorage.com/InfoSight/app#documentation (requires HPE Nimble InfoSight login).
Reference Architecture Page 34
Demonstrate VMware Enterprise PKS as a cloud-native platform on HPE Synergy The Reference Architecture illustrates the deployment of VMware Cloud Foundation NSX-T Workload Domain for VMware Enterprise PKS as the platform for cloud-native applications. This document highlights the implementation of VMware NSX-T as software-defined networking component on HPE Synergy Composable Infrastructure. VMware Cloud Foundation by default provides a software-defined data center (SDDC) architecture by virtualizing compute, storage and network. Using the SDDC Manager, VMware Enterprise PKS is deployed as a platform on the VMware Cloud Foundation NSX-T workload domain for organizations to deploy cloud-native applications.
HPE Synergy comfortably maps all its native components to the VMware SDDC model thus making it an ideal fit for VMware SDDC deployments. HPE Synergy Storage Module D3940 hosts software-defined VMware vSAN storage. HPE Synergy Virtual Connect SE F8 Networking Module along with HPE FlexFabric 5940 Switch provides logical networking components for VMware NSX-T that is a next-generation software–defined networking solution. HPE Synergy powered by HPE OneView composes and recomposes Server Profiles comprising of compute, storage and network for VMware vSphere environments.
The following are the deployment steps to configure VMware Cloud Foundation NSX-T workload domain and VMware Enterprise PKS on HPE Synergy.
Deploy and configure a VMware NSX-T workload domain on HPE Synergy.
Deploy and configure a VMware NSX-T Edge on the VMware Cloud Foundation workload domain for north-south communication.
Configure the VMware NSX-T Routers for East-West and north-south communication from the data center.
Configure and create overlay networks and logical switches for the VMware Enterprise PKS environment.
Deploy and configure a VMware Enterprise PKS as a platform on the VMware Cloud Foundation workload domain.
Reference Architecture Page 35
Figure 22 shows the HPE Synergy master satellite connections for HPE Synergy Frame 1 and 2. HPE Synergy Frame 3 will comprise of two satellite modules that will in turn connect to the master modules in Frame 1 and Frame 2 completing the network loop. For detailed understanding, refer to https://techlibrary.hpe.com/docs/synergy/shared/cabling/GUID-322C6B82-3183-42E6-A01D-B52BE4CF77C7.html.
Figure 22. Network Layout for VMware Enterprise PKS and VMware NSX-T on HPE Synergy
Reference Architecture Page 36
VMware NSX-T architecture VMware NSX-T is designed to address application frameworks and architectures that have heterogeneous endpoints and technology stacks. In addition to vSphere, these environments may include other hypervisors, containers, bare metal, and public clouds. NSX-T allows IT and development teams to choose the technologies best suited for their particular applications. NSX-T is also designed for management, operations, and consumption by development organizations in addition use by IT. Figure 23 shows the VMware NSX-T architecture.
Figure 23. VMware NSX-T architecture
Reference Architecture Page 37
VMware Enterprise PKS Architecture VMware Enterprise PKS builds on Kubernetes, BOSH, VMware NSX-T, and project Harbor to form a production-grade, highly-available container runtime that operates on vSphere and public clouds. With built-in intelligence and integration, VMware Enterprise PKS ties all these open source and commercial modules together, delivering a simple-to-use product for customers, ensuring the customers have the most efficient Kubernetes deployment and management experience possible. Figure 24 shows VMware Enterprise PKS architecture.
Figure 24. VMware Enterprise PKS architecture
For more information on the deployment of VMware Enterprise PKS, refer to https://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50000685enw.
Reference Architecture Page 38
Demonstrate integrated composability with VCF and HPE Synergy using HPE OneView connector for VCF VMware introduced composability service component starting from VCF 3.5 within the SDDC Manager. This composability service communicates with HPE OneView via the HPE OneView connector for VCF. This HPE OneView connector for VCF is developed and will be managed by Hewlett Packard Enterprise and is powered by DMTF Redfish APIs, which is designed to deliver simple and secure management for SDDC. Technically, it acts as a translator for VCF SDDC Manager and HPE OneView.
HPE OneView connector for VCF 0.3.3 HPE OneView connector for VCF 0.3.3 enables composability feature to provide a pure cloud experience in provisioning and deprovisioning of composable resources (compute, storage, and networking resources of HPE Synergy). In this version, the support is extended for external storages – HPE 3PAR StoreServ and HPE Nimble Storage operating on both FC and iSCSI storage protocols. The volume templates of these storage are attached to the Server Profile templates in HPE OneView. Based on the need, the user may choose the required Server Profile template while composing the composable resources from the SDDC Manager.
Figure 25: Workflow of HPE OneView connector for VCF
Reference Architecture Page 39
The connector is to be installed on a Linux instance (virtual or physical) out-of-band to VCF as a prerequisite within the management domain of VMware Cloud Foundation. Details about step-by-step installation, prerequisites and other recommended configurations for HPE OneView connector for VCF can be found at, http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001374enw.
Note If HPE D9340 (vSAN) storage module is part of the VCF workload domain deployment, compose the servers using HPE OneView connector through SDDC Manager as shown in this section.
Once the installation is complete, we need to register HPE OneView connector for VCF within SDDC Manager. To know more about registering HPE OneView connector for VCF with SDDC Manager and to compose servers on HPE Synergy using HPE OneView connector for VCF, reader may refer to the installation document at https://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001374enw.
Demonstrate patching and upgrading in VMware Cloud Foundation SDDC Manager internally has Lifecycle Management (LCM) enabled, which performs automated updates on VMware Cloud Foundation components such as SDDC Manager and its internal services and VMware components such as NSX for vSphere, vCenter Server, VMware ESXi™, Platform Services Controller (PSC), vRealize Suite, NSX-T, and VMware vRealize Suite Lifecycle Manager™. SDDC Manager is configured to communicate with the VMware software repository, if the SDDC Manager VM has internet access and the VMware depot credentials are valid. The high-level update workflow is as follows:
Receive notification of update availability.
Download the update bundle.
Select update targets and schedule the update.
However, the intent of this section is to help administrators to understand how to perform VMware Cloud Foundation upgrade when SDDC Manager does not have access to the internet.
Offline VMware Cloud Foundation update The intent of this section is to describe how to upgrade your VMware Cloud Foundation system if the SDDC Manager VM does not have internet access. The document goes in detail of how to use the Bundle Transfer utility to manually download the bundles from the VMware depot on your local computer with internet access and then upload them to an SDDC Manager VM, and update your VMware Cloud Foundation system.
When to perform offline update VMware Cloud Foundation update needs to be performed only after verifying if the underlying HPE Infrastructure including drivers and firmware are compatible with the version that is going to be installed. Refer to the HPE Synergy firmware and software compatibility matrix for VMware Cloud Foundation guide at https://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001407enw to check if the VCF version is listed as compatible along with the drivers and firmware.
Bundle types Upgrade bundle An upgrade bundle contains bits to update the appropriate Cloud Foundation software components in your management domain or VI workload domain. In most cases, an upgrade bundle must be applied to the management domain before it can be applied to workload domain.
Some upgrade bundles are cumulative bundles. With a cumulative upgrade bundle, you can directly upgrade the appropriate software in your workload domain to the version contained in the cumulative bundle rather than applying sequential upgrades to reach the target version. Cumulative bundles are available only for vCenter Server, Platform Services Controller, and ESXi.
Note You can apply a cumulative bundle to a workload domain only if the target release in the bundle is lower than or at the same version as the management domain. If the cumulative bundle is available for both the management domain and VI workload domains, you must apply it to the management domain before applying it to VI workload domains.
Reference Architecture Page 40
Install bundle VMware Cloud Foundation includes the following install bundles:
• VI workload domain install bundle is used to deploy later versions of the software components instead of the versions in your original Cloud Foundation installation. It includes software bits for vCenter Server and NSX for vSphere.
• Individual install bundles for vRealize products are used for deploying vRealize components.
• NSX-T install bundle is used for deploying an NSX-T based VI workload domain.
• Horizon 7 install bundle is used for creating a Horizon domain.
For more information on upgrading VMware Cloud Foundation (VCF) on HPE Synergy, refer to https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/vcf-39-upgrade/GUID-7992A016-D5E3-4A31-8337-A0611B355C41.html.
Demonstrate stretching a vSAN cluster of a VMware Cloud Foundation management domain VMware vSAN stretched clusters allow stretching of a single data site to two sites for a higher level of availability and inter-site load balancing. This Reference Architecture tells about protecting VMware Cloud Foundation management domain, a special purpose workload domain dedicated to management tasks, which is crucial to business. It contains the following management components:
• SDDC Manager
• vCenter Server and Platform Services Controllers
• vRealize Log Insight
• NSX Manager and controllers
VMware Cloud Foundation management domain is brought up on a single site with no protection or avoidance from disaster and a single point of failure could disrupt application availability. Thus, it becomes important to protect the management domain to ensure business continuity.
To address this issue, we could take advantage of the disaster avoidance and protection solution that vSAN offers in the form of stretched cluster. A vSAN stretched cluster is a specific configuration implemented when disaster/downtime has to be avoided in any situation where business continuity is a critical requirement. The vSAN stretched cluster extends the cluster from one data site to two sites for high availability and load balancing.
Use cases for implementing stretched cluster for VMware Cloud Foundation management domain are:
• Planned maintenance: Perform a planned maintenance on an availability zone without any downtime and then migrate the applications after the maintenance is completed.
• Automated recovery: Stretching a cluster automatically initiates VM restart and recovery and has a low recovery time objective for most unplanned failures.
• Disaster avoidance: With a stretched cluster, you can prevent service outages before an impending disaster.
Reference Architecture Page 41
VMware Cloud Foundation management domain is stretched across two sites (VCF Site1 and VCF Site2) within the region, so the network latency between two sites should be minimum. A third site (vSAN witness Site3) contains a witness host separate from the location of the other two data sites. Figure 35 shows architecture of VMware Cloud Foundation stretched management domain vSAN cluster across two sites in an Active/Active configuration.
Figure 35. Stretched VMware Cloud Foundation management domain vSAN cluster
For more information on stretching a vSAN cluster of a VMware Cloud Foundation management domain, refer to https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/com.vmware.vcf.admin.doc_39/GUID-7B4CC729-20BD-4CC9-B855-B38F02F74D40.html.
Demonstrate ease of monitoring and reporting of VCF infrastructure using HPE OneView for vRealize Operations HPE OneView for VMware vRealize Operations provides an integrated monitoring and reporting tool for VCF infrastructure.
When the HPE OneView for VMware vRealize Operations is installed, the custom HPE OneView dashboards are added to the vRealize Operations custom GUI. The HPE OneView dashboard allows you to proactively monitor HPE Synergy hardware resources and shows the object relationship with other objects in the environments. Proactive monitoring of the HPE Synergy hardware used for VMware Cloud Foundation helps improve productivity and facilitate efficient use of the resources resulting in minimized cost.
HPE OneView for vRealize Operations can be downloaded from Software Depot1, please download “HPE_OneView_for_VMware_vRealize_Ops_2.3_July_2019_Z7550-02544.zip” file and extract the “HPEOneViewAdapter_2.3.0.21_signed.pak”. The details about the installation and configurations can be found at, https://support.hpe.com/hpsc/doc/public/display?docId=a00077762en_us.
1 HPE OneView for vRealize Operations: https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=Z7500-63235
Reference Architecture Page 42
Following are the dashboards available for use:
• HPE OneView Infrastructure dashboard
• HPE OneView networking dashboard
• HPE OneView Servers overview dashboard
• HPE OneView Enclosure overview dashboard
• HPE OneView Uplink Port overview dashboard
The following sections show examples of four among above five dashboards.
HPE OneView Infrastructure dashboard The HPE OneView Infrastructure dashboard provides an overview of the HPE Synergy Infrastructure managed by HPE OneView. It displays the status of HPE OneView managed hardware and allows you to see how the hardware relates to your virtual environment. Figure 36 shows an example of HPE OneView Infrastructure dashboard. It displays the physical and virtual resources that are part of the VCF domain and interrelation between each object in the environments. Administrator gets a quick summary of the environment and relations between the objects. It also shows metric charts demonstrating the performance and usage of the resources.
Figure 36. HPE OneView Infrastructure dashboard
Reference Architecture Page 43
HPE OneView Networking dashboard The HPE OneView Networking dashboard provides an overview of the HPE OneView networking along with its connection to the virtual environment. Selecting an object in the environment overview allows you to see how this object is related to the other objects and generates a graph for each metric collected. Figure 37 shows an example of HPE OneView Networking dashboard with VCF-Node1 server selected and its connections.
Figure 37. HPE OneView Networking dashboard
Reference Architecture Page 44
HPE OneView Servers Overview dashboard The HPE OneView Servers Overview dashboard provides a summary and metrics information pertaining to CPU utilization, temperature, and power utilization of physical servers in the VCF environment. The heatmaps allow you to quickly compare these metrics.
Figure 38. HPE OneView Servers Overview dashboard
Reference Architecture Page 45
HPE OneView Enclosures Overview dashboard The HPE OneView Enclosures Overview dashboard provides summary and metrics information pertaining to the temperature and power utilization of the HPE Synergy Enclosures. The Heatmaps allow you to quickly compare these metrics. Under each Heatmap, there is a comparative representation between the two enclosures. Selecting an enclosure from this list generates a sparkline chart displaying the metric history.
Figure 39 shows HPE OneView Enclosures Overview dashboard with the two enclosures as used in this Reference Architecture solution.
Figure 39. HPE OneView Enclosure Overview dashboard
Reference Architecture Page 46
Demonstrate HPE Synergy firmware upgrade using HPE OneView for VMware vRealize Orchestrator HPE OneView for VMware vRealize Orchestrator helps customers automate complex IT tasks in an extensible and repeatable manner. It provides a predefined collection of HPE tasks and workflows that can be used in vRealize Orchestrator (vRO). The plugin can be added to vRealize Orchestrator seamlessly and various IT tasks can be performed using the workflows available.
Among the available workflows, administrator can use the “Update Cluster Firmware“ workflow to update the HPE Synergy Compute Firmware through HPE OneView and HPE Synergy Service Pack for ProLiant for Day 2 operations.
Following are the steps needed to be performed to update compute firmware using the “Update Cluster Firmware” workflow.
Install HPE iSUT on each of the HPE Synergy Nodes.
Configure the VMware vRealize Orchestrator (vRO).
Install and configure the HPE OneView for vRO plugin.
Configure the vRO Clients for the workflows.
HPE OneView for vRealize Orchestrator can be downloaded from the Software Depot1, please download the “HPE_OneView_for_VMware_vRealize_Orchestration_1.3_October_2019_Z7550-02585.zip” file and extract the “o11npluginhpeov4vro-1.3.0.79.vmoapp” to add that as plug-in with VMware vRealize Orchestrator. Refer to http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50000681enw for more detailed information.
We tested the “Update Cluster Firmware” workflow in the Hewlett Packard Enterprise Engineering lab to update the compute resources firmware inside the HPE Synergy Frame with VMware Cloud Foundation installed. The following sections provides details about the test environment and the process.
Prerequisites • VMware vRealize Orchestrator is installed in the environment.
• VMware vRealize Orchestrator vSphere vCenter plug-in is configured with VMware vCenter(s).
• At least one instance of HPE OneView is installed and configured.
• An HPE OneView account is created and to be used by HPE OneView Management Instance in VMware vRealize Orchestrator.
• The VMware vRealize Orchestrator servers have network access to HPE OneView.
• VMware Cloud Foundation software is installed and configured on HPE Synergy.
• HPE OneView Server Profile template is configured for baseline Service Pack for ProLiant (SPP).
• The vRO workflows will be used to update new SPPs on the HPE Synergy Compute Nodes as available and configure them as baselines for Server Profile template.
1 HPE OneView for vRealize Orchestrator, https://h20392.www2.hpe.com/portal/swdepot/displayProductInfo.do?productNumber=Z7500-63235.
Reference Architecture Page 47
Firmware and software versions Table 15 shows the firmware and software used in the testing process.
Table 15. Firmware and software versions
Name Versions
VMware vRealize Orchestrator 7.6.0.317 Build Number -13020602
HPE OneView for VMware vRealize Orchestrator(OV4VRO) 1.3
HPE Synergy Composer 5.00.02
iSUT sut-esxi6.7-offline-bundle-2.4.5.0-16
VMware Cloud Foundation 3.9.1
vCenter 6.7 U3
Testing the workflow In the Hewlett Packard Enterprise Engineering lab environment, the following two HPE Synergy Service Packs are downloaded, as shown in Table 16.
Table 16: Firmware used in testing
Name Version Size
HPE Synergy Custom SPP 201912 2020 01 24 2020.01.24.00 4.28 GB
HPE Synergy Custom SPP 201912 2020 03 26 2020.03.26.00 4.29 GB
In the server profile template, the Firmware baseline was selected as " HPE Synergy Custom SPP 201912 2020 01 24” as shown in Figure 40.
Figure 40. Firmware baseline in HPE OneView
Reference Architecture Page 48
The firmware of the HPE Synergy Compute Nodes was updated to firmware version “HPE Synergy Custom SPP 201912 2020 03 26” using the workflow “Update cluster Firmware” present in OV4VRO workflows. The following information was necessary to start the “Update cluster Firmware” workflows:
• vCenter Cluster – The vCenter Cluster on which the HPE Synergy host firmware would be upgraded as shown in Figure 41.
Figure 41. vCenter Cluster selection
Reference Architecture Page 49
• HPE OneView Firmware Bundle –The version to which the firmware update would happen as shown in Figure 42.
Figure 42. HPE OneView Firmware Bundle selection
Reference Architecture Page 50
Once the required details are updated, the “Update cluster Firmware” workflow is ready to start as shown in Figure 43.
Figure 43. Start workflow
In HPE OneView, the Server Profile status for each compute resource shows the activity details for the firmware update process as shown in Figure 44.
Figure 44. OneView Screen showing new firmware baseline being applied
Reference Architecture Page 51
Once the Firmware update is complete, it shows the new firmware version “HPE Synergy Custom SPP 201912 2020 03 26” as the firmware baseline as shown in Figure 45. The firmware update state shows as Applied along with the time when it was installed.
Figure 45. HPE OneView showing the new firmware applied
Note Certain firmware components need manual reboot for the complete installation of the firmware. In such scenario, a manual reboot of the host is recommended.
Reference Architecture Page 52
Consolidated architecture deployment of VMware Cloud Foundation VMware Cloud Foundation can be deployed as either a standard architecture model or consolidated architecture model. In the standard architecture model, there is a dedicated management domain that hosts infrastructure virtual machines and at least one workload domain to host user workloads. It also requires a minimum of seven (7) servers, four (4) for management domain and three (3) for workload domain. In the consolidated architecture model, the management and customer workload VMs are part of the same domain or cluster and it leverages vSphere resource pools to provide isolation between management and user workloads.
The consolidated architecture model targets smaller Cloud Foundation deployments, special use cases, and can be deployed starting with four (4) hosts or higher. Management and user workload VMs run together in the same vSphere cluster and the environment is managed from a single vCenter. vSphere resource pools provide isolation between the management and user workloads. Consolidated architecture model does not support NSX-T or automated deployment of Horizon and Enterprise PKS.
Figure 46 shows the VMware Cloud Foundation consolidated architecture deployed on HPE Synergy Frame hosting four to six compute servers.
Figure 46. VMware Cloud Foundation consolidated architecture deployed on HPE Synergy Frame hosting 4 – 6 compute servers
Reference Architecture Page 53
Note VCF consolidated deployment can support up to 64 hosts (to the limits of vCenter). But this solution is tested with four to six hosts. HPE automation also enables four to six hosts for consolidated architecture (targeting Proof-of-Concept scenario and use cases for ROBO, SMB deployments). However, this is not the limitation by VCF SDDC for consolidated deployment.
For more information on deploying VMware Cloud Foundation consolidated architecture on HPE Synergy, see http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a50001373enw.
Summary Hewlett Packard Enterprise and VMware can deliver a software-defined solution running on modular infrastructure across compute, storage, network, security, and cloud management. This Reference Architecture demonstrates a secured and scalable private cloud solutions built using VMware Cloud Foundation on HPE Synergy. It showcases the ability to:
• Integrate composability with VCF and HPE Synergy using HPE OneView connector for VCF.
• Simplify deployment of VI workload domain and operations on VMware Cloud Foundation.
• Simplify firmware updates using HPE OneView for VMware vRealize Orchestrator.
• Build a VCF workload domain with a choice of either HPE 3PAR, HPE Nimble, or HPE Primera for SDDC managed principal storage or out of band managed ancillary storage as secondary choice of storage.
HPE Synergy offers a unique design for running VMware private clouds, and for providing the right IT platform that matches VMware Cloud Foundation characteristics—automated, software-driven, and flexible. HPE Synergy is the only modular infrastructure to run VMware Cloud Foundation deployments, and it provides a foundation for supporting hybrid configurations.
The benefits of using VMware Cloud Foundation on HPE Synergy include:
• Reduced infrastructure complexity and cost.
• Elimination of top-of-rack switching and deployment.
• Availability of rack-scale fabric with HPE Virtual Connect.
• Efficiencies in scaling fabrics across multiple frames.
• Provision and management of the physical fluid resources for SDDC deployments through HPE OneView.
Reference Architecture Page 54
Appendix A: Bill of materials
Note Part numbers are at time of publication and subject to change. The bill of materials does not include complete support options or complete rack and power requirements. For questions regarding ordering, consult with your HPE Reseller or HPE Sales Representative for more details. hpe.com/us/en/services/consulting.html.
Table 17. Bill of materials
Product Qty Product description
Rack and power
P9K10A 1 HPE 42U 600mmx1200mm G2 Enterprise Shock Rack
P9K40A 001 1 HP Factory Express Base Racking Service
H6J85A 1 HPE Rack Hardware Kit
120672-B21 1 HPE Rack Ballast Kit
BW932A 1 HPE 600mm Rack Stabilizer Kit
BW932A B01 1 HPE 600mm Rack include with Complete System Stabilizer Kit
P9S21A 2 HPE G2 Metered/Switched 3Ph 14.4kVA/CS8365C 40A/208V Outlets (12) C13 (12) C19/Vertical NA/JP PDU
HPE Synergy Frames
P06011-B21 2 HPE Synergy 12000 Configure-to-order Frame with 1x Frame Link Module 10x Fans
867796-B21 2 HPE Virtual Connect SE 100Gb F32 Module for Synergy
794502-B23 2 HPE Virtual Connect SE 40Gb F8 Module for Synergy
867793-B21 2 HPE Synergy 50Gb Interconnect Link Module
779218-B21 2 HPE Synergy 20Gb Interconnect Link Module
798096-B21 2 HPE 6x 2650W Performance Hot Plug Titanium Plus FIO Power Supply Kit
872957-B21 2 HPE Synergy Composer 2
876852-B21 2 HPE Synergy 4-port Frame Link Module
804943-B21 2 HPE Synergy Frame 4x Lift Handles
859493-B21 1 HPE Synergy Multi Frame Master1 FIO
804101-B21 4 HPE Synergy Interconnect Link 3m Active Optical Cable
876259-B21 4 HPE Virtual Connect SE 32Gb Fibre Channel Module for Synergy
P08477-B21 4 HPE Virtual Connect SE 16Gb Fibre Channel Module for Synergy
755985-B21 4 HPE Synergy 12Gb SAS Connection Module with 12 Internal Ports
871940-B21 16 HPE Synergy 480 Gen10 Configure-to-order Compute Module
P07351-B21 16 HPE Synergy 480/660 Gen10 Intel Xeon-Gold 6254 (3.1GHz/18-core/200W) Processor Kit
P07351-L21 16 HPE Synergy 480/660 Gen10 Intel Xeon-Gold 6254 (3.1GHz/18-core/200W) FIO Processor Kit
P00920-B21 384 HPE 16GB (1x16GB) Single Rank x4 DDR4-2933 CAS-21-21-21 Registered Smart Memory Kit
P01367-B21 16 HPE 96W Smart Storage Battery (up to 20 Devices) with 260mm Cable Kit
804424-B21 16 HPE Smart Array P204i-c SR Gen10 (4 Internal Lanes/1GB Cache) 12G SAS Modular Controller
804428-B21 16 HPE Smart Array P416ie-m SR Gen10 (8 Int 8 Ext Lanes/2GB Cache) 12G SAS Mezzanine Controller
Reference Architecture Page 55
Product Qty Product description
P02054-B21 16 HPE Synergy 6820C 25/50Gb Converged Network Adapter
777430-B21 16 HPE Synergy 3820C 10/20Gb Converged Network Adapter
777456-B21 12 HPE Synergy 5830C 32Gb Fibre Channel Host Bus Adapter
777452-B21 12 HPE Synergy 3830C 16Gb Fibre Channel Host Bus Adapter
872475-B21 32 HPE 300GB SAS 12G Enterprise 10K SFF (2.5in) SC 3yr Wty Digitally Signed Firmware HDD
845406-B21 10 HPE 100Gb QSFP28 to QSFP28 3m Direct Attach Copper Cable
721064-B21 4 HPE BladeSystem c-Class 40G QSFP+ to 4x10G SFP+ 3m Direct Attach Copper Splitter Cable
861413-B21 4 HPE Synergy Frame Link Module CAT6A 3m Cable
Storage
835386-B21 2 HPE Synergy D3940 12Gb SAS CTO Drive Enclosure with 40 SFF (2.5in) Drive Bays
P09098-B21 12 HPE 400GB SAS 12G Write Intensive SFF (2.5in) SC 3yr Wty Digitally Signed Firmware SSD
P09092-B21 48 HPE 1.6TB SAS 12G Mixed Use SFF (2.5in) SC 3yr Wty Digitally Signed Firmware SSD
757323-B21 2 HPE Synergy D3940 Redundant I/O Adapter
H6Z00A 2 HPE 3PAR StoreServ 8000 4-port 16Gb Fibre Channel Adapter
H6Z07B 1 HPE 3PAR 8440 2N+SW Storage Field Base
K2P90B 8 HPE 3PAR 8000 920GB+SW SFF FE SSD
K2P94B 28 HPE 3PAR 8000 1.8TB+SW 10K SFF HDD
E7Y71A 2 HPE 3PAR StoreServ 8000 SFF(2.5in) Field Integrated SAS Drive Enclosure
Q2S13A 1 HPE 3PAR StoreServ RPS Service Processor
Q8H72A 1 HPE Nimble Storage HF20 Adaptive Dual Controller 10GBASE-T 2-port Configure-to-order Base Array
Q8B69B 1 HPE Nimble Storage HF20/20C Adaptive Array 42TB (21x2TB) FIO HDD Bundle
Q8G27B 1 HPE Nimble Storage NOS Default FIO Software
Q8J30A 1 HPE Nimble Storage HF20 Adaptive Array R2 5.76TB (6x960GB) FIO Cache Bundle
R0N85A 1 HPE Nimble Storage 2x10GBASE-T 2-port and 2x16Gb Fibre Channel 4-port FIO Adapter Kit
R0P84A 2 HPE Nimble Storage IEC 60320 C14 to C19 250V 15Amp 1.8m FIO Power Cord
R3P91A 1 HPE NS AF/HF OEM Trk
N9Z46A 1 HPE Primera 600 2-way Storage Base
N9Z55A 1 HPE Primera A630 2-node Controller
581817-B21 1 HPE Configurator Defined Build Instruction Option
R1P32A 1 HPE Primera 600 without Installation Service
R3R39A 6 HPE Primera 600 1.92TB SAS SFF (2.5in) FIPS Encrypted SSD
N9Z39A 2 HPE Primera 600 32Gb 4-port Fibre Channel Host Bus Adapter
716195-B21 2 HPE External 1.0m (3ft) Mini-SAS HD 4x to Mini-SAS HD 4x Cable
N9Z50A 1 HPE Primera 600 2U 24-disk SFF Drive Enclosure
R3R39A 6 HPE Primera 600 1.92TB SAS SFF (2.5in) FIPS Encrypted SSD
Reference Architecture
© Copyright 2020 Hewlett Packard Enterprise Development LP. The information contained herein is subject to change without notice. The only warranties for Hewlett Packard Enterprise products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. Hewlett Packard Enterprise shall not be liable for technical or editorial errors or omissions contained herein.
Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. Microsoft, Windows, Windows Server and SQL Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. VMware, vSphere, VMware Cloud Foundation, VMware vRealize Automation, VMware NSX, vRealize Orchestrator, vRealize Operations Manager, vRealize Suite Lifecycle Manager and vRealize Log insight are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. VMware Horizon is trademark of VMware, Inc. in the United States and/or other jurisdictions. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. All other third-party trademark(s) is/are the property of their respective owner(s).
a50001399enw, April 2020
Resources and additional links VMware Cloud Foundation 3.9.1, https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9.1/rn/VMware-Cloud-Foundation-391-Release-Notes.html
VMware Cloud Foundation Planning and Preparation Guide, https://docs.vmware.com/en/VMware-Cloud-Foundation/3.9/vcf-39-planprep-guide.pdf
HPE Reference Architecture, hpe.com/info/ra
HPE Synergy, hpe.com/info/synergy
HPE and VMware, hpe.com/partners/vmware
HPE Networking, hpe.com/networking
HPE Servers, hpe.com/servers
HPE Enterprise Information Library, hpe.com/info/convergedinfrastructure
HPE Technology Consulting Services, hpe.com/us/en/services/consulting.html
HPE OneView for VMware vCenter with operation Manager and Log Insight, hpe.com/hpeoneviewforvcenter
To help us improve our documents, please provide feedback at hpe.com/contact/feedback.