“cisco it: scalable enterprise ucsd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkcoc-2009.pdf ·...

“Cisco IT: Scalable Enterprise UCS with ACI – “Any workload anywhere.”

Hugh Flanagan, Senior IT Engineer

Jason Stevens, IT Engineer

BRKCOC-2009

• Introduction

• Challenges of Large Scale UCS Deployments in the Datacenter

• How ACI and UCS Central Allow us to Enhance UCS Infrastructure

• Operational Gains

• Conclusion

Agenda

Introduction

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5BRKCOC-2009

Impact of ACI on Cisco Datacenters

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Hybrid CloudApplication

Centric Cloud

Private Cloud

(Iaas + PaaS)

Home Grown

Automation

Cisco IT’s Internal Cloud Journey

Virtualization

Reduce Cost Deliver Services Faster

UCS

ESX

VM VM VM

VMware Virtualization

• Cisco Intelligent Automation for

Cloud (CIAC)

• Virtual Security Gateway

• Prime Network Services Controller

APIC

Application Centric

Infrastructure

Cisco

Intercloud

Open Source

We Are

HERE

BRKCOC-2009 6

Challenges of Large Scale UCS Deployments in the Datacenter


Cisco IT’s Legacy Design

• Up to 10 Chassis per UCS Domain

• Fault / Migration Domain Limited to Single UCS Domain

• Hardware Maintenance Set Aside = 20%


Physical Connectivity


Datacenter Physical Layout

BRKCOC-2009 10


Legacy UCS Build Work Flow


Legacy UCS Build Work Flow – Continued


Domain Upgrade Maintenance Summary

• Cisco IT has 450 UCS Domains

• Upgrades Complicated by:

• Change Freeze Windows

• Maintenance Windows

• Critical Client Workloads

• This Results in UCS Upgrades Taking up to 2 Years to Complete

13BRKCOC-2009

How ACI and UCS Central Allow us to Enhance UCS Infrastructure


End to End Programmable Infrastructure

Block Storage

Compute

IP File/ IP Block/ IP Object

Storage

Controllers

Resource ManagersOrchestration

(Cloud)

CIAC

ASA

Client

Security

Admin

Compute

Network/Storage

Admins

PaaS

Resources

eACLm

NetworkIntegrated

Security

Application Code

Portal

BRKCOC-2009 15


Fabric InfrastructureRequirements• Flexible Topology

• Multi-Tenant

• Virtual Boundaries

• Physical and Virtual Services

• Easier to Manage

Components

• ACI: 9508, 9396PX, APIC

• UCS: B420m3, B200m4, B200m3

• ESXi + AVS

• Openstack + KVM + OVS

• NetApp CDoT (8.3)

• Citrix SLB

• ASA

• CIAC (eStore & automation)

BRKCOC-2009 16


Next Generation Virtual Compute Design

Legacy Virtual Compute Design

N1KV Domain-1

Physical Leaf pair-1 Physical Leaf pair-2 Physical Leaf pair-3 Physical Leaf pair-4

Logical Pod-2

Logical Pod-3

Logical Pod-4

Logical Pod-1

vCenter Domain-1

vCenter Domain-2

`

vCenter Domain-3

vCenter Domain-4

N1KV Domain-2

N1KV Domain-3

N1KV Domain-4

Virtual Compute Design with ACI

vMotion Across VMM DomainvMotion vMotion vMotion vMotion


Striped Cluster ArchitectureSpine Switches

Leaf Switches

UCS FI

Hypervisor Stripe

Bare Metal Stripe


Challenges with Next Generation Design

• Increased Capacity

• 6.6 Terahertz of CPU Capacity (not including hyper-threading)

• 60 Terabytes of Memory

• Per UCS Domain (80 Blade UCS Domain)

• VLANs are not Able to Support Number of Possible Tenants

• Keeping UCS Domains’ Configuration in Sync

19BRKCOC-2009


Cisco UCS Central

BRKCOC-2009 20


UCS Central Architecture

BRKCOC-2009 21

Web Client

PostgreSQL DB

DMEServices

Framework

UCS Central Virtual Appliance

Hardened Guest OS

UCS Manager #1…

UCS Manager #n

XML API SNMP


Service Profile Overview

RTP ALLN


UCS Central Org. StructureService Profiles UCS Domains


UCS Central – Domain Stripe


VXLAN

• Extends Layer 2 networks across Layer 3 infrastructure

• MAC-to-UDP encapsulation

• VXLAN Tunnel Endpoint (VTEP)

• 50-byte overhead to original frame

• Can support up to 16 million segments

• Cisco IT is using AVS (Application Virtual Switch) for VXLAN connectivity

• VXLAN traffic uses the Infrastructure VLAN configured during initial fabric turn-up

25BRKCOC-2009


VXLAN Packet Format


UCS Central – Ethernet Adapter Policy


Client VMM (VxLAN)


Management/Network Services VMM (VLAN)


OpFlex: An Open Source Framework

OpFlex changes have been upstreamed as part of OpenDaylight.

OpFlex library and MODB are reusable across any device

Rendering agent must be rewritten to specific device

OpFlex

Agent

Open

vSwitch

Datapath

Flow Table

OpenFlow

OVSDB

Managed Objects Store

(MODB)

OVS Render Plugin

(Policy Enforcement)

SW/HW Datapath

OpFlex (Policy Manager)

Open vSwitch changes are all upstream and included in OVS 2.4

Datapath and Linux kernel (3.20) include support for VXLAN-GBP

BRKCOC-2009 30


OpFlex Hypervisor Switch Integration

OpFlex Proxy runs as part of ACI leaf managing interaction with APIC

OpFlex Agent runs on hypervisor in userspace and will be packaged as a separate component

It speaks to OVS through OpenFlow / OSDB interfaces

Available as part of FCS+9 (BMR2 release of ACI)

CISCO AVSMICROSOFT

HYPER-V

OPEN

VSWITCH

OpFlex

Proxy

OpFlex

Proxy

OpFlex

Proxy

OpFlex

Proxy

OpFlex

Proxy

OpFlex

Proxy

THIRD

PARTY

VSWITCH

OPEN

VSWITCHCISCO AVS

OpFlex

Agent

OpFlex

Agent

OpFlex

Agent

OpFlex

Agent

OpFlex

Agent

OpFlex

Agent

BRKCOC-2009 31


Fabric Design

DMZ

Core

Internal

Core

Clustered Data on Tap

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

ESXi

Openstack

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Cisco Nexus 2148T

1GE Fabric Extender

STAT

ID 45 46 47 48 1 2 3 4

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Cisco Nexus 2148T

1GE Fabric Extender

STAT

ID 45 46 47 48 1 2 3 4

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Cisco Nexus 2148T

1GE Fabric Extender

STAT

ID 45 46 47 48 1 2 3 4

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

01

05

10

15

20

25

30

35

40

02

03

04

06

07

08

09

11

12

13

14

16

17

18

19

21

22

23

24

26

27

28

29

31

32

33

34

36

37

38

39

41

42

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Cisco Nexus 2148T

1GE Fabric Extender

STAT

ID 45 46 47 48 1 2 3 4

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

USCS C260 M2

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYSPWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS PWR

SYS

PWR

SYS

PWR

SYS

PWR

SYS

Border Storage

Virtual Compute

Bare Metal

Operational Gains


Hardware Specifications

• Legacy Deployment - B200 M2

• X5680 CPU x2 (6 Cores per CPU)

• 192GB RAM

• VIC 1280 CNA

• Modern Deployment - B200 M4

• E5-2699 v3 CPU x2 (18 Cores per CPU)

• 768GB RAM

• VIC 1380 CNA (Supports VXLAN Offload)

34BRKCOC-2009


Blade Consolidation

• Legacy DC Environment has 2,500 Hypervisor Blades

• 1,500 of which are B200 M2

• ACI, Leveraging B200 M4, Consolidates this to 600 Blades

• 4:1 Consolidation on Average, Achieved Through Higher Density Blades and Larger Migration Domains

• 15:1 Consolidation Seen when Consolidating B200 M1 Environments

35BRKCOC-2009


Domain Upgrade Maintenance

• Time Expended for Legacy

• 2 hours Determining Maintenance Windows

• 4 hours negotiating with clients

• 30 minutes running health checks and verifications

• 1 hour fixing issues to ensure no downtime

• 1.5 hours executing upgrade

• Time Expended for ACI

• 20 minutes evacuating

• 1.5 hours executing upgrade


Datacenter Physical Layout Revisited

BRKCOC-2009 37


• Each UCS Domain is built with a minimal amount of chassis and blades for Day 1 workloads

• Chassis additions are performed when additional capacity is needed for quick turnaround

• Automation built leveraging CIAC, UCS Central, and APICs configures the blades for use

Capacity Additions Under ACI

BRKCOC-2009 38

UCS Domain Evacuation Demo


APIC Cluster

Stretched Fabric Use Case – Single DC Facility

41

Data Hall 1

Transit Leaf Switches

Data Hall 2

APIC APIC APIC

BRKCOC-2009

Best Practices / Lessons Learned


Hitless Upgrades and Failover Testing• All failovers and upgrades: max 2-3 sec connectivity loss

ACI Switch

Application Virtual Switch

UCS Compute

(B-Series)

ESXi/Vmware

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

multiple IP Storage Leaf PairsBorder Leafs

Leaf Pairs

Multiple Compute & Services Leaf Pairs

NAS

DMZ Internal

Services

(physical or virtual)

IP Network

ACI Fabric

Services

(physical or virtual)Compute Compute

Internet

Cisco+Co

rp

APIC APICAPIC

vCenter

NAS


Best Practices and Lessons Learned

• Test/Certify new Features and Code Prior to Production Deployment

• Utilize lab environment for testing

• Create certification process with standard must have capabilities and verification

• Document/track issues found

• Start off with basic functionality and add new features as you go

• Build with Automation in Mind

• Create standard and reusable constructs

• Document naming conventions for various objects to make readability and troubleshooting easier

• Scripting skills will help you on your journey

44BRKCOC-2009



• AVS for vSwitch

• Use AVS and VXLAN where possible

• L4-L7 services required to be on VLAN VMM

• Reduced configuration overhead (No need to trunk down huge amounts of VLANs)

• Better management and visibility when troubleshooting connectivity issues

45BRKCOC-2009



• Upgrades

• Test upgrade process in lab prior to production rollout and check release notes for any important changes

• Maintenance Groups

• Border Leaf Even / Border Leaf Odd

• Separate Compute Leaf Pairs as Capacity Allows (Still into Evens / Odds)

• Config backups

• Schedule secure configuration backups/archives on daily basis

46BRKCOC-2009

Conclusion


Fabric

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

Pod 1 Pod 2 Pod 3 Pod 4

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

41

36

31

26

21

16

11

6

1

42

40

39

38

37

35

34

33

32

30

29

28

27

25

24

23

22

20

19

18

17

15

14

13

12

10

9

8

7

5

4

3

2

Resource Stripe - ACI

Greater Resiliency

Easier Maintenance

Resource pools and applications striped across multiple pods

BRKCOC-2009 48


Flexible AssetsStorage or Compute

Unit 1 Unit 2 Unit 3 Vertical Lock-inInitial placement of applications results in storage and compute

resources physically locked in

Reserve (future & failover) capacity can’t be flexibly used

Stranded Storage and Compute Capacity

Horizontal ScalingApplications can flexibly tap into any available compute or

storage resources

Initially: ACI + VMware + NetApp CDOT

Policy Based Control

Infrastructure Maintenance

Unit 1 Unit 2 Unit 3

ACI

Today

ACI

BRKCOC-2009 49


Any Application, Anywhere

• ACI allows any EPG to be leveraged on any compute resource within fabric

• Using VXLAN allows any compute resource to be immediately capable of running any VM with no further network configuration

• Stretched Fabric allows for logically combining multiple datacenter halls into a single fabric.

• UCS Central allows massively scaling compute capacity while keeping policies and configuration consistent throughout

50BRKCOC-2009


Additional Reading

• Cisco IT ACI Storage Deployment http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Storage_Deployment.html

• Cisco IT ACI Design http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Design.html

• Application Virtual Switch http://www.cisco.com/c/en/us/products/collateral/switches/application-virtual-switch/datasheet-c78-733605.html

• AVS vs DVS Convergence http://www.cisco.com/c/en/us/products/collateral/switches/application-virtual-switch/white-paper-c11-736554.html

51BRKCOC-2009

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Storage_Deployment.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/white_papers/Cisco_IT_ACI_Design.html

http://www.cisco.com/c/en/us/products/collateral/switches/application-virtual-switch/datasheet-c78-733605.html

http://www.cisco.com/c/en/us/products/collateral/switches/application-virtual-switch/white-paper-c11-736554.html


Additional Reading…

• Cisco APIC Controller with Openstack Integration Demo

• https://dcloud-rtp-web-1.cisco.com/dCloud/demoInfo.jsp?d=101570

• UCS Central Demo

• https://dcloud-cms.cisco.com/demo/cisco-ucs-central-1-4-v1

• VXLAN Whitepaper

• http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.html

• Design Guide for L4-L7 Services on AVS

• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/avs/reference-guide/L4-L7-services/Cisco_AVS_L4-L7_White_Paper.html

52BRKCOC-2009

https://dcloud-rtp-web-1.cisco.com/dCloud/demoInfo.jsp?d=101570

https://dcloud-cms.cisco.com/demo/cisco-ucs-central-1-4-v1

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/avs/reference-guide/L4-L7-services/Cisco_AVS_L4-L7_White_Paper.html


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKCOC-2009 53

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKCOC-2009 54

Please join us for the Service Provider Innovation Talk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

11:30 am - 12:30 pm, In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com

Thank you

“cisco it: scalable enterprise ucsd2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkcoc-2009.pdf ·...

Documents