no it left behind - connecting the software-defined data center to multi-modal it

NO IT LEFT BEHIND

Connecting the software-defined data center to

multi-modal IT

1

• Setting the Stage

• BiModal Vs. MultiModal

• Our Approach

• Summary and Lessons Learned

AGENDA

2

18 Years in Business

SETTING THE STAGE

3

Focused on Developing Solutions for the DoD, DHS and other Federal Agencies

• Wide Range of Compliance Challenges

SETTING THE STAGE

4

Diverse Staff

• 300+ Software Engineers

• 150+ Field Personnel

• 100+ Active Projects at any time

SETTING THE STAGE

5

150+ Corporate Systems

• 3rd Party and Homegrown!

SETTING THE STAGE

6

Development• Research and Development• Enhancement• Bug fixes

Life Cycle• Configuration Mgmt• Release Mgmt• Packaging

Infrastructure• Test & Integration• Documentation• Help Desk

Customers• Maintenance• Training• Pro Services

SETTING THE STAGE

7

LANDSCAPE

8

LANDSCAPE

9

LANDSCAPE

10

APPLICATIONS• Unanet, JIRA, SharePoint, Finance, HR…

FABRIC• VMWare, NSX, Laas, Paas, ...

NETWORK• Routers, Phones, VTC, …

COMPUTE• UCS, Towers, Laptops, ...

STORAGE• NetApp, EMC, OneDrive, …

LANDSCAPE

11

12

We have built infrastructure

PROGRESS

13

But building infrastructure is

not the goal.

PROGRESS

• We need to be more Agile to keep up with rapidly changing business needs

• Virtualization Alone Not Sufficient!

• Tenets

NEED FOR CHANGENO MORE ISLANDS

14

Traditional IT focused on Stability and Efficiency

BIMODAL OPTION

15

BIMODAL OPTION

DevOps IT focused on new tools and agile processes

16

PROS: • Reduced risk for ongoing core business activities

• Frees forward looking teams to not worry about the day to day

BIMODAL OPTION

17

PROS: • Reduced risk for ongoing core business activities

• Frees forward looking teams to not worry about the day to day

• Ignores the interdependence between past and future

• Near term increase in resource needs

• Significant cultural impact

• ”haves and have nots”

• Latent talent ignored

BIMODAL OPTION

CONS:

18

• Systems evolve at a natural pace

• Some applications will die off

MULTI-MODAL OPTION

Migrate Everything

19

• Responsiveness and reliability are the same thing!

MULTI-MODAL OPTION

Treat All Systems as Agile

20

• Each piece of the enterprise is at a different place on the continuum and will move forward at a different speed (crawl/walk/run)

• Dynamic on-premise environment that supports flexible, timely and relevant capabilities while adhering to a wide range of compliance challenges

MULTI-MODAL OPTION

DevOps approach spanning the entire enterprise

21

22

HOW ARE WE DOING THIS?

23

APPROACH

Collaborate and Communicate

Fail Fast, Recover Fast and learn

Always Keep Learning

Technologies Small Batch Size

Dsmorse_iss

24

• Ad hoc is bad hoc

• Solution “discovered” on production

• No one person has whole story

• Configuration drift

• Non existent Documentation

• Any manual task can introduce human error

• We shouldn’t have to call you on your on day off

• Goals:

• Repeatable, Repeatable, Repeatable, …

• Known state

• DRY (yeah I used to be a dev)

WHY DEVOPS

Dsmorse_iss

25

• Ad hoc is bad hoc

• Solution “discovered” on production

• No one person has whole story

• Configuration drift

• Non existent Documentation

• Any manual task can introduce human error

• We shouldn’t have to call you on your on day off

• Goals:

• Repeatable, Repeatable, Repeatable, …

• Known state

• DRY (yeah I used to be a dev)

WHY DEVOPS

Dsmorse_iss

28

Frequent small changes are less risky than a Large release

SMALL BATCH SIZE

Dsmorse_iss

29

SMALL BATCH SIZE

Fail Fast: Tracking MTTR

• Small Batch

• Easy to ID change

• Disciplined about ConfigManagement

• Known state before & after

• Automated scripts for backup & deploy

• Scripts also verify expected stat

Anti-fragile thinking to allow us to upgrade live systems during regular hours

Dsmorse_iss

30

• Desired state configuration

ANSIBLE-WHAT GOES WHERE

Dsmorse_iss

31


• It is just a YML file


Dsmorse_iss

32



• Agentless! Uses SSH


Dsmorse_iss

33



• Agentless! Uses SSH

• Idempotent

Idempo-What: “Operations in mathematics and computer science, that can be applied multiple times without changing the result beyond the initial application.” – wikipedia


Dsmorse_iss

34

• Docker containers increase portability

DOCKER, DOCKERDOCKER

Dsmorse_iss

35


• Increase density & VM utilization


Dsmorse_iss

36



• Easier Deployment


Dsmorse_iss

37



• Easier Deployment

• Immutable images removes complexity, changes the way we take care of our apps


Dsmorse_iss

38

SYSTEM DIAGRAM

Dsmorse_iss

39

SYSTEM DIAGRAM

Dsmorse_iss

40

SYSTEM DIAGRAM

Dsmorse_iss

41

A critical part of DevOps is looking at change metrics

• Did we achieve the desired outcome?

• Should we pivot or persevere?

• What problems do we not know we have

COLLABORATE AND COMMUNICATE

Dsmorse_iss

42





Closed systems have been replaced by Open JIRA projects

• All changes are public!


Dsmorse_iss

43





Closed systems have been replaced by Open JIRA projects

• All changes are public!

HipChat was brought in to allow automated flow of relevant data into tailored rooms• Provide context to the sequence of events• Real-time discussion, feedback and change!


Dsmorse_iss

44

Team Development

• Ongoing cross-training and mentoring• Pair programing workstation to enhance

our ability to mentor each other

KEEP LEARNING

Dsmorse_iss

45

Team Development



Git Flow• Pull Requests• Peer Review

KEEP LEARNING

Dsmorse_iss

46

Team Development



Git Flow• Pull Requests• Peer Review

Blameless Retrospectives• Posted to Confluence• 100% transparency

KEEP LEARNING

Dsmorse_iss

47

• Reliable alerts on public facing systems.

• The structures we put in place to support 12 factor apps, can also be applied to our legacy systems to increase commonality

o Decrease cognitive shift

o Embrace changes on both sides

• Growing number of Anisble roles for common tasks

o Firewalls, NTP, SMNP, common tools

• Performed our first few canary rollouts

WHERE ARE WE TODAY?

Dsmorse_iss

48

More of the same!

• Scripting

• Containerization

• Monitoring

• Collaboration

• Next 6 months

Dsmorse_iss

49

More of the same!

• Scripting


• Monitoring

• Collaboration

New Stuff!

• vRealize/NSX for Self-Service

• vSphere integrated containers

• Docker data center

• Next 6 months

Dsmorse_iss

50

More of the same!

• Scripting


• Monitoring

• Collaboration

New Stuff!

• vRealize/NSX for Self-Service

This is working!

NEXT 6 MONTHS

Dsmorse_iss

51

After 9 Months

• Cleared more than 3 years of backlog!

• Migrated from manual changes to scripted changes

• Containerized workloads running on their own

• Containers as a Service

SUMMARY AND LESSONS LEARNED

Dsmorse_iss

52

After 9 Months

• Cleared more than 3 years of backlog!

• Migrated from manual changes to scripted changes

• Containerized workloads running on their own

• Containers as a Service

Lessons Learned

• Establish Metrics Early On

• Change Takes Time

• Expect and Embrace Failure

• Technology Enables, but Not a Panacea

SUMMARY AND LESSONS LEARNED

Dsmorse_iss