devops maturity - testhouse.net · o a large entertainment company mature in devops o a medium...

© Testhouse Ltd. Confidential and Proprietary

DEVOPS MATURITY


About Testhouse

DevOps a brief overview

Challenges with DevOps

Maturity Assessment

Case Studies

o A software house, specialising in travel, implementing DevOps

o A Large entertainment company mature in DevOps

o A medium office space provider starting DevOps

o A Large Bank planning DevOps

o A Large Financial Services mature in DevOps

AGENDA

KEY FACTS


Founded in 2000

Headquartered in London

Full Service Centres(UK, UAE, India)

Partnerships with Vendors(Dynamics CRM & ALM)

Quality Partner of many

Leading Organisations

G-Cloud 9 Supplier

RELEVANT EXPERIENCE

VOLKSWAGEN

FINANCIAL SERVICES

Deployments, migrations & upgrades

Release Management

Team Foundation

Server & VSTS

Onsite, onshore and offshore Flexible blended delivery

Managed Services

Compatibility/Mobility SIT/UAT Pre-built scripts

Functional

Testing

Web & Load performance testing

Application Insights deployment

Performance Testing

& APM

Framework design Automation services

Test

Automation

ALM and DevOps assessments

Continuous integration & deployment

Agile coaching

Agile, ALM & DevOps

Consultancy

Services

@testhouseuk

02

03

01

Bringing the three together to make DevOps4Dynamics

VSTS and DevOps

01

Dynamics 365

02Azure

03

The DevOps methodology and VSTS

The on time provisioning of environments for Dev, Test, Performance Test and Production

Dynamics 365, Dynamics CRM & AX

TESTHOUSE FOCUS

© Testhouse Ltd. Confidential and Proprietary© Testhouse Ltd. Confidential and Proprietary

Waterfall Iterative Agile

Disciplined Agile

DevOps

THE EVOLUTION OF DELIVERY PRACTICES

DEVOPS IS ALL ABOUT EXECUTING WITH SPEED!

8

Line-of-business Customer

• So you can continuously improve• application delivery• environment provisioning• application and environment

delivery process

• Getting ideas into production quickly, in small batches• Getting people to use it• Getting their feedback

Continuous Delivery Continuous Feedback

Continuous Innovation

Adapted culture

• KPIs focusing on end-results: the client satisfaction• Encouraging innovation and collaboration• Supportive of failure to accelerate learning

Dev SCM Build Package Deploy Test Prod


THREE WAYS OF DEVOPSSYSTEMS THINKING


CI / CD

Dev SCM Build Package Deliver Test Deploy

Getting ideas into production quickly, in small batches, Automating the process


THREE WAYS OF DEVOPSFEEDBACK LOOPS




• Getting ideas into production quickly, in small batches• Getting people to use it• Getting their feedback

• Measure and Monitor• Are we building the right product?• Understanding and responding to all customers• Introduce faults to the system


THREE WAYS OF DEVOPSCULTURE OF EXPERIMENTATION




• Getting ideas into production quickly, in small batches• Getting people to use it• Getting their feedback •Experiment and reduce MTTR

• Experiment and introduce failure practice makes perfect• Allocate time to experiment• Allow people to take risks• Introduce faults to the system

T H E M I C R O S O F T D E V O P S S T O R Y

DEVOPS BENEFITS

STRONG IT PERFORMANCE IS

A COMPETITIVE ADVANTAGE

DEVOPS PRACTICES

IMPROVE IT PERFORMANCE

DEPLOY CODE 30X FASTER

HAVE 60X FEWER FAILURES

TOP 15 CHALLENGES


Flexible environment provisioning required by DevOps teams (67%)

There is no coordination of the delivery environment elements (58%)

Governance when applying DevOps (50%) Deployments are largely manual (and error prone) (50%)

There are no or limited DevOps-specific metrics in place (50%)

There is no integrated tools architecture in place (42%)

Insufficient and inconsistent test automation (42%) Test data management (42%)

Testing is not performed early using capabilities such as service virtualisation (33%)

No DevOps vocabulary is in place (33%)

No DevOps vision or strategy defined (33%) Missing or inconsistent mechanisms for getting feedback (33%)

Applying DevOps in a regulated environment (33%) There is a lack of collaboration (and transparency) between business and IT (33%)

There is a lack of collaboration between Dev and Ops (in both directions) (33%)

APPROACH TAKEN – 6 AREAS


DEV

OP

S

APPROACH TAKEN – 3RA STAGES

Reactionary

Reactionary to the skill, ad hoc behaviours and success through luck

Repeatable

Performs training and regular practice to increase success rate

Reliable

Consistency with success most of the time but still room to improve

Aspirational

Ideal state where success is a given due to the level of reliability

DEV

OP

S

DEVOPS MATURITY

Backlog

Evidence

Flow

Production

Schedule & Team

Technical Debt

Target vs Actual

Actual Target

A SOFTWARE HOUSE, SPECIALISING IN TRAVEL, IMPLEMENTING DEVOPS


Background A software house providing software solutions to the tourism and travel industry. Currently going through a DevOps implementation.

Challenges Implemented automated build and deploy using in house built solution. The key individual who built this solution is now left. Currently no knowledge of how this was done. We are currently assisting them with the following

• Re-engineering the automated build and deploy / re –develop the automated build and deploy using TFS

• Testing team waiting for the completed build• Building automated regression testing • Do not promote the build but rebuild each time• No automated approval process• All environments on physical machine

Current state In terms of maturity they are early into the DevOps adoption cycle

A LARGE ENTERTAINMENT COMPANY MATURE IN DEVOPS

Background A large entertainment and excitement company who has implemented DevOps and is very mature.

Challenges As part of their continuous improvement programme asked us to review their DevOps processes and make recommendations

• Very mature and the organisation feels that it has benefitted tremendously after the adoption

• They are not using TFS but tools like Jira and Gitlab therefore not a fully integrated solution, now migrating to TFS

• No Portfolio backlog ( portfolio -> program -> team lead ) hence no ability to view at high level

Current state In terms of maturity they are classified as very mature


A REAL ESTATE INVESTMENT TRUST

Background The company owns and provides office space facilities to businesses.

Challenges Interested in Implementing currently lacks the following• No automated build• No automated testing (functional or performance)• Unit testing for JavaScript but not for .Net code

Current state Currently implementing TFS to provide automated build and deployment and starting Test Automation. Customer is however keen to implement and takes advice seriously.


A LARGE INVESTMENT BANK

Background A large investment bank who are currently releasing four times a year wants to increase the number of releases to eight times a year.

Challenges Increase speed of delivery with increased quality. Mixture of legacy systems, client server systems and web based applications.

• Develop the automated build and deploy using TFS• Building automated regression testing • Implement automated release management• Use of cloud for environment provision with Docker

Combination of the above can achieve their goal by Oct 2018, with the next step to increase the number of releases to once a month, which is an acceptable standard.

Current state In terms of maturity they are early into the DevOps adoption cycle


A LARGE FINANCIAL SERVICES COMPANY

Background Traditionally following waterfall methodology, following the appointment of a CIO, implemented DevOps using opensource technology.

Challenges The change was implemented by stealth and quickly. As a consequence face significant issues. In terms of maturity they are early into the DevOps adoption cycle

Current state


CURRENT STATE - ADVANTAGES

DevOps has been implemented and is reasonably active

Team is receptive to change

Management and team are well aligned

Technically competent team

Collaborative management


© Testhouse Ltd. Confidential and Proprietary© Testhouse Ltd. Confidential and Proprietary

Application

2 weeks train

TestDevPre

ProdUAT Prod

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRxqFQoTCIOetdOxrsgCFYVxPgodS3QJog&url=https://flumes.iei.liu.se/redmine/news/2&psig=AFQjCNFNoUVy4UEtcfXndR42YV3LtVmYLw&ust=1444239516718599

https://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRxqFQoTCIOetdOxrsgCFYVxPgodS3QJog&url=https://flumes.iei.liu.se/redmine/news/2&psig=AFQjCNFNoUVy4UEtcfXndR42YV3LtVmYLw&ust=1444239516718599

ISSUES PRIORITISED


Issue Importance Implementation Priority

Group

Requirements capture volatile High High I

No requirements traceability Medium Medium II

No risk based approach to testing Medium Medium II

Unit tests not automatically generated Low Low III

Scope of unit testing needs to be widened and quality improved High Medium II

Currently no non functional testing implemented Medium Low III

No code scan for security Medium Medium II

No integration testing with API calls carried out Low Low III

No review or testing of SQL statements High Medium II

issues with Fitnesse Fixtures Low Low III

Reference/Model Test Data Set not available Medium High II

The scope of UAT tests are currently too high High High I

Training for staff on nimble toolset is not adequate High High I

Test strategy and guidance High High I

Weak Unit Test Coverage for JavaScript Medium Medium II

Team Communication High High I

FOCUS AREAS- PRIORITIZED


Importance –

categories rated H, M, L based on the impact it will have on the project.

ASSESSMENT SOLUTIONING IMPLEMENTATION INDUSTRIALISATION

• Test Strategy

• Requirements Volatility• Team Communications

• DevOps Training

• High UAT Scope

•Auto generate unit tests

• Integration API level tests

•Non functional testing

• Review current toolset against

commercial tools ( Observation )

• Research into use of SV ( Observation )

• Environments ( Observation )‐ Adopt Build Process, Build & Validate‐ Refine for Virtual v. Physical

•Model Test Data‐ Adopt Creation Process, Create &

Validate

• Capture & Publish Delivery & ROI

Metrics

Ongoing…

• Continuous Process Review and

Improvement

• Implement items from the research

and review from last stage.

Evaluation

& Scoping

August 2017*

Define Efficient

Test Platform

October 2017*Transformation

December 2017*Continuous

Improvement

March 2018*

*Dates are indicative and will be reviewed after each phase

• Model Test Data

• Data Requirements

• Manipulation

• Reuse

• Tooling

• Requirements Traceability

• Risk Based Testing

• Unit Tests• Improve quality of Junit• Setup Jscript unit tests• Setup SQL unit tests

• Security Code Scan

ROADMAP TO ADDRESS KEY CHALLENGES

THANK YOU

Sug [email protected]

www.testhouse.net


devops maturity - testhouse.net · o a large entertainment company mature in devops o a medium...

Documents