continuos delivery in a microservices environment

Continuous Delivery in a microservices environment A success story

Luca Mazzaferro luca.mazzaferro@holidaycheck.com

Outline:

•  About me

•  About HolidayCheck

•  Adopting microservices to enforce continuous delivery

▪  I was born in Rome where I spent most of my life until 2013 when I moved to Munich.

▪  Former physicist, member of ATLAS experiment @CERN since the master degree and during the Ph.D. with physics analysis and computing tasks.

▪  In June 2015, I sold my soul to the business, leaving the public research for happily joining Holidaycheck as DevOps engineer

About Me

About Holidaycheck

1

Holidaycheck means…

Holidays and travel bookings

1

Trustable reviews

Holidaycheck means…

Holidays and travel bookings

2

1

Trustable reviews

Holidaycheck means…

Finding the best price

2

3

Holidays and travel bookings

1

Trustable reviews

Holidaycheck means…

… and much more.

2

3

4

Finding the best price

Holidays and travel bookings

Hotels: 659.000

Dests: 195 Countries 52880 Towns

Reviews: 6.900.000

Photos: 7.470.000

Visit/Month: 13.66 Million

Tour operators: 100

About Holidaycheck Some numbers

Holidaycheck is the largest review, booking and price comparing portal for travel and holidays in German language.

Adopting Microservices The puzzle

Pre HC microservices era: Not so much time ago…

The Deployment Day

Pre HC microservices era: Not so much time ago…

●  All the teams together ●  one big monolith application ●  2 weeks: development ●  1 week: collecting ●  1 day: Deploying and

Testing...

Pre HC microservices era: Not so much time ago…

Consequences: ●  Slow deployment rate ●  High chance of rolling back ●  Inter teams dependencies ●  Team planning difficult ●  Tests difficult ●  High stress

Pre HC microservices era: Not so much time ago…

The Elephant Release

Pre HC microservices era: Not so much time ago…

●  Only one team could deploy ●  Queue system for deployment

(the elephant token) ●  No microservices Consequences: ●  Slow deployments ●  Teams waiting ●  Inter teams dependencies

Adopting Microservices Our solution

Adopting microservices approach First steps

●  Define a migration plan ●  Rethink teams structure and responsibilities ●  New Technology stack

Migration Plan:

1.  Slow migration: from mobile to desktop 2.  One section after another: less traffic to higher 3.  Getting rid of old legacy software (when possible)

Adopting microservices approach

•  Preferred small cross functional teams (5-6 people) •  Devops Culture adopted via in-team devops engineer •  Teams responsible for their own products (microservices) •  Continuous Delivery

Teams structure and responsibilities

PO/TO SM Devs Devops Eng.

Ops Team

Adopting microservices approach

Organiza(onswhichdesignsystems...areconstrainedtoproducedesignswhicharecopiesofthecommunica(onstructuresoftheseorganiza(ons[M.Conway]

Migration Plan The Dust Project

Creation of Dust Team (Ops + Dev): •  Setup the new infrastructure •  Develop tools for microservices

deployments •  Support teams for services

“containerization”

Dust Environment

Migration Plan The Dust Project

Dust Environment

Main advantages:

•  Easily scalable

•  Ready for Continuous

Deployments

•  Services easy to monitor

•  Isolated environment

•  Enforce team autonomy

1 Develop, commit and push

Continuous Delivery @HolidayCheck Our Vision

Build Basic tests (e.g. Unit, Reg.) Deploy on staging env

Staging

Live

Team Server

1

2

3

4

2

3 Sophisticated Tests (integr., accuracy)

4 Deploy in production Smoke tests

Monitoring and Alerting

•  Simple health checks •  Metric based Alerts

•  Logs •  System metrics

Default Monitoring

•  Teams defined dashboards

Custom Monitoring

Another ingredient for Continuous Delivery

•  Required for each service •  Semi-automatically deployed •  As simple as possible •  Used mainly by ops-devops

Engineers

Alerts You can’t survive without

•  Automatically deployed •  Debugging (e.g. Kibana

Dashboards) •  Criticalities discoveries (e.g.

NewRelic Maps) •  Used by devs, ops and

devops

Default Monitoring Makes your life easier

•  Team defined metrics and dashboards

•  Service quality and performances monitoring

•  Data & Business analysis

Custom Monitoring Can drive business decisions

MSs/Containers

100

Teams

12 Deployment Rate

Daily/Weekly

Main languages: Scala/Go/ReactJS

Main Backend Techs ElasticSearch/

MongoDB/Kafka

HC Infrastructure in Numbers

During the migration the old PHP/Java technologies and MySQL/Tomcats backend are in replacement.

Conclusions

The adoption of microservices enforced team autonomy, enabled

the Continuous Delivery and increased the deployment rate.

The entire infrastructure is more stable, reliable, maintainable

and scalable

Conclusions

The adoption of microservices and devops culture enforced team

autonomy, enabled the Continuous Delivery and

increased the deployment rate.

The entire infrastructure is more stable, reliable, maintainable

and scalable

www.holidaycheck.de

HolidayCheck Solutions Neumarkter Str. 61 | D- 81673 München Luca Mazzaferro DevOps Engineer T: Tel +41 (0) 71 686 9000 luca.mazzaferro@holidaycheck.com

Contact.

THANK YOU FOR YOUR ATTENTION!