microservice and continuos delivery
TRANSCRIPT
Continuous Delivery in a microservices environment A success story
Luca Mazzaferro [email protected]
▪ I was born in Rome where I spent most of my life until 2013 when I moved to Munich.
▪ Former physicist, member of ATLAS experiment @CERN during the roaring master degree and Ph.D. years. Mainly focusing in data analysis and grid computing.
▪ In June 2015, I sold my soul to the business, leaving the public research for happily joining HolidayCheck as devops engineer
About Me
1
Trustable reviews
Holidaycheck means…
… and much more.
2
3
4
Finding the best price
Holidays and travel bookings
Hotels: 659.000
Dests: 195 Countries 52880 Towns
Reviews: 6.900.000
Photos: 7.470.000
Visit/Month: 13.66 Million
Tour operators: 100
About HolidayCheck Some numbers
HolidayCheck is the largest review, booking and price comparing portal for travel and holidays in German language.
Pre HC micro-services era: Not so much time ago…
ABC
1 Day 1 Week 2 Weeks
C B A
ABC
Development
Collect &
Merge Test & Deploy
Pre HC micro-services era: Not so much time ago…
Results: ● Low deployment rate ● Frequent full roll backs ● Many inter teams
dependencies ● Hard team planning ● High stress
Pre HC micro-services era: Not so much time ago…
Results:
● Low deployments rate
● Teams queued
● Inter teams dependencies
● Still monolithic application
Pre HC micro-services era: Not so much time ago…
Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands
of users, safely and quickly in a sustainable way.*
A bit better but still far away from here:
* continuousdelivery.com
The migration to micro-services The ingredients
● Migration plan ● New teams structure and responsibilities ● New technology stack
The 1st ingredient
1. Slow migration: from mobile to desktop 2. One section after another: less traffic to higher 3. Getting rid of old legacy software (when possible)
Migration Plan:
• Preferred small cross functional teams (5-6 people) • Devops Culture adopted via in-team devops engineer • Teams responsible for their own products (microservices) • Inter-teams dependencies removed and autonomy enforced
2nd ingredient
PO/TO SM Devs Devops Eng.
Ops Team
Teams structure & reposibilities
Organiza(onswhichdesignsystems...areconstrainedtoproducedesignswhicharecopiesofthecommunica(onstructuresoftheseorganiza(ons[M.Conway]
*
The new technology stack The Dust project
Creation of Dust Team (Ops + Dev): • Setup the new infrastructure
focused to run micro-services • Develop specific tools for micro-
services deployments • Support teams for services
“containerization”
Dust Environment
The new technology stack The Dust project
Dust Environment
Main advantages:
• Easily scalable
• Ready for Continuous
Deployments
• Services easy to monitor
• Isolated environment
• Enforce team autonomy
The new technology stack
• Metric based Alerts
• Logs • System metrics
Default Monitoring
• Teams defined dashboards
Custom Monitoring
New approach to monitoring and alerting
• Required for each service • Based on the user experience
(mainly error rate and response time)
• Customizable by the teams • Automatically deployed
Alerts You can’t survive without
Alerts A simple overview
A
B C
DUST
NGINX metrics
Customizable Metrics
Alerts defined on user experience: - Error rate (5XX, 4XX) - Response Time
On Call Duty
Teams
Traffic
• Automatically deployed • Based on logs and metrics • Useful for
debugging criticalities discovery performances issue discovery
• Accessible by devs and ops.
Default Monitoring Makes your life easier
• One centralize system for all the logs
• Search and visualize • No configuration needed • Easy to integrate in GCE
Default Monitoring Logs
Servers
Microservices
GCE
• Grafana shows all the metrics collected by Prometheus
• Custom metrics • Custom dashboards
Default Monitoring Custom Metrics
Servers
Microservices
GCE
• External service mainly used by micro-services
• Error analysis • Services maps • Metrics monitoring
Default Monitoring NewRelic
Servers
Microservices
GCE
• Team defined metrics and dashboards
• Freedom to chose tools and technologies
• Mainly for service quality and performances monitoring
• Data & Business analysis
Custom Monitoring Can drive business decisions
1 Develop, commit and push
Continuous Delivery @HolidayCheck Our Vision
Staging
Live
Team Server
1
2
3
4
Build Basic tests (e.g. Unit, Reg.) Deploy on staging env
2
3 Sophisticated Tests (integr., accuracy)
4 Deploy in production Smoke tests
* Continuous Monitoring
MSs/Containers
100
Teams
12 Deployment Rate
Daily/Weekly
Main languages: Scala/Go/ReactJS
Main Backend Techs ElasticSearch/
MongoDB/Kafka
HC Infrastructure in Numbers
Currently we are at the end of a migration phase in Google Cloud Engine.
Conclusions The adoption of microservices
enforced team autonomy, enabled the Continuous Delivery and
increased the deployment rate.
The entire infrastructure is more stable, reliable, maintainable
and scalable
A smooth migration in Google has been helped only by the
previous micro-services adoption.
Conclusions The adoption of microservices
enforced team autonomy, enabled the Continuous Delivery and
increased the deployment rate.
The entire infrastructure is more stable, reliable, maintainable
and scalable
A smooth migration in Google has been helped only by the
previous micro-services adoption.
www.holidaycheck.de
HolidayCheck Solutions Neumarkter Str. 61 | D- 81673 München Luca Mazzaferro DevOps Engineer T: Tel +41 (0) 71 686 9000 [email protected]
Contact.