ElasticSearch on AWS

Continuously Deployed, Immutable and Stateful

About me

● Philipp Garbe (@pgarbe)

● Husband

● Father

● AutoScout24

● Former Microsoft-Fanboy

● Now Docker-Fanboy ;)

● Use case: Logging for AS24

● Challenge: How to handle the state?

○ Immutable

○ Continuously Deployed

○ Stateful

● Next challenges

What to expect

Project

Tatsu

UnifiedLogs

CC

STRATEGICGOALSGoals of the business side

ARCHITECTURALPRINCIPLESHigh-Level Principles

DESIGN AND DELIVERY PRINCIPLESTactical measures

REDUCE TIME TO MARKETSpeed, Fast Feedback

COST EFFICIENCYCollect metrics to allow decisions cost vs. value.

SUPPORT DATA-DRIVEN DECISIONSListen to users and validate hypothesis.Provide as many relevant metrics & data as possible.

YOU BUILT IT, YOU RUN ITThe team is responsible for shaping, building, running and maintaining its products. Fast feedback from live and customers helps us to continuously improve.

ORGANIZED AROUND BUSINESS CAPABILITIESBuild teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver.

LOOSELY COUPLEDBy default avoid sharing and tight coupling, except for the big things in common. Don’t create the next monolith.

MACRO AND MICRO ARCHITECTUREClear separation. Autonomous micro services within the rules and constraints of the macro architecture.

AWS FIRSTFavor AWS platform service over managed service, over self-hosted OSS, over self-rolled solutions.

DATA-DRIVEN / METRIC-DRIVENCollect metrics from processes and applications. Analyze, alert and act on them.

ELIMINATE ACCIDENTAL COMPLEXITYStrive to keep it simple. Focus on essential complexity. You build one, you delete one.

AUTONOMOUS TEAMSMake fast local decisions. Be responsible. Know your boundaries. Share findings.

INFRASTRUCTURE AS CODEAutomate everything: Reproducible, traceable and tested.Immutable servers over snowflake servers.

COLLABORATION CULTUREEngineers from all backgrounds work together in collaborative teams as engineers and share responsibilities. No silos.

BE BOLDGo into production early. Value monitoring over tests. Recover and learn. Optimize for MTTR not MTBF.

SECURITY, COMPLIANCE AND DATA PRIVACYSecurity must be included from the beginning and everybody’s concern. Keep data-privacy in mind.

CONTAINMENT AND BOUNDARIESAlign blast radius and vendor lock-in with the boundaries of the organization or business capabilities.

Version 1.0Icons made by Freepik from www.flaticon.com are licensed under CC BY 3.0

https://github.com/autoscout24/scout24-it-principles

INFRASTRUCTURE AS CODEAutomate everything: Reproducible, traceable and tested.

Immutable servers over snowflake servers.

Some numbers

12 nodes (m4.4xlarge, EBS gp2 á 2000 GB)

9 indices (one per day)

648 shards

7 Billion docs (new documents peak: 2,500 docs / sec)

16.99 TB of data

Challenge: How to handle the state?

Immutable Servers

Rake

CloudFormation

ElasticSearch configuration

Continuously Deployed

● One CloudFormation stack per node

● Rolling updates through rake

● Every commit goes to production

Stateful

Keep the data!

Monitoring

● CloudWatch

○ Cron-Lambda sends /_cluster/health metrics to CloudWatch

○ CloudWatch alarms based on these metrics

● DataDog

○ Deep Dive

Next challenges

No AvailabilityZone awareness

Next challenges

Mapping issues

Next challenges

Long-running deployments (2-8h)

?Questions

Philipp Garbe

http://garbe.io

@pgarbe

https://github.com/pgarbe