microservices, continuous delivery, and elasticsearch at capital one
TRANSCRIPT
![Page 1: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/1.jpg)
Capital One
3/8/2017
Microservices, Continuous Delivery, and Elasticsearch at Capital One
Noriaki (Nori) Tatsumi, Bingchen (Ben) Hu, Anne Cather
![Page 2: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/2.jpg)
![Page 3: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/3.jpg)
Security breaches dominate the news
![Page 4: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/4.jpg)
CYBER TECHDATA LAKE
![Page 5: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/5.jpg)
Build vs. buy
• Industry tools only meet ~80% of our requirements
• Vendors’ priorities don’t align with ours
• Elasticsearch is an open source solution
• Open source technology is extensible
![Page 6: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/6.jpg)
30+ data sources, 3B events and 6TB data/day
![Page 7: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/7.jpg)
![Page 8: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/8.jpg)
How we got here
![Page 9: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/9.jpg)
Scale New features NFRs
• More data
• More processing
• Longer data retention
• More consumers
• Alerts console
• Cyber threat intelligence
repository
• And more!
Our initial requirements
• Uptime and DR
• Security
• Compliance
• Data management
![Page 10: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/10.jpg)
The prototype we had
ElasticsearchData Nodes
ElasticsearchMaster Nodes
ElasticsearchClient NodeKibana Fork
w/ SSO Integration
AD SSO
![Page 11: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/11.jpg)
MORE REQUIREMENTS,DELIVERY DATES,
BIGGER TEAMS=
HIGHER COMPLEXITY
![Page 12: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/12.jpg)
Monolith
![Page 13: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/13.jpg)
• Work in parallel
• Do one scope of things well
• Easy to understand and maintain
• Technology stack choice for features and teams
• Quicker, smaller, & independent deploys
• Fault isolation
What we wanted
![Page 14: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/14.jpg)
MICROSERVICES
![Page 15: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/15.jpg)
No SSO Integration!
Embracing microservices
ElasticsearchData Nodes
ElasticsearchMaster Nodes
ElasticsearchClient NodeKibana Fork
w/ SSO Integration
AD SSO
Alerts-API Alerts-UI CTI Repo
![Page 16: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/16.jpg)
• A well known entry point to the system
• Security
• Dynamic routing
• Resiliency
• Latency and fault tolerance
• Monitoring and stats collection
Edge gatewayAlign same qualities to downstream services
![Page 17: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/17.jpg)
• Spring Boot for developer productivity
• JVM-based for production supportability
• Netflix OSS that’s proven microservices technology
Spring CloudFoundation for our web microservices
![Page 18: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/18.jpg)
@SpringBootApplication@EnableAutoConfiguration@EnableZuulProxypublic class EdgeGateway {public static void main(String[] args) throws Exception { SpringApplication.run(EdgeGateway.class, args);}
}
Getting started with Netflix Zuul is easy
Edge gateway
![Page 19: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/19.jpg)
zuul.routes.kibana.path=/kibana/**zuul.routes.kibana.url=https://172.20.10.15:5601
Routing with Zuul
Edge gateway
![Page 20: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/20.jpg)
ElasticsearchClient NodeKibana
ElasticsearchClient NodeKibana
Zuul: the edge gateway
ElasticsearchData Nodes
ElasticsearchMaster Nodes
EdgeGateway
ElasticsearchClient NodeKibana
AD SSO
Alerts API
Alerts UI Reports UI
CyberTechReports Repo
Auth
![Page 21: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/21.jpg)
Asking engineers to maintain IP addresses
![Page 22: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/22.jpg)
• Use cases
• Service connection information lookup
• Automated configuration of load balancing and failover
• Alternatives to Eureka with Spring Cloud
• HashiCorp Consul
• Apache Zookeeper
Discover serviceAutomate orchestration with Netflix Eureka
![Page 23: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/23.jpg)
<application> <name>...</name> <instance> <instanceId>... </instanceId> <hostName>... </hostName> <app>...</app> <ipAddr>...</ipAddr> <status>UP</status> <overriddenstatus>UNKNOWN</overriddenstatus> <port enabled="false">...</port> <securePort enabled="true">...</securePort> <countryId>1</countryId> <dataCenterInfo class="com.netflix.appinfo.AmazonInfo"> <name>Amazon</name> <metadata> <accountId>...</accountId> <local-hostname>... </local-hostname> <instance-id>...</instance-id> <local-ipv4>...</local-ipv4> <instance-type>...</instance-type> <vpc-id>...</vpc-id> <ami-id>...</ami-id> <mac>...</mac> <availability-zone>...</availability-zone> </metadata> </dataCenterInfo> <leaseInfo> <renewalIntervalInSecs>...</renewalIntervalInSecs> <durationInSecs>...</durationInSecs> …..
![Page 24: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/24.jpg)
zuul.routes.kibana.path=/kibana/**zuul.routes.kibana.serviceId=kibana
kibana.ribbon.listOfServers=172.20.10.11:5601,172.20.10.12:5601,172.20.10.13:5601,172.20.10.14:5601ribbon.eureka.enabled=false
Routing with Zuul without Eureka
Discover service
![Page 25: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/25.jpg)
zuul.routes.kibana.path=/kibana/**zuul.routes.kibana.serviceId=kibana
Routing with Zuul with Eureka
Discover service
![Page 26: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/26.jpg)
@SpringBootApplication@EnableDiscoveryClientpublic class Application {
public static void main(String[] args) { SpringApplication.run(Application.class, args);}
}
Making Spring Boot app discoverable with Eureka
Discover service
![Page 27: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/27.jpg)
• Eureka Client (Java)
• Eureka-js-client (JavaScript)
• Eureka REST API (Polyglot)
• *Sidecar/App gateway (Polyglot)
Discover serviceMaking any app discoverable with Eureka
![Page 28: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/28.jpg)
Solving the configuration nightmare
ElasticsearchData Nodes
ElasticsearchMaster Nodes
EdgeGateway
AD SSO
KibanaGateway
ElasticsearchClient Node
KibanaKibana
Gateway
ElasticsearchClient Node
KibanaKibana
Gateway
ElasticsearchClient Node
Kibana
EurekaDiscoveryService
/kibana
Alerts-UI
CyberTechReports UI
Alerts-API
CyberTechReports API
![Page 29: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/29.jpg)
Multi-config Kibanas
ElasticsearchData Nodes
ElasticsearchMaster Nodes
EdgeGateway
AD SSO
KibanaGateway
ElasticsearchClient Node
KibanaKibana
Gateway
ElasticsearchClient Node
KibanaKibana
Gateway
ElasticsearchClient NodeKibana
(Console Off)
KibanaGateway
ElasticsearchClient NodeKibana
(Console On)
AuthorizationService
/kibana
/kibana-admin
![Page 30: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/30.jpg)
Protected Elasticsearch gate
ElasticsearchData Nodes
ElasticsearchMaster Nodes
EdgeGateway
AD SSO
KibanaElasticsearchClient Node
ElasticsearchGateway
KibanaGateway
KibanaElasticsearchClient Node
ElasticsearchGateway
KibanaGateway
Kibana(Console OFF)
ElasticsearchClient Node
ElasticsearchGateway
KibanaGateway
Kibana(Console ON)
ElasticsearchClient Node
ElasticsearchGateway
Kibana-AdminGateway
AuthorizationService
/kibana-admin
/kibana
/esclient
![Page 31: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/31.jpg)
Spring Boot Admin for Spring Cloud microservices
https://github.com/codecentric/spring-boot-admin
![Page 32: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/32.jpg)
Distributed tracing with Spring Cloud Sleuth
https://cloud.spring.io/spring-cloud-sleuth/
![Page 33: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/33.jpg)
Distributed tracing with Spring Cloud Sleuth
![Page 34: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/34.jpg)
Distributed tracing with Spring Cloud Sleuth
![Page 35: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/35.jpg)
• Successes
• Short circuited
• Thread timeouts
• Thread-pool rejections
• Failures/exceptions
• Error percentage
(Rolling 10 second counters)
Circuit breaker monitoring
![Page 36: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/36.jpg)
Crushed it!
Elasticsearch
Kibana
Product delivered and released on time
![Page 37: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/37.jpg)
MICROSERVICES=
PROFIT!
![Page 38: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/38.jpg)
ELASTICSEARCH
OPERATIONS
![Page 39: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/39.jpg)
Cluster on fire!
• Stability issues from end user queries
• Data ingestion latency problems
• Insufficient monitoring
Compliance requiring AMI refresh every 60 days
![Page 40: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/40.jpg)
Finding the causes
• Inconsistent OS, JVM, and Elasticsearch configurations across cluster
• No circuit breakers
• Elasticsearch index templates were missing
• Shards improperly sized
• Incorrect field mappings
• Improper cluster sizing
![Page 41: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/41.jpg)
DEV + OPS
![Page 42: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/42.jpg)
CONTINUOUS DELIVERY
=REQUIREMENT
![Page 43: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/43.jpg)
Configuration management +
Automation
![Page 44: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/44.jpg)
Hello
![Page 45: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/45.jpg)
Hardware Playbook
• Spin up AWS infrastructure
• Tag for purpose
• Configure subnet, security
group, VPC, etc.
Software Playbook
• Install common dependencies
• AWS tags determine software
• Deploy latest artifacts per
environment
Ansible deployment breakdown
![Page 46: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/46.jpg)
Hardware playbook example
roles: - role: servers instances: - name: Elasticsearch_Master instance_type: m4.2xlarge number_of_instances: 3
- name: Elasticsearch_Data instance_type: m4.4xlarge number_of_instances: 100 additional_volume_sizes: [1000, 1000, 1000]
![Page 47: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/47.jpg)
- hosts: tag_{{ ansible_ec2_tag }}_Elasticsearch_Data become: true roles: - role: elasticsearch es_heap_size: '{{ [(ansible_memtotal_mb / 1024) / 2, 16] | min | int }}g' es_plugins: - '{{ es_plugin_license }}' - '{{ es_plugin_marvel_agent }}' - '{{ es_plugin_cloud_aws }}' es_config: cluster.name: '{{ elasticsearch_cluster_name }}' node.name: '{{ ansible_default_ipv4.address }}' node.master: false node.data: true
indices.fielddata.cache.size: 10%indices.breaker.fielddata.limit: 15%indices.breaker.request.limit: 15%indices.breaker.total.limit: 30%network.breaker.inflight_requests.limit: 75%
Software playbook example
![Page 48: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/48.jpg)
./hardware-playbook.yml --extra-vars @dev-vars.yml
./software-playbook.yml --extra-vars @dev-vars.yml
How to use
![Page 49: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/49.jpg)
Monitor everything!Don’t run a black box
![Page 50: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/50.jpg)
• Cloud metrics
• Server metrics
• JVM metrics (even built our own JVM agent)
• Application metrics
• …
What we should monitor
![Page 51: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/51.jpg)
![Page 52: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/52.jpg)
Time-series dashboards with Grafana
![Page 53: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/53.jpg)
ANOTHER SERVICE?
![Page 54: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/54.jpg)
Metrics cluster integration
ElasticsearchCyberLake Nodes
EdgeGateway
AD SSO
KibanaElasticsearchClient Node
ElasticsearchGateway
KibanaGateway
KibanaElasticsearchClient Node
ElasticsearchGateway
KibanaGateway
KibanaElasticsearchClient Node
ElasticsearchGateway
KibanaGateway
KibanaElasticsearch
GatewayKibana-Metrics
GatewayElasticsearchClient Node
/metrics
/kibana
/esclient
ElasticsearchMetrics Cluster
EurekaDiscoveryService
ES query data
ES query data
Service Availability Data
Service Availability Data
![Page 55: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/55.jpg)
PLATFORM STABILITY
![Page 56: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/56.jpg)
TAKEAWAYS
![Page 57: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/57.jpg)
• Microservices architecture works for us
• Increase velocity and reduce maintenance effort
• Elastic stack can integrate easily
• Continuous Delivery must be a requirement
• Monitor everything!
Takeaways
![Page 58: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/58.jpg)
MICROSERVICES+
CONTINUOUS DELIVERY=
PROFIT!
![Page 59: Microservices, Continuous Delivery, and Elasticsearch at Capital One](https://reader035.vdocuments.net/reader035/viewer/2022062503/58e65de81a28ab8d758b4d83/html5/thumbnails/59.jpg)
More Questions?
Visit us at the AMA