implementing microservices tracing with spring cloud and zipkin (spring one)
TRANSCRIPT
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Implementing Microservices Tracing with Spring Cloud and
ZipkinBy Reshmi Krishna
@reshmi9kBy Marcin Grzejszczak
@mgrzejszczak
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
About us
• Software Engineer
• Platform Architect, Pivotal
• Women In Tech Community Member
Twitter : @reshmi9kMeetUp : Cloud Native New Yorkhttp://www.meetup.com/Cloud-Native-New-York/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
About us
• Spring Cloud developer
• Working mostly on Spring Cloud Sleuth and Spring Cloud Contract
• Automation freak
Twitter: @mgrzejszczakBlog: http://toomuchcoding.com
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Agenda• How to correlate logs with Spring Cloud Sleuth?
• How to visualize latency with Spring Cloud Sleuth and Zipkin?
• How to use Pivotal Cloud Foundry to deploy and integrate apps with Spring Cloud Services and Zipkin?
• How to integrate Spring Cloud Sleuth and Zipkin in an existing app?
• How to Investigate issues using PCF Metrics and Zipkin?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
An ordinary system...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
CLICK 200
Everything is awesome
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
CLICK 500
Until it’s not
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
https://tonysbologna.files.wordpress.com/2015/09/mario-and-luigi.jpg?w=468&h=578&crop=1
Time to debug
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
It doesn’t look like this
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
More like this
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
On which server / instancewas the exception thrown?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The answer: distributed tracing• Span
• Trace
• Logs (annotations)
• Tags (binary annotations)
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Is context propagation simple?How do you pass tracing information (incl. Trace ID) between:
• different libraries?
• thread pools?
• asynchronous communication?
• …?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Log correlation with Spring Cloud SleuthWe take care of passing tracing information between threads / libraries / contexts for:• Hystrix• RxJava• Rest Template• Feign• Messaging with Spring Integration• Zuul• ...
If you don’t do anything unexpected there’s nothing you need to do to make Sleuth work. Check the docs for more info.
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Now let’s aggregate the logs!Instead of SSHing to the machines to grep logs lets aggregate them!
• With Cloud Foundry’s (CF) Loggregator the logs from different instances are streamed into a single place
• You can harvest your logs with Logstash Forwarder / FileBeat
• You can use ELK stack to stream and visualize the logs
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Brixton.SR4</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-sleuth</artifactId>
</dependency>
Spring Cloud Sleuth with Maven
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
SERVICE 1/readtimeout
REQUEST
BOOM!
SERVICE 2
REQUEST
BOOM!
REQUEST
BOOM!
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Log correlation with Spring Cloud SleuthDEMO
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Great! We’ve found the exception!But meanwhile....
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
CLICK 200
The system is slow...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
One of the services is slow...
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Which one?How to measure that?
How to visualise latency in a distributed system?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
The answer is:ZIPKIN
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
SPANS SENT TO COLLECTORS
SPANS SENT TO COLLECTORS
STORE IN DB
APP
APP
UI QUERIES FOR TRACE INFO VIA API
How does Zipkin work?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Cloud Sleuth and Zipkin integration
● We take care of passing tracing information between threads / libraries / contexts
● Upon closing of a Span we will send it to Zipkin○ either via HTTP (spring-cloud-sleuth-zipkin)○ or via Spring Cloud Stream (spring-cloud-sleuth-stream)
● You can run Zipkin Spring Cloud Stream Collector as a Spring Boot app (spring-cloud-sleuth-zipkin-stream)○ you can add the dependency to Zipkin UI!
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Cloud Sleuth Zipkin with Maven
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>Brixton.SR4</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-zipkin</artifactId>
</dependency>
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Hold it!
● If I have billion services that emit gazillion spans - won’t I kill Zipkin?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Sampling to the rescue!
● By default Spring Cloud Sleuth sends only 10% of requests to Zipkin
● You can change that by changing the property spring.sleuth.sampler.percentage (for 100% pass 1.0)
● Or register a custom org.springframework.cloud.sleuth.Sampler implementation
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
SERVICE 1/start
REQUEST
RESPONSE
SERVICE 2/foo
SERVICE 3/bar
REQUEST
RESPONSE
REQUEST
RESPONSE
SERVICE 4/baz
REQUEST
RESPONSE
SPRINGONESERVICE/springone
REQUEST
RESPONSE
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
DEMO
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
AGENDA
• How to use Pivotal Cloud Foundry to deploy and integrate apps with Spring Cloud Service and Zipkin?
• How to integrate Spring Cloud Sleuth and Zipkin in an existing app?
• How to Investigate issues using PCF Metrics and Zipkin?
34
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
PIvotal Cloud Foundry & Spring Cloud Services
Applications
Data
Runtime
Middleware
O/S
Virtualization
Servers
Storage
Networking
PCF
IAAS
SPRING CLOUD SERVICESV 1.1.1
CONFIG SERVER
SERVICE REGISTRY
CIRCUIT BREAKER
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Trader App Architecture
Web-UI
Portfolio Accounts
Quotes
DB DB
CONFIG SERVER
SERVICE REGISTRY
CIRCUIT BREAKER
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Integrating Sleuth Stream and Zipkin
SleuthWeb-UI
Quotes
Portofolio
Account
Sleuth
Sleuth
Sleuth
Transport(MQ,HTTP)
Collector
Span Store
Query Server
Zipkin UI
ZIPKIN
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Let’s look at some code & Demo !!!!
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Changes Made
- Add sleuth stream and rabbit binder to build.gradle/pom.xml - Add Sleuth Sampler percentage and logging level for sleuth in config file
(Config Server)
- Deploy Zipkin as a service in PCF- Connect/Bind apps to the same Rabbit MQ instance as zipkin
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
● Log correlation allows you to match logs for a given trace
● Distributed tracing allows you to quickly see latency issues in your system
● Zipkin is a great tool to visualize the latency graph and system dependencies
● Spring Cloud Sleuth integrates with Zipkin and grants you log correlation
● PCF makes integration of your apps and Spring Cloud Sleuth and Zipkin easier
Summary
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
• Code for this presentation (clone and run getReadyForConference.sh - NOTE: you need Vagrant!) : https://github.com/marcingrzejszczak/vagrant-elk-box/tree/presentation
• Sleuth’s documentation:http://cloud.spring.io/spring-cloud-sleuth/spring-cloud-sleuth.html• Repo with Spring Boot Zipkin server: https://github.com/openzipkin/zipkin-java• Zipkin deployed to PCF for Brewery Sample app: http://docsbrewing-zipkin-server.cfapps.io • Pivotal Web Services trial : https://run.pivotal.io/• Trader Application : https://github.com/reshmik/cf-SpringBootTrader.git• Zipkin deployed as PCF
https://github.com/reshmik/Zipkin/tree/master/spring-cloud-sleuth-samples/spring-cloud-sleuth-sample-zipkin-stream
• PCF on your laptop : https://docs.pivotal.io/pcf-dev/
Links
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Learn More. Stay Connected.
https://gitter.im/spring-cloud/spring-cloud-sleuth
How to Properly Blame Things for Causing Latency: An Introduction to Distributed Tracing and Zipkin by Adrian Cole
@springcentralspring.io/blog
@pivotalpivotal.io/blog
@pivotalcfhttp://engineering.pivotal.io