observability beyond logging for java microservices

Observability beyond logging for Java Microservices

Luke Marsden, Developer Experience

@lmarsden

Developers

Users

Developers

Users

observability

Ship features fasterDevelopers

Users

Fix problems faster

Developers

Users

Developers

Users

competitiveness = speed of iteration

Containers & microservices• Docker is a good fit for microservices • Any non-trivial app runs on multiple hosts • How to manage containers across hosts? • “Container orchestration” • Kubernetes, Docker Swarm…

• Provide networking, service discovery etc

Once you’ve got your app in prod, how do you know how it’s doing?

Observability

Observability = Logging +

Monitoring + Tracing +

Visualization

Monitoring• Prometheus is a labelled time-series database • Labels are key-value pairs • A time-series is [(timestamp, value), …]

• lists of timestamp, value tuples • values are just floats, but you represent counters, gauges,

histograms, etc – PromQL lets you make sense of them • So the data type of Prometheus is

• {key1=A, key2=B} —> [(t0, v0), (t1, v1), …] • …

time

counter

13

13233336

t1 t2 t3 t4 t5 t6 t7 t8 t9

1 2 3 13 23 33 34 35 36t1-3 t2-4 t3-5 t4-6 t5-7 t6-8 t7-9

1 2 3 13 23 33 342 3 13 23 33 34 353 13 23 33 34 35 36

counter[3s]

time

rate(counter[3s])

1

5

10

t3 t4 t5 t6 t7 t8 t9

1 5.5 10 10 5.5 1 1

Monitoring – instrumentation• Whitebox / blackbox • Prometheus client library

Example – Prometheus

Tracing• Tracing allows you

to map causality through multiple layers of microservices

Tracing• Define spans – a tree structure of events • Annotate with timing info & messages

frontend_request

session_lookup user_lookup

user_db

Tracing• Most useful when something goes wrong! • Ever had a mysterious 500?

frontend_request

session_lookup user_lookup

user_db

Tracing• Spans are communicated when services RPC to

each other; each span has a unique ID • e.g. spans are injected into HTTP headers • and extracted on the other side

• Spans are propagated internally in contexts • Tracing-instrumented code reports to tracing

server, which then re-assembles causality

OpenTracing• Standard API for tracing implementations • Instrument your code once with

OpenTracing libraries, swap in and out different tracing implementations

• Example tracing implementations: Zipkin, Appdash, LightStep …

OpenTracing – instrumentation

https://github.com/opentracing/opentracing-java

Zipkin example

Visualization• Draw a live map of your

application • Understand interactions

between different components

• Not tracing individual requests, but shows overall structure

• Drill down into details • Example: Weave Scope

meetup.com/pro/Weave/

Join us on Slack – weave.works/help

Thanks! Questions?

We are hiring!DX in San Francisco

Engineers in London & SF

weave.works/company/hiring

observability beyond logging for java microservices

Internet