![Page 1: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/1.jpg)
Max. width
Min. height
Max. height
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Microservices with Kafka
László-Róbert ALBERT Dan BALESCU Casablanca • 02 November 2016
![Page 2: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/2.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Róbert
Dan
![Page 3: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/3.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
ING Software Development Center
Bucharest
3
![Page 4: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/4.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
ING Software Development Center
Bucharest
4
IT 5,7% of GDP with 100k ppl
Agriculture 4,4% of GDP with 1.8 million ppl
![Page 5: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/5.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
5
FAULT
TOLERANCE
PE
RFO
RM
AN
CE
RELIABILITY SECURITY
AVAILABILITY HIGH
SCALABILITY
REPLICATION
CH
AN
GE
SELF HEALING
![Page 6: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/6.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
6
![Page 7: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/7.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
7
![Page 8: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/8.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
8
Account Manager
![Page 9: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/9.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
9
Account Manager
Core Banking
![Page 10: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/10.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
10
Account Manager
Core Banking
Transfer
![Page 11: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/11.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
11
Account Manager
Core Banking
Transfer
![Page 12: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/12.jpg)
12
Source: http://www.slideshare.net/dberkholz/ how-microservices-are-redefining- modern-application-architecture
![Page 13: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/13.jpg)
13
Source: http://www.slideshare.net/dberkholz/ how-microservices-are-redefining- modern-application-architecture
![Page 14: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/14.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
14
![Page 15: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/15.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
500 million messages / day
15
![Page 16: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/16.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
500 million messages / day 700 billion messages / day
16
![Page 17: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/17.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
500 million messages / day 700 billion messages / day 1.4 trillion messages / day
17
![Page 18: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/18.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
500 million messages / day 700 billion messages / day 1.4 trillion messages / day
18
![Page 19: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/19.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
19
Kafka ActiveMQ RabbitMQ
![Page 20: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/20.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
20
Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc
Kafka ActiveMQ RabbitMQ
![Page 21: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/21.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
21
Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
![Page 22: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/22.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
22
Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
![Page 23: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/23.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
23
Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
Transactions
![Page 24: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/24.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
24
Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
Transactions
Filtering
![Page 25: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/25.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Messaging Brokers Features
25
Support for multiple protocols - Advanced Message Queueing Protocol - Streaming Text Oriented Messaging Protocol - Message Queuing Telemetry Transport - Etc
Kafka ActiveMQ RabbitMQ
Java Messaging System
Java Authentication and Authorization Service
Simple Authentication and Security Layer
Transactions
Filtering
Persistence
![Page 26: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/26.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
26
Pe
rfo
rma
nce
Features
![Page 27: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/27.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
27
Messaging Brokers Performance
Message producing Message consumption
LinkedIn – Kafka: a Distributed Messaging System for Log Processing
![Page 28: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/28.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
28
Pe
rfo
rma
nce
Features
![Page 29: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/29.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
29
Usecases of Kafka
Consumer
Microservice A Microservice B Transport API
Producer
![Page 30: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/30.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
30
Usecases of Kafka
Consumer
Microservice A Microservice B Transport API
Producer
Consumer
Producer
![Page 31: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/31.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
31
Usecases of Kafka
Microservice A Microservice B Transport API
Producer
Consumer
Microservice C
Consumer
![Page 32: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/32.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
32
System Availability & Performance
![Page 33: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/33.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
33
System Availability & Performance
![Page 34: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/34.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
34
Scalability
![Page 35: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/35.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
35
Scalability
![Page 36: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/36.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
36
Usecases of Kafka
Microservice A ELK stack Kafka Consumer Kafka Appender
K
A
F
K
A
![Page 37: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/37.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
© ING 2016
37
Producer
// build a 'default' factory based on your-implicit-kafka-producer.properties
// - the properties file contains the list of brokers to connect to
ProducerFactory factory = ProducerFactory.builder().build();
// create a message producer with a Payload that sends the messages to topic ‘test’
Producer<Message<Payload>> producer =
factory.createMessageProducer("test", Payload.class);
// send a Message { header: Map<String, String>; payload: Payload }
producer.send(
Message.<Payload> builder()
.header("applicationId", "sample-producer")
.header("timestamp", now())
.payload(new Payload("/some/file/path"))
.build()
);
![Page 38: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/38.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
38
Producer
Native Kafka Producer
API
your-implicit-producer.properties
your-security.properties
![Page 39: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/39.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
39
Consumer // build a 'default' factory – your-implicit-kafka-consumer.properties
// - the properties contains the list of brokers from which it will receive messages
ConsumerFactory factory = ConsumerFactory.builder().build();
// obtain a consumer instance that can process Message<Payload> messages
Consumer<Message<Payload>> consumer = factory.createMessageConsumer(Payload.class);
// prepare the message (Message { header: Map<String, String>; payload: Payload } ) handler
ReceiverHandler<Message<Payload>> messageReceiver =
new ReceiverHandler<Message<Payload>>() {
@Override
public void onReceive(Message<Payload> message) {
System.out.printf("message received: %s%n", message);
}
};
// register the handler to receive messages from the topic named “test”
consumer.registerHandler("test", messageReceiver);
© ING 2016
![Page 40: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/40.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
40
Consumer
Native Kafka Consumer
API
your-implicit-consumer.properties
your-security.properties
![Page 41: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/41.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
41
KafkaAppender
ELK
KAFKA
Microservice A
Logback
File Appender
Kafka Appender
Kafka Producer Logstash
Fordwarder
![Page 42: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/42.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
43
Mutual TLS - your-security.properties
# The location of the key store file.
kafka.tls.keystore.location=/full/path/to/a/keystore.jks
# The store password for the key store file.
kafka.tls.keystore.password=some-good-password
# The file format of the key store file.
kafka.tls.keystore.type=JKS
# The password of the private key in the key store file
kafka.tls.key.password=the-private-keys-password
# The location of the trust store file.
kafka.tls.truststore.location=/path/to/truststore.p12
# The password for the trust store file.
kafka.tls.truststore.password=some-other-good-password
# The file format of the trust store file
kafka.tls.truststore.type=PKCS12
© ING 2016
![Page 43: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/43.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
44
Serialize
Producer
1
Metadata
N
Metadata
1 N
Transformer Serializer
Metadata Metadata
KAFKA 1 2 3
![Page 44: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/44.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
45
Filter
Microservice A
Kafka Producer
Microservice B
Kafka Consumer
KAFKA
Topic A
Kafka Consumer Kafka Producer
Metadata
Topic B
Filter
![Page 45: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/45.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
Transformer
46
Transformer
Microservice A Kafka
Producer
Microservice B Kafka
Consumer
KAFKA
Topic A Topic B
1
Kafka Consumer Kafka Producer
Metadata
1 2
1 2
2
![Page 46: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/46.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
47
Forward & Dispatch
Microservice A Kafka
Producer
Microservice B Kafka
Consumer
KAFKA
Topic A
Topic C
Kafka Consumer
Metadata
Topic B
Topic N
Kafka Producer
1
1
1 1
Microservice D Kafka
Consumer
Microservice Z Kafka
Consumer
Forward & Dispatch
![Page 47: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/47.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
48
Replay
0 1 2 3 i N N-1
beginning i end
Microservice A
![Page 48: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/48.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
49
Pe
rfo
rma
nce
Features
![Page 49: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/49.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
50
Pe
rfo
rma
nce
Features
FIL
TE
RIN
G
DIS
PA
TC
H
![Page 50: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/50.jpg)
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
No content below the grey line
51
Pe
rfo
rma
nce
Features FI
LT
ER
ING
D
ISP
AT
CH
![Page 51: Devoxx Morocco 2016 - Microservices with Kafka](https://reader034.vdocuments.net/reader034/viewer/2022042605/586f8fcf1a28ab54768b7765/html5/thumbnails/51.jpg)
Max. width
Min. height
Max. height
ING Orange
RGB= 255, 98, 0
ING Light Grey
RGB= 168, 168, 168
ING Indigo
RGB= 82, 81, 153
ING Sky
RGB= 96, 166, 218
Colour Guidelines
ING Fuchsia
RGB= 171, 0, 102
ING Lime
RGB= 208, 217, 60
ING Leaf
RGB= 52, 150, 81
ING Mid Grey
RGB= 118, 118, 118
Text Colour
RGB= 51, 51, 51
Use microservices with Kafka … … code ahead of the curve
László-Róbert ALBERT Dan BALESCU Casablanca • 02 November 2016
@danbalescu https://www.linkedin.com/in/albertlaszlorobert