databases onions - goto conference · integrating microservices via rpc • avoids problems of...
TRANSCRIPT
![Page 1: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/1.jpg)
@tlberglund
Commit Logs Microservices
in an age of
DatabasesOnionsare like
![Page 2: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/2.jpg)
Normal Applications(i.e., monoliths)
![Page 3: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/3.jpg)
Monoliths are hard to think about.
![Page 4: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/4.jpg)
Monoliths are hard to change.
![Page 5: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/5.jpg)
Re-integration?
![Page 6: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/6.jpg)
Re-integration
![Page 7: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/7.jpg)
There are no good ways to integrate microservices.
![Page 8: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/8.jpg)
There are no-good ways to integrate microservices.
![Page 9: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/9.jpg)
Filesystem
![Page 10: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/10.jpg)
Database
![Page 11: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/11.jpg)
Integrating Microservices through the database
• Easy. “I have a database and I know how to use it.” • Problem: eventually causes services to co-mingle. • Results in violating the “bounded context.” • Great to use inside a service boundary! • Terrible for sharing data or negotiating change.
![Page 12: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/12.jpg)
RPC
![Page 13: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/13.jpg)
Integrating microservices via RPC
• Avoids problems of database integration • Feels natural given imperative programming
sensibilities • Aligns with the request/response paradigm • Problem: cascading failures • Question: how do you debug this system? • Answer: you build a log.
🤔
![Page 14: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/14.jpg)
Events?
![Page 15: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/15.jpg)
Events.
![Page 16: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/16.jpg)
What’s an event?
![Page 17: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/17.jpg)
A shared narrative describing the
evolution of the business over
time.
![Page 18: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/18.jpg)
A combination of:• Notification • State transfer
![Page 19: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/19.jpg)
Also, events are immutable.
![Page 20: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/20.jpg)
Kafka Basics
![Page 21: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/21.jpg)
What is a Streaming Platform?
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
![Page 22: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/22.jpg)
Kafka’s Distributed Log
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
![Page 23: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/23.jpg)
The log is a simple idea
Messages are added at the end of the log
Old New
![Page 24: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/24.jpg)
Consumers have a position all of their own
Sally is here
George is here
Fred is here
Old New
Scan Scan
Scan
![Page 25: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/25.jpg)
Only Sequential Access
Old NewRead to offset & scan
![Page 26: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/26.jpg)
Shard data to get scalability
Producer (1) Producer (2) Producer (3)
Cluster of machines Partitions live on
different machines
Messages are sent to different partitions
![Page 27: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/27.jpg)
Linearly Scalable Architecture
Consumers
Producers
KAFKA
Single topic: - Many producers machines - Many consumer machines - Many Broker machines No Bottleneck!!
![Page 28: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/28.jpg)
The Connect API
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
![Page 29: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/29.jpg)
Ingest / Output to practically any data source
Kafka Connect
Kafka ConnectKafka
![Page 30: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/30.jpg)
Stream Processing in Kafka
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
![Page 31: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/31.jpg)
SELECT card_number, count(*) FROM authorization_attempts WINDOW (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
KSQL: an engine for continuous computation
![Page 32: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/32.jpg)
Kafka Streams: a Java API for the same
public static void main(String[] args) { StreamsBuilder builder = new StreamsBuilder(); builder.stream(”caterpillars")
.map((k, v) -> coolTransformation(k, v))
.to(“butterflies”);
new KafkaStreams(builder.build(), props()).start(); }
![Page 33: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/33.jpg)
Microservices
![Page 34: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/34.jpg)
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine
Let’s build microservices on Kafka
![Page 35: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/35.jpg)
Suppose we have these services
CustomerService
ShippingService
![Page 36: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/36.jpg)
Many services share the same core facts
OrdersCustomers
Catalog
Most services live
in here
![Page 37: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/37.jpg)
Kafka works as a Backbone for Services to exchange Events
Kafka
Notification
State
![Page 38: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/38.jpg)
Recall that events wear two hats
Notification State
![Page 39: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/39.jpg)
ECommerce Microservices (with RPC)
Submit Order
shipOrder() getCustomer()
Orders Service
Shipping Service
Customer Service
Webserver • Orders Service calls Shipping Service to tell it to ship item.
• Shipping service looks up address to ship to (from Customer Service)
• No Kafka 😢
![Page 40: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/40.jpg)
Refactoring Orders and Shipping
Message Broker (Kafka)
Submit Order
Order Created
getCustomer()RPC
Notification
Orders Service
Shipping Service
Customer Service
Webserver
KAFKA
• Orders Service no longer knows about the Shipping service (or any other service). Events are fire and forget.
![Page 41: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/41.jpg)
Refactoring Customers
Customer Updated
Submit Order
Order Created
State
Orders Service
Shipping Service Customer
Service
Webserver
KAFKA
• Call to Customer service is gone.
• Instead data is replicated, as events, into the shipping service, where it is queried locally.
![Page 42: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/42.jpg)
Events are the key to scalable service ecosystems
Sender has no knowledge of who consumes the event they send. This decouples the system.
Orders Service
![Page 43: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/43.jpg)
![Page 44: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/44.jpg)
What’s a database anyway?
![Page 45: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/45.jpg)
SQL
![Page 46: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/46.jpg)
Tables
![Page 47: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/47.jpg)
Storage Engine
![Page 48: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/48.jpg)
Commit Log
![Page 49: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/49.jpg)
Consider this simple system
POST
GET
Load
Ba
lanc
er
ORD
ERS
ORD
ERS
OV
TOPI
C
Order ValidationsKAFKA
INVENTORY
Orders
Inventory
Fraud Service
Order Details Service
Inventory Service
(see previous figure)
Order Created
Order Validated
Orders View Q in CQRS
Orders Service C is CQRS
Find the code online: https://github.com/confluentinc/kafka-streams-examples/tree/3.3.0-post/src/main/java/io/confluent/examples/streams/microservices
![Page 50: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/50.jpg)
What are these things?
POST
GET
Load
Ba
lanc
er
ORD
ERS
ORD
ERS
OV
TOPI
C
Order ValidationsKAFKA
INVENTORY
Orders
Inventory
Fraud Service
Order Details Service
Inventory Service
(see previous figure)
Order Created
Order Validated
Orders View Q in CQRS
Orders Service C is CQRS
Find the code online: https://github.com/confluentinc/kafka-streams-examples/tree/3.3.0-post/src/main/java/io/confluent/examples/streams/microservices
![Page 51: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/51.jpg)
You are not just writing microservices.
![Page 52: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/52.jpg)
You are building an inside-out database.
![Page 53: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/53.jpg)
And that is a good thing.
![Page 54: Databases Onions - GOTO Conference · Integrating microservices via RPC • Avoids problems of database integration • Feels natural given imperative programming sensibilities •](https://reader030.vdocuments.net/reader030/viewer/2022040913/5e89a9b783dc3d428f3104f7/html5/thumbnails/54.jpg)
THANK YOUhttp://confluent.io/ksql
https://slackpass.io/confluentcommunity
@tlberglund