real-time financials with microservices and functional … · with microservices and ... - written...
TRANSCRIPT
![Page 1: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/1.jpg)
Real-time Financials with Microservices and Functional ProgrammingVitor Guarino [email protected] @ura1a https://nubank.com.br/
![Page 2: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/2.jpg)
MAIN PRODUCT Live since September 2014
![Page 3: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/3.jpg)
A TECHNOLOGY DRIVEN APPROACH TO FINANCIAL SERVICES
![Page 4: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/4.jpg)
CONTINUOUS DELIVERY
![Page 5: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/5.jpg)
MICROSERVICES
![Page 6: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/6.jpg)
INDEPENDENTLY AND CONTINUOUSLY DEPLOYABLE
![Page 7: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/7.jpg)
DECOUPLED AND EASY TO REPLACE
![Page 8: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/8.jpg)
BOUNDED BY CONTEXT AND INDEPENDENTLY DEVELOPED
![Page 9: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/9.jpg)
WHAT HAPPENS WHEN WE NEED TO COMBINE DATA ACROSS SEVERAL SERVICES?
ESPECIALLY IN REAL-TIME
![Page 10: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/10.jpg)
- Running on AWS, 2 AZs, config as code, immutable infra, horizontally scalable, sharded by customers
SERVICE ARCHITECTURE
- Producer/Consumer to Kafka
- REST APIs
REST
- Written in Clojure (functional)
- Persistence with Datomic
![Page 11: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/11.jpg)
DATOMIC
- Immutable, append-only database
- ACID on writes (atomic, consistent, isolated, durable)
Lucas Cavalcanti & Edward Wible - Exploring four hidden superpowers of Datomic
- A database that works a lot like
![Page 12: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/12.jpg)
The Problem
![Page 13: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/13.jpg)
WE HAVE OVER 90 SERVICES
![Page 14: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/14.jpg)
THE PROBLEM:A LOT OF BUSINESS LOGIC DEPENDS ON DATA
ACROSS MANY SERVICES
Should I authorize a purchase? Should I block a card? Should I charge interest?
Purchases InterestChargebacksPayments Currencies
![Page 15: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/15.jpg)
THE PROBLEM:WE ARE SHOWING THESE NUMBERS TO THE
CUSTOMER IN REAL TIME
![Page 16: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/16.jpg)
THE PROBLEM:NO CANONICAL DEFINITION OF OUR KEY
NUMBERS
- Ad-hoc definitions created by analysts and engineers
- Analysis vs. operational definition gap
- Nubank, investors, customers, and regulators are all worried about the same numbers.
![Page 17: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/17.jpg)
A BALANCE SHEET IS THE CANONICAL WAY OF REPRESENTING FINANCIAL INFO
- We can apply generally accepted accounting principles (verifiable, unbiased)
- Conservation of money (every credit should have a debit)
- One of the original event-sourced systems
LIABILITYASSET
EQUITY
![Page 18: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/18.jpg)
THE MODEL
- Entry: represents a debit and a credit to two book-accounts
- Meta-entity: it’s a reference to the external entity that originated the event
- Movement: a collection of entries. Maps one Kafka message to one db transaction
- Book-account: A customer owned balance sheet accountex: cash, prepaid, late, payable
-Algebraic Models For Accounting Systems by Salvador Cruz Rambaud and José Garcia Pérez
- Balance: cumulative sum of entries of a book account
![Page 19: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/19.jpg)
Double-entry accountingservice
![Page 20: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/20.jpg)
OUR GOAL FOR OUR ACCOUNTING LEDGER (aka DOUBLE-ENTRY SERVICE)
- High availability to other services, clients, and analysts in real-time
- Resilient to distributed systems craziness
- Traceability of when and why we were inconsistent (strong audit trail)
- Event-driven, via kafka. (we could subscribe to existing topics)
![Page 21: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/21.jpg)
THE IDEAL FLOW
f(payload)
MOVEMENT[ ]
EVENT
ACID transaction
- Event ordering doesn’t matter
- No mutable state
- Needs to guarantee all events are consumed
- Thread safe
![Page 22: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/22.jpg)
{:purchase {:id (uuid) :amount 100.0M :interchange 1M :post-date "2016-12-01"}}
Initial Balances: Current Limit R$ 1000, Current Limit Offset R$ 1000
Final Balances: Current Limit: R$ 900, Current Limit: Offset R$ 900Settled Purchase: R$ 100, Payable: R$ 99, Interchange Revenue: R$ 1
[{:entry/id (uuid) :entry/amount 100.0M :entry/debit-account :asset/settled-purchase :entry/credit-account :liability/payable :entry/post-date "2016-12-01" :entry/movement new-purchase}
recognizereceivable/payable
{:entry/id (uuid) :entry/amount 100M :entry/debit-account :liability/current-limit :entry/credit-account :asset/current-limit :entry/post-date "2016-12-01" :entry/movement new-purchase}
reduce limit
{:entry/id (uuid) :entry/amount 1M :entry/debit-account :liability/payable :entry/credit-account :pnl/interchange-revenue :entry/post-date "2016-12-01" :entry/movement new-purchase}]
recognizerevenue
![Page 23: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/23.jpg)
WE CAN'T GUARANTEE CONSISTENCY, BUT WE CAN MEASURE IT
f(payload)
MOVEMENT[ ]
ACID transaction
- Kafka Lag
- Service downtime
- Processing time
produced-at vs. consumed-at
post-date vs. produced-at
consumed-at vs. db/txInstant
![Page 24: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/24.jpg)
PURE FUNCTIONS OF THE PAYLOAD WON'T ALWAYS WORK
![Page 25: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/25.jpg)
The Stateful Flow
![Page 26: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/26.jpg)
{:payment {:id (uuid) :amount 150.00M :post-date "2016-12-01"}}
[{:entry/id (uuid) :entry/amount 100.0M :entry/debit-account :asset/cash :entry/credit-account :asset/late :entry/post-date "2016-12-01" :entry/movement new-payment}
amortize debt
{:entry/id (uuid) :entry/amount 100M :entry/debit-account :asset/current-limit :entry/credit-account :liability/current-limit :entry/post-date "2016-12-01" :entry/movement new-payment}
increase limit
{:entry/id (uuid) :entry/amount 50M :entry/debit-account :asset/cash :entry/credit-account :liability/prepaid :entry/post-date "2016-12-01" :entry/movement new-payment}]
recognizeprepaid amount
Initial Balances: Current Limit: R$ 900, Current Limit Offset: R$ 900Late: R$ 100, Payable: R$ 99, Interchange Revenue: R$ 1
Final Balances: Current Limit: R$ 1000, Current Limit Offset: R$ 1000Cash: R$ 150, Prepaid R$ 50, Payable: R$ 99, Interchange Revenue: R$ 1
![Page 27: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/27.jpg)
{:payment {:id (uuid) :amount 150.00M :post-date "2016-12-01"}}
[{:entry/id (uuid) :entry/amount 100.0M :entry/debit-account :asset/cash :entry/credit-account :asset/late :entry/post-date "2016-12-01" :entry/movement new-payment}
amortize debt
{:entry/id (uuid) :entry/amount 100M :entry/debit-account :asset/current-limit :entry/credit-account :liability/current-limit :entry/post-date "2016-12-01" :entry/movement new-payment}
increase limit
{:entry/id (uuid) :entry/amount 50M :entry/debit-account :asset/cash :entry/credit-account :liability/prepaid :entry/post-date "2016-12-01" :entry/movement new-payment}]
recognizeprepaid amount
Initial Balances:
Late: R$ 100
Final Balances:
Cash: R$ 150, Prepaid R$ 50
![Page 28: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/28.jpg)
THE STATEFUL FLOW
- Movements in the past will modify all future balances
- Adapters are a function of the event payload AND current balances
- Balances can’t change during calculations
- Can’t allow for data to be corrupted depending on the order of the events
INVARIANTS
![Page 29: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/29.jpg)
INVARIANTS
- Some balances can’t coexist (no late alongside prepaid)
- We can establish invariants that must hold true at all times
- Some balances can’t be negative (cash)
- Some can’t be positive (credit-loss)
![Page 30: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/30.jpg)
THE STATEFUL FLOW
EVENT
f(payload, )
ACID transaction
f( )VALID STATE?
FIX VIOLATION
INVARIANT VIOLATIONS?
NO
MOVEMENT[ ]Cr: Late Dr: CashR$ 150
Cr: LateDr: CashR$ 150
MOVEMENT WITH CORRECTION[ ]Cr: Prepaid
Dr: LateR$ 50
Initial Balances: Late: R$ 100
Final Balances: Cash: R$ 150, Prepaid R$ 50
[ ]VIOLATIONS YES
Negative Late
Balance
![Page 31: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/31.jpg)
CHALLENGES
![Page 32: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/32.jpg)
CHALLENGES
- Fixing invariants logic is extremely complex.
- Datomic indexing is tested until 10 billion facts.
- Datomic isn’t the best option for analytical workload, especially with sharded dbs
- Other services bugs may generate incorrect entries that will need to be fixed
![Page 33: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/33.jpg)
GENERATIVE TESTING
- We generate random events from our schemas (bill, purchases, payments, etc)
- Write a function that describes a property that should always hold true instead of describing input and expected output,
- Properties that should hold true are the same invariants that are guaranteed in prod
- Embed the least amount of domain logic assumptions
![Page 34: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/34.jpg)
GENERATIVE TESTING
(def balances-property (prop/for-all [account (g/generator Account) events (gen/vector (gen/one-of [(g/generator Purchase) (g/generator Payment) ...]))] (->> datomic (consume-all! account events) :db-after (balances-are-positive!)))
(fact (tc/quick-check 500 balances-property) => (th/embeds {:result true}))
(ns double-entry.controllers.rulebook-test (:require [midje.sweet :refer :all] [clojure.test.check.properties :as prop] [clojure.test.check :as tc] [schema-generators.generators :as g] [clojure.test.check.generators :as gen]))
![Page 35: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/35.jpg)
MONITORING / REPLAY HISTORY TOOLING
- Other services have republish endpoints (same payload and meta data as original thanks to datomic)
- We set sanity checks to make sure events aren’t missing
- We have an endpoint that can retract all entries for a customer(resets business timeline, but not DB)
![Page 36: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/36.jpg)
SHARDING BY CUSTOMER / TIME
- No cross customer entries allows for per customer sharding
- We shard the database by time fairly often.
- simple representation of the end state of the customer at a time shard: final balance of each of the book accounts
- As time passes, any single customer’s db will approach infinite datoms
![Page 37: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/37.jpg)
ETL
extract logs
facts to table(one per entity type)
tables stored
applies functions to generate balances
balances on redshift*
* also accessible through metabase
![Page 38: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/38.jpg)
The Result
![Page 39: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/39.jpg)
Text placeholder
2015-01 2015-04 2015-07 2015-10 2015-01 2016-03
REAL TIME BALANCE SHEET
![Page 40: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/40.jpg)
2 TIMELINES
ACTUAL (DB) TIME audit trail / Datomic log “when did we know”day 0 day 30 day 90
BUSINESS TIME official version of events uses business-relevant “post dates” can correct after the fact
day 0 day 30
![Page 41: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/41.jpg)
WHAT WE LIKE
- Canonical definition of our most important numbers
- Financial analysis applied at a the customer level in real-time
- Business-specific invariants provide safety
- Generative testing finds real bugs
- Ability to replay history for a customer without losing data
- Shardable by time and by customer
- Extensible to other products (some don’t require stateful approach)
- Inconsistency traceability allows us to react to it
![Page 42: Real-time Financials with Microservices and Functional … · with Microservices and ... - Written in Clojure (functional) - Persistence with Datomic. DATOMIC - Immutable, append-only](https://reader036.vdocuments.net/reader036/viewer/2022081613/5fbe89ff4c9fe44ced530063/html5/thumbnails/42.jpg)
42
nubank.com.br/jobs
@ura1a
https://gist.github.com/ura1a to get snippets of our domain!
THANK YOU!