rt analytics with storm and redis @ big data bucharest meetup

RT Analytics with Data Streaming Felix Crisan / Co-Founder & CTO NETOPIA[mobilPay, web2sms, kartela.ro]

About Us

- Largest SMS Aggregator in RO (+100MM/year) - Largest mobile payments processor in RO (own wallet and SMS)- Second largest online card processing gateway in RO (+2M trx / year)

RT Analytics @ mobilPay

Is it BigData?

What do we use it for

- Operational Insights- Transaction Monitoring- Health Checks- Business Dashboards (internal for now, GA later this year)

Storm one-pager

- Streaming processor, Apache Incubator (http://storm.incubator.apache.org/)- Zookeeper/Nimbus(+ZeroMQ/Netty)- Scalable / No Data Loss (guaranteed)- Fault - Tolerant- Topologies (~ MR job) / Streams / Spouts (data sources) / Bolts (data processors)

Redis one-pager

- In-Memory (disk persistent) KV Store (http://redis.io)- Values can be complex types (strings, hashes, lists, sets, sorted sets)- Single-threaded (!) but high-performance (500k req/s)- Atomic operations

(Simplified) Data Model

Our Data Streaming setup

What Happens

- Transaction details are put in a RabbitMQ queue (state, errors, values etc)- A Storm Topology reads the details and aggregates

- per Merchant / Service - per Status / Error / Currency- per time interval (5min, H, D, M)

Why did we choose this setup?

So what’s the result?

Know Your Queries

Better insights

Clear(er) views

Better predictions

KYC

Summary

Thank you!felix@mobilpay.com