realtime bigdata use cases for retail

Catch Them in the Act: CEP for Realtime Ecommerce

InfluenceSrinath Perera

Director, ResearchWSO2 Inc.

Retail• World 7B People

– 6.5B have a phone– 2.5B online

• Global E-commerce itself was 1.5 Trillion market (215$/year for everyone in the world).

Usecases

Recommendations • One of the main retail

scenarios, can be extended to targeted marketing and deals

• Search let us make sense of unstructured Web

• In retail, there are too many choices, Recommendations help us make a decision.

A Main driver of traffic (e.g. 70% amazon site is devoted for recommendations, 70% Netflix sales though recommendations)

Finding Recommendation

• Problem: based on customer activities (purchases, ratings, views) etc., find out products that are of interest to him– Done by finding similar items or similar users (common

case)

• There are often Millions of customers and even millions of items

Recommendations (Collaborative Filtering)

What if you can freeze time!!• Most solutions are overnight • Think how you would buy something!

Research a bit and buy, often overnight is too late.

• But not all trends takes time – People change their mind – Trends move fast – React to what customer is doing (do not let him

move away)

• At CEP speed 400k/sec, if each event takes a second, it takes 4 days to pass a second in real world!!

Big data Processing Technologies Landscape

Real-time Analytics

• Idea is to process data as they are received in streaming fashion

• Used when we need – Very fast output – Lots of events (few 100k to millions)– Processing without storing (e.g. too much data)

• Two main technologies– Stream Processing (e.g. Strom,

http://storm-project.net/ )– Complex Event Processing (CEP)http://wso2.com/products/complex-event-processor/

Complex Event Processing (CEP)• Sees inputs as Event streams SQL like Queries language • Supports Filters, Windows, Join, Patterns and Sequences • Fast, 300K+/sec over network per node!

from p=PINChangeEvents#win.time(3600) join t=TransactionEvents[p.custid=custid][amount>10000]

#win.time(3600)return t.custid, t.amount;

Lambda Architecture with WSO2 Products

WSO2 Big Data Platform

Demo Usecases

• Operations Dashboard • Automatic Offers• Recommendations

based on personal trends

• Recommendations based on overall trends

Architecture

Demo

Usecase 1: Dashboard

• CEP will do realtime analytics – Most queries are look very SQL like

• Then push updates to Dashboard.

from transactionStream[metadata== 'purchase' or metadata== 'select']#window.time(1 min)

select tag, count(*) as count group by tag

Recommendations (Collaborative Filtering)

Realtime Recommendations

• As part of Collaborative filtering, data is broken into similar groups– E.g. Via collaborative filtering – This is done batch style, e.g. once a day

• For realtime matching, we will search only within that group

Usecase 2 & 3: Realtime Recommendations with CEP

Usecase 4: Automatic Offers

• Often pretty straight forward CEP scenario! Track and trigger when conditions match.

define partition transactionStream.customerID{from transactionStream#window.time(1 week)

having sum(amount) > 20 or count (*) > 3

insert into OfferStream}

Usecases

Thank You