ZiggridProcessing Data in Near Real-Time Using Couchbase

Christopher Tse (Head of R&D, McGraw-HIll Education) Gareth Powell, Ph. D. (Chief Scientist, Ziniki Network)CouchConf SF 2013 - Sep 13, 2013

@ McGraw-Hill Education

+

Research & Development

@ McGraw-Hill Education

+

Research & Development

Leveraging EmberJS, a JavaScript MVC framework to rethink the teaching and learning experiences on the Web and on mobile devices

HTML

Collecting and analyzing multiple streams of student engagement, performance, and demographics for dashboards.

Data

FACT

Dimension

DimensionDimension

Dimension Dimension

Action

Collections

EdSense: Real-time Reactions

Learning Style

Engagement

User Intents

Recommendations ReactionActivity Log

Previous

ly

Achievements Efficacy

Action

Collections

EdSense: Real-time Reactions

Learning Style

Engagement

User Intents

Recommendations

Reaction

Activity Log

Previous

ly

Achievements Efficacy

Learning Portal• Designed and built as a

collaboration between MHE Labs and Couchbase

• Serves as proof-of-concept and testing harness for Couchbase + ElasticSearch integration

• Available for download and further development as open source code

http://github.com/couchbaselabs/learningportal 

Unveiled during CouchConf SF 2012

SQL

Building Content-Driven Apps

Couchbase and Elasticsearch for content and metadata stores

Interactive content-driven apps need to store and share di!erent kinds of content and metadata. Whether you’re designing an online publication, a product catalog, a website landing page, or other digital content, it’s likely you want to support rich, personalized content for millions of users worldwide.

To handle the needs of these types of applications, you’ll want to consider a NoSQL database. In this paper, you’ll learn what makes Couchbase Server a good "t for apps that store and serve content and metadata.

What content-driven apps need

Your database is critical for your app because it is a central store for data and metadata. That means your app needs a database that lets you:

Store unstructured content and metadata. The vast majority of data generated today is unstructured, with a variety of new and changing data types. So, if you’re building a content catalog, you may store tens of millions of di!erent objects in a variety of forms – unstructured content and metadata that may be hierarchical, sparse, free-form text, or varying length. Keeping your app fresh means being able to add new types of data quickly, without dealing with the complexity and time it takes to change a schema. Storing content in a RDBMS requires knowing ahead of time the exact structure of the data your app needs to store, a requirement that is hard to meet for content and metadata apps that must support new data types as they emerge.

Hi!

SQL ETL

SQLSome-sort-of query language

ETLTo extract, transformand load in steps

We mean:

So we can: Declaratively express the logic for the machine to calculate and process

But: Processing complex, multi-layered queries upon request can be slow

Store the results from the intermediate or final steps of our calculations

Stored data gets out-of-sync with reality. And refresh is often expensive

When we say:

SQL ETL

SQL ETLLogic

SQL ETLLogic Steps

SQL ETLLogic Steps

Fresh Data

SQL ETLLogic Steps

Fresh Data Fast Access

SQLETLLogic Stepsin

Fresh Data Fast Access&

FRPLogic Stepsin

Fresh Data Fast Access&

Introduc

ing

Functional Reactive ProgrammingFRP

Functional reactive programming (FRP) is a programming paradigm for reactive programming using the building blocks of functional programming.

The key traits of FRP are:

• The concept of "behaviors" or "signals" which model values that vary over continuous time.

• The concept of "events" which have occurrences at finitely many points in time.

• A means to change the FRP system in response to events, generally termed "switching".• The separation of evaluation details such as sampling rate from the reactive model.

An additional common but contentious trait is a notion of consistency when ordering events (not just within one stream). Variants include synchrony and glitch freedom.

The semantic model of FRP in side-effect free languages is typically in terms of continuous functions, and typically over time. In contrast, integration with a host language that has side-effects is typically given in terms of data flow or dependency graphs by extending the typical operational semantics to manipulate and use them.

WTF is FRP?

Functional reactive programming (FRP) is a programming paradigm for reactive programming using the building blocks of functional programming.

The key traits of FRP are:

• The concept of "behaviors" or "signals" which model values that vary over continuous time.

• The concept of "events" which have occurrences at finitely many points in time.

• A means to change the FRP system in response to events, generally termed "switching".• The separation of evaluation details such as sampling rate from the reactive model.

An additional common but contentious trait is a notion of consistency when ordering events (not just within one stream). Variants include synchrony and glitch freedom.

The semantic model of FRP in side-effect free languages is typically in terms of continuous functions, and typically over time. In contrast, integration with a host language that has side-effects is typically given in terms of data flow or dependency graphs by extending the typical operational semantics to manipulate and use them.

TL;DR

WTF is FRP?

Hint

Excel is FRP

Excel is FRPFunctionalEvery cell is either is a value or a f(x) that generates a value

Excel is FRPFunctional

Reactive

Every cell is either is a value or a f(x) that generates a value

If you change one cell, all the other cells that refer to it changes immediately

Excel is FRPFunctional

Reactive

Every cell is either is a value or a f(x) that generates a value

If you change one cell, all the other cells that refer to it changes immediately

Excel is FRPFunctional

Reactive

Programming

Every cell is either is a value or a f(x) that generates a value

If you change one cell, all the other cells that refer to it changes immediately

Yes, you are programming when you are create a model in an Excel spreadsheet

Start with a simple sum()Adding numbers within one worksheet

Excel is FRP

Start with a simple sum()

Add more tabs

Adding numbers within one worksheet

To reflect higher level aggregates

Excel is FRP

Start with a simple sum()

Add more tabs

Draw fancy graphs

Adding numbers within one worksheet

To reflect higher level aggregates

That visualizes the valuable aggregates

Excel is FRP

The world runs on Excel. :)

The world runs on Excel. : )

What if...

Cells inside Sheets Documents in JSONData Model:

Calculating: When you open the file

Visualization: Supported chart types

All the time in the cloud

Anything drawable in HTML5

Instead of... We have...

=SUM(A1:B10) function Sum() { ... }Language:

What if...

Cells inside Sheets Documents in JSONData Model:

Calculating: When you open the file

Visualization: Supported chart types

All the time in the cloud

Anything drawable in HTML5

Instead of... We have...

=SUM(A1:B10) function Sum() { ... }Language:

Ziggrid is FRP

f(x) f(x)

f(x)

Ziggrid is FRPStores values in JSON

Specifies f(x) in JSON

Inside a Couchbase cluster

Also builds a dependency graph

f(x) f(x)

f(x)

Ziggrid is FRPStores values in JSON

Specifies f(x) in JSON

Inside a Couchbase cluster

Also builds a dependency graph

Push data out via JSONSo clients can render data in HTML5, etc

Ziggrid is FRPStores values in JSON

Specifies f(x) in JSON

Push data out via JSON

Inside a Couchbase cluster

Also builds a dependency graph

So clients can render data in HTML5, etcf(x) f(x)

f(x)

“The Ziggurat”

Ziggrid is FRPStores values in JSON

Specifies f(x) in JSON

Push data out via JSON

Inside a Couchbase cluster

Also builds a dependency graph

So clients can render data in HTML5, etcf(x) f(x)

f(x)

“The Ziggurat”

JS N

Layers of the Ziggurat

Raw Events

Enhanced Events

Summaries

Rankings

Correlations

Snapshots

Composites

Gareth Powell, Ph. D.Functional Programming ExpertWrote doctorate thesis on Haskell

Gareth Powell, Ph. D.Functional Programming ExpertWrote doctorate thesis on HaskellBaseball Fanatic

Example: Baseball Data Analysis Model

Raw Events

Enhanced Events

Summaries

Rankings

Correlations

Snapshots

Composites

Plate AppearancesPlayer

SituationOutcome

Player Totals

Correlate vs Situation

Snapshots of Player Totals

Player Profile

Snapshots of Correlation

Game Results

Leaderboards (HR, AVG, PROD)

Win / Loss Record

LIVE DEMO

Beane Counter Architecture

HTML5 Data Tables and SVG VisualizationEmber.js + D3.js via WebSockets

Mid

dlew

are

Fron

t-end

Model Description, Calculation, and Event ChainingJava via Memcached Protocol

Back

end

Raw and Aggregated Data Storage and IndexingCouchbase JSON Store + Incremental MapReduce

ZiggridModels

• Data model described in JSON structure

{ "name": "plateAppearance", "fields": [ { "name": "team", // The team identifier from the Retrosheet Event file "type": "string", "key": true }, { "name": "player", // The player identifier from the Retrosheet Event file "type": "string", "key": true }, { "name": "season", // Year represented as YYYY "type": "string", "key": true }, { "name": "dayOfYear", // 1-365, proxy for which game it was "type": "number", "key": true }, { "name": "inning", // 1-9 for regular innings "type": "number", "key": true },

...}

JS N

{ "enhanced": "situation", "from": "plateAppearance", "enhance": { "player": "player", "season": "season", "dayOfYear": "dayOfYear", "atbat": { "op": "+", "args": [{ "op": "*", "args": [ 3, "inning" ] }, "outs", -3 ] }, "bases": "bases", "lead": { "op": "group", "value": { "op": "ifelse", "test": "home", "true": { "op": "-", "lhs": "homeScore", "rhs": "awayScore" }, "false": { "op": "-", "lhs": "awayScore", "rhs": "homeScore" } }, "dividers": [ -3, -1, 0, 2 ], // (-inf, -3], (-3, -1], (-1, 0], (0, 2], "moreThan": 3 // (2,inf) },

ZiggridAlgorithms

• Data model described in JSON structure

• Define all calculation via communative and associative operators

JS N

{ "composeInto": "profile", "from": "correlate_on_situation_groupedBy_player_and_season", "key": [ "player/", { "field": "player" } ], "fields": { "clutchness": "correlation" } }, { "leaderboard": "hotness", "from": "snapshot_playerSeasonToDate", "groupby": [ [ "season", "dayOfYear" ] ], "sortby": [ "average" ], "order": "desc", "values": [ "player" ] }, { "composeInto": "profile", "from": "snapshot_playerSeasonToDate", "key": [ "player/", { "field": "player" } ], "fields": { "hotness": "average" } } ...]

ZiggridComposites

• Data model described in JSON structure

• Define all calculation via communative and associative operators

• Projecting data via composite definition

JS N

https://github.com/Ziniki-Network/Ziggrid

Ziggrid is 100%Open SourceLet’s work together!

Future Improvements

Using Couchbase View Engine to do more of the processing in the database via Incremental MapReduce. Currently, only the leaderboards are computed using views.

GREATER SCALABILITY

Expand the functions support by Ziggrid to perform transformation, statistical calculations typical of Big Data analysis, and even ones for machine learning.

Allow in-browser development of new models using a subset of data. We need to finish developing a pure JavaScript-based Ziggrid processing engine.

Using UPR protocol to be notified of changes in inside Couchbase to allow more immediate, and thus more real-time propagation of events up the Ziggurat.

EASIER MODELDEVELOPMENT

REDUCEDLATENCY

DEEPERANALYTICS

Hadoop

HadoopBig Data

HadoopBig DataBut Slow

Zebrasfor

Thanks to 2 members of the Ember.js Core Team Who helped us design and code the sexy Ember + D3.js + WebSockets front-end

@machty

@stefanpenner

Questions?

@christseFollow me on Twitter