NIST BIG DATA WGReference Architecture Subgroup

Draft

Co-chairs:Orit Levin (Microsoft)James Ketner (AT&T)Don Krapohl (Augmented Intelligence)

August 8, 2013

From the previous subgroups call:Next Steps and AIs• Deliverable I: Write the White Paper draft showing one or more (e.g., Data Flow and Stack

approaches) using the same or similar terminology• AI: Chairs will start the draft of the document incorporating the submissions to the Ref Arch subgroup. Is

it still relevant?• AI: Close cooperation between “Ref Arch” and “Def&Tax” sub-groups to produce the Output: taxonomy

for the RA diagrams with definitions for major entities/blocks; Input: M-0057. Partially Done.

• Deliverable II: A draft of a single RA requires more discussion and inputs based on the work of all sub-groups• AI: Chairs will start the draft of the document incorporating the findings of the Ref Arch subgroup. To be

discussed today.• AI: Review the latest contributions to the Ref Arch and incorporate their findings (See email from Yuri

Demchenko / University of Amsterdam) Done.• AI: Close cooperation with the “Use Cases” and “Security” sub-groups to identify the areas of focus for

“zooming” into their architecture Still TBD

2NIST Big Data WG / Ref Arch Sub-group8/8/2013

Reference Architecture ObjectivesNo change• Addresses a broad range of stakeholders (e.g., data owners, industries,

academia, policy makers)• Wide scope:• Encompasses the whole data life cycle or in the ecosystem• Can be applied to different use cases (including various verticals)• Represents different system architectures (e.g., an enterprise data warehouse,

distributed cloud-based system using multiple service providers)• Focus• Potentially with initial focus on the Big Data analytics and tools• Assists in identifying security and privacy issues

• Agnostic to any specific technologies

38/8/2013 NIST Big Data WG / Ref Arch Sub-group

Draft Agreement / Rough ConsensusNo change

• Transformation includes• Processing functions• Analytic functions• Visualization functions

• Data Infrastructure includes• Data stores• In-memory DBs• Analytic DBs

NIST Big Data WG / Ref Arch Sub-group 4

Sources

Transformation

Usage

Data

Infra

stru

ctur

e

Secu

rity

Man

agem

ent

Clou

d Co

mpu

ting

Net

wor

k

8/8/2013

RA: Possible Next Level Of Details for DiscussionThe ideas are based on various input documents to NIST BD subgroups; nothing has been finalized; additional input is solicited

NIST Big Data WG / Ref Arch Sub-group 5

Secu

rity

Man

agem

ent

Clou

d Co

mpu

ting

Net

wor

k

8/8/2013

Sources

Transformation

Usage

Data Infrastructure

Collection

Export

Curation

Pre-analytics

Visualization

Data

Man

ager

Stor

age

(disk

, mem

ory,

etc.

)

File

Sys

tem

Spec

ializ

ed A

bstr

actio

ns

Data Mining

Analytics

Buffe

r Man

ager

Map

Red

uce

Rela

tiona

l DB

NoS

QL

DB

Aggregation

Integration

Transfer

Search

Statistics

RT Analytics

VOLUMEVARIETY

VELOCITY

Streaming

Interactive An.

Batch Analytics

Next Steps and AIs• Deliverable I: Write the White Paper draft showing one or more (e.g., Data Flow

and Stack approaches) using the same or similar terminology• AI: Chairs will start the draft of the document incorporating the submissions to the Ref

Arch subgroup. Is it still relevant?• AI: Close cooperation between “Ref Arch” and “Def&Tax” sub-groups to produce the

Output: taxonomy for the RA diagrams with definitions for major entities/blocks; Input: M-0057. Continue.

• Deliverable II: A draft of a single RA requires more discussion and inputs based on the work of all sub-groups• AI: Chairs will start the draft of the document incorporating the findings of the Ref Arch

subgroup. To be discussed today.• AI: Close cooperation with the “Use Cases” and “Security” sub-groups to identify the

areas of focus for “zooming” into their architecture Remaining Action Item

6NIST Big Data WG / Ref Arch Sub-group8/8/2013

Backup Slides

7NIST Big Data WG / Ref Arch Sub-group8/8/2013