Reference Architecture Subgroup

NIST Big Data Public Working Group

Reference Architecture Subgroup

September 30, 2013

Co-chairs:Orit Levin MicrosoftJames Ketner AT&TDon Krapohl Augmented Intel

Reference Architecture Subgroup 2

Agenda

• Deliverable #1: White Paper: Survey of Existing Big Data RAs

• Deliverable #2: NIST Big Data Reference Architecture• Next Steps

Reference Architecture Subgroup

NIST SurveyBig Data Architecture Models

Input Document M0151

Reference Architecture Subgroup 4

List Of Surveyed Architectures

• Vendor-neutral and technology-agnostic proposals– Bob Marcus ET-Strategies– Orit Levin Microsoft– Gary Mazzaferro AlloyCloud– Yuri Demchenko University of Amsterdam

• Vendors’ Architectures– IBM– Oracle– Booz Allen Hamilton– EMC– SAP– 9sight– LexusNexis

Reference Architecture Subgroup 5

Vendor-neutral and Technology-agnostic Proposals

Data Processing Flow

M0039

Data Transformation Flow

M0017

IT StackM0047

Reference Architecture Subgroup 6

Vendor-neutral and Technology-agnostic Proposals

Data Processing Flow

M0039

Data Transformation Flow

M0017

IT StackM0047

Reference Architecture Subgroup 7

Vendor-neutral and Technology-agnostic Proposals

Data Processing Flow

M0039

IT StackM0047

Data Transformation Flow

M0017

Reference Architecture Subgroup 8

Vendor-neutral and Technology-agnostic Proposals

Data Transformation Flow

M0017

IT StackM0047

Data Processing Flow

M0039

Reference Architecture Subgroup 9

Draft Agreement / Rough Consensus

• Transformation includes– Processing functions– Analytic functions– Visualization functions

• Data Infrastructure includes– Data stores– In-memory DBs– Analytic DBs

Sources

Transformation

Usage Data

In

frast

ruct

ure

Secu

rity

Man

ag

em

en

t

Clo

ud

Com

pu

tin

g

Netw

ork

Reference Architecture Subgroup

NIST BIG DATAReference Architecture

Input Document M0226

Reference Architecture Subgroup 11

• A superset of a “traditional data” system

• A representation of a vendor-neutral and technology-agnostic system

• A functional architecture comprised of logical roles

• Applicable to a variety of business models– Tightly-integrated enterprise

systems– Loosely-coupled vertical

industries

• A business architecture representing internal vs. external functional boundaries

• A deployment architecture

• A detailed IT RA of a specific system implementation

All of the above will be developed in the next stage in the context of specific use cases.

What the Baseline Big Data RAIs Is Not

Reference Architecture Subgroup

Main Functional Blocks

12

Big Data Application Provider

System Orchestrator

Data

C

on

su

mer

Data

P

rovid

er

Big Data Framework Provider

• Application Specific• Identity Management &

Authorization• Etc.

• Discovery of data• Description of data• Access to data• Code execution on data• Etc.

• Discovery of services• Description of data• Visualization of data• Rendering of data• Reporting of data• Code execution on data• Etc.

• Analytic processing of data• Machine learning• Code execution on data et situ• Storage, retrieval, search, etc.

of data• Providing computing

infrastructure• Providing networking

infrastructure• Etc.

Reference Architecture Subgroup

Big Data Lifecycle

13

Big Data Application Provider

System Orchestrator

DATA

DATA

Visualization Access

AnalyticsCuration Collection D

ata

C

on

su

mer

Data

P

rovid

er

Big Data Framework Provider

DA

TA

Reference Architecture Subgroup

Big Data Frameworks

14

Big Data Application Provider

Visualization Access

AnalyticsCuration Collection

System Orchestrator

DATA

DATA

Data

C

on

su

mer

Data

P

rovid

er

Horizontally Scalable (VM clusters)

Vertically Scalable

Horizontally Scalable

Vertically Scalable

Horizontally Scalable

Vertically Scalable

Processing Frameworks (analytic tools, etc.)

Platforms (databases, etc.)

Infrastructures

Physical and Virtual Resources (networking, computing, etc.)

DA

TA

Big Data Framework Provider

Reference Architecture Subgroup

Bringing Tools to the Data

15

Big Data Application Provider

Visualization Access

AnalyticsCuration Collection

System Orchestrator

DATASW

DATASW

Data

C

on

su

mer

Data

P

rovid

er

Horizontally Scalable (VM clusters)

Vertically Scalable

Horizontally Scalable

Vertically Scalable

Horizontally Scalable

Vertically Scalable

Big Data Framework ProviderProcessing Frameworks (analytic tools, etc.)

Platforms (databases, etc.)

Infrastructures

Physical and Virtual Resources (networking, computing, etc.)

DA

TA S W

Reference Architecture Subgroup

Ma

na

ge

me

nt

Se

cu

rit

y &

P

riv

ac

y

16

Big Data Application Provider

Visualization Access

AnalyticsCuration Collection

System Orchestrator

DATASW

DATASW

INFORMATION VALUE CHAIN

IT V

AL

UE

C

HA

IN

Data

C

on

su

mer

Data

P

rovid

er

Horizontally Scalable (VM clusters)

Vertically Scalable

Horizontally Scalable

Vertically Scalable

Horizontally Scalable

Vertically Scalable

Big Data Framework ProviderProcessing Frameworks (analytic tools, etc.)

Platforms (databases, etc.)

Infrastructures

Physical and Virtual Resources (networking, computing, etc.)

DA

TA S W

Reference Architecture Subgroup 17

Outline

Executive Summary1 Introduction2 Big Data System Requirements3 Conceptual Model4 Main Components

4.1 Data Provider4.2 Big Data Application Provider4.3 Big Data Framework Provider4.4 Data Consumer4.5 System Orchestrator

5 Management5.1 System Management5.2 Lifecycle Management

6 Security and Privacy7 Big Data TaxonomyAppendix A: Terms and DefinitionsAppendix B: AcronymsAppendix C: ReferencesAppendix D: Deployment Considerations1 Big Data Framework Provider1.1 Traditional On-Premise Frameworks1.2 Cloud Service Providers

Reference Architecture Subgroup 18

Summary

• Summary– The NIST Big Data functional reference architecture (RA

v.1.0) is available for review as input document M0226.

• Next Steps– Continue the editorial and alignment effort– Map generic Big Data use cases to RA– Map specific collected Big Data cases to RA

Let’s exchange additional ideas this afternoon at the breakout session!