dice & cloudify – quality big data made easy

DICEHorizon 2020 Research & Innovation ActionGrant Agreement no. 644869http://www.dice-h2020.eu

Funded by the Horizon 2020Framework Programme of the European Union

DICE: quality Big Data made easy

Matej Artač @matej_artacXLAB Research @xlab_research

Michele Guerriero, Damian A. TamburriPolitecnico di Milano

2

Agenda

o Introductiono About DICE

Developing Data-Intensive Cloud Applications with Iterative Quality Enhancements

o DICERo TOSCAo Cloudifyo DICE delivery tools

3

Introduction

4

Building blocks for DIAs today

Coordinator (Kafka)

Orchestrator (Hadoop Cluster)

Data Store

Batch Layer

Speed Layer

Serving LayerServing LayerServing Layer

Data Source

Data Source

Distributedcomputation

Data streaming

HDFS

Distributed storage

Lambda architecture Cloud infrastructure

5

What problems EU SMEs face?

Traditional market: Legacy software systems

Customers with legacydata now ask for Big Data technologies

Growth in sight, but …

Learning curves

Initial prototyping

Risk of failure

(+ others…)

Fast-paced market

Traditional approach to deployment

o Spend time studying documentation

o Use trial and error to set up a working cluster

o Use incompatible public cookbooks

o Repeat for each changeo Keep the Big Data cluster

fixed for fear of breaking it

days

7

What is in the box?

o DICER model your application create a TOSCA blueprint

o Cloudify Pure-play orchestration & automation

o DICE deployment tool alternative front-end for Cloudify

o TOSCA library worries about deploying Big Data

services so that you don’t have to

8

DICEDeveloping Data-Intensive Cloud Applications with Iterative

Quality Enhancements

9

The Rapid Growth of Big Data

o Software market rapidly shifting to Big data 27% compound annual growth rate through 2017 (IDC) Popular technologies such as Spark, Hadoop, and NoSQL

boost Big Data adoption and revenues from new services

Business issue: 65% of Big data projects still fail (CapGemini’15)

Source: IDC Source: Wikibon

10

DevOps toolchains in innovation

Application Release Automation

Continuous Delivery

DevOps Toolchain

11

DICE Mission and Partners

ICT 9 Call/2014 – Software engineering 9 partners (Academia & SMEs), 7 EU countries

Mission: support SMEs in developing high-quality cloud-based data-intensive applications (DIAs)

(IEAT)

(IMP)

(PMI)

(ZAR) (NETF)

(XLAB)

(ATC)

(FLEXI)(PRO)

12

Ingredients of the DICE approach

o DevOps

o Model-Driven EngineeringDev Ops Dev Ops

Analysis

Deployment blueprint

13

DICE incremental modeling and analysis

DICE Platform Independent Model (DPIM)

DICE Technology Specific Model (DTSM)

DICE Deployment Specific Model (DDSM)

is implemented by

is deployed onto

TOSCA blueprint

Analysis

Analysis

Analysis & Optimization

M2M transformation

M2M transformation

M2T transformation

DICE Methodology

14

DICE deployment, monitoring and testing

Deployment

Testbed

Monitoring

Fault Injection

Quality Testing

Trace Checking

Enhancement

Anomaly

Detection

Running DIA

Comp

MW

VM

Running DIA

Comp

MW

VM

Configurationoptimization

TOSCA blueprint DICE M

ethodology

15

DICE architecture

IDEbased on

Eclipse

Profile

Simulation

Optimization

Verification

Repository & CI

Configuration Optimization

Delivery

Running DIA

Comp

Running DIA

Comp

Monitoring

Trace Checking

Enhancement

Anomaly Detection

Fault Injection(Resilience)

Quality Testing

MW

VMMW

VM

MW

VMRunning

DIA Comp

DICER

16

XLAB delivery tools

IDEbased on

Eclipse

Profile

Simulation

Optimization

Verification

Repository & CI

Configuration Optimization

Delivery

Running DIA

Comp

Running DIA

Comp

Monitoring

Trace Checking

Enhancement

Anomaly Detection

Fault Injection(Resilience)

Quality Testing

MW

VMMW

VM

MW

VMRunning

DIA Comp

DICER

17

DICERCreate actionable deployment diagrams of

Data Intensive Applications

18

DICER

o Assisted Component-based infrastructure designo 100% automation of model transformations

19

DICER in action: Assisted modelling!

1. <<Jerry: I’m modelling this DDSM thingie… I don’t know what I’m doing!>>

2. <<DICE: Hey Jerry! You’re missing this piece here… And there!>>

…

20

DICER in action: Assisted modelling!

3. <<DICE: ok that’s better now… Carry on…>>

21

OASIS TOSCAOASIS Topology and Orchestration Specification

for Cloud Applications

22

What is TOSCA?

o Open standardo Enabling a unique Cloud

eco-systemo Supported by a large

and growing number of international industry leaders

Associated Companies

TOSCA is an Intent Model which is declarative (integration points for imperative)

TOSCA Domain-Specific Language

Information ModelsTypically, used to model a constrained

domain that can be described by a closed set of entity types, properties,

relationships and operations.

Data ModelsTypically, describe the structure (format), enabling manipulation (via interfaces) of the data stored in data management systems assuring integrity.

• Topology• Composition • Requirements - Capabilities• State (Nodes, Relationships)• Lifecycle (Management)• Policy

Intent Model Adds:

• Structure • Format• interfaces

• Types, Relationships• Properties• Operations

TOSCA can work with imperative scripts (e.g., Ansible, Chef, Bash, Ant, etc.)

TOSCA can include other data models (e.g., JSON, YANG)

24

Tier (Group Type)

TOSCA is used first and foremost to describe the topology of the deployment view for cloud applications and services

Topology – Nodes and Relationships

source_resource

Node_Type_A

target_resource

Node_Type_B

Requirement

connect_relationship

ConnectsToCapability

Nodes - are the resources or components that will be materialized or consumed in the deployment topology

Relationships express the dependencies between the nodes (not the traffic flow)

Requirement - Capability Relationships can be customized to match specific source requirements to target capabilities

GroupsCreate Logical, Management or Policy groups (1 or more nodes)

Node templates to describe components in the topology structure

Relationship templates to describe connections, dependencies, deployment ordering

25

Application Tier (container)

Application Tier

(container)

Composition – different service templates can be “wired” together

Logging/Monitoring Tier (ELK)

nodejs

WebServer

app_server

Compute

paypal_pizzastore

WebApplication

collectd

logstashSoftwareComponent

Requirements

Container

Capabilitieslog_endpoint

logstash_server

Compute

CapabilitiesContainer

elasticsearchSoftwareComponent

RequirementsContainer

Capabilitiessearch_endpoint

elasticsearch _server

Compute

Capabilities

kibanaSoftwareComponent

Requirements

Container

kibana_server

Compute

Capabilities

search_endpoint

ConnectsTo

HostedOn HostedOn HostedOn

ConnectsTo

mongo_dbms

DBMS

mongo_server

Compute

mongo_db

Database

rsyslog

search_endpoint

ContainerContainer

ConnectsTo

Enabling the description of complex, multi-tier (hybrid) Cloud applications

26

Cloudify

27

Introducing CloudifyOpen | Extensible |

Simple

REPEATABLE

28

ApplicationBlueprint(TOSCA)

IaaSPlugins

ContainerPlugins

Conf MgmtPlugins

● Provision● Configure● Monitor● Manage

Monitoring &Alarming

29

30

Cloudify Key Aspects

Open SourceOpen source is key to drive innovation and create superb quality software.

Open StandardOpen standard-based TOSCA Spec for application blueprints allows vendor neutrality, and enables collaboration.

Future ProofTry new emerging technologies while using stable in place existing ones.

31

The only constant is change” -Unknown

32

DICE Delivery Tools

33

Components of Delivery Tools

RESTful API

IaaS

Web GUITechnology

Library

34

Delivery Service

Container 1

Blueprint A

Platform params

Container 2

Blueprint B

Blueprint B.2

Platform params

Container 15

Blueprint B.2

35

TOSCA technology library

o A plug-in for Cloudifyo A single import line in the TOSCA blueprinto Node types + Chef cookbooks for Big Data serviceso Unified across supported IaaS vendors

©DICE

36

Deploy your own Big Data services

DevOps approach:o Describe your Big Data

cluster and application in a blueprint

o Store and maintain the blueprint in your VCS with the application’s code: IasC

o Rely on orchestrators and configuration managers for executing deployments

hours

42

Demo – DevOps

43

Conclusion

44

Conclusion

o DICE tools remove barriers to Big Datao DICE technology library simplifies blueprintso TOSCA blueprints describe infrastructure as codeo Enabled Continuous Integration and Continuous

Delivery

45

Links

o Cloudify: http://getcloudify.org o DICE H2020: http://www.dice-h2020.eu/

o DICE deployment service:https://github.com/dice-project/DICE-Deployment-Service

o Big Data blueprint examples:https://github.com/dice-project/DICE-Deployment-Examples

o DICER:https://github.com/dice-project/DICER

46

Follow us

o DICE project: @diceh2020o Cloudify:

@CloudifySource – ilanadl@getcloudify.orgo User groups:

https://groups.google.com/forum/#!forum/cloudify-users o Webinars:

http://getcloudify.org/webinars.html

o Matej Artač: @matej_artac – matej.artac@xlab.sio XLAB Research: @xlab_research

47

Q & A