spring xd - today and tomorrow

SPRINGONE2GX WASHINGTON, DC

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spring XD – Today and Tomorrow By Mark Pollack

@markpollack

Safe Harbor Statement

The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.

2

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Contents

•  Big Data ecosystem driving new features

•  Cloud Native principles driving next generation architecture

3

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Since last year…

4

Release Date Features

1.0 7/30/15 Stream & Batch processing, Analytics, Machine Learning DSL, Shell Runtime: Standlone FT, YARN

1.1 2/15/15 Kafka Message Bus Reactor, RxJava, Spark, Python processing. Sqoop Integration Role based authorization

1.2 6/17/15 Flo UI Performance Improvements Ambari HAWQ/GPDB Integration

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Flo Overview

•  HTML 5 canvas application

•  Drag and Drop interface

•  Health Monitoring

•  Visualization of Metrics

•  Also available for SI

5

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Python

•  XD 1.0 provides evaluation of models in PMML format

•  Java Evaluator Predictive Model Markup Language (PMML)

•  JPMML-Evaluator

•  https://github.com/jpmml/jpmml-evaluator

•  Python has rich libraries for numerical analysis, machine learning

•  Numpy, scikit

•  XD 1.1 – Python in stream definitions

6

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Apache Kafka

•  “Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.”

•  Why it is popular?

•  Fast – can easily saturate network

•  Store tons of data – TBs

•  Replay

•  Clustered/HA

•  Partitioning –

o  Topics have multiple partitions

o  You divide the data based on the value of a field in the data

o  Can subscribe to specific partitions

7

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 8

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

RxJava

•  Reactive X – http://reactivex.io/

•  “An API for asynchronous programming with observable streams”

•  “ReactiveX is a combination of the best ideas from the Observer pattern, the Iterator pattern, and functional programming”

•  Java Version - NetflixOSS project: https://github.com/ReactiveX/RxJava

•  Functional style programming with asynchronous events

•  Map, filter, reduce

•  Functional transformations are a natural way of describing stream processing

9

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spark

•  Large scale data processing system

•  RDD

•  Resilient distributed data set

•  Immutable

•  Partitioned across machines

•  Computation

•  DAG of connected RDDs

•  Split DAGs into stages to compute

•  Batch and Stream style processing

10

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spark Streaming

11

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Word Count

12

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Move to Data Flow

•  Analogy with tomcat/servlet containers

•  Some downsides

•  New advantages (right now)

•  Don’t need to run hsqldb if you are only doing stream processing

•  Can scale up/down individual modules in a stream at runtime.

•  Resource allocation per module (memory)

•  Better ‘composition’ to co-locate modules

•  Entire stream can be deployed to avoid message hops

13

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

What’s next…

14

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spring Cloud Data Flow

15

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Motivations

•  Parent/Child class load issues, inadvertent spral of xd/lib

•  Use cases where streams should avoid any network hops

•  Additional requirments

•  Scale up/down, upgrade/downgrade versions

•  Resource allocaiton

•  Would require ‘us’ to add these features to the runtime….

•  Many other people are doing this…

•  Develop a SPI to delegate to other systems to launch, scale, monitor, etc.

16

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 17

Modules -> ‘Microservice’ Applications Runtime -> SPI (Lattice, CF, YARN, local)

Analogy with Web Apps and servlet contaier

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 18

Container Container

Spring XD 1.x

Host

Spring Cloud Stream

•  Modules run inside containers •  Container is a Boot app •  Modules are ApplicationContexts

•  Modules are executable Boot apps •  Easier to use Spring Cloud features •  Easier to test, consistent dev experience •  Portable and “cloud-native”

M

Host

M

M M

M M M

M M

M

ZooKeeper Spring Cloud Data Flow (optional) SPI Implementation

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 19

•  OOTB modules •  Executable Boot Apps

•  Boot for Spring Integration and Batch •  Auto-Configuration, Bindings

XD

Container

Modules

Admin

Spring Cloud Data Flow

Spring Cloud Stream

Modules

Spring Cloud Task

Modules

Spring Cloud Stream

Spring Cloud Task

•  REST API, Shell, UI •  Module Deployer SPI:

•  Singlenode •  Lattice •  YARN •  Cloud Foundry

XD 1.x --> Spring Cloud Data Flow

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

spring-cloud-dataflow

20

shell ui

admin (REST API)

dsl-parser

stream-repository

module-registrymodule-deployer

local-deployer cloudfoundry-deployer

yarn-deployer lattice-deployer

task-repository

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 21

http://cloud.spring.io/spring-cloud-dataflow/

http://cloud.spring.io/spring-cloud-stream/

Learn More. Stay Connected.

@springcentral Spring.io/video

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

STOP! Download Fonts Now.

PLEASE DOWNLOAD AND INSTALL THESE FONTS BEFORE CREATING YOUR PRESENTATION. You can download the fonts here…

https://www.dropbox.com/s/na8jewnsp8dp659/SpringOne-2015-Fonts.zip?dl=0

Fonts included in the ZIP file:

•  Proxima Nova (body text)

•  Domine (headlines)

http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-mac

http://www.fontspring.com/support/installing/how-do-i-install-fonts-on-my-windows-pc

22

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Brand Colors

23

R: 241 G: 241 B: 241

R: 153 G: 153 B: 153

R: 109 G: 179 B: 63

Spring Brand Colors

R: 52 G: 48 B: 45

SpringOne 2GX 2015 Theme Colors

R: 83 G: 83 B: 83

R: 12 G: 48 B: 100

R: 156 G: 209 B: 121

R: 204 G: 204 B: 204

R: 38 G: 97 B: 168

R: 114 G: 163 B: 211

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Code Slide

24

// This is Arial: 18 pt or higher please public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; } … }

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Safe Harbor Statement

The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward looking information in this presentation.

25

If you are a pivotal employee and your talk contains forward facing information, please include this slide at the beginning or conclusion of the presentation. Remove this text box before presenting.

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

LOGOS & ICONS

26

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Logos

27

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Event Logos and Icons

28

Logo on dark Logo on light Date logo on dark Date logo on light

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spring Project Icons

29

Spring Framework

Spring Security

Spring Data

Spring Batch

Spring Integration

Spring Reactor

Spring AMQP

Spring Hateoas

Spring Mobile

Spring Android

Spring Social

Groovy Spring Web Services

Spring Web Flow

Spring XD

Spring Boot

Spring LDAP

Grails

PNG and PSD files for all Spring projects can be found here… https://www.dropbox.com/sh/5rbbt8v6evvm7xm/AAAZmhgf0zglGHZUWYhPvojFa?dl=0

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spring Tool Suite Icons

30

Spring Tool Suite Groovy / Grails Tool Suite

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Spring Document Icons

31

Spring Framework Tools Support

Guides Reference App Tutorials

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Additional Spring Icons

32

Continuous Integration

Metrics Forums / Discussion

Blog @ Issues

All Author Broadcast Date News Event

Releases Apple Linux Engineering Windows

Professional Support

Community Support

Location

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Icon Guidelines

•  These icons can be used when Spring icons are not appropriate.

•  Max icon size is 2.08 inches – anything larger and the icon will be blurry

•  Icons are transparent and can be placed over images and other objects

•  To use an icon, simply choose, copy and paste

•  200x200 PNG files can be found here… https://www.dropbox.com/sh/wy65b4l5t4fv9b4/AAD1LMHcOtwJSTbNu23zqywka?dl=0

33

Max icon size: 2.08 inches

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 34

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/ 35

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

CHARTS & TABLES

36

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Sample Pie Chart

37

1st Qtr

2nd Qtr

3rd Qtr

4th Qtr

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Sample Bar Chart

38

0

20

40

60

80

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

North

South

East

West

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Sample Line Chart

39

0

20

40

60

80

100

1st Qtr 2nd Qtr 3rd Qtr 4th Qtr

North

South

East

West

Unless otherwise indicated, these s l ides are © 2013-2015 Pivotal Software, Inc. and l icensed under a Creat ive Commons Attr ibut ion-NonCommercial l icense: ht tp: / /creat ivecommons.org/ l icenses/by-nc/3.0/

Sample Table

40

2007 2008 2009 2010

Category 1 2.4 6.4 4.1 6.6

Category 2 8.2 4.5 3.2 3.8

Category 3 4.6 3.2 1.9 9.6

Category 4 6.7 3.3 3.4 2.2

Category 5 4.3 5.6 7.1 3.4