integrating apache nifi and apache flink

30
Page 1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved Integrating Apache NiFi and Apache Flink Feb 4 th 2016 Bryan Bende – Member of Technical Staff

Upload: hortonworks

Post on 06-Jan-2017

5.276 views

Category:

Technology


2 download

TRANSCRIPT

Page 1 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Integrating Apache NiFi and Apache Flink

Feb 4th 2016

Bryan Bende – Member of Technical Staff

Page 2 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Outline

• Introduction to NiFi

• NiFi Site-To-Site

• Flink + NiFi Integration

• Use Case Discussion

Page 3 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

About Me

• Member of Technical Staff at Hortonworks

• Apache NiFi Committer & PMC Member since June 2015

• Contributed NiFi + Flink Streaming Integration

• Twitter: @bbende / Blog: bryanbende.com

Page 4 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Introduction to Apache NiFi

Page 5 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Apache NiFi• Powerful and reliable system to process and

distribute data

• Directed graphs of data routing and transformation

• Web-based User Interface for creating, monitoring, & controlling data flows

• Highly configurable - modify data flow at runtime, dynamically prioritize data

• Data Provenance tracks data through entire system

• Easily extensible through development of custom components

[1] https://nifi.apache.org/

Page 6 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi - TerminologyFlowFile

• Unit of data moving through the system• Content + Attributes (key/value pairs)

Processor• Performs the work, can access FlowFiles

Connection• Links between processors• Queues that can be dynamically prioritized

Process Group• Set of processors and their connections• Receive data via input ports, send data via output ports

Page 7 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi - User Interface

• Drag and drop processors to build a flow• Start, stop, and configure components in real time• View errors and corresponding error messages• View statistics and health of data flow• Create templates of common processor & connections

Page 8 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi - Provenance

• Tracks data at each point as it flows through the system

• Records, indexes, and makes events available for display

• Handles fan-in/fan-out, i.e. merging and splitting data

• View attributes and content at given points in time

Page 9 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi - Queue Prioritization

• Configure a prioritizer per connection

• Determine what is important for your data – time based, arrival order, importance of a data set

• Funnel many connections down to a single connection to prioritize across data sets

• Develop your own prioritizer if needed

Page 10 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi - Extensibility

Built from the ground up with extensions in mind

Service-loader pattern for…• Processors• Controller Services• Reporting Tasks• Prioritizers

Extensions packaged as NiFi Archives (NARs)• Deploy NiFi lib directory and restart• Provides ClassLoader isolation• Same model as standard components

Page 11 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi - Architecture

OS/Host

JVM

Flow Controller

Web Server

Processor 1 Extension N

FlowFileRepository

ContentRepository

ProvenanceRepository

Local Storage

OS/Host

JVM

Flow Controller

Web Server

Processor 1 Extension N

FlowFileRepository

ContentRepository

ProvenanceRepository

Local Storage

OS/Host

JVM

Flow Controller

Web Server

Processor 1 Extension N

FlowFileRepository

ContentRepository

ProvenanceRepository

Local Storage

OS/Host

JVM

NiFi Cluster Manager – Request Replicator

Web Server

MasterNiFi Cluster Manager (NCM)

OS/Host

JVM

Flow Controller

Web Server

Processor 1 Extension N

FlowFileRepository

ContentRepository

ProvenanceRepository

Local Storage

SlavesNiFi Nodes

Page 12 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi Site-To-Site

Page 13 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi Site-To-Site

• Direct communication between two NiFi instances

• Push to Input Port on receiver, or Pull from Output Port on source

• Communicate between clusters, standalone instances, or both

• Handles load balancing and reliable delivery

• Secure connections using certificates (optional)

Page 14 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Site-To-Site Push

• Source connects Remote Process Group to Input Port on destination

• Site-To-Site takes care of load balancing across the nodes in the cluster

NCM

Node 1

Input Port

Node 2

Input Port

Standalone NiFi

RPG

Page 15 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Site-To-Site Pull

• Destination connects Remote Process Group to Output Port on the source

• If source was a cluster, each node would pull from each node in cluster

NCM

Node 1

RPG

Node 2

RPG

Standalone NiFi

Output Port

Page 16 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Site-To-Site Client

• Code for Site-To-Site broken out into reusable module• https://github.com/apache/nifi/tree/master/nifi-commons/nifi-site-to-site-client

• Can be used from any Java program to push/pull from NiFi

Java Program

Site-To-Site Client

Node 1

Output Port

NCM

Node 2

Output Port

Page 17 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Flink + NiFi Integration

Page 18 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Flink + NiFi Integration

• Use Site-To-Site Client in Flink Streaming

• NiFiSource to pull data from NiFi Output Port

• NiFiSink to push data to NiFi Input Port

• NiFiDataPacket to represent data to/from NiFi (think FlowFile)

public interface NiFiDataPacket {

byte[] getContent();

Map<String, String> getAttributes();

}

Page 19 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi Source Example

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder() .url("http://localhost:8080/nifi") .portName("Data for Flink") .requestBatchCount(…) .buildConfig();

SourceFunction<NiFiDataPacket> nifiSource = new NiFiSource(clientConfig);

DataStream<NiFiDataPacket> streamSource = env.addSource(nifiSource);

Page 20 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

NiFi Sink Example

StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

SiteToSiteClientConfig clientConfig = new SiteToSiteClient.Builder()

.url("http://localhost:8080/nifi") .portName("Data from Flink") .buildConfig();

// Creates a NiFiDataPacket from incoming data of a given type// Here we are creating NiFiDataPackets for each StringNiFiDataPacketBuilder<String> dpb = ...

DataStreamSink<String> dataStream = ... .addSink(new NiFiSink<>(clientConfig, dpb));

Page 21 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Use Case Discussion

Page 22 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Drive Data to Flink for Analysis

NiFi Flink

NiFi

NiFi

• Drive data from sources to central data center for analysis

• Tiered collection approach at various locations, think regional data centers

Edge

Edge

Core

Page 23 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Dynamically Adjusting Data Flow

• Push analytic results from Flink back to NiFi

• Push results back to edge locations/devices to change behavior

NiFi Flink

NiFi

NiFi

Edge

Edge

Core

Page 24 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

1. Logs filtered by level and sent from Edge -> Core

2. Flink produces new filter levels based on rate & sends back to core

3. Edge polls core for new filter levels & updates filtering

Example: Dynamic Log Collection

Core NiFiFlink

Edge NiFiLogs Logs

New Filters

Logs Output Log Input Log Output

Result Input Store Result

Service Fetch ResultPoll Service

Filter

New Filters

New Filters

Poll

Analytic

Page 25 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Dynamic Log Collection – Edge NiFi

Page 26 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Dynamic Log Collection – Core NiFi

Page 27 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Dynamic Log Collection – Flink StreamingStreamExecutionEnvironment env = ...

SiteToSiteClientConfig clientConfig = getSourceConfig(props);DataStream<NiFiDataPacket> streamSource = env.addSource(new NiFiSource(clientConfig));

int windowMs = ...LogLevelFlatMap logLevelFlatMap = new LogLevelFlatMap(...);

DataStream<LogLevels> counts = streamSource.flatMap(logLevelFlatMap) .timeWindowAll(Time.of(windowSize, TimeUnit.MILLISECONDS)) .apply(new LogLevelWindowCounter());

double rate = ...SiteToSiteClientConfig sinkConfig = getSinkConfig(props);NiFiDataPacketBuilder<LogLevels> builder = new DictionaryBuilder(window, rate);

counts.addSink(new NiFiSink<>(sinkConfig, builder));

Page 28 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Dynamic Log Collection – Full Flow

NiFi Flink

NiFi

NiFi

Edge

Edge

Core

Logs

Logs

Logs

New Filters

New Filters

New Filters

Page 29 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Summary

• Use NiFi to drive data from sources to Flink

• Leverage Flink results to adjust your dataflows

Sources• [1] https://nifi.apache.org/

Resources• https://github.com/bbende/nifi-streaming-examples• https://github.com/apache/flink/tree/master/flink-examples/flink-examples-streaming• https://flink.apache.org/news/2015/02/09/streaming-example.html

Contact Info: • Email: [email protected]• Twitter: @bbende

Page 30 © Hortonworks Inc. 2011 – 2015. All Rights Reserved

Thank you