CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
Marius Bogoevici @mariusbogoevici
CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
• Use cases: • predictive maintenance, fraud detection, QoS, IoT • enterprise integration meets ‘Big Data’
• Specific requirements • High throughput, low latency • Grouping, ordering, windowing
• Middleware adaptations: beyond JMS, Rabbit MQ • Kafka, Amazon Kinesis, Google Pub/Sub, Azure
Event Hubs • Brokerless: Aeron, ZeroMQ
Spring I/O 2016
CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
• Architectural style advantages: • bounded context, development agility
• Process-specific resource tuning: • memory, CPU, instance count
• Messaging vs. HTTP • Discoverability • Availability • Consistency across heterogenous resources
Spring I/O 2016
CLOUD-NATIVE STREAMING AND EVENT-DRIVEN MICROSERVICES
• Modern platforms • Resource management • memory, CPU, instance count
• Scaling up/down • Health monitoring and failover • Routing and load balancing
Spring I/O 2016
ApacheYARNApacheMesos Kubernetes
Spring I/O 2016
dataflow:> stream create demo --definition “http | file”
Goal
Spring Cloud Stream Spring Cloud Data Flow
Spring I/O 2016
Spring Cloud Data Flow
Spring Cloud Stream
Spring Cloud Task
2015
Spring Cloud Stream
• Event-driven microservice framework • Built on battle-tested components • Spring Boot: full-stack standalone
applications • Spring Integration: Messaging, EIP
patterns, connectors • Opinionated primitives • Pluggable middleware abstractions
Spring I/O 2016
Spring Cloud Stream in a nutshell
Spring I/O 2016
…. 10000 ft nutshell
Spring I/O 2016
Durable publish-subscribe
• Fits both streaming and event-driven use cases
Spring I/O 2016
Consumer Groups
• Borrowed from Kafka, applied to all binders • Groups of competing consumers within the pub-sub
destination • Used for scaling and partitioning
Spring I/O 2016
Partitioning
• Required for stateful processing • Outputs specify a partitioning strategy • Inputs consuming from specific partition
Spring I/O 2016
Programming model
@EnableBinding + Binder implementation
Apache Kafka
Gemfire
Spring I/O 2016
DEMO
Spring Cloud Data Flow
• Orchestration: • DSL for Stream topologies • REST API • Shell • UI
• Portable Deployment SPI • OOTB apps for common integration use-
casesSpring I/O
2016
Stream DSL: pipes and filters
Stream definition
Boot Apps build on top of Spring Ecosystem
httpfile = http | file
|
Spring I/O 2016
‘pipe’, or how to communicate Rabbit, Kafka, Gemfire
Spring Cloud Deployer• SPI for deploying applications to modern runtimes
• Local (for testing) • Cloud Foundry • YARN • Kubernetes • Mesos + Marathon
• Different resources supported • Spring Boot Uberjars • Docker Images
• Different locations • File system, HDFS, HTTP, Maven, Docker Hub
Spring I/O 2016
Spring Cloud Data Flow
Pla$orm
Spring I/O 2016
Spring I/O 2016
DEMO
Timeline
• Spring Cloud Stream • 1.0.0.GA May 10, 2016 • 1.1.0.M1 July 2016 • 1.1.0.GA Q3 2016
• Spring Cloud Data Flow • 1.0.0.M3 May 7, 2016 • 1.0.0.RC1 End of May, 2016
Spring I/O 2016
Spring Cloud Stream 1.1.0.M1
• Reactive APIs: Reactor, RxJava • Kafka 0.9 • Schema evolution support
Spring I/O 2016
Q&A Marius Bogoevici
twitter: @mariusbogoevici