slide guss 2014 events hub - future of integration
TRANSCRIPT
#JSS2014
Les journées
SQL Server 2014
Un événement organisé par GUSS
#JSS2014
Les journées
SQL Server 2014
Un événement organisé par GUSS
Session « Le futur de l’intégration »
Michel Hubert – MVP Azure
Radoine Douhou
#JSS2014
• Etat des lieux.
• Futur de l’intégration.
• Solutions d’intégration orientées Cloud.
• Solutions d’intégration orientées Events & IoT.
• Démo.
Agenda
#JSS2014
Evolution of Integration
Once upon a Time
Point to point (“Spaghetti”)
Integration broker (EAI/B2B)
Enterprise Service Bus
HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
#JSS2014
BizTalk Server
Partners
Evolution of Integration
Microsoft BizTalk Server
#JSS2014
BizTalk Server ESB toolkit
e-Commerce
Suppliers
EDI
EDIFACT / AS2
WS
RESTFul
Market place
warehouse
Magasins
SI Back Office
Orchestration B2B
BI
SSIS
SSRS
SSAS
Microsoft BizTalk Server
Common scenario
#JSS2014
FUTUR DE L’INTEGRATION
Nouveaux usages
#JSS2014
Point to point (“Spaghetti”)
Integration broker (EAI/B2B)
Enterprise Service Bus What’s next?
?HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
Business partner
HR
CRM E-commerce
ERP
Future of Integration
What’s next ?
#JSS2014
Future of
Integration
Mobiles
Events
Messaging
EAI/ESB
Business Process Automation
Telemetry
Analytics
BizTalk Server
BizTalk Services
Service Bus
Queuing
Events Hub
Stream Analytics
Hybrid Connection
B2B
Future of Integration
Integration = Cloud + Mobile + IoT
IoT
SaaS
#JSS2014
SOLUTIONS D’INTEGRATION ORIENTEES
CLOUD
BizTalk Server / BizTalk Services
#JSS2014
Upgrade to BizTalk Server 2013 R2
with confidence
Accelerate vertical solutions with
standards and platform support
Extend existing solutions to Azure
Service Bus Adapters
RESTFul/Json Adapter
BizTalk Server 2013 R2
Cloud Ready
#JSS2014
BizTalk Server
Microsoft Azure
PartnersBizTalk Services
Service Bus
Future of Integration
Federated Enterprise Service Bus
#JSS2014
Upgrade to BizTalk Server 2013 R2
with confidence
Accelerate vertical solutions with
standards and platform support
Extend existing solutions to Azure
Develop and test integration
solutions in the cloud without re-
writes
Clear up on-premises infrastructure
capacity and provision in minutes
instead of weeks
Service Bus Adapters
RESTFul/Json Adapter
BizTalk Server 2013 Virtual Machines
BizTalk Server 2013 R2
Cloud Ready
#JSS2014
App owner Datacenter admin
Customer data center
Microsoft Azure data center
LOB app Active Directory
BizTalk LOB app
SQL
Agility for app owners Control for IT pros
BizTalk Server 2013 R2
Develop in Azure
#JSS2014
Upgrade to BizTalk Server 2013 R2
with confidence
Accelerate vertical solutions with
standards and platform support
Extend existing solutions to Azure
Develop and test integration
solutions in the cloud without re-
writes
Clear up on-premises infrastructure
capacity and provision in minutes
instead of weeks
Enable hybrid extension of current
on-premises BizTalk Server
deployments with Azure BizTalk
Services
Power new cloud-hosted business-
to-business, Internet of Things, and
EDI capabilities
Service Bus Adapters
RESTFul/Json Adapter
BizTalk Server 2013 Virtual Machines
Microsoft Azure BizTalk Services
BizTalk Server 2013 R2
Cloud Ready
#JSS2014
Microsoft Azure BizTalk Services (MABS)
#JSS2014
Equivalent to BizTalk ESB itinerary Transforms
Enrich
Validation XML and xsd schema
MABS
Bridge principle
#JSS2014
Hybrid Connections
Web Sites
Mobile Services
Corporate network
Microsoft SQL Server
Hybrid Connection
Other published
resourcesHybrid Connection
Manager
#JSS2014
Use Cases
Microsoft Azure
On-premises
AS2 SFTP HTTP
? !
Home office Branch office Regional office
#JSS2014
SOLUTIONS D’INTEGRATION ORIENTEES
EVENTS / IOT
Service Bus, Event Hub, Stream Analytics
#JSS2014
Azure Service Bus
Relay
Queue
Topic
Notification Hub
Event Hub
NAT and Firewall Traversal Service
Request/Response Services
Unbuffered with TCP Throttling
Many publishers and many consumers to
communicate over a FIFO like channel.
(Competing consumers and Queue-based
Load leveling scenarios)
Pub / Sub communication channel. Each
Consumer subscribes to a copy of message
High-scale notification distribution
Most mobile push notification services
Millions of notification targets
Azure Service Bus
Cloud Messaging Broker
#JSS2014
Service Bus Events Hub
Event Producers
Azure Event Hub
> 1M Producers> 1GB/sec
Aggregate Throughput
Up to 32 partitions via portal, more on
request
Part
itio
ns
Direct
PartitionKey
Hash
Throughput Units:
• 1 ≤ TUs ≤ Partition Count
• TU: 1 MB/s writes, 2 MB/s reads
Consumer Group(s)
Receivers
AMQP 1.0
Credit-based flow control
Client-side cursors
Offset by Id or Timestamp
#JSS2014
Event Producers
> 1M Producers> 1GB/sec
Aggregate Throughput
Direct
PartitionKey
Hash
• Consistent security model with Service Bus,
extended by publisher policies
• Publisher protocols :
– HTTPS : Short lived, low throughput :
– AMQP : Long lived, High throughput :
• Publish modes :
– Directely to a PartitionId.
– Automatic hash-based distribution by
PartitionKey or Publisher Identity
• Stream or batch
Event Hub
Producers
#JSS2014
• 1 Partition : ordered sequence of events
that is held. A partition can be thought of
as a “commit log.”
• Segmentation of the event stream for
scale-out :
– Parallelism on both Publishing and Polling
sides.
• Default 16, minimum 8, self-service maximum
32 :
– 1024 partitions via Azure Support.
Azure Event HubPart
itio
ns
Event Hub
Partition principle
#JSS2014
• view (state, position, or offset) of an entire
Event Hub
• Publish/subscribe mechanism of Event
Hubs is enabled through consumer groups.
• Enable multiple consuming applications to
each have a separate view of the event
stream, and to read the stream
independently at its own pace and with its
own offsets
Consumer Group(s)
Event Hub
Consumer Group
#JSS2014
• Receive from partitions via consumer
groups
• Client-side cursors allowing to freely
process and repeatedly reprocess the
available retained event stream based on
offsets or timestamps.
• Using .NET API or using generic AMQP 1.0
client (e.g. Apache Proton-C/J)
ID, Time, [Data]
ID, Time, [Data]
ID, Time, [Data]
Time
Id
Event Hub
Consumers
#JSS2014
Stream Analytics
Data Source
Collect Process
Consume
Deliver
Event Inputs- Event Hub
- Azure Blob
Transform- Temporal joins
- Filter
- Aggregates
- Projections
- Windows
- Etc.
Enrich
Correlate
Outputs- SQL Azure
- Azure Blobs
- Event Hub
☁
BI
Dashboards
Predictive
Analytics
Azure
Storage
• Temporal Semantics
• Guaranteed delivery
• Guaranteed up time
Azure Stream Analytics
Reference Data- Azure Blob
#JSS2014
• Easily filter, project, aggregate, join streams, add static data with streaming data, detect patterns or lack of patterns with a few lines of SQL
• Development and debugging experience through Azure Portal
• Built-in monitoring trough Azure Portal
• No hardware acquisition and maintenance
• Up and running in a few clicks (and within minutes) => Bypasses deployment expertise
• Elasticity of Azure for scale up or scale down
• Distributed, scale-out architecture
• Integrated with highly-
scalable publish-
subscriber Events Hub.
• Transform, augment,
correlate, temporal
operations.
• Detect patterns
and anomalies in
streaming data
• Correlate streaming with
reference data
Stream Analytics
Value added
#JSS2014
CLICK STREAM ANALYSIS
#JSS2014
Demo architecture
e-Commerce
web site
Log file
Events Producer
ETW
Azure Event Hub
Stream Analytics
Tumbling window
Collect
Transform
SQL Azure
Deliver
Power View
Real Time
analysis
AMQP Client
#JSS2014
Merci à nos sponsors
#JSS2014#JSS2014
Les évaluations des sessions,
c’est important !!
…Et en plus on peut gagner
des cadeaux
#JSS2014
http://GUSS.Pro/jss
#JSS2014#JSS2014