fast big data ingest into sap hana
TRANSCRIPT
![Page 1: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/1.jpg)
1confidential
Fast Big Data Ingestinto SAP HANA
Denis KingSeptember, 2016
![Page 2: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/2.jpg)
2confidential
Speaker Introductiono SVP Field Operations, Solace Inc.
o Working @ Solace for 12 years
o Many years working in Capital Markets, Telco and Government industries
o Focus mainly on networking and middleware
![Page 3: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/3.jpg)
3confidential
Enterprise IT circa 1990
Application#1
Application#n
Imagine……achieving HA
…application overhead…Apps going offline
…spanning datacentres
COMPLEX & !SCALE
![Page 4: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/4.jpg)
4confidential
Enter the Enterprise Bus
Application#1
Application#n
Connect once, the bus handles everything
![Page 5: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/5.jpg)
5confidential
Big Data, the Apache way…
“How can we get all of that data into HANA/Hadoop?”
![Page 6: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/6.jpg)
6confidential
Big Data Ingestion the Apache + Kafka way…
API
API
API
API
API
API
API
API
API
“How can we get all of that data into Hadoop?”
![Page 7: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/7.jpg)
7confidential
Big Data Team Enterprise Architecture Team
![Page 8: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/8.jpg)
8confidential
Open Source (Rabbit) Kafka0
50
100
150
200
250
300
350
400
450
Message Broker vs Kafka Throughput
Open Source JMS Kafka Solace VMR0
50
100
150
200
250
300
350
400
450
500
Open Source (Rabbit) Kafka Solace VMR Solace 35600
500
1000
1500
2000
2500
3000
3500
4000
o 1 server‐ 1 message broker vs
10 Kafka flows
o 1K messageso Java client APIs
Test configurations:
K m
essa
ges/
sec
![Page 9: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/9.jpg)
9confidential
Big Data – The Simpler Approach
“Subscribe”
![Page 10: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/10.jpg)
10CONFIDENTIAL
Big Data Lake MEET Big Data RIVER
![Page 11: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/11.jpg)
11confidential
Big Data Lake meet Big Data RIVER
BigDataLake
Enterprise Big Data River
AMQP
JMS
MQTT
REST
JMSAMQP
![Page 12: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/12.jpg)
12CONFIDENTIAL
So…what does an Enterprise Big Data RIVER need?
![Page 13: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/13.jpg)
13confidential
Multi-Protocol, Multi QoS, Multi-pattern1234
![Page 14: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/14.jpg)
14confidential
OpenPub/Sub
Req/ReplyWeb/Streaming
RESTfulWAN
JMSAMQPMQTTREST
PersistentNon-
persistentLow
LatencyHigh/Low Volume
Data Movement
Linking applications, devices and people across
any cloud, any platform, anywhere around the world.
Any Protocol Any QoS Any Pattern
![Page 15: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/15.jpg)
15confidential
Event Driven, across datacenters, private, public clouds. Universal fabric
1234
1
![Page 16: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/16.jpg)
16confidential
Distributed Big Data Rivero Large enterprise moving
workloads to the cloud, refreshing IT strategy
o Running workloads across public cloud, private cloud and on-premise systems
o Big Data lake subscribes to any data from any source
PublicClouds
On Premise
Private Cloud
App App App
App App App
PaaS(HCP)
IaaS
App App App
PublicCloud
App App App
PublicCloud
![Page 17: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/17.jpg)
17confidential
Robustness HA & DR with e2e Security
1234
![Page 18: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/18.jpg)
18confidential
Big Data River Security
o Authentication, Authorization‐ Kerberos, LDap, Radius, SSL‐ ACLs (Topics & IP), Role based access
o Encryption‐ Transport level SSL
o Five 9’s High Availability
o Flexible Disaster Recovery for async/sync replication
![Page 19: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/19.jpg)
19confidential
Scale data bursts with“shock absorption”
1234
![Page 20: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/20.jpg)
20confidential
Shock Absorbing the RIVER from the LAKE
Capacity &Availability Limits
Big Data RIVERShock Absorber
NetworkProcessingStorage
OutagesUpgrades
InconsistentAggregate
InputStream
![Page 21: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/21.jpg)
21confidential
Big Data RIVER..deeper look
Big Data
PrivateCloud
PublicCloud
App 2App 1 App 3 App 4 App 6App 5 App 7 App 8
FrontOffice
RiskManagement
TradingEngines
Enterprise Data River
Compliance& Settlement
Easily extend your enterprise bus to the cloud
And capture enterprise events
![Page 22: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/22.jpg)
22confidential
Big Data Case Study: Citibank Post Trade Buso “Rio” – the post
trade data river• Global, multi geo
post trade bus
• Lambda Architecture• Feeds Hadoop for analytics,
reporting and compliance• Feeds KDB for real time insights• Traditionally feeds into Netezza:
“Ocean” as data warehouse
o Solace is the Big Data RIVER “Post Trade Bus”
o 600M orders, trades, RFQs etc hit Solace every day,
o Solace feeds Hadoop, Netezza and KDB at different speeds for Analytics, AML, Risk, Compliance
![Page 23: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/23.jpg)
23confidential
Other Big Data Rivers
![Page 24: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/24.jpg)
24CONFIDENTIAL
Oh…one last thing….
![Page 25: Fast Big Data Ingest into SAP HANA](https://reader036.vdocuments.net/reader036/viewer/2022062400/58a1bae41a28ab537c8b638b/html5/thumbnails/25.jpg)
25confidential
Big Data River
Enterprise Big Data River
AMQP
JMS
MQTT
REST
JMSAMQP
MQTT
SAP now supports nativeingest over MQTT…