bigdata processing in the cloud – guest lecture - university of applied sciences rapperswil -...
DESCRIPTION
TRANSCRIPT
![Page 1: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/1.jpg)
© 2013 IBM Corporation1
BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14
Romeo Kienzler
IBM Innovation Center
Source: http://res.sys-con.com/story/oct12/2398990/Cloud_BigData_468.jpg
![Page 2: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/2.jpg)
© 2013 IBM Corporation2
What is BIG data?
![Page 3: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/3.jpg)
© 2013 IBM Corporation3
What is BIG data?
![Page 4: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/4.jpg)
© 2013 IBM Corporation4
What is BIG data?
Big Data
Hadoop
![Page 5: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/5.jpg)
© 2013 IBM Corporation5
What is BIG data?
Business Intelligence
Data Warehouse
![Page 6: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/6.jpg)
© 2013 IBM Corporation6
Map-Reduce → Hadoop → BigInsights
![Page 7: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/7.jpg)
© 2013 IBM Corporation7
BigData UseCases● Google Index
● 40 X 10^9 = 40.000.000.000 => 40 billion pages indexed● Will break 100 PB barrier soon● Derived from MapReduce● now “caffeine” based on “percolator”
● Incremental vs. batch● In-Memory vs. disk
![Page 8: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/8.jpg)
© 2013 IBM Corporation8
BigData UseCases● CERN LHC
● 25 petabytes per year● Facebook
● Hive Datawarehouse● 300 PB, growing 600 TB / d● > 100 k servers
● Genomics● Enterprises
● Data center analytics (Logflies, OS/NW monitors, ...)● Predictive Maintenance, Cybersecurity
● Social Media Analytics● DWH offload● Call Detail Record (CDR) data preservation
http://www.balthasar-glaettli.ch/vorratsdaten/
![Page 9: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/9.jpg)
© 2013 IBM Corporation9
BigData Analytics
![Page 10: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/10.jpg)
© 2013 IBM Corporation10
BigData Analytics – Predictive Analytics
"sometimes it's not who has the best algorithm that wins; it's who has the most data."
(C) Google Inc.
The Unreasonable Effectiveness of Data¹
¹http://www.csee.wvu.edu/~gidoretto/courses/2011-fall-cp/reading/TheUnreasonable%20EffectivenessofData_IEEE_IS2009.pdf
No Sampling => Work with full dataset => No p-Value/z-Scores anymore
![Page 11: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/11.jpg)
© 2013 IBM Corporation11
Data Parallelism
![Page 12: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/12.jpg)
© 2013 IBM Corporation12
Aggregated Bandwith between CPU, Main Memory and Hard Drive
1 TB (at 10 GByte/s)
- 1 Node - 100 sec
- 10 Nodes - 10 sec
- 100 Nodes - 1 sec
- 1000 Nodes - 100 msec
![Page 13: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/13.jpg)
© 2013 IBM Corporation13
Fault Tolerance / Commodity Hardware
AMD Turion II Neo N40L (2x 1,5GHz / 2MB / 15W), 8 GB RAM,
3TB SEAGATE Barracuda 7200.14
< CHF 500
100 K => 200 X (2, 4, 3) => 400 Cores, 1,6 TB RAM, 200 TB HD
MTBF ~ 365 d > 1,5 d
Source: http://www.cloudcomputingpatterns.org/Watchdog
![Page 14: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/14.jpg)
© 2013 IBM Corporation14
![Page 15: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/15.jpg)
© 2013 IBM Corporation15
![Page 16: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/16.jpg)
© 2013 IBM Corporation16
HDFS – Hadoop File System
![Page 17: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/17.jpg)
© 2013 IBM Corporation17
![Page 18: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/18.jpg)
© 2013 IBM Corporation18
![Page 19: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/19.jpg)
© 2013 IBM Corporation19
![Page 20: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/20.jpg)
© 2013 IBM Corporation20
![Page 21: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/21.jpg)
© 2013 IBM Corporation21
![Page 22: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/22.jpg)
© 2013 IBM Corporation22
![Page 23: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/23.jpg)
© 2013 IBM Corporation23
![Page 24: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/24.jpg)
© 2013 IBM Corporation24
![Page 25: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/25.jpg)
© 2013 IBM Corporation25
![Page 26: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/26.jpg)
© 2013 IBM Corporation26
![Page 27: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/27.jpg)
© 2013 IBM Corporation27
![Page 28: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/28.jpg)
© 2013 IBM Corporation28
![Page 29: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/29.jpg)
© 2013 IBM Corporation29
![Page 30: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/30.jpg)
© 2013 IBM Corporation30
![Page 31: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/31.jpg)
© 2013 IBM Corporation31
![Page 32: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/32.jpg)
© 2013 IBM Corporation32
![Page 33: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/33.jpg)
© 2013 IBM Corporation33
![Page 34: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/34.jpg)
© 2013 IBM Corporation34
![Page 35: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/35.jpg)
© 2013 IBM Corporation35
Map-Reduce
Source: http://www.cloudcomputingpatterns.org/Map_Reduce
![Page 36: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/36.jpg)
© 2013 IBM Corporation36
![Page 37: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/37.jpg)
© 2013 IBM Corporation37
![Page 38: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/38.jpg)
© 2013 IBM Corporation38
![Page 39: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/39.jpg)
© 2013 IBM Corporation39
![Page 40: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/40.jpg)
© 2013 IBM Corporation40
![Page 41: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/41.jpg)
© 2013 IBM Corporation41
![Page 42: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/42.jpg)
© 2013 IBM Corporation42
![Page 43: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/43.jpg)
© 2013 IBM Corporation43
![Page 44: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/44.jpg)
© 2013 IBM Corporation44
![Page 45: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/45.jpg)
© 2013 IBM Corporation45
![Page 46: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/46.jpg)
© 2013 IBM Corporation46
![Page 47: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/47.jpg)
© 2013 IBM Corporation47
![Page 48: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/48.jpg)
© 2013 IBM Corporation48
![Page 49: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/49.jpg)
© 2013 IBM Corporation49
![Page 50: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/50.jpg)
© 2013 IBM Corporation50
![Page 51: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/51.jpg)
© 2013 IBM Corporation51
![Page 52: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/52.jpg)
© 2013 IBM Corporation52
![Page 53: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/53.jpg)
© 2013 IBM Corporation53
![Page 54: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/54.jpg)
© 2013 IBM Corporation54
![Page 55: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/55.jpg)
© 2013 IBM Corporation55
![Page 56: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/56.jpg)
© 2013 IBM Corporation56
![Page 57: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/57.jpg)
© 2013 IBM Corporation57
![Page 58: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/58.jpg)
© 2013 IBM Corporation58
![Page 59: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/59.jpg)
© 2013 IBM Corporation59
![Page 60: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/60.jpg)
© 2013 IBM Corporation60
![Page 61: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/61.jpg)
© 2013 IBM Corporation61
![Page 62: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/62.jpg)
© 2013 IBM Corporation62
![Page 63: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/63.jpg)
© 2013 IBM Corporation63
![Page 64: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/64.jpg)
© 2013 IBM Corporation64
![Page 65: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/65.jpg)
© 2013 IBM Corporation65
![Page 66: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/66.jpg)
© 2013 IBM Corporation66
![Page 67: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/67.jpg)
© 2013 IBM Corporation67
![Page 68: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/68.jpg)
© 2013 IBM Corporation68
![Page 69: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/69.jpg)
© 2013 IBM Corporation69
![Page 70: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/70.jpg)
© 2013 IBM Corporation70
![Page 71: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/71.jpg)
© 2013 IBM Corporation71
![Page 72: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/72.jpg)
© 2013 IBM Corporation72
![Page 73: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/73.jpg)
© 2013 IBM Corporation73
![Page 74: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/74.jpg)
© 2013 IBM Corporation74
![Page 75: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/75.jpg)
© 2013 IBM Corporation75
![Page 76: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/76.jpg)
© 2013 IBM Corporation76
![Page 77: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/77.jpg)
© 2013 IBM Corporation77
What role is the cloud playing here?
![Page 78: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/78.jpg)
© 2013 IBM Corporation78
“Elastic” Scale-Out
Source: http://www.cloudcomputingpatterns.org/Continuously_Changing_Workload
![Page 79: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/79.jpg)
© 2013 IBM Corporation79
“Elastic” Scale-Out
of
![Page 80: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/80.jpg)
© 2013 IBM Corporation80
“Elastic” Scale-Out
of
CPU Cores
![Page 81: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/81.jpg)
© 2013 IBM Corporation81
“Elastic” Scale-Out
of
CPU Cores Storage
![Page 82: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/82.jpg)
© 2013 IBM Corporation82
“Elastic” Scale-Out
of
CPU Cores Storage
![Page 83: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/83.jpg)
© 2013 IBM Corporation83
“Elastic” Scale-Out
of
CPU Cores Storage Memory
![Page 84: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/84.jpg)
© 2013 IBM Corporation84
“Elastic” Scale-Out
of
CPU Cores Storage Memory
![Page 85: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/85.jpg)
© 2013 IBM Corporation85
“Elastic” Scale-Out
linear
Source: http://www.cloudcomputingpatterns.org/Elastic_Platform
![Page 86: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/86.jpg)
© 2013 IBM Corporation86
“Elastic” Scale-Out
linear
Source: http://www.cloudcomputingpatterns.org/Elastic_Platform
![Page 87: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/87.jpg)
© 2013 IBM Corporation87
BigData Scale-Out
How do Databases Scale-Out?
![Page 88: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/88.jpg)
© 2013 IBM Corporation88
BigData Scale-Out
How do Databases Scale-Out?
![Page 89: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/89.jpg)
© 2013 IBM Corporation89
How do Databases Scale-Out?
Shared Disk Architectures
![Page 90: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/90.jpg)
© 2013 IBM Corporation90
How do Databases Scale-Out?
Shared Disk Architectures
![Page 91: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/91.jpg)
© 2013 IBM Corporation91
How do Databases Scale-Out?
Shared Nothing Architectures
![Page 92: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/92.jpg)
© 2013 IBM Corporation92
Born on the cloud Databases
Source: http://www.constructioncloudcomputing.com/wp-content/uploads/2010/10/dreamstime_7360880-480x300.jpg
Source: http://www.cloudcomputingpatterns.org/Execution_Environment
![Page 93: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/93.jpg)
© 2013 IBM Corporation93
Google AppEngine
Google App Engine is a Platform as a Service (PaaS) offering that lets you build and run applications on Google’s infrastructure. App Engine applications are easy to build, easy to maintain, and easy to scale as your traffic and data storage needs change. With App Engine, there are no servers for you to maintain. You simply upload your application and it’s ready to go.
Source: http://www.cloudcomputingpatterns.org/Platform_as_a_Service_%28PaaS%29
![Page 94: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/94.jpg)
© 2013 IBM Corporation94
Google AppEngine Database Services
![Page 95: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/95.jpg)
© 2013 IBM Corporation95
![Page 96: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/96.jpg)
© 2013 IBM Corporation96
IBM BlueMix
BlueMix is a Platform as a Service Cloud, based on Cloud Foundry, employing Enterprise grade services enriched with IBM Software and hosted at SOFTLAYER
![Page 97: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/97.jpg)
© 2013 IBM Corporation97
IBM BlueMix, a Cloudfoundry runtime
Linux VM
Linux VM
CodeRuntime
Framework+
Droplet
Linux VMContainer Container Container
SQL
Push
SSO
Services:
...
DropletDroplet
![Page 98: BigData processing in the cloud – Guest Lecture - University of Applied Sciences Rapperswil - 29.4.14](https://reader034.vdocuments.net/reader034/viewer/2022051609/547ec4a85906b5bf718b46d8/html5/thumbnails/98.jpg)
© 2013 IBM Corporation98
● Summary
● BigData is born on the cloud
● Cloud facilitates resource provisioning, configuration and deployment
● Highly innovative area
● Technology
● UseCases
● Links
● http://en.wikipedia.org/wiki/MapReduce
● http://www.se-radio.net/2013/12/episode-199-michael-stonebraker/
● Sign up for the free BlueMix beta
● http://bluemix.net
● Come to the BlueMix Days
● http://bit.ly/1lsIY8J
● Use our software
● Biginsights: http://www.ibm.com/software/data/infosphere/biginsights/quick-start/