1 modern approaches of customer’s dream distribution across the cluster evgenij kozhevnikov,...
TRANSCRIPT
![Page 1: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/1.jpg)
1
Modern Approaches of Customer’s Dream Distribution Across the ClusterEvgenij Kozhevnikov, Samara
AUGUST 4, 2015
![Page 2: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/2.jpg)
2
About me
1+ years of production experience in BigData– Edmunds.com– BigData CC
3+ years of development experience in BigData– Hadoop– Spark– Storm– Akka
6+ years of development experience– Java EE, IBM Websphere– Spring
![Page 3: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/3.jpg)
3
Successful Business - Growing Business
![Page 4: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/4.jpg)
4
Growing Business – Growing Load
![Page 5: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/5.jpg)
5
Our software should be ready to grow with business
1. Pay for your needs, not for plans
2. Growth doesn’t require any changes in application
3. Where one growing app, there are some growing apps
![Page 6: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/6.jpg)
6
Our software should be ready to grow with business
![Page 7: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/7.jpg)
7
Our software should be ready to grow with business
• Caching Proxy• CDN
![Page 8: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/8.jpg)
8
Our software should be ready to grow with business
• Caching Proxy• CDN
• NoSQL• Distributed cache
![Page 9: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/9.jpg)
9
Our software should be ready to grow with business
• Caching Proxy• CDN
• NoSQL• Distributed cache
![Page 10: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/10.jpg)
10
Does Edmunds need cluster solution?
1. What trends we have now?2. Is quality of the vehicle catalog is enough?3. Is our ad efficient?4. What results of A/B testing do we get?5. What can we recommend to our clients?6. Where is the car that client needs?7. How many leads were sent to the dealer?8. Is the dealer successful?9. Are our visitors not robots?10. What revenue do we have in this year?11. Are we growing?12. Are our dealers growing?
![Page 11: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/11.jpg)
11
1. What trends we have now?2. Is quality of the vehicle catalog is enough?3. Is our ad efficient?4. What results of A/B testing do we get?5. What can we recommend to our clients?6. Where is the car that client needs?7. How many leads were sent to the dealer?8. Is the dealer successful?9. Are our visitors not robots?10. What revenue do we have in this year?11. Are we growing?12. Are our dealers growing?
It’s not a competitive advantageAll competitors do that
Does Edmunds need cluster solution?
![Page 12: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/12.jpg)
12
• Need in fast access to the whole amount of data
• Historical data is important as a new one
• Support dynamically extended hardware resources
• Be able to run some independent applications on the same cluster
• Each application run require specific amount of resources
• Need in convenient monitoring tool and fault-tolerance of the system
• Code should be readable and distributed algorithms should be supportable
Does Edmunds need cluster solution?
Growing amount of data
Amount of tasks growth
![Page 13: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/13.jpg)
13
MAPREDUCEYARN
Hadoop-based solutions
![Page 14: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/14.jpg)
14
MapReduce across YARN
Node Node
Node Node
Node
Node
Node
Node
![Page 15: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/15.jpg)
15
MapReduce across YARN
ResourceManager
NameNode
Resource Manager
Name Node
Node
Node
Node
Node
![Page 16: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/16.jpg)
16
MapReduce across YARN
StandbyResourceManager
ActiveResource Manager
HadoopClient
MR Application
Master
NameNode
Data NodeMR Executor
Data NodeMR Executor
![Page 17: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/17.jpg)
17
SPARKYARN
Hadoop-based solutions
![Page 18: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/18.jpg)
18
Spark across YARN
StandbyResourceManager
ActiveResource Manager
SparkClient
MR Application
Master
NameNode
Data NodeSpark Executor
Data NodeSpark Executor
![Page 19: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/19.jpg)
19
SPARKMESOS
Mesosphere-based solutions
![Page 20: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/20.jpg)
20
Spark across YARN
StandbyMesos Master
ActiveMesosMaster
SparkClient
Spark Scheduler
NameNode
Data NodeSpark Executor
Data NodeSpark Executor
![Page 21: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/21.jpg)
21
1 2 3 4
WHAT NEXT
Myriad
• YARN on Mesos
• Efficient access to Hadoop resources
• Dynamic nature of Mesos
Kubernetes
• Resource Manager for docker-based infrastructure
• Solution from Google
Akka Cluster
• Efficient model for vertical and horizontal scaling
• Freedom of choosing the way of distribution
Task-specific tools
• Apache Storm
• Hive/Pig/Cascading…
• NoSQL solutions
• Kafka/Sqoop/Flume…
• Chef/Puppet/Ansible…
• Docker/Rocket/CoreOS
• Data Science
![Page 22: 1 Modern Approaches of Customer’s Dream Distribution Across the Cluster Evgenij Kozhevnikov, Samara AUGUST 4, 2015](https://reader035.vdocuments.net/reader035/viewer/2022070416/5697c0231a28abf838cd3ee1/html5/thumbnails/22.jpg)
22
Modern Approaches of Customer’s Dream Distribution Across the Cluster
Evgenij Kozhevnikov, Samara