BIG CONTAINERS, BIG ORCHESTRATION, BIG DATA ?· BIG CONTAINERS, BIG ORCHESTRATION, BIG DATA William…

Download BIG CONTAINERS, BIG ORCHESTRATION, BIG DATA ?· BIG CONTAINERS, BIG ORCHESTRATION, BIG DATA William…

Post on 09-Sep-2018

218 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>BIG CONTAINERS, BIG ORCHESTRATION, BIG DATAWilliam Benton Red Hat, Inc.@willb</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>BACKGROUND</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>Mesos</p><p>WHAT OUR CLUSTER LOOKED LIKE IN 2014</p><p>Networked POSIX FS</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>1</p><p>2</p><p>3</p><p>4</p><p>1</p><p>1</p><p>2</p><p>3</p><p>3</p><p>4</p></li><li><p>Analytics is no longer a separate workload.Analytics is an essential component of modern data-driven applications.</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>OUR GOALS</p><p>git</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>FORECAST</p><p>Spark and microservices</p><p>Architectures for analytics and applications</p><p>Scheduling and storage</p><p>Future work (and how to get involved)</p></li><li><p>SPARK AND MICROSERVICES</p></li><li><p>Apache Spark is a fast and general framework for distributed data processing.</p></li><li><p>Resilient Distributed Datasets are partitioned, lazy, and immutable homogeneous collections.</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>RESILIENT DISTRIBUTED DATASETS</p><p>1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16</p><p>2 3 4 6 7 8 10 11 121 5 9 13 14 15 16</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>RESILIENT DISTRIBUTED DATASETS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>1 2 3 x: x % 2 != 0 x: x * 3FILTER MAP</p><p> x: [x, x+1]</p><p>FLATMAP</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>3 x: x % 2 != 0 x: x * 3FILTER MAP</p><p> x: [x, x+1]</p><p>FLATMAP</p><p>3 4 9 10COLLECT</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>1 2 3 x: x % 2 != 0 x: x * 3FILTERMAP</p><p> x: [x, x+1]</p><p>FLATMAP</p><p>3 4 9 10SAVE AS TEXT FILECACHE</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>executor1</p><p>1 2 3</p><p>executorn</p><p>10 11 12</p><p>cluster manager</p><p>2 4 6 20 22 24</p><p> x: x * 2 x: x * 2</p><p>driver</p><p>CACHCACH</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>Spark core</p><p>Graph SQL ML Streaming</p><p>ad hoc Mesos YARN</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>Spark core</p><p>Graph SQL ML Streaming</p><p>ad hoc Mesos YARNk8s</p></li><li><p>A microservice architecture employs lightweight, modular, and typically stateless components with well-defined interfaces and contracts.</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>BENEFITS OF MICROSERVICE ARCHITECTURES</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>BENEFITS OF MICROSERVICE ARCHITECTURES</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>BENEFITS OF MICROSERVICE ARCHITECTURES</p><p>2 + 2 5</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>MICROSERVICES AND SPARK</p><p>executor</p><p>1 2 3</p><p>executor</p><p>4 5 6</p><p>executor</p><p>7 8 9</p><p>executor</p><p>10 11 12</p><p>master</p><p> x: x * 22 4 6 8 10 12 14 16 18 20 22 24</p><p> x: x * 2 x: x * 2 x: x * 2 x: x * 2</p></li><li><p>ARCHITECTURES FOR ANALYTICS AND APPLICATIONS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>APPLICATION RESPONSIBILITIES</p><p>archive</p><p>trainmodels</p><p>transform</p><p>transform</p><p>transform</p><p>aggregate</p><p>events</p><p>databases</p><p>file, object storage</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>APPLICATION RESPONSIBILITIES</p><p>archive</p><p>trainmodels</p><p>transform</p><p>transform</p><p>transform</p><p>aggregate</p><p>events</p><p>databases</p><p>file, object storage</p><p>management</p><p>web and mobile</p><p>reporting</p><p>developer UI</p></li><li><p>LEGACY ARCHITECTURES</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>transactionprocessing</p><p>CONVENTIONAL DATA WAREHOUSE</p><p>transformevents</p><p>UI business logic</p><p>RDBMS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>transactionprocessing</p><p>CONVENTIONAL DATA WAREHOUSE</p><p>transformevents</p><p>UI business logic</p><p>RDBMS analytic processing</p><p>RDBMS</p><p>analysis</p><p>interactive queryreporting</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>HADOOP-STYLE DATA LAKE</p><p>HDFS</p><p>events</p><p>HDFS HDFS HDFS HDFS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>HADOOP-STYLE DATA LAKE</p><p>HDFS</p><p>compute</p><p>events</p><p>HDFS</p><p>compute</p><p>HDFS</p><p>compute compute compute</p><p>HDFS HDFS</p></li><li><p>MODERN ARCHITECTURES</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>serving layerspeed layer</p><p>THE LAMBDA ARCHITECTURE</p><p>events</p><p>batch layer</p><p>UIfederate</p><p>(precise)analysistransform</p><p>(imprecise)analysistransform</p><p>DFS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>queue for raw data topic</p><p>THE KAPPA ARCHITECTURE</p><p>events</p><p>transform analysis</p><p>queue for preprocessed data topic</p><p>queue for analysis results topic</p><p>reporting end-user UI</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>DATA FEDERATION IN THE COMPUTE LAYER</p><p>aggregate</p><p>trainmodels</p><p>archive</p><p>events</p><p>databases</p><p>file, object storage</p><p>management</p><p>web and mobile</p><p>reporting</p><p>developer UItransform</p><p>transform</p><p>transform</p></li><li><p>PRACTICALITIES AND POTENTIAL PITFALLS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>Cluster scheduler</p><p>SIDEBAR: THE MONOLITHIC SPARK ANTIPATTERN</p><p>Shared FSSpark executor</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>Spark executor</p><p>Resource manager</p><p>app 1 app 2</p><p>app 4app 3</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>OpenShift</p><p>ONE CLUSTER PER APPLICATION</p><p>Object storesapp 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>app 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>Databases</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>OpenShift</p><p>app 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>app 1</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>OpenShift</p><p>app 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>app 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>POSIX FS</p><p>HDFS HDFS</p><p>HDFS HDFS</p><p>HDFS</p><p>HDFS</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>OpenShift</p><p>app 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>app 1 app 2</p><p>app 5app 4</p><p>app 3</p><p>app 6</p><p>object store</p><p> interoperability fine-grained AC many implementations</p><p> consistency model performance</p></li><li><p>For the workloads from Facebook and Bing, we see that 96% and 89% of the active jobs respectively can have their data entirely fit in memory, given an allowance of 32GB memory per server for caching</p><p>PACMan: Coordinated Memory Caching for Parallel Jobs. G. Ananthanarayanan et al., in Proceedings of NSDI 12.</p></li><li><p>Recent studies have shown that reading data from local disks is only about 8% faster than reading it from remote disks over the network [and] this 8% number is decreasing.</p><p>Tom Phelan, The Elephant in the Big Data Room: Data Locality is Irrelevant for Hadoop (goo.gl/MnCKuM)</p><p>http://goo.gl/MnCKuM</p></li><li><p>Three out of ten hours of job runtime were spent moving files from the staging directory to the final directory in HDFSWe were essentially compressing, serializing, and replicating three copies for a single read.</p><p>Apache Spark @Scale: a 60+ TB production use caseFacebook Engineering Blog Post</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>executor1 executornCACHCACH</p><p>driver</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>COLOCATED COMPUTE AND STORAGE: YAGNI</p><p>Disk locality is just another kind of caching, but memory is much faster than disk and working set sizes typically fit in cluster memory after ETL.</p><p>The I/O-heavy behavior of frameworks designed for colocated compute and storage performs worse than iterative processing in memory. </p><p>Colocating compute and storage prevents independent scale-out of compute and turns cattle into pets.</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>BUT IF YOU DOOpenShift</p><p>app 1 app 2 app 3app 1 app 2 app 3</p><p>Storage</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>BUT IF YOU DOOpenShift</p><p>app 1 app 2 app 3app 1 app 2 app 3</p><p>Storage Storage Storage</p></li><li><p>PLAYING ALONG AT HOME</p></li><li><p>COMMONS GATHERINGSeattle | November 7#OCGathering2016</p><p>TRY IT OUT YOURSELF</p><p>Enabling Spark on OpenShift: https://github.com/radanalyticsio </p><p>Video demo: https://vimeo.com/189710503 </p><p>Meet the teams at lunch!</p><p>https://github.com/radanalyticsiohttps://vimeo.com/189710503</p></li><li><p>@willb willb@redhat.com https://chapeau.freevariable.com</p><p>THANKS!</p>mailto:willb@redhat.com?subject=https://chapeau.freevariable.com</li></ul>