Page 1 © Hortonworks Inc. 2014

OpenShift scheduling docker containers in YARN via Kubernetes

Ali BajwaPartner Solutions EngineerOct 2014

Page 2 © Hortonworks Inc. 2014

Static website Web frontend User DB Queue Analytics DB

Development VM QA server Public Cloud Contributor’s laptop

Docker: a shipping container system for code M

ultip

licity

of S

tack

sM

ultip

licity

of

hard

war

e en

viro

nmen

ts

Production Cluster

Customer Data Center

Do services and apps interact

appropriately?

Can I migrate

smoothly and quickly

…that can be manipulated using standard operations and run consistently on virtually any hardware platform

An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…

Page 3 © Hortonworks Inc. 2014

Why are Docker containers lightweight?

HDP 2.1Hortonworks Data Platform

I/O performance comparable to Bare Metal

Page 4 © Hortonworks Inc. 2014

Kubernetes – container orchestrator

• Service for container cluster management• Allows deploying and managing applications running

on multiple hosts using docker• Open sourced by Google• Supports GCE, CoreOS, Azure, vSphere• Used to manage Docker containers as a default

implementation• Master – maintain state of Kubernetes Server runtime

• Scheduler, API server, registries, storage• Minions – represent the host were containers created

• Kubelet – manage pod and container lifecycle

Page 5 © Hortonworks Inc. 2014

OpenShift

• Red Hat’s platform as a service forapplications in the cloud that supportsboth public and private cloud• Provides high level abstraction for applications on top

of containers allowing easy scaling, service discovery, and deployment

• Enable Docker image authors to easily deliver reusable application components, including highly available databases, monitoring and log aggregation tools, service discovery platforms, and prepackaged web applications

• Allow developers to deeply customize their runtime environments while preserving operational support at scale for those applications

Page 6 © Hortonworks Inc. 20146

Kubernetes/YARN Integration

YARN Node Manager

YARN Node Manager

Kubernetes AppMaster

Page 7 © Hortonworks Inc. 2014

Demo: OpenShift scheduling docker container in YARN

Running in docker

Page 8 © Hortonworks Inc. 2014

Demo: 1. Launch Docker container on YARN via OpenShift

#openshift kube create pods –c demo.json

#cat demo.json

Page 9 © Hortonworks Inc. 2014

Demo: 2. Enter into container and confirm web server started

#openshift kube create pods –c demo.json

Page 10 © Hortonworks Inc. 2014

Demo: 3. Webapp on Openshift interacting with Hadoop

Page 11 © Hortonworks Inc. 2014

Demo: 4. Launch other workloads on YARN

YARN allows Big Data and other workloads to run side by side on same cluster

Page 12 © Hortonworks Inc. 2014

Questions?