build your own 4 node virtualized hadoop cluster

Overall Setup

Network Configuration

Network Configuration and Host Setup

Inspect Network Configuration

Inspect Network Configuration (Cont.)

Storage Capacity

Increasing Online Storage Capacity - ESXi

Increasing Online Storage Capacity – ESXi (Cont.)

Increasing Online Storage Capacity – LVM

Increasing Online Storage Capacity – LVM (Cont.)

Increasing Online Storage Capacity – LVM (Cont.)

Disable some services and update Yum

Some Prep Work

Check for previous or unsuccessful installed CDH

Check for previous or unsuccessful installed CDH (Cont.)

Some log files that you might want took at

Some Ports to keep track of

Starting CDH Installation

Point Your Browser to Your Server http://<host_server_address>:7180

Specify List of Hosts of Your CDH Cluster

Specify List of Hosts of Your CDH Cluster (Cont.)

Select CDH 5.4 Repository Version

JDK Installation Options

Enable Single User Mode (in my case I did not)

Provide SSH Login CredentialsUsing private key or root password

Wait for the Installation progressDepending on the size of the cluster and setup - it might take around +20 minutes

Once the Installation is Successful>|Click Continue

Distribution of CDH Parcels Will Start Downloading and Installing on the all Nodes

Hosts Inspection (Refer to Linux Prep Work section in case you have some failures)

Hosts Inspection (Cont.)(Scroll down the page to make all are fine)

Select Types of Services and Roles you want to include

Start Your Role Assignments(Make sure you have at least 3 Zookeeper hosts assigned )

View Role Assignment by Host

Hive, CDH Reports Manager and Oozie Database Setup

Review the Changes Before You Continue

The Progress Screen for the Service Deployment

The Progress Screen for the Service Deployment (Cont.)

Cluster Setup is Complete

What did CDH install?

Check Status

Check to see if you can login to the manager server

Some Spark Related Directories and Files

Running Sample Spark

Spark URL Job Monitor

Spark Job Detail

Spark Job Detail

Running Spark in--deploy-mode client -- master yarn

Running Spark in--deploy-mode client -- master local