zerostack cloudera deployment guide · zerostack has added cloudera manager to the app store. as a...
TRANSCRIPT
Intelligent Cloud Platform
@ZeroStackInc
www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of
ZeroStack, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
ZeroStack Cloudera
Deployment Guide
Version 1.0
Admin Guide Ver. 3.0
2
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
The ZeroStack App Store is an integrated application catalog available out-of-the-box hosting a wide range of application templates that can be deployed on ZeroStack with a single click. It is a simple and easy way to enable rapid application development and delivery. The App store is included as part of the ZeroStack Intelligent Cloud Platform.
Many different types of applications are available on the App Store including
CI/CD tools such as Jenkins
Big data applications such as Hadoop and Spark
SQL and NoSQL databases
Monitoring and data analysis tools such as ELK, Splunk
Application servers such as Apache and Nginx
Orchestration tools such Terraform
Container tools such as Kubernetes and Docker
ZeroStack has added Cloudera Manager to the App Store. As a starting point for a Hadoop installation, the App Store Cloudera template automates the deployment of the Cloudera controller node, a secondary name node, and two data nodes one of which also doubles as secondary named node.
Deploying Cloudera using the App Store follows generally the same process used for other applications, as described in the previous section. Cloudera, however, is a more complex application than many of the other applications in App Store and will require considerably more application knowledge and configuration.
1.1. Verify System Resources for Cloudera Deployment
The current implementation of the Cloudera deployment does not perform any pre-checks on available resources or quota on the target Z-Block. Given the large quantity of system resources required by Cloudera, there may not be sufficient resources to successfully complete the deployment, which could leave the Cloudera application in an unusable state. Before deploying please verify there are sufficient system resources available and the project in which the application is being deployed has enough resource quota assigned to it. Please also check the hardware requirements guide to determine what resources you need to deploy and configure Cloudera: Hardware requirements guide for Cloudera deployment
ZeroStack will be adding a system resource pre-check in a future product release.
1.2. Importing the Cloudera Template
The Cloudera template can be found in App Store and should be added to “My Library”, as described above.
Admin Guide Ver. 3.0
3
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
1.3. Deploying the Cloudera Application
Import the templates into My Library and then click “Deploy App”. Please note the default-image for this deployment is Ubuntu 16.04. If it is available under Recommend Image tab use that one, other wise pick a Ubuntu 16.04 image from Other Images tab.
Note: In order to ensure you are using the latest version of the deployment template, we recommend you delete the existing template and import to My Library again. A future version of ZeroStack will check for new versions and alert you to reimport new versions automatically.
Admin Guide Ver. 3.0
4
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
Initiate deployment from the App Store and provide the required configuration parameters. Key points to note:
Use all lowercase for the name of you application
The default flavor that is selected is m1.xlarge. You will need this capacity for a real-world workload. Note you can always resize this flavor in the planning tab of the project. Lower flavor sizes are not recommended.
Create a private network (if one doesn’t exist, please see previous sections) that will be used for communication between the Cloudera nodes. The public network is used to connect to the controller and login to the environment through a browser. The data nodes are not accessible from the external network.
You can pick any root password for the MySQL DB.
Select the volume sizes (default value is 100 GB) you need for the Controller, secondary named node, and the data nodes. Please note that only 3 data nodes (remember one of the data nodes also is used as a secondary named node) will be provisioned for this initial deployment. Once Cloudera is up and running, you can perform additional operations such as adding more data nodes directly from the Cloudera Manager UI.
Note: this currently requires that you manually create a VM in ZeroStack and provide the IP address to Cloudera Manager for adding the node. ZeroStack also has a template to install and add additional data nodes to the current cluster. (Please contact ZeroStack for additional information on this template)
For best performance pick the SSD option for the volume type, which provides local SSD access to the nodes in question. You can also pick HDD, which is the local hard disk.
Note: Note that typically cloudera HDFS using its replication but that would work only across the disks on the host in which the VM is deployed. If you need higher level of resilience you can also chose SSD with replication or HDD with replication volume, which will provide protection against host failures as well. However, this
will have a performance impact.
Admin Guide Ver. 3.0
5
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
The deployment is fully automated – the template orchestrates the creation of all the virtual machines and the associated volumes and security groups. These virtual machines and their volumes are distributed on different hosts to ensure high performance with local SSD access and to prevent resource contention.
ZeroStack downloads the Cloudera Manager bits from the source repository. Depending on the speed of the Internet connect, it can take some time to deploy all the necessary resources. We also allow installation of bits from local repository (Please contact ZeroStack if you are interested in this option). Once the deployment is completely successfully, you will see the application information as shown below. The green dot represents a successful deployment.
Click View details on the application panel to see the page of the resources deployed and URL link to the Cloudera manager on port 7180:
Admin Guide Ver. 3.0
6
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
1.4. Access the Cloudera Manager
Pointing to the access URL on the browser will get you to the login screen of full deployed and functional Cloudera Manager
From here, the Cloudera manager’s full capabilities are available to the end user including deploying all the big data applications such as Hadoop, Spark, etc and operational
Admin Guide Ver. 3.0
7
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
management of the virtual nodes. If more capacity is required, ZeroStack can create more data nodes, which can be added to the cluster from the Cloudera Manager UI.
Cloudera Manager also manages the installation Cloudera application parcels.
The selected parcels will be deployed automatically
Admin Guide Ver. 3.0
8
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
Cluster inspector will automatically run checks to test the health of the installation:
Once the packages are installed, it’s time to install the core services you need for your Big Data project. Select the appropriate combination of services and hit Continue:
Admin Guide Ver. 3.0
9
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
You can change the default host assignments to services in the next screen:
Admin Guide Ver. 3.0
10
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
You will need to provide database crendetials on the next screen:
Please note: You get the credentials for various Cloudera services (hue, hive,etc) in the following file /etc/cloudera-scm-server/db-properties on the master node. Login in the master node, obtain the name, username, and password, and enter into the next screen:
The services and your cloudera is ready once test connection to the database passes.
References:
ZeroStack User Guide: https://zerostack.zendesk.com/hc/en-us/articles/360005399053-User-Guide-3-1
Blog on Big Data pain points and how to solve them: https://www.zerostack.com/8-cloud-big-data-pain-points-and-how-to-solve-them/
Video on the cloudera deployment:
https://www.zerostack.com/how-to-deploy-cloudera-big-data-apps-in-few-clicks/
White paper on ZeroStack IT infrastructure for Big Data Deployment https://www.zerostack.com/wp-
content/uploads/2018/07/WhitePaper_ZeroStack_Big-Data-v4.pdf?utm_source=ZeroStack&utm_medium=Website
ZeroStack Security Guide: http://go.zerostack.com/rs/673-DPL-387/images/WhitePaper_ZeroStack_SecurityGuide.pdf
Admin Guide Ver. 3.0
11
@ZeroStackInc [email protected] www.zerostack.com
Copyright © 2018 ZeroStack, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. ZeroStack is a registered trademark or trademark of ZeroStack,
Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
ZeroStack Low Level Architecture and Key Features for Admins: http://go.zerostack.com/rs/673-DPL-387/images/WhitePaper_ZeroStack_Low-Level-Architecture.pdf
ZeroStack platform data sheet: https://www.zerostack.com/wp-content/uploads/2018/03/ZeroStack-ICP-DataSheet-Updated.pdf