elasticsearch logstash and kibana (elk) vm for hitachi ... · pdf...

Hitachi Hyper Scale-Out Platform (HSP)Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide

MK-95HSP033-0009 November 2016

© 2016 Hitachi, Ltd. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopying and recording, or stored in a database or retrieval system forcommercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Data SystemsCorporation (collectively, “Hitachi”). Licensee may make copies of the Materials provided that any suchcopy is: (i) created as an essential step in utilization of the Software as licensed and is used in no othermanner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials."Materials" mean text, data, photographs, graphics, audio, video and documents.

Hitachi reserves the right to make changes to this Material at any time without notice and assumes noresponsibility for its use. The Materials contain the most current information available at the time ofpublication.

Some of the features described in the Materials might not be currently available. Refer to the most recentproduct announcement for information about feature and product availability, or contact Hitachi DataSystems Corporation at https://support.hds.com/en_us/contact-us.html.

Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicableHitachi agreements. The use of Hitachi products is governed by the terms of your agreements with HitachiData Systems Corporation.

By using this software, you agree that you are responsible for:

1) Acquiring the relevant consents as may be required under local privacy laws or otherwise fromauthorized employees and other individuals to access relevant data; and

2) Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance withrelevant laws.

Notice on Export Controls. The technical data and technology inherent in this Document may besubject to U.S. export control laws, including the U.S. Export Administration Act and its associatedregulations, andmay be subject to export or import regulations in other countries. Reader agrees tocomply strictly with all such regulations and acknowledges that Reader has the responsibility to obtainlicenses to export, re-export, or import the Document and any Compliant Products.

Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries.

AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eServer, FICON, FlashCopy,IBM, Lotus, MVS, OS/390, PowerPC, RS6000, S/390, System z9, System z10, Tivoli, z/OS, z9, z10, z13,z/VM, and z/VSE are registered trademarks or trademarks of International Business Machines Corporation.

Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft,the Microsoft Corporate Logo, MS-DOS, Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQLServer, Visual Basic, Visual C++, Visual Studio, Windows, the Windows logo, Windows Azure, WindowsPowerShell, Windows Server, the Windows start button, andWindows Vista are registered trademarks ortrademarks of Microsoft Corporation. Microsoft product screen shots are reprinted with permission fromMicrosoft Corporation.

All other trademarks, service marks, and company names in this document or web site are properties oftheir respective owners.

https://support.hds.com/en_us/contact-us.html

Contents

Preface vIntended audience viProduct version viRelease notes viDocument revision level viDocument conventions viiAccessing product documentation viiGetting help viiiComments viii

Quick start 1Before you begin 2Procedure 2

Deploy the ELKVMs onHSP 5Before you begin 6Procedure 6

Deploy the ELK template 7Installation and configuration script 8

Appendix A: Logstash Configuration File 11Logstash default configuration file 12

Appendix B: KVM image details 13ELK VM software bundle 14Licenses used 15

Contents iii

Elasticsearch, Logstash, and Kibana (ELK) VM Deployment Guide

iv Contents


Preface

This document describes and provides instructions for deploying theElasticsearch, Logstash, and Kibana (ELK) VM template on Hitachi HyperScale-Out Platform (HSP).

This Preface includes the following information:

❒ Intended audience

❒ Product version

❒ Document revision level

❒ Document conventions

❒ Accessing product documentation

❒ Getting help

❒ Comments

Preface v


Intended audience

This document is intended for system administrators, Hitachi Data Systemsrepresentatives, and authorized service providers who need to deploy theELK VM template on Hitachi Hyper Scale-Out Platform (HSP).

Readers of this document should be familiar with the following:

• ELK administration

• Linux operating system and working in a restricted shell environment

• Site-specific network information

Product version

This document applies to Hyper Scale-Out Platform release 1.2.0.037 orlater.

Release notes

The release notes for this product are available on Hitachi Data SystemsSupport Connect: https://support.hds.com/en_us/contact-us.html.Read the release notes before installing and using this product. They maycontain requirements or restrictions that are not fully described in thisdocument or updates or corrections to this document.

Document revision level

Revision Date Description

MK-95HSP033-00 09 November 2016 Initial release

Intended audience

vi Preface



Document conventionsThis document uses the following typographic conventions:

Convention Description

Bold Indicates:

Text in a window or dialog box, such as menus, menu options,buttons, and labels. For example:On the Add Pair dialog box, clickOK.

Text appearing on screen or entered by the user. For example:The -split option…

<Italic> in anglebrackets

Indicates a variable, which is a placeholder for site- or installation-specific details that you need to provide. For example:copy <source-file> <target-file>

monospace Indicates text that is displayed on the screen or text that you needto enter. For example:

# pairdisplay -g oradb

Also, the name of a directory, folder, or file. For example:

The horcm.conf file…

This document uses the following icons to draw attention to information:

Icon Meaning Description

Important Calls attention to information that is essential to thecompletion of a task.

Caution Warns that failure to take or avoid a specified action couldresult in adverse conditions or consequences (for example,loss of access to data).

Accessing product documentation

Product documentation is available on Hitachi Data Systems SupportConnect: https://knowledge.hds.com/Documents. Check this site for themost current documentation, including important updates that may havebeen made after the release of the product.

Document conventions

Preface vii


https://knowledge.hds.com/Documents

Getting help

Hitachi Data Systems Support Portal is the destination for technicalsupport of products and solutions sold by Hitachi Data Systems. To contacttechnical support, log on to Hitachi Data Systems Support Connect forcontact information: https://support.hds.com/en_us/contact-us.html.

Hitachi Data Systems Community is a global online community for HDScustomers, partners, independent software vendors, employees, andprospects. It is the destination to get answers, discover insights, and makeconnections. Join the conversation today! Go to community.hds.com,register, and complete your profile.

Comments

Please send us your comments on this document:

[email protected]

Include the document title and part number, including the revision (forexample, -01), and refer to specific sections and paragraphs wheneverpossible. All comments become the property of Hitachi Data Systems.

Thank you!

Getting help

viii Preface


https://support.hds.com/


https://community.hds.com/welcome

http://community.hds.com/welcome

mailto:[email protected]

Quick start

This chapter is a summary of the procedure for deploying an ELK cluster onHSP.

This summary is intended as a quick reference for users that are familiarwith manually deploying VMs on HSP.

The full procedure is described starting at Deploy the ELK VMs on HSP

❒ Before you begin

❒ Procedure

Quick start 1


1

Before you begin

Verify that you have:

• An initialized HSP cluster running at least HSP 1.2.0.037. Ensure thatyour HSP cluster has its DNS domain name, DNS server and NTP serverconfigured correctly.

• Downloaded copy of the qcow2 file containing the repository-vm for ELK.This file is available from Technical Information Service Center (TISC),and may be called something like ELK_HSP-1.1.1_r1.qcow2

• A Linux client

Procedure

1. Add the ELK template to the HSP cluster.

2. Deploy the ELK VMs. One or more ELK VM can be deployed to form theELK cluster.

3. Log into the first ELK VM instance as root.

ssh <ELK-VM Instance IP address>

Username: rootPassword: smrace1

4. Run the following script to install, setup, and start the ELK services:

/opt/escale/install/install_elk.py -m 3 -d 2

Where:Value for m = sets the number of master nodes in the ELK cluster.In the example above this is set to 3 master nodes.

Value of d = sets the number of data nodes in the ELK cluster.In the example above this is set to 2 data nodes.

2 Quick start


Before you begin

5. When the script has finished running create a Logstash configurationLogstash default configuration fileat:

/etc/logstash/conf.d/

Note: The default Logstash configuration file can be seen in Appendix A:Logstash Configuration File.See the www.elastic.co website for details on creating your own Logstashconfiguration file.

6. Check that all of the services are running using the following commands:

service elasticsearch statusservice logstash statusservice kibana status

Quick start 3


Procedure

http://www.elastic.co/

4 Quick start


Deploy the ELK VMs on HSP

This chapter describes the procedure for deploying the ELK cluster. At theend of this procedure the VM instances will be deployed and ready to beused.

❒ Before you begin

❒ Procedure

Deploy the ELK VMs on HSP 5


2

Before you begin

Verify that you have:

• An initialized HSP cluster running at least HSP 1.2.0.037. Ensure thatyour HSP cluster has its DNS domain name, DNS server and NTP serverconfigured correctly.

• Downloaded copy of the qcow2 file containing the repository-vm for ELK.This file is available from Technical Information Service Center (TISC),and may be called something like ELK_HSP-1.1.1_r1.qcow2.

• Linux client.

Procedure

First, add the vm-template to the HSP cluster.

1. Mount a share from the HSP cluster on your Linux system:

linux# mount <HSP Cluster Virtual IP Address>:/<myshare> /mnt/hspWhere /myshare is the name of the available share.

2. Copy the qcow2 images from the ISOs to the share:

linux# cp /mnt/iso/ELK_HSP-1.1.1_r1.qcow2 /mnt/hsp

3. On the HSP cluster, add the vm-template:

admin@Node003:~$ hspadm vm-template add --name elk_template1 --image-format qcow2 --image-path <myshare>:/ELK_HSP-1.1.1_r1.qcow2

4. Verify this was successful:

admin@Node003:~$ hspadm vm-template list

6 Deploy the ELK VMs on HSP


Before you begin

Deploy the ELK template

1. Deploy the vm-template. In this example we are creating a 5 node ELKcluster.

admin@Node003:~$ hspadm vm-instance-group add -–name hsp_elk --num-instances 5 --vm-template elk_template1 --vm-size medium

2. Verify this was successful:admin@Node001:~$ hspadm vm-instance list --template elk_template1



Procedure

Installation and configuration script

Logging into the first VM instance launches a script that installs, configures,and starts the Elasticsearch, Logstash, and Kibana services on all of the ELKVM instances.

Note: The script installs the Logstash service on a data node but does notstart it. You must create you own Logstash configuration file to start theLogstash service. The default configuration file can be seen in Appendix A:Logstash Configuration File.See the www.elastic.co website for details on creating your own Logstashconfiguration file.

1. SSH to the first ELK VM instance (in this example 10.129.71.21 isused):

ssh 10.129.71.21 (VM instance address)

Username: rootPassword: smrace1

2. Run the following script to install, setup, and start the ELK services:

/opt/escale/install/install_elk.py -m 3 -d 2

Where:Value for m = sets the number of master nodes in the ELK cluster.In the example above this is set to 3 master nodes.

Value of d = sets the number of data nodes in the ELK cluster.In the example above this is set to 2 data nodes.



Procedure


3. When the script has finished running create a Logstash configuration at:

/etc/logstash/conf.d/

Note: The default Logstash configuration file can be seen in Appendix A:Logstash Configuration File.See the www.elastic.co website for details on creating your own Logstashconfiguration file.

4. Check that all of the services are running using the following commands:

service elasticsearch statusservice logstash statusservice kibana status



Procedure


Logstash Configuration File

This appendix contains the Logstash default configuration file.

You must create you own Logstash configuration file to start the Logstashservice.

See the www.elastic.co website for details on creating your own Logstashconfiguration file. The configuration file must be created at:

/etc/logstash/conf.d/.

❒ Logstash default configuration file

Appendix A: Logstash Configuration File 11


A


Logstash default configuration file

12 Appendix A: Logstash Configuration File


Logstash default configuration file

KVM image details

Elasticsearch, Logstash, and Kibana (ELK) deployment begins with a KVMbased VM image.

The package consists of one VM image:

• ELK_HSP-1.1.1_r1 ELK VM software bundle

❒ ELK VM software bundle

❒ Licenses used

Appendix B: KVM image details 13


B

ELK VM software bundle

• ELK VM with Elasticsearch, Logstash, and Kibana repositories.

¡ CentOS 7.2 x64 - Linux Operating system

¡ Installed software

– Oracle JDK 1.8 jdk-8u60-linux-x64 – Java virtual machine

– Python 2.7.5 – Python interpreter (for scripting language)

¡ Packaged Software

– Elasticsearch 2.3.3

– Logstash 2.3.2

– Kibana 4.5.1

– Lucene 5.5.0

14 Appendix B: KVM image details


ELK VM software bundle

Licenses used

This media kit includes software covered under the following licenses:

Apache License, Version 2.0 (the "License")

The Platform Components comprising the Hortonworks Data Platform (HDP)are released under the Apache 2.0 License. Unless required by applicablelaw, software distributed under the Apache License is distributed on an “ASIS” BASIS.

Licensed under the Apache License, Version 2.0 (the "License"); you maynot use this file except in compliance with the License. You may obtain acopy of the Apache License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS, WITHOUTWARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.

Optional Add-Ons and 3rd-Party Components

While HDP is provided under the terms of the Apache 2.0 License, it is alsocommonly used with 3rd-Party Components and Optional Add-Ons(example Hive ODBC Driver). When you choose to use those components, itis recommended you read and understand the licensing terms specific toeach of those components. Also note that Add-Ons can be downloadedseparately from HDP and each comes with instructions on how to install andleverage the features alongside HDP.

Trademarks:

Apache and the Apache feather logo are trademarks of The Apache SoftwareFoundation

CentOS is a trademark of Red Hat, Inc.

Hortonworks Data Platform (HDP) is trademark of Hortonworks Inc.

Python is a registered trademark of the Python Software Foundation

Appendix B: KVM image details 15


Licenses used

http://www.apache.org/licenses/LICENSE-2.0

16 Appendix B: KVM image details


Hitachi Data Systems

Corporate Headquarters2845 Lafayette StreetSanta Clara, California 95050-2627U.S.A.www.hds.com

Regional Contact Information

Americas+1 408 970 [email protected]

Europe, Middle East, and Africa+44 (0) 1753 [email protected]

Asia Pacific+852 3189 [email protected]

MK-95HSP033-00

http://www.hds.com/

http://[email protected]/

