streamanaltix docker installation guidedocs.streamanalytix.com/2.1.6/pdf/docker.pdf · docker...

StreamAnalytix Docker Installation Guide 2.1.6

© 2017 Impetus Technologies, Inc.

STREAMANALYTIX DOCKER

INSTALLATION GUIDE 2.1.6

GO VIRTUAL Start running the StreamAnalytix Image on Docker.


© 2017 Impetus Technologies, Inc. of 32

Introduction Welcome to StreamAnalytix! StreamAnalytix platform enables enterprises to analyze and respond to events in real-time at Big Data scale. With its unique multi-engine architecture, StreamAnalytix provides an abstraction that offers a flexibility to execute data pipelines using a stream processing engine of choice depending upon the application use-case, taking into account the advantages of Storm or Spark Streaming based upon processing methodology (CEP, ESP) and latency. Key Features:

• Multi-engine platform with support for Apache Spark Streaming and Apache Storm • Visual Pipeline Designer for rapid application development • Rich array of drag-and-drop operators for data source, sink, transformation, and analytics • Built-in dashboards for real-time data visualization • Configuration, Management, and Monitoring through an easy-to-use web interface

About This Guide This guide describes how to install StreamAnalytix Docker Engine.

For More Information Please visit www.streamanalytix.com To give us your feedback on your experience with the application and report bugs or problems, mail us at [email protected] To receive updated documentation in the future please register yourself at www.streamanalytix.com We welcome your feedback.

http://www.streamanalytix.com/

mailto:[email protected]

http://www.streamanalytix.com/



Terms & Conditions

This manual, the accompanying software and other documentation, is protected by U.S. and international copyright laws, and may be used only in accordance with the accompanying license agreement. Features of the software, and of other products and services of Impetus Technologies, may be covered by one or more patents. All rights reserved.

All other company, brand and product names are registered trademarks or trademarks of their respective holders. Impetus Technologies disclaims any responsibility for specifying which companies own which marks or which organizations.

USA Los Gatos Impetus Technologies, Inc. 720 University Avenue, Suite 130 Los Gatos, CA 95032, USA Ph.: 408.252.7111, 408.213.3310 Fax: 408.252.7114 © 2016 Impetus Technologies, Inc., All rights reserved.

If you have any comments or suggestions regarding this document, please send them via e-mail to [email protected]

mailto:[email protected]



Table of Contents Introduction ............................................................................................................................................... 2

About This Guide .................................................................................................................................... 2

For More Information ............................................................................................................................ 2

Terms & Conditions ............................................................................................................................ 3

About This Guide ................................................................................................................................ 5

Docker Overview ................................................................................................................................ 5

Prerequisites....................................................................................................................................... 5

Running StreamAnalytix Image on Docker Engine for the First Time ................................................... 5

Accessing StreamAnalytix .................................................................................................................... 13

Stop the container ............................................................................................................................... 18

Linux distributions ............................................................................................................................ 18

Windows and Mac ............................................................................................................................ 18

Running an Existing StreamAnalytix Container ................................................................................... 18


Windows and Mac ............................................................................................................................ 19

Remove the StreamAnalytix image ..................................................................................................... 19


Windows and Mac ............................................................................................................................ 20

Debug Notes ........................................................................................................................................ 20

Stop, Start and Restart StreamAnalytix Services ................................................................................. 21

Supervisord....................................................................................................................................... 21

Appendix .............................................................................................................................................. 23

Configure StreamAnalytix Connections ........................................................................................... 23

Remove the StreamAnalytix image ..................................................................................................... 29

Best Practices ................................................................................................................................... 29

Important URLs ................................................................................................................................ 31



About This Guide

This guide takes you through the fundamentals of running StreamAnalytix by using Docker Engine and integrating it into your environment.

Docker Overview

Docker provides a way to run applications securely, isolated in a container, packaged with all its dependencies and libraries. Containers are lightweight and run without the load of a hypervisor. You can run multiple applications that all rely on different libraries and environments on a single kernel, without interference. This guide is broken into major sections that take you through learning the basics of installing and running StreamAnalytix Docker Image.

Prerequisites

To use the StreamAnalytix Docker Image with Docker Engine, your system should meet the following requirements: 1. Install Docker Engine.

• Download the Docker Engine from here: https://docs.docker.com/engine/installation/

• Supported OS Windows Requires Microsoft Windows 10 Professional or Enterprise 64-bit Mac Requires Apple MacOS, Yosemite 10.10.3 or above Linux distributions Like Ubuntu 2. Minimum Hardware requirements:

Note: The StreamAnalytix Docker is not supported on Internet Explorer

System Requirements

Component Minimum Requirement CPU 64-bit machine with a chip that supports virtualization.

Number of cores: 4 Minimum.

RAM 16 GB (Minimum 16 GB for Virtual Machine) Supported Browsers

Firefox 39.0+ Google Chrome 23.0+ Safari 9.1

Running StreamAnalytix Image on Docker Engine for the First Time Listed below are instructions for running the StreamAnalytix Image on Linux distributions, Windows and Mac:

https://docs.docker.com/engine/installation/



Linux distributions

Start Docker Open new terminal and enter below command.

sudo service docker start Where, Property Description storage-driver Storage driver to use. storage-opt=[] Storage driver options. dm.basesize Specify the Base size; Base size is the maximum size that a

container/image can grow to.

1. Pull the StreamAnalytix Image from Docker hub. Open a terminal and run the below command.

sudo docker pull impetus/streamanalytix

2. List images.

Open a terminal and run the below command. sudo docker images

3. Run the impetus/streamanalytix image. Run the image in different ways listed below.

Option 1: Run the image with random IP and random ports. Run the below command. sudo docker run -i -t –P impetus/streamanalytix Where, Property Description -t Allocate a pseudo-TTY. -i Keep STDIN open even if not attached.



-P Publish all exposed ports to random ports. Option 2: Run the image with static IP and random ports. If you want to allocate a static IP to the container, create a user-defined network so that static IP can be assigned to StreamAnalytix Container. Use the below command to create a user defined network named saxnet. sudo docker network create --driver=bridge --subnet=10.10.0.0/16 saxnet

Run the impetus/streamanalytix image by entering the below command. sudo docker run --net saxnet --ip 10.10.0.8 -i -t –P Where, Property Description --net Name of the user defined network. --ip IP to be allocated to the container. Option 3: Run the image with Static IP and on specific ports. To map a port of the container to a specific port on the host machine, run the below command. sudo docker run -i -t -p 127.0.0.1:8088:8088 -p 127.0.0.1:50070:50070 -p 127.0.0.1:50075:50075 -p 127.0.0.1:50090:50090 -p 127.0.0.1:15672:15672 -p 127.0.0.1:9090:9090 -p 127.0.0.1:9001:9001 -p 127.0.0.1:9200:9200 -p 127.0.0.1:8080:8080 -p 127.0.0.1:8090:8090 -p 127.0.0.1:2222:22 -p 127.0.0.1:8042:8042 -p 127.0.0.1:8011:8011 -p 127.0.0.1:16010:16010 impetus/streamanalytix Where, -p to map a port of the container to specific port on the host. Using any of the above options, when the impetus/streamanalytix image comes in running state, you will get the CentOS terminal on which StreamAnalytix and all other services are installed. Look for StreamAnalytix is running, on the CentOS terminal, this means that StreamAnalytix web application is up now and you can access it on http://127.0.0.1:8090/StreamAnalytix

http://127.0.0.1:8090/StreamAnalytix



4. To list the Containers, open terminal of host machine and run the below command. sudo docker ps

For Windows and Mac

1. Start Docker: Windows:

o Double click icon Docker for Windows.

o Open Docker settings and set resources: Memory: 10 GB minimum Core: 2 cores minimum



Mac:

o Double click icon Docker

o Open Docker settings and set resources: Memory: 10 GB minimum

Core: 2 cores minimum



Windows output:

1. Pull the StreamAnalytix Image from Docker hub. Open command prompt\Terminal and run the below command.

docker pull impetus/streamanalytix

2. List images.

docker images

Mac output:



3. Run the impetus/streamanalytix image. The image can be run in various ways listed below:

Option 1: Run the image with random IP and random ports Run the below command. docker run -i -t –P impetus/streamanalytix Where, Property Description -t Allocate a pseudo-TTY. -i Keep STDIN open even if not attached. -P Publish all exposed ports to random ports. Option 2: Run the image with Static IP and random ports If you want to allocate a static IP to the container, create a user-defined network so that static IP can be assigned to StreamAnalytix Container. Use the below command to create a user defined network named saxnet. docker network create --driver=bridge --subnet=10.10.0.0/16 saxnet

Run the impetus/streamanalytix image by entering the below command. docker run --net saxnet --ip 10.10.0.8 -i -t –P Where, Property Description --net Name of the user defined network. --ip IP to be allocated to the container. Option 3: Run the image with Static IP and on specific ports. If you want to map a port in the container to specific port on the host machine, run the below command. docker run -i -t -p 127.0.0.1:8088:8088 -p 127.0.0.1:50070:50070 -p 127.0.0.1:50075:50075 -p 127.0.0.1:50090:50090 -p 127.0.0.1:15672:15672 -p 127.0.0.1:9090:9090 -p 127.0.0.1:9001:9001 -p 127.0.0.1:9200:9200 -p 127.0.0.1:8080:8080 -p 127.0.0.1:8090:8090 -p 127.0.0.1:2222:22 -p 127.0.0.1:8042:8042 -p 127.0.0.1:8011:8011 -p 127.0.0.1:16010:16010 impetus/streamanalytix Where,



Property Description -p to map a port of the container to specific port on the host. When the impetus/streamanalytix image comes in running state using any of the above options, you will get the CentOS terminal on which StreamAnalytix and all other services are installed. Look up for “StreamAnalytix is running“, on the CentOS terminal, this means that StreamAnalytix application is up now and you can access it on http://127.0.0.1:8090/StreamAnalytix Windows Output :

Mac Output:

4. To list the Containers, open terminal of host machine and run the below command.

http://127.0.0.1:8090/StreamAnalytix



docker ps Windows Output:

Mac Output:

Accessing StreamAnalytix 1. Access StreamAnalytix welcome page.

URL - http://127.0.0.1:8090/StreamAnalytix/

http://127.0.0.1:8090/StreamAnalytix/



2. Upload License.



3. Confirm Upload.

4. Click Continue.



5. Click on Start Here.



5. Login into the application using the following credentials. Superuser login username – superuser Password - superuser.

6. When running pipeline on StreamAnalytix container, you may get the container id at the

place of IP when opening some links, such as: • Application URL for the running pipeline.

• Alternatively, even for accessing logs of applications.

In any such case, replace the container id by the IP of the container. You can check the IP with the ifconfig command on CentOS terminal.



Stop the container Linux distributions

Exit the impetus/streamanalytix container by typing exit on the CentOS terminal. Open terminal of host machine and run the below command.

sudo docker ps

Copy the container Id for impetus/streamanalytix and run the below command to stop the container.

sudo docker stop “container Id”

Windows and Mac

Exit the impetus/streamanalytix container by typing exit on the CentOS terminal. Open terminal of host machine and run the below command.

docker ps

Copy the Container Id for impetus/streamanalytix and run the below command to stop the container.

docker stop “container Id”

Running an Existing StreamAnalytix Container

Linux distributions

Open terminal of host machine and run the below command.

sudo docker ps

Copy the container Id for impetus/streamanalytix and run the below command.

sudo docker start -i "containerID"



Windows and Mac

Open terminal of host machine and run the below command.

docker ps

Copy the container Id for impetus/streamanalytix and run the below command.

docker start -i "containerID"

Remove the StreamAnalytix image

Linux distributions

1. The below command is run to stop the container.

sudo docker stop “container Id”

Copy the container Id from the results of the below command.

sudo docker ps

2. The below command is run to remove the container.

sudo docker rm “container Id”


sudo docker rm “container Id”

3. The below command is run to remove the image.

sudo docker rmi “image Id”

Copy the image Id from the results of the below command.

sudo docker images



Windows and Mac

1. The below command is run to stop the container.

docker stop “container Id”


docker ps

2. The below command is run to remove the container.

docker rm “container Id”


docker rm “container Id”

3. The below command is run to remove the image.

docker rmi “image Id”

Copy the image Id from the results of the below command.

docker images

Debug Notes 1. For all debug and log related information check the following paths in StreamAnalytix image:

• Supervisord logs - /var/log/supervisor/supervisord.log • Path to tomcat logs – /usr/local/StreamAnalytix/server/tomcat/logs • Path to storm worker logs - /usr/local/apache-storm-0.10.0/logs • Spark pipelines will be submitted over yarn, spark pipelines UI and logs can accessed from

http://127.0.0.1:8088 2. All services are installed in the StreamAnalytix image using root as a user with Username: root Password: root

http://127.0.0.1:8088/



StreamAnalytix location on CentOS is /usr/local/StreamAnalytix 3. By default, the following components are installed and configured on StreamAnalytix image:

• Apache Storm - 0.10.0 • Elasticsearch - 2.2.0 • Graphite – 0.9.15 • Kafka – 2.10-0.9.0.1 • PostgreSQL – 9.3 • RabbitMQ – 3.3.5 • Tomcat – 7.0 • Zookeeper – 3.4.6 • Hadoop- 2.7.1 • Hbase-1.2.4

4. Following components are also supported, but not installed by default on StreamAnalytix image:

• Apache Solr - 4.10.2 • Apache Cassandra • Apache ActiveMQ • Amazon S3

5. Following features are not supported, on StreamAnalytix image:

• Spark Data Lineage • Storm Data Lineage • Spark Inspect Pipeline • Spark Scala processor • Storm Hive Emitter

Stop, Start and Restart StreamAnalytix Services Supervisord

Supervisord is preinstalled and can be used to change the status of the services. To access Supervisord, use URL http://127.0.0.1:9001/ Note: Access Supervisord service ONLY to stop/start/restart individual StreamAnalytix services.

http://127.0.0.1:9001/



Appendix Configure StreamAnalytix Connections

Prerequisites

1. All StreamAnalytix Persistence components must be in operation, including Apache Hadoop/HBase or Cassandra.

2. Indexing component SOLR must be setup and in operation. 3. Update the /etc/hosts file with IP and host mapping for the machines hosting

Persistence/Indexing nodes. 4. Restart StreamAnalytix (Tomcat) when the below configuration changes are complete from

SupervisorD Interface

Note: For Docker, StreamAnalytix by default supports Elasticsearch as indexing so it comes preconfigured with the docker image.

Configuration

Update the /etc/hosts file with IP and host mapping for the machines hosting.

Create connection

Only the superuser has the privilege to create connections while other users can select and use these connections. Login as a superuser, go to Connections and follow the steps mentioned below to create connections.

HBase To create a connection on HBase, perform the following steps:

1. Click the Add Connection button. The system displays a connection window.



2. Select HBase and enter the remote connection details: • Connection name: Name of the connection. (e.g. HBase) • HDFS User: HDFS user name. (e.g. sax) • Zookeeper Hosts: Host names of the machines where HBase is installed. • Zookeeper Port: Port of the machine where Zookeeper is available. (e.g. 2181) • Client Retries Number. Number of retries. (e.g. 1) • Zookeeper Recovery Retry: Number of retries. (e.g. 1) • Zookeeper Parent Node: Parent node name HBase. (e.g. /hbase)



3. To test the connection, click the Test Connection button on the connection window. The system displays one of the following:

• Connection Available



• Connection Unavailable

Cassandra Similarly, for other connections as well, you can Add Connection and Test Connection. Provide the following details.

a. Component Type: The Component (Cassandra) b. Connection name: Name of the connection (e.g. Cassandra) c. Hosts: Hosts and ports of the machine where Cassandra is installed. d. Connection Retries: Connection Retries (e.g. 1)



If you click the Authentication Enabled check box, the system will prompt you to enter username and password. This enables authentication and displays the required Cassandra properties for its connection. HDFS For HDFS connection, choose HDFS from the component list and provide the following details.

a. Component Type: Name of the Component (HDFS) b. Connection name: Name of the connection (e.g. HDFS Connection) c. FS URI: File System URI of the machine where HDFS installed d. Username: User name of the HDFS (e.g. sax)



Solr

On Solr connection tab, provide the following details: a. Connection name: Name of the connection. (e.g. AnSolr) b. Zookeeper Hosts: Host names and ports of the machine.

Kafka On Kafka connection tab, provide the following details:

a. Component Type: Name of the Component. (Kafka) b. Connection Name: Name of the connection. (e.g. Default) c. ZK Hosts: Hosts name and ports of Zookeeper machine. d. Kafka Brokers: Hosts name and ports of Kafka machine.



Remove the StreamAnalytix image To remove the StreamAnalytix Docker image, perform the following steps: 1. The below command is run to remove the container. sudo docker rm “container Id” Copy the container Id from the results of the below command. sudo docker ps

2. The below command is run to remove the image. sudo docker rmi “image Id” Copy the image Id from the results of the below command.

sudo docker images

Best Practices



Spark pipelines are memory intensive so in order to run multiple business pipelines, you can modify the memory and cores for a particular pipeline, while saving the pipeline, by default yarn is configured with 10 GB memory and 12 cores for running spark pipeline in impetus/streamanalytix container. In order to run more number of spark pipelines you can modify the yarn.scheduler.capacity.maximum-am-resource-percent property in capacity-scheduler.xml at /usr/local/hadoop-2.7.1/ etc/hadoop/capacity-scheduler.xml in the Docker container and restart Hadoop services from SupervisorD. In order to run alerts and monitoring, respective Superuser pipelines should be brought up. To start monitoring on the Spark pipelines, perform the steps below: 1. Login as a Superuser. 2. Go to Data Pipeline link on the menu. 3. Select the Spark Tab. 4. Play the SparkMetricsWriterPipeline and SparkMonitoringMetricPipeline to enable monitoring.

To start monitoring and run alerts on the Storm pipeline, perform the steps below: 1. Login as SuperUser. 2. Go to Data Pipeline link on the menu. 3. Select the Storm Tab. 4. For Alerts, play the AlertPipeline. 5. For Monitoring, play the GraphiteWriterPipeline and MonitorMetricPipeline to enable monitoring.



Note: Do not start Storm and Spark pipelines together. To receive emails, perform the following steps to setup the email server: 1. Login as Superuser. 2. Go to configuration link from the menu. 3. Select Environment Tab. 4. Select Activiti. 5. Specify values for Host, User, Default Sender email and Alert Sender Email.

Important URLs

• Apache Storm http://127.0.0.1:9021/index.html

• Graphite http://127.0.0.1:8080/

• StreamAnalytix http://127.0.0.1:8090/StreamAnalytix/

• RabbitMQ Server http://127.0.0.1:15672/

• ElasticSearch http://127.0.0.1:9200/_plugin/head/

• Supervisord http://127.0.0.1:9001/

• Hbase http://127.0.0.1:16010/

http://127.0.0.1:9021/index.html

http://127.0.0.1:8080/

http://127.0.0.1:8090/StreamAnalytix/

http://127.0.0.1:15672/

http://127.0.0.1:9200/_plugin/head/

http://127.0.0.1:9001/

http://127.0.0.1:16010/



• Hadoop

o Primary Namenode http://127.0.0.1:50070

o Secondary Namenode http://127.0.0.1:50090

o Datanode http://127.0.0.1:50075

o Yarn UI http://127.0.0.1:8088

http://127.0.0.1:50070/

http://127.0.0.1:50090/

http://127.0.0.1:50075/

streamanaltix docker installation guidedocs.streamanalytix.com/2.1.6/pdf/docker.pdf · docker...

Documents