tibco activespaces4 on gke k8s cloud · cloud. from apis and systems to devices and people, we...

TIBCO Software Inc.

Global Headquarters

3307 Hillview Avenue

Palo Alto, CA 94304

Tel: +1 650-846-1000

Toll Free: 1 800-420-8450

Fax: +1 650-846-1005

www.tibco.com

TIBCO fuels digital business by enabling better decisions and faster, smarter actions through the TIBCO Connected Intelligence Cloud. From APIs and systems to devices and people, we interconnect everything, capture data in real time wherever it is, and augment the intelligence of your business through analytical insights. Thousands of customers around the globe rely on us to build compelling experiences, energize operations, and propel innovation. Learn how TIBCO makes digital smarter at www.tibco.com.

TIBCO ActiveSpaces™ on GKE This document describes how to configure and run the TIBCO ActiveSpaces 4 in a Google Kubernetes Engine (GKE) environment.

Version 1.0 July 2019 Initial Document

©2019 TIBCO Software Inc. All Rights Reserved. 2

Copyright Notice COPYRIGHT© 2019 TIBCO Software Inc. All rights reserved.

Trademarks TIBCO, the TIBCO logo, TIBCO ActiveSpaces are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries. All other product and company names and marks mentioned in this document are the property of their respective owners and are mentioned for identification purposes only.

Content Warranty The information in this document is subject to change without notice. THIS DOCUMENT IS PROVIDED "AS IS" AND TIBCO MAKES NO WARRANTY, EXPRESS, IMPLIED, OR STATUTORY, INCLUDING BUT NOT LIMITED TO ALL WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. TIBCO Software Inc. shall not be liable for errors contained herein or for incidental or consequential damages in connection with the furnishing, performance or use of this material.

For more information, please contact:

TIBCO Software Inc. 3303 Hillview Avenue Palo Alto, CA 94304 USA


Table of Contents

1 Overview ................................................................................................................................ 5 1.1 GKE/AS4 Architecture ..................................................................................................................... 5

1.1.1 Development Architecture .......................................................................................................... 5 1.1.2 Production Architecture .............................................................................................................. 5

1.2 Supported Versions ........................................................................................................................ 6 1.3 Prerequisites .................................................................................................................................. 6 1.4 Prepare Local Environment ............................................................................................................. 6 1.5 Prepare Preliminary Google Account and Kubernetes Configuration ............................................... 7

2 Google GKE Setup .................................................................................................................. 8 2.1 Create a New Google Kubernetes Engine (GKE) Cluster ................................................................... 8 2.2 Configuring Kubectl to connect to Google Kubernetes Engine ......................................................... 9 2.3 Configure the Google Container Registry ...................................................................................... 10

2.3.1 Load the TIBCO images into the Docker Registry ....................................................................... 10 2.3.2 Create the Google Container Registry ....................................................................................... 10 2.3.3 Tag and Push the Docker Images to GCR ................................................................................... 10

3 Configuring ActiveSpaces 4 in GKE ........................................................................................ 12 3.1.1 Configuring AS4 for Kubernetes ................................................................................................ 12 3.1.1 Stopping or Deleting the AS4 processes .................................................................................... 17

4 Accessing and Testing AS4 .................................................................................................... 19 4.1 Accessing AS4 internally ............................................................................................................... 19 4.2 Accessing AS4 Externally ............................................................................................................... 20 4.3 Monitoring ActiveSpaces .............................................................................................................. 21


Table of Figures FIGURE 1 - KUBERNETES CLUSTER CREATION EXAMPLE ............................................................................................................. 9 FIGURE 2 - CONFIGURE KUBECTL ........................................................................................................................................ 9 FIGURE 3 - VERIFY CONNECTING TO THE KUBERNETES CLUSTER ................................................................................................ 10 FIGURE 4 - TAG THE AS4 DOCKER IMAGE ........................................................................................................................... 11 FIGURE 5 – AS4-LB-SMALL.YAML EXAMPLE ......................................................................................................................... 14 FIGURE 6 - EXAMPLE OF AS4 STORAGE AND LB PODS RUNNING IN GKE ..................................................................................... 14 FIGURE 7 - RUNNING AS4 ENVIRONMENT ........................................................................................................................... 17 FIGURE 8 - TO STOP AND START THE AS4 STATEFULSETS ........................................................................................................ 17 FIGURE 9 - ACCESS TIBDG EXAMPLE ................................................................................................................................... 19 FIGURE 10 - ACCESS OPERATIONS APP EXAMPLE ................................................................................................................... 20 FIGURE 11 - EXTERNAL ACCESS WITH TIBDG EXAMPLE............................................................................................................ 20 FIGURE 12 - ACCESS AS4 EXTERNALLY FROM THE OPERATIONS APPLICATION ............................................................................... 21 FIGURE 13 - FTL MONITOR-START EXAMPLE ........................................................................................................................ 22 FIGURE 14 - IMPORTING THE AS4 DASHBOARDS ................................................................................................................... 22 FIGURE 15 - ACTIVESPACES GIRD ACTIVITY DASHBOARD ........................................................................................................ 23


1 Overview

This document will outline how to configure the TIBCO ActiveSpaces® v4.1.0 in a Kubernetes cluster on Google’s Kubernetes Engine (GKE) on the Google Cloud. Running TIBCO AS4 on GKE involves:

• Configuring the Google Kubernetes Engine (GKE) for TIBCO ActiveSpaces in the Google Cloud.

• Configuring the Google Container Registry (GCR) for the Docker® image registry, and hosting the AS4.1.0 and FTL5.4.1 Docker images in GCR.

• Configuring and creating Kubernetes containers based on the Docker images for the individual components

1.1 GKE/AS4 Architecture

Using this document, two different architectures can be produced. A smaller configuration suitable for development/testing, or a larger fault-tolerant configuration suitable for production. Essentially, they are the same, with the larger environment consisting of a F/T Realm Server, and additional copy sets, nodes, and proxies.

1.1.1 Development Architecture The development architecture created will contain:

• One (1) VPC • GKE cluster will consist of six (6) nodes with a minimum of 15 GB of RAM and 4 CPUs. • Can be created in one or multiple zones in a region. • SSD persisted disks for all AS4/FTL configuration and data. • GCR Registries for all containers • ELB (Kubernetes) for external access to the Realm Services and the AS4 Proxies. • One (1) AS4 Copyset • Two (2) AS4 Nodes (1 – replica set) • Three (3) AS4 Statekeepers • Two (2) AS4 Proxies • One (1) FTL Realm Server

1.1.2 Production Architecture The created architecture created will contain:

• One (1) VPC • AKS cluster will consist of nine (9) nodes with a minimum of 15 GB of RAM and 4 CPUs.

Larger cluster nodes can be used. • Can be created in one or multiple zones in a region. Three (3) zones in the same region is

recommended. • SSD persisted disks for all AS4/FTL configuration and data • GCR Registries for all containers


• ELB (Kubernetes) for external access to the Realm Services and the AS4 Proxies. • Two (2) AS4 Copysets • Four (4) AS4 Nodes (2 – replica sets) • Three (3) AS4 Statekeepers • Four (4) AS4 Proxies • Two (2) FTL Realm Servers – (FTL 5.4 Primary and Secondary configuration)

1.2 Supported Versions

The steps described in this document are supported for the following versions of the products and components involved:

• TIBCO ActiveSpaces 4.1.0 • TIBCO FTL 5.4.1 • Docker Community/Enterprise Edition should be most recent version, (at least 18.09.2), to

address recent security vulnerabilities • Kubernetes 1.12 or newer

Note: This version will not work with FTL 6.x.

1.3 Prerequisites

The reader of this document must be familiar with:

• Docker concepts • Google cloud console, the Google Cloud CLI (gcloud) • Kubernetes installation and administration • Kubernetes CLI, kubectl • TIBCO AS4 installation and configuration • All necessary downloads discussed in the next section • The appropriate TIBCO license(s), if required.

1.4 Prepare Local Environment

The following infrastructure should already be in place:

• A Linux or macOS machine equipped for building Docker images • The following software must already be downloaded to the Linux or macOS machine

equipped for building Docker images.

Note: All software must be for Linux!

• TIBCO ActiveSpaces v4.1 Docker images which are part of the TIBCO AS4 installation package. The Enterprise Edition or the Community Edition can be used. Download the EE from edelivery.tibco.com , or the CE at https://www.tibco.com/products/messaging-event-processing


• The tibas4_gke_files.zip. The zip file contains all of the necessary Kubernetes build files. Download from https://community.tibco.com/wiki/tibcor-messaging-article-links-quick-access

• Create a directory, place tibas4_gke_files.zip in the directory. • Unzip tibas4_gke_files.zip. • Copy all of the as-*.xz and ftl-*.xz Docker images from the TIBCO AS4 installation

package to tibas4_aks_files/docker directory.

1.5 Prepare Preliminary Google Account and Kubernetes Configuration

Use the following to prepare the preliminary environment to install the TIBCO AS4 on GKE.

• An active Google Cloud account is required. If necessary, create an account at https://console.cloud.google.com/and follow the on-screen instructions.

• Install the Google CLI on the workstation used.

• Install Docker on the workstation to load the TIBCO AS4 images.

• Install the kubectl command-line tool to manage and deploy applications to Kubernetes in GKE from a workstation.


2 Google GKE Setup

2.1 Create a New Google Kubernetes Engine (GKE) Cluster

A new Kubernetes cluster must be created in GKE. Use the following to build a new Kubernetes Service in the Google cloud. This should be created via the Google Cloud console.

• Sign into the Google Cloud at https://console.cloud.google.com/ • In the left-hand side of the Google console, select Compute > Kubernetes Engine. • Select a Create cluster. • Provide a new Kubernetes Cluster Name, Location type, Region, Master Kubernetes

version. o Cluster name can be anything, but should represent the use, such as as4gke. o Location type can be Zonal or Regional. Zonal will keep all nodes on one Zone.

Regional will spread nodes across multiple zones. For development/testing either can be used. For production, Regional is highly recommended.

o Select which region to run the Kubernetes cluster in. o Version select must be at least 1.12.8. The Default is recommended.

• For the default-pool, click on more options • Select the minimum node count of (6 or 9), depending on if you are building a development

or production environment. Note: For regional deployments, this would be 2 or 3. • Click on Enable autoscaling. Select the maximum number of nodes. It is recommended

to be +1 over the minimum. • For Node Pools, select the node size. Recommend at a minimum - 4 vCPU / 15 GB RAM,

n1-standard-4. Can be larger, but not smaller. • Select 30 GB for the local disk size. All AS4 data will be on a persisted SSD disk. • The rest of the selections can be left at their defaults if desired. • Click on Create. It will take several minutes to complete.


Figure 1 - Kubernetes Cluster creation example

2.2 Configuring Kubectl to connect to Google Kubernetes Engine

With GKE, the Kubernetes command line tool, kubectl, will be used to configure the Kubernetes cluster for AS4 on GKE. It is also possible to use Google Cloud Shell. This document will be based on using kubectl for configuration. After the Kubernetes cluster has been built, click on Connect for the newly create cluster. Copy, paste, and run the gcloud CLI access command. The following is an example of setting the command line access for kubectl. Note: The Google Project ID can be found by clicking on Google Cloud Platform in the upper left hand corner in the Google Cloud Console.

Figure 2 - Configure Kubectl

Use kubectl get nodes as shown in the following example to verify connecting to the cluster.


Figure 3 - Verify connecting to the Kubernetes Cluster

2.3 Configure the Google Container Registry

New GCR registry(s) must be created to host the TIBCO AS4.1.0/FTL 5.4.1 Docker images. Use this section to create the necessary GCR registry(s). Seven (7) Docker images need to be tagged and pushed to GCR. All images can be in the same registry, or separated in to individual registries. This document will create a registry for each of the Docker images.

2.3.1 Load the TIBCO images into the Docker Registry Load the the TIBCO AS4/FTL Docker images into the local Docker registry.

• Change directory to the directory where the AS4/FTL Docker images are located. • Load the the Docker images in the local registry:

docker load -i as-tibdg-4.1.0.dockerimage.xz docker load -i as-tibdgnode-4.1.0.dockerimage.xz docker load -i as-tibdgkeeper-4.1.0.dockerimage.xz docker load -i as-tibdgproxy-4.1.0.dockerimage.xz docker load -i as-tibdgadmind-4.1.0.dockerimage.xz docker load -i as-operations-4.1.0.dockerimage.xz docker load -i ftl-tibrealmserver-5.4.1.dockerimage.xz

2.3.2 Create the Google Container Registry The Google Container Registries must be created. These can be done for the command line using gloud.

• First, run the following command to configure Docker to use the GCR:

gloud auth configure-docker

• Follow the steps in https://cloud.google.com/container-registry/docs/pushing-and-pulling to create the Google Container Registries. It is recommended to create a registry for each of the seven (7) AS4/FTL Docker images. Note the names of your registries, and your Google Project ID.

2.3.3 Tag and Push the Docker Images to GCR Once the Google Container Registries have been created, the Docker images can be tagged and pushed to GCR. Tag the image and push the Docker images to the GCR registry using the URL of the appropriate registry


• Tag the image and push the Docker image to the GCR. In the following examples, us.gcr.io is where the tibdgnode registry was created. Note: Name of Docker image may differ depending on setup.

Figure 4 - Tag the AS4 Docker image

• Push the AS4 Docker image to GCR. In the following example, the tibdgnode image will be tagged latest in GCR.

• Tag and push the six (6) remaining Docker images to GCR similar to the tibdgnode example show above.


3 Configuring ActiveSpaces 4 in GKE

After the FTL and AS4 Docker images are pushed to GCR, Kubernetes can be configured to run the AS4/FTL containers.

3.1.1 Configuring AS4 for Kubernetes There are different templates used depending on if the development or production environment is being configured. In the tibas4_gke_files/kubernetes directory, there will be five templates; two for the development environment (small), and two for the production environment (large), and a fifth template, as4-storage.yaml, which is used by both to create the required persisted storage. The as4-storage.yaml file will create the persisted SSD storage. This can be modified to be standard hard drives, if desired. See the Google/Kubernetes documentation for details. The as4-gke-small.yaml and the as4-gke-large.yaml are used to create the bulk of the environment. The as4-lb-small.yaml and the as4-lb-large.yaml will create the Kubernetes load balancers for the environment. The only difference with the load balancer files is the number of Realm Server and proxy load balancers created. The main configuration file and the LB configuration file will need minor changes. The storage configuration file requires no modifications.

3.1.1.1 Load Balancer Configuration Following, is the tibas4_gke_files/kubernetes/as4-lb-small.yaml used to configure the load balancers. The are three load balancers created for the development environment, while there six created in the production environment. The change required is exactly the same for both. The trusted IP range will determine what IP addresses can connect to the load balancer. NOTE: The load balancer file must be configured and applied first. No other changes should be made other than the trusted IP range. If port number changes are made, the port changes must also be made in the as4-gke yaml file. It will take a few minutes for all of the load balancers to be created.

apiVersion: v1 kind: Service metadata: labels: name: realmserver-lb name: realmserver-lb spec: externalTrafficPolicy: Cluster ports: - name: realm nodePort: 30080 port: 30080 targetPort: 30080 protocol: TCP - name: admind


nodePort: 30081 port: 30081 targetPort: 30081 protocol: TCP - name: ftl nodePort: 30083 port: 30083 targetPort: 30083 protocol: TCP selector: com.tibco.datagrid.service: realmserver sessionAffinity: None type: LoadBalancer loadBalancerSourceRanges: - <your trusted IP range in the form of 0.0.0.0/0> (1) status: loadBalancer: {} --- apiVersion: v1 kind: Service metadata: labels: app: tibdg name: proxy-0-lb name: proxy-0-lb spec: externalTrafficPolicy: Cluster ports: - name: proxy-0 nodePort: 30086 port: 30086 targetPort: 8555 protocol: TCP selector: com.tibco.datagrid.service: tibdgproxy sessionAffinity: None type: LoadBalancer loadBalancerSourceRanges: - <your trusted IP range in the form of 0.0.0.0/0> (1) status: loadBalancer: {} --- apiVersion: v1 kind: Service metadata: labels: app: tibdg name: proxy-1-lb name: proxy-1-lb spec: externalTrafficPolicy: Cluster ports: - name: proxy-1


nodePort: 30087 port: 30087 targetPort: 8555 protocol: TCP selector: com.tibco.datagrid.service: tibdgproxy sessionAffinity: None type: LoadBalancer loadBalancerSourceRanges: - <your trusted IP range in the form of 0.0.0.0/0> (1) status: loadBalancer: {}

Figure 5 – as4-lb-small.yaml Example

(1): The trusted IP address range to connect to the load balancer.

3.1.1.2 Applying the LB and Storage configurations in Kubernetes Once the AS4 LB yaml files has been updated, this file, along with the storage yaml files can be applied using kubectl to GKE. Use kubectl apply –f as4-storage.yaml,as4-lb-small.yaml to apply both files. If will take several minutes for the external IP Addresses to be created. Use kubectl get storageclass,svc to verify the storage classes and load balancers are available. Do not continue until the load balancers have been assigned an External IP address similar to the following example. These IP addresses are required for the main AS4 yaml file.

Figure 6 - Example of AS4 storage and LB pods running in GKE

3.1.1.3 AS4 configuration Once there are load balancer IP addresses for the proxies, the main configuration file as4-gke-small.yaml can be configured. On the again, changes to the as4-gke-large.yaml files are the same, just additional locations for modification. The only changes required is to the proxy_client_listen_external_host for proxy-0 and proxy-1 and the GCR image for each container. Note: The example below is not the entire yaml file. Ensure the GCR image is changed in all locations.


--- # This is used to populate the grid configuration. apiVersion: v1 kind: ConfigMap metadata: name: tibdg-conf labels: app: tibdg data: conf.tibdg: | grid create copyset_size=2 proxy_client_listen_port=8555 copyset create cs-01 node create --copyset cs-01 --dir /data/cs-01-node-0 cs-01-node-0 node create --copyset cs-01 --dir /data/cs-01-node-1 cs-01-node-1 keeper create --dir /data/keeper-0 keeper-0 keeper create --dir /data/keeper-1 keeper-1 keeper create --dir /data/keeper-2 keeper-2 proxy create proxy_mirroring_listen_port=8556 proxy_client_listen_external_host=<proxy-0 external IP address> proxy_client_listen_external_port=30086 proxy-0 (1) proxy create proxy_mirroring_listen_port=8557 proxy_client_listen_external_host=<proxy-1 external IP address> proxy_client_listen_external_port=30087 proxy-1 (1) table create t1 key long column create t1 value string ... --- apiVersion: apps/v1 kind: StatefulSet metadata: name: realmserver labels: com.tibco.datagrid.service: realmserver app: tibdg spec: serviceName: realmserver replicas: 1 selector: matchLabels: com.tibco.datagrid.service: realmserver template: metadata: name: realmserver labels: com.tibco.datagrid.service: realmserver app: tibdg spec: containers: - name: realmserver


image: <your GCR registry for the FTL 5.4 realm server container> (2) imagePullPolicy: Always resources: requests: cpu: 1 memory: 2Gi limits: cpu: 2 memory: 4Gi args: - '--ftl' - '*:30083' - '--gui' - '*:30085' - '--http' - '*:30080' volumeMounts: - mountPath: /data name: realmserver-data - name: tibdgadmind image: <your GCR registry for the AS4 admind container> (2) imagePullPolicy: Always resources: requests: cpu: .25 memory: 1Gi limits: cpu: .5 memory: 2Gi args: [ '-r', 'http://realmserver:30080', '-l', ':30081' ] volumeClaimTemplates: - metadata: name: realmserver-data spec: accessModes: - ReadWriteOnce storageClassName: as4-ssd resources: requests: storage: 5Gi

(1): The proxy_client_listen_external_host. For the the development (small) configuration, this is proxy-0 and proxy-1. For the large configuration, this will also include proxy-2 and proxy-3. Insert the value of the External IP for the matching load balancer. An example would be 40.124.10.20. (2): The name and location of the Google Container Registry (GCR) where the FTL/AS4 Docker containers are located. Ensure the proper permissions are set. The image maybe something


different than latest, depending on how it was tagged in Docker. Change all locations referencing the GCR. Note: If port numbers or other changes are made, the port number changes must also be made in the as4-lb yaml file. If is not recommended to make other changes unless, you are very familiar with Kubernetes. After making the necessary modifications, use kubectl apply –f as4-gke-small.yaml to apply the configuration to Kubernetes. Note: This can take a few minutes, and there will be some pod restarts. This is normal. Use kubectl get pods to get the status of the pods. Wait until all pods are running, and the tibdgconfig pod has completed before continuing.

Figure 7 - Running AS4 environment

3.1.1 Stopping or Deleting the AS4 processes To stop all of the processes without deleting them, use the kubectl scale operation to set its number of replicas to 0. For example: > kubectl scale --replicas=0 statefulset proxy,keeper,cs-01-node,realmserver

To start the process again, set its number of replicas back to their original values. Node is two, keeper is three, proxy is two, and realmserver is one. For example: > kubectl scale --replicas=3 statefulset keeper

Figure 8 - To Stop and Start the AS4 Statefulsets

To delete the all of the statefulsets and service entirely, use the kubectl delete operation: kubectl delete –f as4-gke-small.yaml,as4-lb-small.yaml,as4-storage.yaml


All of the corresponding pods, statefulsets, storage classes, and LB services will be deleted. The PVC and PV will not be deleted, nor will the corresponding data. To delete the data, PV, and PVC, use the following: Kubectl delete pvc,pv --all


4 Accessing and Testing AS4

This section will outline testing ActiveSpaces 4 running in GKE. Access can be tested internally and externally.

4.1 Accessing AS4 internally

Testing access internally can be done by trying to access tibdg and running the operations sample application. The External IP Address the realmserver-lb is all that is needed. Use: docker run –rm –it as-tibdg:4.1.0 –r http://<IP of RS LB>:30080 status. Substitute the location of the as-tibdg:4.1.0 for your Docker registry, and your Realm Server LB external address.

Figure 9 - Access Tibdg example

To test with the operations apps. Use the following: docker run –rm -it as-operations:4.1.0 –r http://<IP of RS LB>:30080


Figure 10 - Access Operations app example

4.2 Accessing AS4 Externally

Accessing ActiveSpaces can be from an external site. To do this test, ActiveSpaces 4 must be installed on a workstation. You will still need the External IP Address for the Realm Server LB. To test with tibdg:

• Change directory to /opt/tibco/as/4.1/bin • Run ./tibdg –r http://<external IP of the RS LB>:30080 status. You should see something

similar to the following example.

Figure 11 - External Access with Tibdg example

Testing with the operations application is similar: • Change directory to /opt/tibco/as/4.1/samples/bin


• Run ./operations –r http://<external IP for the RS LB>. You should get the operations menu. Feel free to add/get/delete data. This will verify that AS4 is active and available from an external source.

Figure 12 - Access AS4 externally from the Operations application

4.3 Monitoring ActiveSpaces

AS4 can be monitored utilizing the FTL5.4/AS4.1 monitoring utilities from an external location. You will need the External IP Address for the Realm Server LB. Note: The Enterprise editions of FTL 5.4.1 and AS 4.1.0 are required for the monitoring components.

• On the workstation that was used to build the GKE environment, install both FTL5.4.1 and ActiveSpaces 4.1.0 following their respective installation guides.

• Once both products are installed, change directory to /opt/tibco/ftl/5.4/monitoring • Run ./monitor-start.py operations –r http://<external IP for the RS LB>.

Note: On a Mac (Darwin 18.6.0) it may be necessary to modify monitor-start.py, and remove releases from line3. You should get output similar to the following example:


Figure 13 - FTL monitor-start example

• Change directory to /opt/tibco/as/4.1/monitor/scripts, and run ./import-activespaces-dashboards. This will import the AS4 dashboards into the FTL monitor. This only has to be done once. The output should be similar to the following example:

Figure 14 - Importing the AS4 dashboards

• Using a web browser, go to http://<machine running the monitoring software>:3000 • Login in using admin/admin. The user password can be changed, as well as securing the

monitor. See the FTL 5.4 and ActiveSpaces 4.1 documentation for details. • Click on the Home to get the Dashboards, and then click on the ActiveSpaces Grid Activity

dashboard. You should get something similar to the following example:


Figure 15 - ActiveSpaces Gird Activity Dashboard

• Feel free to use the operations sample application to enter data into the grid, while monitoring ActiveSpaces. The rates will change as data is added/deleted.

• Use monitor-stop.py to stop the monitoring software.

tibco activespaces4 on gke k8s cloud · cloud. from apis and systems to devices and people, we...

Documents