red hat mobile application platform 4.1 mbaas ......mbaas runs in a docker container. those...

Red Hat Customer ContentServices

Red Hat Mobile Application Platform4.1MBaaS Administration andInstallation Guide

For Red Hat Mobile Application Platform 4.1

Red Hat Mobile Application Platform 4.1 MBaaS Administration andInstallation Guide

For Red Hat Mobile Application Platform 4.1

Legal Notice

Copyright © 2017 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative CommonsAttribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA isavailable athttp://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you mustprovide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinitylogo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and othercountries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United Statesand/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union andother countries.

Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related toor endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marksor trademarks/service marks of the OpenStack Foundation, in the United States and other countriesand are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed orsponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

AbstractThis document provides guides related to installation and administration of the RHMAP 4.x MBaaSon OpenShift 3.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table of Contents

CHAPTER 1. RHMAP 4.X MBAAS1.1. OVERVIEW1.2. ARCHITECTURE OF THE MBAAS1.3. SECURITY CONSIDERATIONS

CHAPTER 2. INSTALLING THE MBAAS2.1. INSTALLATION STEPS

CHAPTER 3. PREPARING NODES FOR MBAAS INSTALLATION3.1. OVERVIEW3.2. PREREQUISITES3.3. PROCEDURE3.4. NEXT STEPS

CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 34.1. OVERVIEW4.2. PREREQUISITES4.3. AUTOMATIC INSTALLATION4.4. MANUAL INSTALLATION4.5. CREATING AN MBAAS TARGET4.6. NEXT STEPS

CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OF THE MBAAS AND CLOUD APPS5.1. OVERVIEW5.2. PREREQUISITES5.3. ADJUSTING RESOURCE USAGE OF THE MBAAS5.4. ADJUSTING RESOURCE USAGE OF CLOUD APPS5.5. SETTING RESOURCE REQUESTS AND LIMITS5.6. USING CLUSTER METRICS TO VISUALIZE RESOURCE CONSUMPTION

CHAPTER 6. SETTING UP SMTP FOR CLOUD APP ALERTS6.1. OVERVIEW6.2. PREREQUISITES6.3. CONFIGURING SMTP SETTINGS IN FH-MBAAS6.4. VERIFYING SMTP SETTINGS6.5. TROUBLESHOOTING

CHAPTER 7. USING MONGODB IN AN RHMAP 4.X MBAAS7.1. OVERVIEW7.2. ACCESSING DATA IN THE MONGODB IN THE MBAAS

CHAPTER 8. CENTRALIZED LOGGING FOR MBAAS COMPONENTS8.1. OVERVIEW8.2. ACCESSING LOGS THROUGH KIBANA WEB CONSOLE8.3. IDENTIFYING ISSUES IN AN MBAAS

CHAPTER 9. MONITORING THE MBAAS WITH COCKPIT9.1. OVERVIEW9.2. INSTALLATION9.3. VIEWING THE CONTAINERS ON AN OPENSHIFT NODE9.4. VIEWING MULTIPLE HOSTS SIMULTANEOUSLY

CHAPTER 10. MONITORING THE MBAAS WITH NAGIOS10.1. OVERVIEW10.2. PREREQUISITES

4444

66

8888

10

11111112122122

23232323242526

272727272727

292929

30303031

3333333434

353535

Table of Contents

1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10.2. PREREQUISITES10.3. DEPLOYING NAGIOS TO OPENSHIFT10.4. USING NAGIOS

CHAPTER 11. TROUBLESHOOTING THE RHMAP 4.X MBAAS11.1. OVERVIEW11.2. CHECK THE HEALTH ENDPOINT OF THE MBAAS11.3. ANALYZE LOGS11.4. COMMON PROBLEMS

CHAPTER 12. KNOWN ISSUES IN THE RHMAP 4.0 MBAAS12.1. OVERVIEW12.2. DELETING OPENSHIFT SSH KEY BREAKS CLOUD APP DEPLOYMENTS12.3. CLOUD APP ANALYTICS DATA DOES NOT UPDATE12.4. INCORRECT DEPLOYMENT STATUS INDICATED BY PROGRESS BAR FOR SUCCESSFULDEPLOYMENTS12.5. MONGODB DOESN’T WORK AFTER A RESTART12.6. MONGODB POD STARTING IN REMOVED STATE

353536

3737373838

49494949

494950

Red Hat Mobile Application Platform 4.1 MBaaS Administration and Installation Guide

2

Table of Contents

3

CHAPTER 1. RHMAP 4.X MBAAS

1.1. OVERVIEW

Red Hat Mobile Application (RHMAP) 4.0 has a hybrid deployment model — the Core, the MBaaS,and the Build Farm are deployed in different locations.

Development and management of apps occurs in the multi-tenant cloud instance of the RHMAPCore hosted by Red Hat.

Application data, runtime, and integrations are deployed to the RHMAP MBaaS installed in aprivate or public instance of OpenShift Enterprise 3.

The Build Farm is deployed separately from the Core and the MBaaS and is shared between allinstances of RHMAP. Third-party Linux, Windows, and Apple server hosting providers are usedto support building client app binaries for all platforms.

The Mobile Backend-as-a-Service (MBaaS) is a core component of RHMAP – the back-end platformhosting containerized cloud applications in conjunction with database storage (MongoDB). Thecloud applications deployed in an MBaaS can make use of RHMAP APIs, such as datasynchronization, caching, or push notifications, and integrate with enterprise systems or other cloudservices.

1.2. ARCHITECTURE OF THE MBAAS

The RHMAP MBaaS 4.0 is built on top of several technologies, including OpenShift Enterprise 3,Kubernetes, Docker, and Red Hat Software Collections. The MBaaS consists of severalcomponents, each running in its own Docker container. Similarly, every cloud app deployed to theMBaaS runs in a Docker container. Those containers are deployed and orchestrated by Kubernetes.

In the MBaaS, the users can configure multiple isolated runtime and storage environments tosupport software development life-cycle stages, such as development, testing, and production. Eachenvironment can host multiple cloud apps.

1.3. SECURITY CONSIDERATIONS


4

Since the MBaaS is not hosted in Red Hat’s public multi-tenant cloud, the data transmitted betweenthe mobile device and the cloud app does not pass through any servers operated by Red Hat or anyother third party. Private data from back-end systems is transmitted directly between mobile devicesand the MBaaS.

The following data still resides in the RHMAP Core:

User names and passwords of RHMAP accounts

Master database of the Core, with entries for projects, apps, and their IDs

Git repositories hosting the source code of client and cloud apps

App store containing the built binaries of client apps

CHAPTER 1. RHMAP 4.X MBAAS

5

CHAPTER 2. INSTALLING THE MBAAS

This guide provides a detailed overview of the steps required to get from purchasing a Red HatMobile Application Platform (RHMAP) 4.x subscription to a working installation of an MBaaSconnected to your RHMAP domain.

2.1. INSTALLATION STEPS

After Red Hat receives the purchase order for a Red Hat Mobile Application Platform (RHMAP) 4.xsubscription, a member of the sales team internally requests a new RHMAP domain for access to aninstance of the RHMAP Core hosted by Red Hat.

Once the domain is created, a representative of the Red Hat Customer Enablement team willinstruct you to install the MBaaS, which involves installation of Red Hat Enterprise Linux (RHEL),installation of OpenShift Enterprise, provisioning of the MBaaS in the OpenShift cluster, andinstallation of other optional components.

2.1.1. Getting Access to RHMAP Core

The following steps for getting access to RHMAP Core are performed by a representative of the RedHat Customer Enablement team:

1. Create a domain.

The domain, such as customername.redhatmobile.com, hosts the RHMAP Core for asingle customer.

2. Create an administrator account.

An RHMAP administrator account is created in the domain, and the customer’s technicalcontact receives an activation e-mail which allows access to the domain using the newaccount.

2.1.2. Preparing Nodes for MBaaS Installation

The following steps are covered in the Preparing Nodes for MBaaS Installation guide.

1. Install RHEL on each cluster node as per the RHEL Installation Guide. You can DownloadRHEL from the Red Hat Customer Portal.

2. Register each node with Red Hat Subscription Manager (RHSM).

Follow step 2 in the Preparing Nodes for MBaaS Installation guide.

3. Install OpenShift as per the OpenShift Installation and Configuration guide.

Follow the considerations for infrastructure sizing to determine how many nodes toconfigure in your OpenShift cluster.

4. Enable Docker on each node to access container images of RHMAP components hosted inthe Red Hat Docker registry.

Follow step 4 in the Preparing Nodes for MBaaS Installation guide.


6

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/index.html

https://access.redhat.com/downloads/content/69/ver=/rhel---7/7.2/x86_64/product-software

https://docs.openshift.com/enterprise/3.2/install_config/index.html

https://access.redhat.com/node/2456121

2.1.3. Installing the MBaaS

After the nodes in the cluster have RHEL installed, OpenShift installed, and are registered withRHSM, you can proceed to the main installation step.

1. Follow the Provisioning an RHMAP 4.x MBaaS in OpenShift 3 guide.

As part of the provisioning process, OpenShift automatically downloads RHMAP containerimages from the Red Hat Docker registry.

2. Adjust system resource usage of MBaaS components.

If the MBaaS components are deployed on dedicated nodes in your cluster (separate fromcloud apps), we strongly recommend that you adjust the resource limits of MBaaScomponents to take full advantage of the available system resources.

Follow the Adjusting System Resource Usage of the MBaaS and Cloud Apps guide fordetailed steps.

2.1.4. Installing Additional Features

1. Set up cloud app alerts.

Setting Up SMTP for Cloud App Alerts

2. Set up centralized logging.

Centralized Logging for MBaaS Components

3. Set up monitoring.

Monitoring the MBaaS with Cockpit

Monitoring the MBaaS with Nagios

CHAPTER 2. INSTALLING THE MBAAS

7

CHAPTER 3. PREPARING NODES FOR MBAASINSTALLATION

3.1. OVERVIEW

Before installation of the MBaaS, you must first register each node in the cluster with the Red HatSubscription Manager (RHSM), and install OpenShift 3.

The registration enables OpenShift to access the Docker container images of RHMAP componentshosted in the Red Hat Docker registry.

3.2. PREREQUISITES

Access to an RHMAP domain and an RHMAP administrator account. See Installing the MBaaSfor more information.

3.3. PROCEDURE

1. Install RHEL.

Install RHEL on each machine that will serve as a node in the OpenShift cluster backing theMBaaS. Follow the RHEL Installation Guide.

2. Register all cluster nodes using RHSM and attach the nodes to the RHMAP subscription.

Perform the following procedure for each node in the cluster.

a. Register the node with RHSM.

Replace <username> and <password> with the user name and password for yourRed Hat account.

Registering to: subscription.rhn.redhat.com:443/subscriptionThe system has been registered with ID: abcdef12-3456-7890-1234-56789012abcd

b. List the available subscriptions.

c. Find the pool ID for an RHMAP subscription and attach it.

You will see output similar to the following:

Successfully attached a subscription for: {ProductName}

sudo subscription-manager register --username=<username> --password=<password>

sudo subscription-manager list --available

sudo subscription-manager attach --pool=<pool_id>


8

https://registry.access.redhat.com/

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Installation_Guide/index.html

3. Install OpenShift.

See the Installation and Configuration guide in the OpenShift documentation for detailedinstallation procedure.

Follow the considerations for infrastructure sizing to determine how many nodes toconfigure in your OpenShift cluster.

Note

In the OpenShift Installation and Configuration guide:

Skip steps 1, 2 and 3 in section 2.2.4.1. Software Prerequisites – Registeringthe Hosts, which describe the same registration process that is alreadycovered in this guide in step 1.

Choose the default, RPM-based installation method. See section 2.3. RPM vs.Containerized for more details.

4. Enable Docker to access container images of RHMAP components.

Perform the following procedure for each node in the cluster.

a. Enable the rhel-7-server-optional-rpms repository.

Repository 'rhel-7-server-optional-rpms' is enabled for this system.

b. Install the RHSM plugin subscription-manager-plugin-container.

c. Run rhsmcertd-worker to refresh the local certificate store.

rhsmcertd-worker must be run as the superuser, otherwise it may fail to workwithout a warning.

To verify that the certificates were downloaded, check the contents of the /etc/docker/certs.d/ directory.

/etc/docker/certs.d/ now contains directories access.redhat.com and registry.access.redhat.com.

sudo subscription-manager repos --enable=rhel-7-server-optional-rpms

sudo yum install subscription-manager-plugin-container

sudo /usr/libexec/rhsmcertd-worker

ls -l /etc/docker/certs.d/ | grep access.redhat.com

CHAPTER 3. PREPARING NODES FOR MBAAS INSTALLATION

9

https://access.redhat.com/documentation/en/openshift-enterprise/version-3.1/installation-and-configuration/


https://access.redhat.com/documentation/en/openshift-enterprise/version-3.1/installation-and-configuration/#install-config-install-rpm-vs-containerized

drwxr-xr-x. 2 root root 67 Jun 01 10:30 access.redhat.comdrwxr-xr-x. 2 root root 67 Jun 01 10:30 registry.access.redhat.com

After registering each node with RHSM, downloading the entitlement certificates, and installingOpenShift, you can proceed to installation of the MBaaS.

3.4. NEXT STEPS

Provisioning an RHMAP 4.x MBaaS in OpenShift 3


10

CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS INOPENSHIFT 3

4.1. OVERVIEW

An OpenShift 3 cluster can serve as an MBaaS target and host your Cloud Apps and CloudServices. This guide provides detailed steps to deploy the RHMAP 4.x MBaaS on an OpenShift 3cluster.

You can choose a simple automated installation to preview and test the MBaaS, or follow themanual installation steps for a fully supported production-ready MBaaS:

Automatic Installation

You can quickly try the RHMAP 4.x MBaaS by choosing the automatic installation.

The following limitations apply to the automatically installed MBaaS:

not suitable for production use

single replica for each MBaaS component

single MongoDB replica with no persistent storage

Manual Installation

For production use, follow the manual installation procedure, which results in an MBaaS with thefollowing characteristics:

suitable for production use

three replicas defined for each MBaaS component (with the exception of fh-statsd)

three MongoDB replicas with a 50GB persistent storage requirement each

nodeSelectors of mbaas_id=mbaas1, mbaas_id=mbaas2, and mbaas_id=mbaas3 forthe MongoDB replicas

4.2. PREREQUISITES

This guide assumes several prerequisites are met before the installation:

All nodes in the cluster must be registered with the Red Hat Subscription Manager and haveRHMAP entitlement certificates downloaded. See Preparing Nodes for MBaaS Installation fordetailed steps.

An existing OpenShift Enterprise installation, version 3.2

The OpenShift master and router must be accessible from the RHMAP Core.

A wildcard DNS entry must be configured for the OpenShift router IP address.

A dedicated OpenShift user for MBaaS administration. This guide refers to this user as theMBaaS administrator. The user must have the authorization to:

create projects;

CHAPTER 4. PROVISIONING AN RHMAP 4.X MBAAS IN OPENSHIFT 3

11

create any objects associated with a project (for example, Services, DeploymentConfigs,Routes).

An OpenShift user with the admin role is needed for the manual installation. For example, thedefault system:admin user. This guide refers to this user as the OpenShift administrator.

For information on installation and management of an OpenShift cluster and its users, see the officialOpenShift documentation.

4.3. AUTOMATIC INSTALLATION

The automatic installation of an MBaaS in OpenShift 3 results in the MBaaS components beinginstalled on nodes chosen by the OpenShift scheduler. Only a single instance of each componentruns at any time and thus makes the MBaaS susceptible to downtime in case of failure of a singlenode. The data of the MongoDB database is not backed by any permanent storage and is thereforetransient.

There are no setup steps required before the automatic installation. Refer to Creating an MBaaSTarget to continue the installation.

Note

In order for automatic MBaaS installation to work, the OpenShift SDN must be configuredto use the ovs-subnet SDN plugin (this is the default). If it is not set to this, refer toNetwork Configuration.

4.4. MANUAL INSTALLATION

The manual installation of an MBaaS in OpenShift 3 results in a resilient three-node cluster withMBaaS components spread across all three nodes, MongoDB replica set spread over three nodes,and the MongoDB data backed by persistent volumes.

The installation consists of several phases. Before the installation, you must prepare your OpenShiftcluster:

Set up persistent storage - you need to create Persistent Volumes with specific parameters inOpenShift.

Label the nodes - nodes need to be labeled in a specific way, to match the node selectorsexpected by the OpenShift template of the MBaaS.

Network Configuration - configuring the SDN network plugin used in OpenShift so that CloudApps can communicate with MongoDB in the MBaaS.

Warning

The automatic installation procedure must not be used in production environments. Youshould only use this procedure for evaluation purposes, since the provided template isnot optimized for resiliency and stability required in production environments. Follow themanual installation steps for production use.


12

https://docs.openshift.com/enterprise

After the OpenShift cluster is properly configured:

Install the MBaaS from a template

Verify the installation

4.4.1. Before The Installation

The manual installation procedure poses certain requirements on your OpenShift cluster in order toguarantee fault tolerance and stability.

4.4.1.1. Network Configuration

Cloud Apps in an MBaaS communicate directly with a MongoDB replica set. In order for this to work,the OpenShift SDN must be configured to use the ovs-subnet SDN plugin. For more detailedinformation on configuring this, see Migrating Between SDN Plug-ins in the OpenShift Enterprisedocumentation.

4.4.1.1.1. Making Project Networks Global

Alternatively, if you cannot use the ovs-subnet SDN plugin, you will need to make the network ofthe MBaaS project global after installation. For details on how to do this with your MBaaS project,see Making Project Networks Global in the OpenShift Enterprise documentation.

4.4.1.2. Persistent Storage Setup

An MBaaS running on OpenShift 3 contains a MongoDB replica set. The replica set members usepersistent storage for the directory where the database data is stored.

The OpenShift template that is used to create the project and resources for the MBaaS requires:

At least one PersistentVolume (PV) resource in an Available state for each of the threeMongoDB replica members.

Each PersistentVolume has at least 50GB of space.

For detailed information on PersistentVolumes and how to create them, see Persistent Storage inthe OpenShift Enterprise documentation.

4.4.1.3. Apply Node Labels

By applying labels to OpenShift nodes, you can control which nodes the MBaaS components,MongoDB replicas, and cloud apps will be deployed to.

This section describes the considerations for:

Section 4.4.1.3.1, “Labelling for MBaaS components”

Section 4.4.1.3.2, “Labelling for MongoDB replicas”

Cloud apps get deployed to nodes labeled with the default nodeSelector, which is usually set to type=compute (defined in the OpenShift master configuration).

You can skip this entire labeling section if your OpenShift cluster only has a single schedulablenode. In such case, all MBaaS components, MongoDB replicas, and cloud apps will necessarily runon that single node.


13

https://docs.openshift.com/enterprise/3.2/install_config/configuring_sdn.html#migrating-between-sdn-plugins

https://docs.openshift.com/enterprise/3.2/admin_guide/pod_network.html#making-project-networks-global

https://docs.openshift.com/enterprise/3.2/architecture/additional_concepts/storage.html

4.4.1.3.1. Labelling for MBaaS components

It is recommended, but not required, to deploy the MBaaS components to dedicated nodes, separatefrom other applications (such as RHMAP cloud apps).

Refer to Infrastructure Sizing Considerations for Installation of RHMAP MBaaS for therecommended number of MBaaS nodes and cloud app nodes for your configuration.

For example, if you have 12 nodes, the recommendation is:

Dedicate three nodes to MBaaS and MongoDB.

Dedicate three nodes to cloud apps.

To achieve this, apply a label, such as type=mbaas to the three dedicated MBaaS nodes.

Then, when creating the MBaaS project, as described later in Section 4.4.2, “Installing the MBaaS”,set this label as the nodeSelector.

You can check what type labels are applied to all nodes with the following command:

NAME STATUS AGE TYPEose-master Ready,SchedulingDisabled 27d masterinfra-1 Ready 27d infrainfra-2 Ready 27d infraapp-1 Ready 27d computeapp-2 Ready 27d computeapp-3 Ready 27d computembaas-1 Ready 27d mbaasmbaas-2 Ready 27d mbaasmbaas-3 Ready 27d mbaas

In this example, the deployment would be as follows:

Cloud apps get deployed to the three dedicated cloud app nodes app-1, app-2, and app-3.

The MBaaS components get deployed to the three dedicated MBaaS nodes mbaas-1, mbaas-2, and mbaas-3 (if the nodeSelector is also set on the MBaaS Project).

4.4.1.3.2. Labelling for MongoDB replicas

In the production MBaaS template, the MongoDB replicas are spread over three MBaaS nodes. Ifyou have more than three MBaaS nodes, any three of them can host the MongoDB replicas.

To apply the required labels (assuming the three nodes are named mbaas-1, mbaas-2, and mbaas-3):

oc label node mbaas-1 type=mbaasoc label node mbaas-2 type=mbaasoc label node mbaas-3 type=mbaas

oc get nodes -L type

oc label node mbaas-1 mbaas_id=mbaas1oc label node mbaas-2 mbaas_id=mbaas2oc label node mbaas-3 mbaas_id=mbaas3


14


You can verify the labels were applied correctly by running this command:

NAME STATUS AGE MBAAS_ID10.10.0.102 Ready 27d <none>10.10.0.117 Ready 27d <none>10.10.0.141 Ready 27d <none>10.10.0.157 Ready 27d mbaas310.10.0.19 Ready,SchedulingDisabled 27d <none>10.10.0.28 Ready 27d mbaas110.10.0.33 Ready 27d <none>10.10.0.4 Ready 27d <none>10.10.0.99 Ready 27d mbaas2

See Updating Labels on Nodes in the OpenShift documentation for more information on how toapply labels to nodes.

4.4.1.3.2.1. Why are MongoDB replicas spread over multiple nodes?

Each MongoDB replica is scheduled to a different node to support failover.

For example, if an OpenShift node failed, data would be completely inaccessible if all threeMongoDB replicas were scheduled on this failing node. Setting a different nodeSelector for eachMongoDB DeploymentConfig, and having a corresponding OpenShift node in the clustermatching this label will ensure the MongoDB pods get scheduled to different nodes.

In the production MBaaS template, there is a different nodeSelector for each MongoDB DeploymentConfig:

mbaas_id=mbaas1 for mongodb-1



Excerpt of DeploymentConfig of mongodb-1

{ "kind": "DeploymentConfig", "apiVersion": "v1", "metadata": { "name": "mongodb-1", "labels": { "name": "mongodb" } }, "spec": { ... "template": { ... "spec": { "nodeSelector": { "mbaas_id": "mbaas1" }

oc get nodes -L mbaas_id


15

https://docs.openshift.com/enterprise/3.2/admin_guide/manage_nodes.html#updating-labels-on-nodes

4.4.2. Installing the MBaaS

In this step, you will provision the MBaaS to the OpenShift cluster from the command line, based onthe MBaaS OpenShift template.

First, download the latest version of the MBaaS OpenShift template.

1. In the Studio, navigate to the Admin > MBaaS Targets section. Click New MBaaS Target.

2. Choose OpenShift 3 as Type.

3. At the bottom of the page, click Download Template and save the template file fh-mbaas-template-3node.json. You may now close the New MBaaS Target screen.

Using the downloaded template, provision the MBaaS in the OpenShift cluster from the commandline. For general information about the OpenShift CLI, see CLI Operations in the OpenShiftEnterprise documentation.

1. Create a new project.

Log in as the MBaaS administrator. You will be prompted for credentials.

Create the project:

2. Set the node selector of the project to target MBaaS nodes.

This ensures that all MBaaS components are deployed to the dedicated MBaaS nodes.

Note

If you’ve chosen not to have dedicated MBaaS nodes in Section 4.4.1.3.1,“Labelling for MBaaS components”, skip this step.

Log in as the OpenShift administrator. You will be prompted for credentials.

oc login <public URL of the OpenShift master>

Warning

The name of the OpenShift project chosen here must have the suffix -mbaas.The part of the name before -mbaas is used later in this guide as the ID of theMBaaS target associated with this OpenShift project. For example, if the ID ofthe MBaaS target is live, the OpenShift project name set here must be live-mbaas.

oc new-project live-mbaas



16

https://docs.openshift.com/enterprise/3.2/cli_reference/basic_cli_operations.html

Set the openshift.io/node-selector annotation to type=mbaas in the project’snamespace:

Note

You may need to add this annotation if it is missing.

apiVersion: v1kind: Namespacemetadata: annotations: openshift.io/node-selector: type=mbaas...

Log back in as the MBaaS administrator if you had to switch users for the above steps. Youwill be prompted for credentials.

3. Start the installation.

Create all the MBaaS resources from the template.

After all the resources are created, you should see output similar to the following:

--> Deploying template fh-mbaas for "fh-mbaas" With parameters: MONGODB_FHMBAAS_USER=u-mbaas ...

--> Creating resources ... Service "fh-mbaas-service" created Service "fh-messaging-service" created Service "fh-metrics-service" created Service "fh-statsd-service" created Service "mongodb-1" created Service "mongodb-2" created Service "mongodb-3" created DeploymentConfig "fh-mbaas" created DeploymentConfig "fh-messaging" created DeploymentConfig "fh-metrics" created DeploymentConfig "fh-statsd" created PersistentVolumeClaim "mongodb-claim-1" created PersistentVolumeClaim "mongodb-claim-2" created PersistentVolumeClaim "mongodb-claim-3" created DeploymentConfig "mongodb-1" created DeploymentConfig "mongodb-2" created DeploymentConfig "mongodb-3" created

oc edit ns live-mbaas


oc new-app -f fh-mbaas-template-3node.json


17

Pod "mongodb-initiator" created Route "mbaas" created--> Success Run 'oc status' to view your app.

It may take a minute for all the resources to get created and up to 10 minutes for all thecomponents to get to a Running status.

4.4.3. Verifying The Installation

1. Verify the Services.

Each MBaaS component defines a Service, which load-balances and proxies traffic to theunderlying pods.

To verify that all the services of the MBaaS have been created, enter the followingcommand:

The output should look similar to the following:

NAME CLUSTER_IP EXTERNAL_IP PORT(S) SELECTOR AGEfh-mbaas-service 172.30.208.89 <none> 8080/TCP name=fh-mbaas 1mfh-messaging-service 172.30.194.171 <none> 8080/TCP name=fh-messaging 1mfh-metrics-service 172.30.65.222 <none> 8080/TCP name=fh-metrics 1mfh-statsd-service 172.30.161.128 <none> 8080/TCP,8081/UDP name=fh-statsd 1mmongodb-1 None <none> 27017/TCP name=mongodb-replica-1 1mmongodb-2 None <none> 27017/TCP name=mongodb-replica-2 1mmongodb-3 None <none> 27017/TCP name=mongodb-replica-3 1m

Verify that the output contains a service for all MBaaS components and a service for eachMongoDB replica.

Each service forwards traffic to one or more pods. During the MBaaS creation, someintermediate pods get created, which are responsible for deploying or setting up other pods.These intermediate pods have a -deploy or -build suffix. If these intermediate pods fail,the installation can not proceed. You can try viewing logs of the intermediate pods to identifythe cause of the failure.

2. Verify that the MongoDB replica set is configured correctly.

a. Verify that the status of the mongodb-initiator pod is Completed.

oc get svc

oc get pod mongodb-initiator


18

NAME READY STATUS RESTARTS AGEmongodb-initiator 0/1 Completed 0 5d

b. Verify that each MongoDB replica has all replica set members configured.

Enter the following command to list the replica set members of each MongoDBreplica:

Each replica should have the same three members listed in its array of replica setmembers — mongodb-1, mongodb-2, and mongodb-3.

If there are not exactly three members in the array of replica set members of eachnode, refer to the Common Problems section of the troubleshooting guide for help.

3. Verify the pods.

To verify that all the pods are running, enter the following command:


NAME READY STATUS RESTARTS AGEfh-mbaas-1-h511r 1/1 Running 0 53mfh-mbaas-1-hg5ub 1/1 Running 0 54mfh-mbaas-1-uwpl6 1/1 Running 0 53mfh-messaging-1-3wxap 1/1 Running 0 53mfh-messaging-1-j5asf 1/1 Running 0 53mfh-messaging-1-yh8hn 1/1 Running 0 54mfh-metrics-1-f5ems 1/1 Running 0 53mfh-metrics-1-faihq 1/1 Running 0 53mfh-metrics-1-vleqs 1/1 Running 0 54mfh-statsd-1-36hw0 1/1 Running 0 54mmongodb-1-1-12l8b 1/1 Running 0 54mmongodb-2-1-hwmzx 1/1 Running 0 54mmongodb-3-1-bl12r 1/1 Running 0 54mmongodb-initiator 0/1 Completed 0 54m

Verify that all Pods are in a Running state, with the exception of the mongodb-initator,which should be in a Completed state. If any Pod is in a different state, they may requiremore time, or may have an issue starting up. A stream of events for the namespace,including any issues with scheduling and creating pods, pulling images and any otherpotential issues, can be viewed using oc get events -w.

4. Verify the Route

To verify the MBaaS route is exposed, enter the following command:

for j in `(for i in 1 2 3;do oc get po -l deploymentconfig=mongodb-$i -o name;done) | sed -e 's|pod/||'`; do echo "## $j ##" && echo mongo admin -u admin -p \${MONGODB_ADMIN_PASSWORD} --eval "printjson$rs.conf\($.members\)" | oc rsh --shell='/bin/bash' $j; done

oc get pods

oc get routes mbaas


19


NAME HOST/PORT PATH SERVICE LABELS INSECURE POLICY TLS TERMINATIONmbaas live-mbaas.example.com fh-mbaas-service Allow edge

5. Ping the health endpoint.

If all services are created, all pods are running, and the route is exposed, the MBaaS healthendpoint can be queried as follows:

The endpoint responds with health information about the various MBaaS components andtheir dependencies. If there are no errors reported, the MBaaS is ready to be configured foruse in the Studio. Successful output will resemble the following:

curl `oc get route mbaas --template "{{.spec.host}}"`/sys/info/health

{ "status": "ok", "summary": "No issues to report. All tests passed without error", "details": [ { "description": "Check Mongodb connection", "test_status": "ok", "result": { "id": "mongodb", "status": "OK", "error": null }, "runtime": 33 }, { "description": "Check fh-messaging running", "test_status": "ok", "result": { "id": "fh-messaging", "status": "OK", "error": null }, "runtime": 64 }, { "description": "Check fh-metrics running", "test_status": "ok", "result": { "id": "fh-metrics", "status": "OK", "error": null }, "runtime": 201 }, { "description": "Check fh-statsd running", "test_status": "ok",


20

After verifying that the MBaaS is installed correctly, you must create an MBaaS target for the newMBaaS in the Studio.

4.5. CREATING AN MBAAS TARGET

1. In the Studio, navigate to the Admin > MBaaS Targets section. Click New MBaaS Target.

2. As Type, choose OpenShift 3.

3. Fill in the following information

MBaaS Id - a unique ID for the MBaaS, for example, live. The ID must be equal to theOpenShift project name chosen in the Installing the MBaaS section, without the -mbaassuffix.

OpenShift Master URL - the URL of the OpenShift master, for example, https://master.openshift.example.com:8443

OpenShift Username, OpenShift Password - username and password of anOpenShift user which can create projects and any objects associated with a project (forexample, services, DeploymentConfigs, Routes). For the manual installation, enter theusername and password of the dedicated MBaaS admin user.

OpenShift Router DNS - a wildcard DNS entry of the OpenShift router, for example, *.cloudapps.example.com

If you’ve chosen the manual installation procedure, uncheck Automatic MBaasInstallation and fill in these two additional fields:

MBaaS Service Key

Equivalent to the value of the FHMBAAS_KEY environment variable, which isautomatically generated during installation. To find out this value, enter the followingcommand:

Alternatively, you can find the value in the OpenShift Console, in the Details tab of the fh-mbaas deployment, in the Env Vars section.

MBaaS URL

A URL of the route exposed for the fh-mbaas-service, including the https protocolprefix. This can be retrieved from the OpenShift web console, or by running the followingcommand:

"result": { "id": "fh-statsd", "status": "OK", "error": null }, "runtime": 7020 } ]}

oc env dc/fh-mbaas --list | grep FHMBAAS_KEY


21

https://master.openshift.example.com:8443

4. Click Save MBaaS and you will be directed to the MBaaS Status screen. If you chose anautomatic installation, it can take several minutes before the status is reported back. For amanual installation, the status should be reported back in less than a minute.

Once the process of creating the MBaaS has succesfully completed, you can see the new MBaaS inthe list of MBaaS targets.

In your OpenShift account, you can see the MBaaS represented by a project.

4.6. NEXT STEPS

Create an Environment - you must create at least one environment for the MBaaS to be usableby Cloud Apps and Cloud Services

Adjusting System Resource Usage of the MBaaS and Cloud Apps - we strongly recommendthat you adjust the system resource usage of MBaaS components as appropriate for yourproduction environment

Optional: Set up centralized logging - deploy a centralized logging solution based onElasticSearch, Fluentd, and Kibana to debug issues with the MBaaS

echo "https://"$(oc get route/mbaas -o template --template {{.spec.host}})


22

https://access.redhat.com/documentation/en/red-hat-mobile-application-platform/4.1/single/product-features/#creating-an-environment

CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OFTHE MBAAS AND CLOUD APPS

5.1. OVERVIEW

In the RHMAP 4.x MBaaS based on OpenShift 3, each cloud app and and each MBaaS componentruns in its own container. This architecture allows for granular control of CPU and memoryconsumption. A fine level of control of system resources helps to ensure efficient use of nodes, andto guarantee uninterrupted operation of critical services.

An application can be prepared for various situations, such as high peak load or sustained load, bymaking decisions about the resource limits of individual components. For example, you could decidethat MongoDB must keep working at all times, and assign it high, guaranteed amount of resources.At the same time, if the availability of a front-end Node.js server is less critical, the server can beassigned less initial resources, with the possibility to use more resources when available.

5.2. PREREQUISITES

The system resources of MBaaS components and cloud apps in the MBaaS can be regulated usingthe mechanisms available in OpenShift – resource requests, limits, and quota. Before proceedingwith the instructions in this guide, we advise you to read the Quotas and Limit Ranges section in theOpenShift documentation.

5.3. ADJUSTING RESOURCE USAGE OF THE MBAAS

The RHMAP MBaaS is composed of several components, each represented by a single containerrunning in its own pod. Each container has default resource requests and limits assigned in theMBaaS OpenShift template. See the section Overview of Resource Usage of MBaaS Componentsfor a complete reference of the default values.

Depending on the deployment model of the MBaaS, you may have to adjust the resource limits andrequests to fit your environment. If the MBaaS components are deployed on the same nodes as thecloud apps, there is no adjustment required.

However, when the MBaaS components are deployed on nodes dedicated to running the MBaaSonly, it is strongly recommended to adjust the resource limits to take full advantage of the availableresources on the dedicated nodes.

5.3.1. Calculating the Appropriate Resource Requests and Limits

Note

This section refers to CPU resources in two different terms – the commonly used termvCPU (virtual CPU), and the term millicores used in OpenShift documentation. The unit of1 vCPU is equal to 1000 m (millicores), which is equivalent to 100% of the time of oneCPU core.

The resource limits must be set accordingly for your environment and depend on the characteristicsof load on your cloud apps. However, the following rules can be used as a starting point foradjustments of resource limits:

CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OF THE MBAAS AND CLOUD APPS

23

https://docs.openshift.com/enterprise/3.2/dev_guide/compute_resources.html

Allow 2 GiB of RAM and 1 vCPU for the underlying operating system.

Split the remainder of resources in equal parts amongst the MBaaS Components.

5.3.1.1. Example

Given a virtual machine with 16 GiB of RAM and 4 vCPUs, we allow 2 GiB of RAM and 1 vCPU forthe operating system. This leaves 14GB RAM and 3 vCPUs (equal to 3000 m) to distribute amongstthe 5 MBaaS components.

14 GiB / 5 = 2.8 GiB of RAM per component

3000 m / 5 = 600 m per component

In this example, the resource limit for each MBaaS component would be 2.8 GiB of RAM and 600millicores of CPU. Depending on the desired level of quality of service of each component, set theresource request values as described in the section Quality of service tiers in the OpenShiftdocumentation.

5.3.2. Overview of Resource Usage of MBaaS Components

The following table lists the components of the MBaaS, their idle resource usage, default resourcerequest, and default resource limit.

MBaaScomponent

Idle RAMusage

RAMrequest

RAMlimit

Idle CPUusage

CPUrequest

CPUlimit

fh-mbaas 160 MiB 200 MiB 800 MiB <1% 200 m 800 m

fh-messaging 160 MiB 200 MiB 400 MiB <1% 200 m 400 m

fh-metrics 120 MiB 200 MiB 400 MiB <1% 200 m 400 m

fh-statsd 75 MiB 200 MiB 400 MiB <1% 200 m 400 m

mongodb 185 MiB 200 MiB 1000MiB

<1% 200 m 1000 m

5.4. ADJUSTING RESOURCE USAGE OF CLOUD APPS

The resource requests and limits of cloud apps can be set the same way as for MBaaScomponents. There is no particular guideline for doing the adjustment in cloud apps.

5.4.1. Overview of Resource Usage of Cloud App Components


24

https://docs.openshift.com/enterprise/3.2/dev_guide/compute_resources.html#quality-of-service-tiers

Cloud appcomponent

Idle RAMusage

RAMrequest

RAMlimit

Idle CPUusage

CPUrequest

CPUlimit

nodejs-frontend 125 MiB 500 MiB 1 GiB <1% 100 m 500 m

redis 8 MiB 100 MiB 500 MiB <1% 100 m 500 m

5.5. SETTING RESOURCE REQUESTS AND LIMITS

The procedure for setting the resource requests and limits is the same for both MBaaS componentsand cloud app components.

Open the DeploymentConfig of a component, for example fh-mbaas:

The DeploymentConfig contains two resources sections with equivalent values: one in the spec.strategy section, and another in the spec.template.spec.containers section. Setthe cpu and memory values of requests and limits as necessary, making sure the values stayequivalent between the two sections, and save the file.

apiVersion: v1kind: DeploymentConfigmetadata: name: fh-mbaas ...spec: ... strategy: resources: limits: cpu: 800m memory: 800Mi requests: cpu: 200m memory: 200Mi ... spec: containers: ... resources: limits: cpu: 800m memory: 800Mi requests: cpu: 200m memory: 200Mi

oc edit dc fh-mbaas

CHAPTER 5. ADJUSTING SYSTEM RESOURCE USAGE OF THE MBAAS AND CLOUD APPS

25

5.6. USING CLUSTER METRICS TO VISUALIZE RESOURCECONSUMPTION

It is possible to view the immediate and historical resource usage of pods and containers in the formof donut charts and line charts using the Cluster Metrics deployment in OpenShift. Refer to EnablingCluster Metrics in the OpenShift documentation for steps to enable cluster metrics.

Once cluster metrics are enabled, in the OpenShift web console, navigate to Browse > Pods andclick on the component of interest. Click on the Metrics tab to see the visualizations.


26

https://docs.openshift.com/enterprise/3.2/install_config/cluster_metrics.html

CHAPTER 6. SETTING UP SMTP FOR CLOUD APP ALERTS

6.1. OVERVIEW

Each cloud app can automatically send alerts by e-mail when specified events occur, such as whenthe cloud app gets deployed, undeployed, or logs an error. See Alerts & Email Notifications for moreinformation.

For the RHMAP 4.x MBaaS based on OpenShift 3, the e-mail function is not available immediatelyafter installation. You must configure an SMTP server to enable e-mail support.

6.2. PREREQUISITES

An RHMAP 4.x MBaaS running in OpenShift Enterprise 3

An account on an SMTP server through which notification alerts can be sent

An email address where alerts should be sent

A deployed Cloud App

6.3. CONFIGURING SMTP SETTINGS IN FH-MBAAS

The FH_EMAIL_SMTP and FH_EMAIL_ALERT_FROM environment variables in the fh-mbaasDeploymentConfig need to be set, using the below commands:

oc project <mbaas-project-id>oc env dc/fh-mbaas FH_EMAIL_SMTP="smtps://username:password@localhost" FH_EMAIL_ALERT_FROM="[email protected]"

After modifying the DeploymentConfig, a redeploy of the fh-mbaas pod should be triggeredautomatically. Once the pod is running again, you can verify the changes.

6.4. VERIFYING SMTP SETTINGS

1. In the Studio, navigate to a deployed Cloud App.

2. Go to the Notifications > Alerts section.

3. Click Create An Alert .

4. In the Emails field, enter your e-mail address.

5. Click Test Emails.

You should receive an e-mail from the e-mail address set as FH_EMAIL_ALERT_FROM.

6.5. TROUBLESHOOTING

If the test email fails to send, verify the SMTP settings in the running fh-mbaas Pod.

oc env pod -l name=fh-mbaas --list | grep EMAIL

CHAPTER 6. SETTING UP SMTP FOR CLOUD APP ALERTS

27

https://access.redhat.com/documentation/en/red-hat-mobile-application-platform/4.1/single/product-features/#alerts-email-notifications

It may help to view the fh-mbaas logs while attempting to send an email, looking for any errorsrelated to SMTP or email.

oc logs -f fh-mbaas-<deploy-uuid>

Ensure the Cloud App you are using to send a test mail with is running. If the test email sends OK,but fails to arrive, check it hasn’t been placed in your spam or junk folder.


28

CHAPTER 7. USING MONGODB IN AN RHMAP 4.X MBAAS

7.1. OVERVIEW

In an RHMAP 4.x MBaaS based on OpenShift 3, all components of the MBaaS run within a singleOpenShift project, together with a shared MongoDB replica set. Depending on how the MBaaS wasinstalled, the replica set runs either on a single node, or on multiple nodes, and may be backed bypersistent storage. The recommended production-grade MongoDB setup for an MBaaS has 3replicas, each backed by persistent storage.

Each cloud app deployed to the MBaaS has its own OpenShift project. However, the database of acloud app is created in the shared MongoDB instance. Therefore, all management operations on thepersistent data of cloud apps and the MBaaS, such as backup, or replication can be centralized. Atthe same time, the data of individual cloud apps is isolated in separate databases.

7.2. ACCESSING DATA IN THE MONGODB IN THE MBAAS

A simple way to store data is using the $fh.db API, which provides methods for create, read,update, delete, and list operations. See the $fh.db API documentation for more information.

If you need the full capability of a native MongoDB driver, or want to use another library to accessthe data, such as Mongoose, you can use the connectionString method of the $fh.db API toretrieve the connection string to the MongoDB instance:

Note

To avoid concurrency issues, we recommend using either the $fh.db API or a directconnection to the database, but not both at the same time.

$fh.db({ "act" : "connectionString"}, function(err, connectionString){ console.log('connectionString=', connectionString);});

CHAPTER 7. USING MONGODB IN AN RHMAP 4.X MBAAS

29

https://access.redhat.com/documentation/en/red-hat-mobile-application-platform/4.1/single/cloud-api/#fh-db

CHAPTER 8. CENTRALIZED LOGGING FOR MBAASCOMPONENTS

8.1. OVERVIEW

Logging output from RHMAP MBaaS components can be aggregated and accessed through a webconsole when using an MBaaS backed by OpenShift Enterprise 3 (OSEv3). Aggregated logging isenabled by deploying an EFK logging stack to your OSEv3 instance, which consists of the followingcomponents:

Elasticsearch indexes log output collected by Fluentd and makes it searchable.

Fluentd collects standard output of all containers.

Kibana is a web console for querying and visualizing data from Elasticsearch.

To enable this functionality, follow the official OpenShift guide Aggregating Container Logs.

8.2. ACCESSING LOGS THROUGH KIBANA WEB CONSOLE

The Kibana web console is where logs gathered by Fluentd and indexed by Elasticsearch can beviewed and queried. You can access the Kibana web console via the OpenShift web console, ordirectly by its URL configured through the KIBANA_HOSTNAME in the deployment procedure.

8.2.1. Viewing Logs of a Single Pod

If you have configured loggingPublicURL in step 8 of the deployment procedure, the OpenShiftweb console allows you to view the log archive of a particular pod.

1. In the OpenShift web console, select a project, and look for the deployment named fh-mbaas.

2. Click on the Pods circle.

3. Choose one of the pods to inspect.

4. Click on the Logs tab.

5. Click on the View Archive button at the top right corner to access the logs of the chosen podin the Kibana web console.

Note

By default, Kibana’s time filter shows the last 15 minutes of data. If you don’t see anyvalues, adjust the Time filter setting to a broader time interval.

8.2.2. Accessing Kibana Directly

You can access the Kibana web console directly at https://KIBANA_HOSTNAME, where KIBANA_HOSTNAME is the host name you set in step 4 of the deployment procedure.


30

https://www.elastic.co/products/elasticsearch

http://www.fluentd.org/

https://www.elastic.co/products/kibana

https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging.html

https://kibana_hostname

8.2.3. Configuring an Index Pattern

When accessing the Kibana web console directly for the first time, you are presented with the optionto configure an index pattern. You can also access this configuration screen in the Settings tab. Bydefault, there is an index pattern in the format <MBaaS ID>-mbaas.*, matching the ID of thedeployed MBaaS target.

To make queries more efficient, you can restrict the index pattern by date and time.

1. Select the Use event times to create index names

2. Enter the following pattern in the Index name or pattern input text field. For example:

3. You will see output similar to the following below the input field

4. Click Create to create the index based on this pattern.

5. You can now select this newly created index in the Discover tab when doing searches, aswell as in other parts, such as the Visualizations tab.

8.3. IDENTIFYING ISSUES IN AN MBAAS

If you suspect that an error of an MBaaS component may be the cause of an issue, you can useKibana’s Discover tab to find the root of the problem. The following steps describe the generalprocedure you can follow to identify issues.

1. Select the index for the MBaaS target you are interested in

Use the dropdown just below the input bar in the Discover view to list all available indices.An index is similar to a database in relational database systems. Select which index yoursearches will be performed against.

2. Select a time interval for your search

Click the Time Filter (clock icon) and adjust the time interval. Initially, try a broader search.

3. Perform a simple search

To search for all error events, perform a simple search for error in the Discovery field. Thiswill return the number of hits within the chosen time interval.

4. Select the msg or message field to be displayed

On the left hand side of the Discover view is a list of fields. From this list you can selectfields to display in the document data section. Selecting a field replaces the _source fieldin the document data view. This enables you to see any error messages and might help yourefine your original search if needed. You can also select more fields to help you locate theissue.

5. Narrow down the time interval

[onprem-mbaas.]YYYY.MM.DD

Pattern matches 100% of existing indices and aliasesonprem-mbaas.2016.02.04onprem-mbaas.2016.02.05

CHAPTER 8. CENTRALIZED LOGGING FOR MBAAS COMPONENTS

31

https://www.elastic.co/guide/en/elasticsearch/guide/current/_indexing_employee_documents.html#_indexing_employee_documents

The histogram shows search hits returned in the chosen time interval. To narrow down thesearch in time you have the following options:

Click on a bar in the histogram to narrow down the search to that bar’s time interval.

Select a time window in the date histogram by clicking and dragging between thestart/end time you are interested in.

6. Inspect the document data

Once you narrow down the search, you can inspect the document data items. Apart from themsg and message fields, you might be interested in kubernetes_pod_name to see thepod a message originates from.

8.3.1. Viewing All Debug Logs for an MBaaS Component

If searching for error messages doesn’t help, you can try looking into debug logs of individualMBaaS components.

1. Select the index for the MBaaS target that you are interested in

2. Start a new search

Click on the New Search button to the left of the search input bar, which looks like adocument with a plus sign.

3. Search an MBaaS component for all debug messages

For example, to search for all debug messages of the fh-messaging component, enter thefollowing query:

If you know some part of the error message, you can specify that as part of the search:

You can narrow down your search further by time, as described in step 5 above.

As a reference, the following are the Bunyan log levels:

type: bunyan && level: 20 && kubernetes_container_name: "fh-messaging"

type: bunyan && level: 20 && kubernetes_container_name: "fh-messaging" && "Finished processing"

TRACE = 10;DEBUG = 20;INFO = 30;WARN = 40;ERROR = 50;FATAL = 60;


32

CHAPTER 9. MONITORING THE MBAAS WITH COCKPIT

9.1. OVERVIEW

System resources of nodes and containers in the MBaaS on OpenShift 3 can be monitored andmanaged using Cockpit.

Cockpit is a system administration tool, that provides insights into how nodes and containers areperforming. It lets you monitor current values and adjust limits on system resources, control lifecycleof container instances, and manipulate container images. For more information about Cockpit, referto the official web site of the Cockpit Project and its Documentation.

9.2. INSTALLATION

For most OpenShift 3 instances, Cockpit is most likely already installed on all nodes. This is not thecase if your nodes use the RHEL Atomic Host, where Cockpit needs to be installed manually.

To check whether Cockpit is installed in your OpenShift cluster, try visiting the URL of the Cockpitweb interface:

http://<master node host>:9090

If there’s no response to the request, Cockpit is most likely not installed.

9.2.1. Installing Cockpit Manually

1. Install Cockpit on nodes.

The following three steps must be repeated for each node you wish to monitor in yourOpenShift cluster.

2. Log in to the node.

3. Install Cockpit packages.

4. Enable and start the Cockpit service.

5. Create a Cockpit system user on master.

To log in to the Cockpit web interface, you will have to provide the username and passwordof an operating system user existing on the OpenShift master node. This guide refers to thisuser as the Cockpit system user. To allow Cockpit to access system resources, performoperations on Docker containers and Kubernetes resources, the Cockpit system user must:

be in the docker group;

ssh <node host>

yum install cockpit cockpit-docker

systemctl enable cockpit.socketsystemctl start cockpit.socket

CHAPTER 9. MONITORING THE MBAAS WITH COCKPIT

33

http://cockpit-project.org/

http://cockpit-project.org/guide/latest/

be able to log in to other nodes using ssh;

be able to perform Kubernetes operations.

Create the Cockpit system user on the master node, or modify an existing user to have thenecessary privileges.

9.3. VIEWING THE CONTAINERS ON AN OPENSHIFT NODE

Navigate to the Cockpit dashboard for a node in a web browser (port 9090 by default) and log in asthe Cockpit system user. To see all containers deployed on that node, click Containers in the left-hand side menu.

You can filter the list to only display running containers, using the dropdown menu above the list ofcontainers. This view lets you see the RAM and CPU usage of all running containers.

If you select an MBaaS node, you will see the containers for all MBaaS components. Clicking on acontainer will show the current logs, CPU shares, and RAM usage. In the Tools menu on the lefthand side, you can get terminal access into the node for further investigation.

9.4. VIEWING MULTIPLE HOSTS SIMULTANEOUSLY

Cockpit can connect to multiple hosts from a single Cockpit session. This can be useful to compareresource usage of two or more machines in the same dashboard. See Multiple Machines in theCockpit documentation for more information.


34

http://cockpit-project.org/guide/latest/feature-machines.html

CHAPTER 10. MONITORING THE MBAAS WITH NAGIOS

10.1. OVERVIEW

To monitor the status of the MBaaS and its components, you can deploy the Nagios monitoringsoftware to your OpenShift cluster. This guide provides steps for deployment and usage of Nagios.

10.2. PREREQUISITES

An RHMAP MBaaS running in Openshift Enterprise 3

An account on an SMTP server through which notification alerts can be sent

An email address where alerts should be sent

A PersistentVolume in Openshift, with 1Gi storage. Refer to the Persistent Storage Setupsection of the MBaaS installation guide, and Persistent Storage in the OpenShift documentationfor information on how to achieve this.

10.3. DEPLOYING NAGIOS TO OPENSHIFT

Using the oc command, change to the MBaaS project in OpenShift by running the following afterchanging <mbaas-project-id> to your MBaaS project:

The following command will create the necessary resources. Edit the environment variableparameters based on your SMTP server configuration and other details:

To learn more about the available template parameters, enter the following command:

oc project <mbaas-project-id>

oc new-app -f \https://raw.githubusercontent.com/feedhenry/mbaas-monitoring/4.0.8-8/nagios.yml \-p SMTP_SERVER=localhost,\SMTP_USERNAME=username,\SMTP_PASSWORD=password,\SMTP_TLS=auto,\[email protected],\MBAAS_ADMIN_EMAIL=root@localhost,\NAGIOS_USER=nagiosadmin,\NAGIOS_IMAGE_NAME=<nagios-image-name>,\NAGIOS_IMAGE_VERSION=<nagios-image-version>,\MBAAS_ROUTER_DNS=$(oc get route mbaas -o template --template {{.spec.host}}),\NAGIOS_HOST=$(printf "nagios-%s" `oc get route mbaas -o template --template {{.spec.host}}`)

oc process --parameters -f https://raw.githubusercontent.com/feedhenry/mbaas-monitoring/4.0.8-8/nagios.yml

CHAPTER 10. MONITORING THE MBAAS WITH NAGIOS

35

https://docs.openshift.com/enterprise/3.2/architecture/additional_concepts/storage.html

For the MBAAS_ROUTER_DNS and NAGIOS_HOST parameters, the above example is recommendedto get sane default values.

The NAGIOS_IMAGE_NAME and NAGIOS_IMAGE_VERSION are optional properties to the templateand allow for overriding the default image name and version if required.

After the command finishes, you can check the status of the deployment to see if Nagios is ready foruse – the status of the Nagios pod must be Running.

10.4. USING NAGIOS

Navigate to the Nagios web console, which is exposed at NAGIOS_HOST set during creation. Forexample:

https://nagios.apps.feedhenry.io

The username and password for login are the values of the NAGIOS_USER and NAGIOS_PASSWORDenvironment variables respectively in the 'nagios' DeploymentConfig.

Once the Nagios web console loads in a web browser, the status of the MBaaS components can beviewed by navigating to the Services tab in the panel on the left.

For further details on using Nagios, see the online Nagios documentation.

oc get pods

NAME READY STATUS RESTARTS AGEnagios-1-g7u1t 1/1 Running 0 29m


36

https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/4/en/toc.html

CHAPTER 11. TROUBLESHOOTING THE RHMAP 4.XMBAAS

11.1. OVERVIEW

This document provides information on how to identify, debug, and resolve possible issues that canbe encountered during installation and usage of the RHMAP MBaaS 4.0 on OpenShift 3.

In Common Problems, you can see a list of resolutions for problems that might occur duringinstallation or usage of the MBaaS.

Note

See also Known Issues for a list of workarounds for issues that currently exist in theRHMAP MBaaS 4.0 and will be fixed in upcoming versions.

11.2. CHECK THE HEALTH ENDPOINT OF THE MBAAS

The first step to check whether an MBaaS is running correctly is to see the output of its healthendpoint. The HTTP health endpoint in the MBaaS reports the health status of the MBaaS and of itsindividual components.

From the command line, enter the following command:

If the MBaaS is running correctly without any errors, the output of the command should be similar tothe following, showing a "test_status": "ok" for each component:

If there are any errors, the output will contain error messages in the result.error object of the details array for individual components. Use this information to identify which component is failingand to get information on further steps to resolve the failure.

You can also see a HTTP 503 Service Unavailable error returned from the health endpoint. This canhappen in several situations:

curl https://<MBaaS URL>/sys/info/health

{ "status": "ok", "summary": "No issues to report. All tests passed without error", "details": [ { "description": "Check fh-statsd running", "test_status": "ok", "result": { "id": "fh-statsd", "status": "OK", "error": null }, "runtime": 6 },...}

CHAPTER 11. TROUBLESHOOTING THE RHMAP 4.X MBAAS

37

The MBaaS hasn’t finished deploying.

The URL of the MBaaS is not reachable on port 80. Check your network configuration.

Provisioning of the MBaaS has failed. See the Prerequisites and Before the Installation (formanual installation) sections of the Provisioning an MBaaS in Red Hat OpenShift Enterprise 3guide and make sure your OpenShift cluster fulfills all the listed requirements.

Alternatively, you can see the result of this health check in the Studio. Navigate to the Admin >MBaaS Targets section, select your MBaaS target, and click Check the MBaaS Status.

If there is an error, you are presented with a screen showing the same information as describedabove. Use the provided links to OpenShift Web Console and the associated MBaaS Project inOpenShift to help with debugging of the issue on the OpenShift side.

11.3. ANALYZE LOGS

To see the logging output of individual MBaaS components, you must configure centralized loggingin your OpenShift cluster. See Centralized Logging for MBaaS Components for a detailedprocedure.

The section Identifying Issues in an MBaaS provides guidance on discovering MBaaS failures bysearching and filtering its logging output.

11.4. COMMON PROBLEMS

11.4.1. A replica pod of mongodb-service is replaced with a new one

11.4.1.1. Summary

The replica set is susceptible to down time if the replica set members configuration is not up to datewith the actual set of pods. There must be at least two members active at any time, in order for anelection of a primary member to happen. Without a primary member, the MongoDB service won’tperform any read or write operations.

A MongoDB replica may get terminated in several situations:

A node hosting a MongoDB replica is terminated or evacuated.

A re-deploy is triggered on one of the MongoDB Deployment objects in the project – manually orby a configuration change.

One of the MongoDB deployments is scaled down to zero pods, then scaled back up to one pod.

To learn more about replication in MongoDB, see Replication in the official MongoDBdocumentation.


38

https://docs.mongodb.org/manual/replication/

11.4.1.2. Fix

The following procedure shows you how to re-configure a MongoDB replica set into a fullyoperational state. You must synchronize the list of replica set members with the actual set ofMongoDB pods in the cluster, and set a primary member of the replica set.

1. Note the MongoDB endpoints.

Make note of the list of endpoints. It is used later to set the replica set membersconfiguration.

2. Log in to the oldest MongoDB replica pod.

List all the MongoDB replica pods.

In the output, find the pod with the highest value in the AGE field.

In this case, it is mongodb-1-1-4nsrv with an age of 19 hours.

Log in to the pod using oc rsh.

3. Open a MongoDB shell on the primary member.

oc get ep | grep mongo

NAME ENDPOINTS AGEmongodb-1 10.1.2.152:27017 17hmongodb-2 10.1.4.136:27017 17hmongodb-3 10.1.5.16:27017 17h

oc get po -l name=mongodb-replica

NAME READY STATUS RESTARTS AGEmongodb-1-1-4nsrv 1/1 Running 0 19hmongodb-2-1-j4v3x 1/1 Running 0 3hmongodb-3-2-7tezv 1/1 Running 0 1h

oc rsh mongodb-1-1-4nsrv

mongo admin -u admin -p ${MONGODB_ADMIN_PASSWORD} --host ${MONGODB_REPLICA_NAME}/localhost

MongoDB shell version: 2.4.9connecting to: rs0/localhost:27017/admin[...]Welcome to the MongoDB shell.For interactive help, type "help".For more comprehensive documentation, see


39

4. List the configured members.

Run rs.conf() in the MongoDB shell.

5. Ensure all hosts have either PRIMARY or SECONDARY status.

Enter the following command. It may take several seconds to complete.

There must be exactly one PRIMARY node. All the other nodes must be SECONDARY. If amember is in a STARTUP, STARTUP2, RECOVERING, or UNKNOWN state, try running theabove command again in a few minutes. These states may signify that the replica set isperforming a startup, recovery, or other procedure potentially resulting in an operationalstate.

11.4.1.3. Result

After applying the fix, all three MongoDB pods will be members of the replica set. If one of the threemembers terminates unexpectedly, the two remaining members are enough to keep the MongoDBservice fully operational.

11.4.2. MongoDB doesn’t respond after repeated installation of the MBaaS

http://docs.mongodb.org/Questions? Try the support group http://groups.google.com/group/mongodb-userrs0:PRIMARY>

rs0:PRIMARY> rs.conf(){ "_id" : "rs0", "version" : 56239, "members" : [ { "_id" : 3, "host" : "10.1.0.2:27017" }, { "_id" : 4, "host" : "10.1.1.2:27017" }, { "_id" : 5, "host" : "10.1.6.4:27017" } ]}

rs0:PRIMARY> rs.status().members.forEach(function(member) {print(member.name + ' :: ' + member.stateStr)})

mongodb-1:27017 :: PRIMARYmongodb-2:27017 :: SECONDARYmongodb-3:27017 :: SECONDARYrs0:PRIMARY>


40

11.4.2.1. Summary

Note

The described situation can result from an attempt to create an MBaaS with the samename as a previously deleted one. We suggest you use unique names for every MBaaSinstallation.

If the mongodb-service is not responding after the installation of the MBaaS, it is possible thatsome of the MongoDB replica set members failed to start up. This can happen due to a combinationof the following factors:

The most likely cause of failure in MongoDB startup is the presence of a mongod.lock lock fileand journal files in the MongoDB data folder, left over from an improperly terminated MongoDBinstance.

If a MongoDB pod is terminated, the associated persistent volumes transition to a Releasedstate. When a new MongoDB pod replaces a terminated one, it may get attached to the samepersistent volume which was attached to the terminated MongoDB instance, and thus getexposed to the files created by the terminated instance.

11.4.2.2. Fix

Note

SSH access and administrator rights on the OpenShift master and the NFS server arerequired for the following procedure.

Note

This procedure describes a fix only for persistent volumes backed by NFS. Refer toConfiguring Persistent Storage in the official OpenShift documentation for generalinformation on handling other volume types.

The primary indicator of this situation is the mongodb-initiator pod not reaching the Completedstatus.

Enter the following command to see the status of mongodb-initiator:

If the status is any other than Completed, the MongoDB replica set is not created properly. If mongodb-initiator stays in this state too long, it may be a signal that one of the MongoDB podshas failed to start. To confirm whether this is the case, check logs of mongodb-initiator usingthe following command:

oc get pod mongodb-initiator

NAME READY STATUS RESTARTS AGEmongodb-initiator 1/1 Running 0 5d

oc logs mongodb-initiator


41

https://docs.openshift.com/enterprise/3.1/install_config/persistent_storage/index.html

If the above message is the last one in the output, it signifies that some of the MongoDB pods arenot responding.

Check the event log by running the following command:

If the output contains a message similar to the following, you should continue with the belowprocedure to clean up the persistent volumes:

FailedMount {kubelet ip-10-0-0-100.example.internal} Unable to mount volumes for pod "mongodb-1-1-example-mbaas": Mount failed: exit status 32

The following procedure will guide you through the process of deleting contents of existingpersistent volumes, creating new persistent volumes, and re-creating persistent volume claims.

1. Find the NFS paths.

On the OpenShift master node, execute the following command to find the paths of allpersistent volumes associated with an MBaaS. Replace <mbaas-project-name> with thename of the MBaaS project in OpenShift.

Example output:

2. Delete all contents of the found NFS paths.

Log in to the NFS server using ssh.

Execute the following command to list contents of the paths. Replace <NFS paths> withthe list of paths from the previous step, separated by spaces.

=> Waiting for 3 MongoDB endpoints ...mongodb-1mongodb-2mongodb-3=> Waiting for all endpoints to accept connections...

oc get ev

list=$(oc get pv | grep <mbaas-project-name> | awk '{ print $1}');for pv in ${list[@]} ; do path=$(oc describe pv ${pv} | grep Path: | awk '{print $2}' | tr -d '\r') echo ${path}done

/nfs/exp222/nfs/exp249/nfs/exp255

for path in <NFS paths> ; do echo ${path} sudo ls -l ${path} echo " "done


42

Example output:

If the listed contents of the paths resemble the output shown above, delete all contents ofthe found NFS paths. Replace <NFS paths> with the list of paths from step 1, separatedby spaces.

3. Re-create persistent volumes.

/nfs/exp222-rw-------. 1001320000 nfsnobody admin.0-rw-------. 1001320000 nfsnobody admin.ns-rw-------. 1001320000 nfsnobody fh-mbaas.0-rw-------. 1001320000 nfsnobody fh-mbaas.ns-rw-------. 1001320000 nfsnobody fh-metrics.0-rw-------. 1001320000 nfsnobody fh-metrics.ns-rw-------. 1001320000 nfsnobody fh-reporting.0-rw-------. 1001320000 nfsnobody fh-reporting.nsdrwxr-xr-x. 1001320000 nfsnobody journal-rw-------. 1001320000 nfsnobody local.0-rw-------. 1001320000 nfsnobody local.1-rw-------. 1001320000 nfsnobody local.ns-rwxr-xr-x. 1001320000 nfsnobody mongod.lockdrwxr-xr-x. 1001320000 nfsnobody _tmp

/nfs/exp249drwxr-xr-x. 1001320000 nfsnobody journal-rw-------. 1001320000 nfsnobody local.0-rw-------. 1001320000 nfsnobody local.ns-rwxr-xr-x. 1001320000 nfsnobody mongod.lockdrwxr-xr-x. 1001320000 nfsnobody _tmp

/nfs/exp255drwxr-xr-x. 1001320000 nfsnobody journal-rw-------. 1001320000 nfsnobody local.0-rw-------. 1001320000 nfsnobody local.ns-rwxr-xr-x. 1001320000 nfsnobody mongod.lockdrwxr-xr-x. 1001320000 nfsnobody _tmp

Warning

Make sure to back up all data before proceeding. The following operation mayresult in irrecoverable loss of data.

for path in <NFS paths>do if [ -z ${path+x} ] then echo "path is unset" else echo "path is set to '$path'" cd ${path} && rm -rf ./* fidone


43

Log in to the OpenShift master node using ssh.

Navigate to the directory which contains the YAML files that were used to create thepersistent volumes in the section Creating NFS-backed PersistentVolumes for theMongoDB replica set members of the guide Provisioning an MBaaS in Red Hat OpenShiftEnterprise 3.

Execute the following command to delete and re-create the persistent volumes. Replace <mbaas-project-name> with the name of the MBaaS project in OpenShift.

The persistent volumes are now re-created and in Available state.

Note

The re-created persistent volumes will not be used by OpenShift again for thesame persistent volume claims. Make sure you have at least three additionalpersistent volumes in Available state.

4. Re-create persistent volume claims for MongoDB.

Create three JSON files, with the following names:

mongodb-claim-1.json



Copy the following contents into each file. Change the metadata.name value to match thename of the file without the suffix. For example, the contents for the mongodb-claim-1.json file are as follows:

list=$(oc get pv | grep <mbaas-project-name> | awk '{ print $1}');for pv in ${list}; do oc delete pv ${pv} oc create -f ${pv}.yamldone

{ "kind": "PersistentVolumeClaim", "apiVersion": "v1", "metadata": { "name": "mongodb-claim-1" }, "spec": { "accessModes": ["ReadWriteOnce"], "resources": { "requests": { "storage": "50Gi" } } }}


44

Enter the following command to re-create the persistent volume claims.

5. Verify that mongodb-initiator proceeds with initialization.

Enter the following command to see the logs of mongodb-initiator.

After mongodb-initiator completes its work, the log output should contain the followingmessage, indicating that the MongoDB replica set was successfully created.

11.4.2.3. Result

The MongoDB service is fully operational with all three replicas attached to their persistent volumes.The persistent volumes left in Released state from the previous installation are now in the Availablestate, ready for use by other persistent volume claims.

11.4.3. MongoDB replica set stops replicating correctly

11.4.3.1. Summary

If some of the MBaaS components start to crash, this may be because they can not connect to aprimary member in the MongoDB replica set. This usually indicates that the replica set configurationhas become inconsistent. This can happen if a majority of the member pods get replaced and havenew IP addresses. In this case, data cannot be written to or read from MongoDB replica set in theMBaaS project.

To verify the replica set state as seen by each member, enter the following command in the shell ofa user logged in to OpenShift with access to the MBaaS project:

For a fully consistent replica set, the output for each member would contain a members object listingdetails about each member. If the output resembles the following, containing the "ok" : 0 valuefor some members, proceed to the fix in order to make the replica set consistent.

for pvc in mongodb-claim-1 mongodb-claim-2 mongodb-claim-3; do oc delete pvc ${pvc} oc create -f ${pvc}.jsondone

oc logs mongodb-initiator -f

=> Successfully initialized replSet

for i in `oc get po -a | grep -e "mongodb-[0-9]\+" | awk '{print $1}'`; do echo "## ${i} ##" echo mongo admin -u admin -p \${MONGODB_ADMIN_PASSWORD} --eval "printjson$rs.status\($\)" | oc rsh --shell='/bin/bash' $idone

## mongodb-1-1-8syid ##MongoDB shell version: 2.4.9connecting to: admin{ "startupStatus" : 1,


45

11.4.3.2. Fix

You can make the replica set consistent by forcing a re-deploy.

1. Note the MongoDB endpoints which are in an error status.

Example:

```bashNAME READY STATUS RESTARTS AGEmongodb-1-1-pu0fz 1/1 Error 0 1h

```

2. Force a deploy of this Pod

11.4.3.3. Result

The replica starts replicating properly again and dependent MBaaS components start working again.

11.4.4. An MBaaS component fails to start because no suitable nodes arefound

11.4.4.1. Summary

If some of the MBaaS components are not starting up after the installation, it may be the case thatthe OpenShift scheduler failed to find suitable nodes on which to schedule the pods of those MBaaScomponents. This means that the OpenShift cluster doesn’t contain all the nodes required by the

"ok" : 0, "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"}## mongodb-2-1-m6ao1 ##MongoDB shell version: 2.4.9connecting to: admin{ "startupStatus" : 1, "ok" : 0, "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"}## mongodb-3-2-e0a11 ##MongoDB shell version: 2.4.9connecting to: admin{ "startupStatus" : 1, "ok" : 0, "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"}

oc get po

oc deploy mongodb-1 --latest


46

MBaaS OpenShift template, or that those nodes don’t satisfy the requirements on system resources,node labels, and other parameters.

Read more about the OpenShift Scheduler in the OpenShift documentation.

To verify that this is the problem, enter the following command to list the event log:

If the output contains one of the following messages, you are most likely facing this problem – thenodes in your OpenShift cluster don’t fulfill some of the requirements.

Failed for reason MatchNodeSelector and possibly others

Failed for reason PodExceedsFreeCPU and possibly others

11.4.4.2. Fix

To fix this problem, configure nodes in your OpenShift cluster to match the requirements of theMBaaS OpenShift template.

Apply correct labels to nodes.

Refer to Apply Node Labels in the guide Provisioning an MBaaS in Red Hat OpenShift Enterprise3 for details on what labels must be applied to nodes.

Make sure the OpenShift cluster has sufficient resources for the MBaaS components, cloudapps, and cloud services it runs.

Configure the machines used as OpenShift nodes to have more CPU power and internalmemory available, or add more nodes to the cluster. Refer to the guide on Overcommitting andCompute Resources in the OpenShift documentation for more information on how containersuse system resources.

Clean up the OpenShift instance.

Delete unused projects from the OpenShift instance.

Alternatively, it is also possible to correct the problem from the other side — change the deploymentconfigurations in the MBaaS OpenShift template to match the setup of your OpenShift cluster.

To list all deployment configurations, enter the following command:

oc get ev

Warning

Changing the deployment configurations may negatively impact the performance andreliability of the MBaaS. Therefore, this is not a recommended approach.

oc get dc

NAME TRIGGERS LATESTfh-mbaas ConfigChange 1fh-messaging ConfigChange 1fh-metrics ConfigChange 1


47

https://docs.openshift.com/enterprise/3.2/admin_guide/scheduler.html#default-scheduler-policy

https://docs.openshift.com/enterprise/3.2/admin_guide/overcommit.html

https://docs.openshift.com/enterprise/3.2/dev_guide/compute_resources.html

To edit a deployment configuration, use the oc edit dc <deployment> command. Forexample, to edit the configuration of the fh-mbaas deployment, enter the following command:

You can modify system resource requirements in the resources sections.

Changing a deployment configuration triggers a deployment operation.

11.4.4.3. Result

If you changed the setup of nodes in the OpenShift cluster to match the requirements of the MBaaSOpenShift template, the MBaaS is now fully operational without any limitation to quality of service.

If you changed the deployment configuration of any MBaaS component, the cluster should now befully operational, with a potential limitation to quality of service.

fh-statsd ConfigChange 1mongodb-1 ConfigChange 1mongodb-2 ConfigChange 1mongodb-3 ConfigChange 1

oc edit dc fh-mbaas

...resources: limits: cpu: 800m memory: 800Mi requests: cpu: 200m memory: 200Mi...


48

CHAPTER 12. KNOWN ISSUES IN THE RHMAP 4.0 MBAAS

12.1. OVERVIEW

This document describes issues that currently exist in the RHMAP 4.0 MBaaS and will be fixed inupcoming releases.

12.2. DELETING OPENSHIFT SSH KEY BREAKS CLOUD APPDEPLOYMENTS

For each cloud app, an SSH key pair is created in RHMAP so that OpenShift can use it to clone theproject and perform a build. If the key is deleted, then cloud apps can no longer be deployed fromthe Studio.

12.3. CLOUD APP ANALYTICS DATA DOES NOT UPDATE

Data in the Reporting section of a project and in the Aggregated Analytics section of the Studio maystop updating intermittently.

To fix this issue:

1. Navigate to the Admin > MBaaS Targets section.

2. Select the MBaaS hosting the cloud apps which manifest this problem.

3. Click Check the MBaaS Status.

4. Click MBaaS Project in OpenShift.

5. In the OpenShift project screen, find the fh-messaging-service, and restart itsdeployment by scaling it down to zero pods, and back up to one pod.

After the pod restarts, the analytics data is updated.

12.4. INCORRECT DEPLOYMENT STATUS INDICATED BYPROGRESS BAR FOR SUCCESSFUL DEPLOYMENTS

The progress bar of the deployment status for cloud apps and MBaaS services sometimesincorrectly indicates ongoing operation — showing blue color and moving bars instead of a solidgreen color — even after the cloud app or MBaaS service is already succesfully deployed toOpenShift.

To ensure the cloud app or service is fully deployed and running correctly, visit the Current HostURL found in the Deploy section of the cloud app or service and verify the status manually.

12.5. MONGODB DOESN’T WORK AFTER A RESTART

If two or more of the MongoDB pods are shut down, the MongoDB service will stop working until thereplica set is manually corrected. Refer to the section MongoDB doesn’t respond after repeatedinstallation of the MBaaS of the document Troubleshooting the RHMAP MBaaS 4.0 for the manualcorrection steps. After applying the procedure, the MongoDB should be fully operational.

CHAPTER 12. KNOWN ISSUES IN THE RHMAP 4.0 MBAAS

49

12.6. MONGODB POD STARTING IN REMOVED STATE

When a MongoDB pod is restarted, it can sometimes fail to resolve the host name of its associatedservice. As a result, the MongoDB instance fails to join the replica set and enters the REMOVEDstate.

The log of the MongoDB pod contains a message similar to the following:

Locally stored replica set configuration does not have a valid entry for the current node;waiting for reconfig or remote heartbeat;Got "NodeNotFound: No host described in new configuration 3 for replica set rs0 maps to this node" while validating { ... replicaset config json omitted ... }

To fix this issue:

1. Log into the master node of the OpenShift cluster.

2. Connect to MongoDB.

3. Enter the following command in the MongoDB shell:

The replica set reconfiguration may take up to several minutes. Afterwards, all replica set membersreturn to the PRIMARY or SECONDARY state.

oc rsh $(oc get pods | grep "mongodb-1" | grep -v "deploy" | awk '{print $1}')

mongo -u admin -p ${MONGODB_ADMIN_PASSWORD} admin

rs.reconfig(rs.config(), {force: true});

Warning

Running the command can lead to rollback of committed writes.


50

red hat mobile application platform 4.1 mbaas ......mbaas runs in a docker container. those...

Documents