soa cloud services disaster recovery - oracle · soa cloud service disaster recovery on oci...
TRANSCRIPT
SOA Cloud Service Disaster Recovery on OCI
Production and DR in the Cloud
O R A C L E W H I T E P A P E R | J U N E 2 0 2 0
SOA CLOUD SERVICE DISASTER RECOVERY ON OCI
Table of Contents
Introduction 1
Disaster Protection Considerations for Oracle SOA Cloud Service 3
Service Level Requirements 5
Security Requirements 5
Configuration Requirements 6
SOACS/MFTCS DR Deployment 8
Set up Details 9
1. Assign virtual hostname for the frontend OTD/LBaaS/OCILBR and update
frontend address in primary site 9
2. Update t3/RMI urls (if used) 11
3. Setup Secondary Database 12
3.1 Option 1) Configuring the Data Guard using the OCI Console 13
3.2 Option 2) Configuring the Data Guard manually 14
4. Provision the secondary SOACS System 14
5. Assign virtual hostname for the frontend OTD/LBaaS/OCILBR in the
secondary site 16
6. Download and run the Disaster Recovery Setup (DRS) tool 16
7. OPTIONAL: Verify full switchover process 19
SOACS/MFTCS DR Lifecycle Procedures 20
Switchover 20
Failover 20
Applying domain configuration changes to the system 21
SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
a) Repeating domain configuration changes in both sites 21
b) Using DBFS for configuration propagation 22
Scaling the SOACSDR system 27
How to recreate the dbfs wallet 27
Conclusion 28
Appendix A - Setting up Data Guard manually 29
Appendix B – Using a Single Tenancy to create a DR system 35
Appendix C – Cloud backups in DB Systems 37
Appendix D – Using Enterprise Manager Site Guard to manage the SOACS DR
switchovers 39
Appendix E – Summary of networking requirements for DR Setup 39
1 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Introduction
Oracle’s Maximum Availability Architecture (Oracle MAA) is the best practices blueprint for data
protection and availability of Oracle products (Database, Fusion Middleware, Applications) deployed on
on-premises, private, public or hybrid clouds. Implementing Oracle Maximum Availability Architecture
best practices is one of the key requirements for any Oracle deployment. Oracle Fusion Middleware and
Oracle Databases include an extensive set of high availability features, such as process death detection
and restart, clustering, server migration, clusterware integration, GridLink, load balancing, failover,
backup and recovery, rolling upgrades, and rolling configuration changes, which protects application
deployments from unplanned downtime and minimize planned downtime.
Oracle Service-Oriented Architecture Cloud Service (SOACS) is a Platform as a Service (PaaS)
computing platform solution for running Oracle SOA Suite in the cloud. Oracle Managed File Transfer
Cloud Service (MFTCS) is a high performance, standards-based, end-to-end managed file gateway.
These two computing platforms use Oracle Compute Cloud Service, Oracle Database Cloud Service
and Oracle Java Cloud Service as its basic infrastructure. Both require an Oracle Database to store
Oracle Platform Security Services information, instance tracking, composite and document metadata
and other Oracle FMW Infrastructure schemas. In a typical Oracle SOA deployment the application data
(such as application-specific schemas, jms stores etc.) and the SOA-specific schemas are stored in the
same database for transactional consistency and simplified administration reasons. In a SOACS/MFTCS
instance an Oracle Database Cloud Service instance is used to store these schemas. All Oracle SOA
deployments need protection from unforeseen disasters and natural calamities. This protection is also
required for systems deployed in the Cloud and it needs to address both the middle tier (Oracle SOA
Suite Cloud Service) and the data tier (Database Cloud Service). The solution involves setting up a
standby system at a geographically different Oracle cloud data center than the production site. The
standby system may have equal or fewer services and resources compared to the production site. The
standby system is normally in a passive mode and is activated when the primary site becomes
unavailable. This deployment model is sometimes referred to as an active-passive model. This model is
usually adopted when the two sites are connected over a WAN and network latency does not allow
clustering across the two sites.
Oracle SOA Cloud Service (SOACS) provides the best Recovery Time Objective (RTO) and Recovery
Point Objective (RPO) by utilizing high availability and disaster protection capabilities provided by Oracle
Fusion Middleware and Oracle Database. While there are some unique considerations to a cloud
disaster recovery (DR) configuration, it follows the same Oracle MAA best practices as any Oracle
Fusion Middleware (FMW) and Oracle Database deployment. This Oracle MAA blueprint details Oracle
2 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
MAA Best Practices and provides a procedural overview for deploying DR for SOA Cloud Service. Oracle
SOA Cloud Service Disaster Recovery solution is achieved by replicating a limited set of configuration
files that are required to bootstrap SOA components. The application may require additional
configuration files to be replicated. Options are provided in this paper to suit different application
paradigms. Disaster protection for Oracle Database Cloud Service used by Oracle SOA Cloud Service
is provided through Oracle Data Guard.
In this paper, SOA Cloud Service and DB Systems on Oracle Cloud Infrastructure are used. The
configuration screens and setup steps provided in the different sections are based on the capabilities
provided by this new IaaS platform. Refer to the OCI Classic SOACSDR whitepaper for enabling disaster
protection in older IaaS versions. For SOA on Marketplace, refer to SOA on OCI Market Place Disaster
Recovery whitepaper.
This paper is intended for a technical audience having knowledge of Oracle WebLogic Server, Oracle
FMW SOA, Oracle Database, Data Guard and Oracle Database backup and recovery. This paper also
assumes a basic understanding of services offered on the Oracle Cloud1.
1 https://cloud.oracle.com/home
3 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Disaster Protection Considerations for Oracle SOA Cloud Service
Oracle SOA Cloud Service uses DBCS (Oracle Database Cloud Service) to host the metadata and SOA instance
information. There are different database models and services available in Oracle Cloud. This paper precisely focus
and uses Database Systems VM on OCI for the examples and the configuration used. There are two different
Software Editions available for protecting Oracle Database Cloud Services against disasters:
Data Guard utilizing Enterprise Edition Service or High Performance Service.
Active Data Guard utilizing the Extreme Performance Service.
Oracle Active Data Guard allows, besides providing all the features included with Data Guard, the use of a standby
database for read operations. For more information on Data Guard and Active Data Guard please refer to the Data
Guard home page on the Oracle Technology Network and the Active Data Guard white paper2.
Oracle FMW SOA Suite can take advantage of the automatic block repair, fast incremental backups, fast rolling
upgrades, Far Sync, and most features of Active Data Guard. However, SOA servers cannot, however, use Oracle
Active Data Guard ‘s read-only query capabilities. This is because Oracle SOA Servers cannot run in read-only
mode. As soon as a SOA Server starts up, it connects to the database and tries to process business flows (i.e. they
“write” to the database right away). The only way to start the SOA servers in the standby site is by either 1)
performing a switchover/failover of the database or 2) by converting the standby database to a snapshot standby
which makes the standby read-writable temporarily for testing purposes. Redo received from the primary are stored
but not applied. The changes that are done are discarded at the end of testing where it can resume applying the
redo from the primary database. It needs to be noted that additional storage is required for the fast recovery area
when a standby is in snapshot mode (to hold archived redo received from the primary production database for later
use and current redo and flashback logs generated by the snapshot standby).
In a SOACS Disaster Recovery (DR) set up, a snapshot standby can be used to replicate configuration changes that
were applied in the production site when WLS domain configuration is not changed too frequently. The procedure
(which will be described later on in detail) consists in converting the standby database to a snapshot standby and
applying again the changes using the WLS Admin Console in the secondary location. This updates the file system
artifacts (ear files, deployment plans etc.) to be the same as in the primary site. It is also a best practice to
periodically place the standby in read/write mode (using Data Guard Snapshot Standby) to validate its readiness to
support read-write production workloads. The snapshot standby, however, needs to be used carefully with SOA
servers since undesired processing of pending instances may occur when the SOA servers have access to a
dehydration store. A snapshot standby may also be used for a final level of pre-production functional and
performance testing of patches and upgrades since the DR system is frequently sized similar to the production
system. Please refer to the Oracle documentation for additional details on Data Guard Snapshot Standby3.Beyond
this, the Oracle Cloud provides all backend infrastructure and capabilities required for disaster recovery should the
primary database become unavailable for any reason. This includes:
1. Ability to monitor the standby database and alert on major issues.
2. Ability to activate the standby to validate DR readiness and then convert back to a synchronized standby.
3. Utilization of essentially the same Oracle MAA best practices as on-premises deployments. This paper
describes the few considerations that are specific to cloud deployments.
2 http://www.oracle.com/technetwork/database/availability/active-data-guard-wp-12c-1896127.pdf 3 http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/sbydb&id=GUID-B140C38B-DE01-4252-8422-7154018DDFEC
4 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
4. Ability to switchover (planned event) or failover (unplanned event) production to the standby database in the
cloud during a planned maintenance or an unplanned outage. Once the failed primary database is repaired, it
can be resynchronized with the new production database in the cloud and then roles can be switched back to
the original status.
5. Ability to failover both the SOA/middle tier and database to enable production applications to run in the Oracle
Cloud when there is a complete site outage.
In the middle tier, the architecture of a SOACS/MFTCS DR system consists of two SOACS instances deployed in
the same sites as the Database Service instances. Each SOACS instance uses a SOA Cluster, an Administration
Server and OTD/LBaaS/OCILBR4 as front-end load balancer. A single Database (Database Cloud Service) instance
is used to host both the SOACS schemas (MDS, composite deployment, SOAINFRA schemas etc.) the JMS
persistent stores, the Transaction Logs persistent stores and any application specific data. Figure 1 describes this
architecture:
Figure 1: SOACS/MFTCS DR architecture
This topology can be created in different ways. Each approach has some implications. This paper describes the set
up process that has been considered to provide the greatest advantages from the Availability, Provisioning overhead
and Lifecycle perspective. Refer to the following sections for the recommend approach.
4 Although the examples and tests in this paper use Oracle Traffic Director (OTD) as front end load balancer, the same virtual server, certificates and
routing changes described in the document apply to Load Balancer Classic (LbaaS) and Oracle Cloud Infrastructure Load Balancing service (OCILBR)
PROVIDED SOACS supports that precise Web Tier component.
5 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Service Level Requirements
Oracle SOA Cloud Service is a user-managed environment. The user must determine service level expectations for
availability, data protection, and performance that are practical for a given configuration and application. Service
Levels must be established for each of three dimensions relevant to disaster recovery that are applicable to any
Data Guard configuration:
Availability: Recovery Time Objective (RTO) describes the maximum acceptable downtime should an
outage occur. This includes time required to detect the outage and to failover the database, the Web tier
and SOA servers so that service is resumed.
Data Protection: Recovery Point Objective (RPO) describes the maximum amount of data loss that can
be tolerated. In SOA’s case this is especially related to transaction logs, JMS messages and SOA instance
information which all resides in the same database. The actual achievable RPO depends upon:
» Available network bandwidth.
» Network reliability.
» Data Guard transport method used: either asynchronous for near-zero data loss protection, or
synchronous for zero data loss protection.
Performance: Database and Middle Tier response time may be different after failover if less capacity –
compute, memory, I/O, etc., are provisioned at the standby system than in the primary system. This occurs
when users purposefully under-configure standby resources to reduce cost (accepting reduced service
level while in DR mode). MAA best practices recommend configuring symmetrical capacity at both primary
and standby in the web, application and database tiers so there is no change in response time after
failover. Rapid provisioning available with the cloud can enable a middle ground where less capacity is
initially deployed, but where the new primary is rapidly scaled-up should a failover be required.
Note: Independent of the service levels related to DR, all database instances created in the Oracle cloud conform to
the service descriptions defined by the applicable Database Cloud Service5.
Security Requirements
Oracle MAA best practice recommends using Oracle Transparent Data Encryption (TDE) to encrypt primary and
standby databases at rest. Conversion to TDE enables automatic encryption at rest for all DATA/INDEX tablespaces
and encryption-in-flight of user data redo changes during replication to the cloud. Oracle Net encryption is also
required for encryption-in-flight for other redo changes that are not encrypted by TDE (e.g. data from unencrypted
tablespaces such as SYSTEM and SYSAUX). When DBFS is used to maintain WLS domain configuration, using the
appropriate encryption at the DB level guarantees also the security of configuration propagation across sites.
5 http://www.oracle.com/us/corporate/contracts/paas-iaas-public-cloud-2140609.pdf
6 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Note: Data Guard and Active Data Guard use redo-based replication – a process that transmits redo generated by a
primary database and applies those changes to a standby database using continuous media recovery. This means
that primary and standby databases are block for block identical copies of each other. Using TDE to encrypt a
standby database also requires that the primary database be encrypted with TDE.
Using TDE to protect data is an important part of improving the security of the system. Users should however be
aware of certain considerations when using any encryption solution, including:
Additional CPU overhead: Encryption requires additional CPU cycles to calculate encrypted and
decrypted values. TDE minimizes the overhead by taking advantage of database caching capabilities and
leveraging hardware acceleration for AES on Intel and SPARC CPUs. Most TDE users see little
performance impact on their production systems after enabling TDE. Please refer to the Oracle Database
Advanced Security Guide6 for more details.
Lower data compression: Encrypted data compresses poorly because it must reveal no information
about the original plain text data. Thus, any compression applied to TDE encrypted data will have low
compression ratios. Hence, redo transport compression should not be enabled when TDE encryption is
used. However, when TDE is used in conjunction with Oracle database compression technologies such as
Advanced Compression or Hybrid Columnar Compression, compression is performed before the
encryption occurs, and the benefits of compression and encryption are both achieved.
Key management: Encryption is only as strong as the key used to encrypt. Furthermore, the loss of the
encryption key is tantamount to losing all data protected by that key. If encryption is enabled on a few
databases, keeping track of the key and its lifecycle is relatively easy. As the number of encrypted
databases grows, managing keys becomes an increasingly difficult problem. For users with a large
number of encrypted databases, it is recommended that Oracle Key Vault7 on-premise be used to store
and manage TDE master keys.
Configuration Requirements
Beyond these runtime-related aspects, the following considerations are key to the SOACS/MFTCS DR set up:
The SOACS instance name in primary & standby should be the same. The instance name is used to
construct the hostnames, WLS server names and WLS domain name where SOACS will run. Using the
same instance name guarantees consistency and is required for the recovery of JMS messages and
Tlogs. It also simplifies customizations and operations in both sites. However, in Oracle SOA Cloud
Service, the same instance name cannot be used more than once in a single tenancy for the same service
type. i.e. it is not possible to use two different SOACS instances with the same name in the same account.
If the system is being created from scratch and there is flexibility in the name that will be used for the
primary SOACS instance, an instance name sharing the first 8 characters in the primary and standby
instances can be used to overcome this limitation. (SOACS WebLogic domain and WebLogic server
names are limited to eight characters. This limitation, however, does not exist for the SOACS Cloud
instance name). In this case the automated Data Guard configuration provided by DBCS and a single
tenancy can be used to configure a SOACS DR system- Refer to Appendix B for an example of the
instance names and approach that can be used to avoid the need of two different tenancies. In all other
6 http://www.oracle.com/pls/topic/lookup?ctx=en/database/oracle/oracle-database/12.2/asoag&id=GUID-81BF34FA-6044-47D4-BF31-12DEF178BA6B
7 http://www.oracle.com/us/products/database/security/key-vault/overview/index.html
7 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
cases (i.e. when the primary system is not using an instance name of 8 or more than 8 characters) the
following will apply TO CONFIGURE SOACS/MFTCS DR:
1. TWO DIFFERENT TENANCIES NEED TO BE USED
2. THE AUTOMATED DATAGUARD CONFIGURATION OPTION PROVIDED BY DBCS CAN
NOT BE USED
A database instance is created automatically when you create the Oracle Database Cloud Service that
SOACS needs. When configuring manually the Data Guard on the standby site, the database initially
created cannot be used as a Data Guard standby database. This database will be deleted during the DR
set up process.
Note that the DR setup described in this document has not been certified with RAC databases.
Note also that Oracle Autonomous Processing (ATP) is out of the scope of this document. ATP does not
provide cross-region disaster recovery protection yet so it can’t be used for Oracle SOACS disaster
recovery topologies.
MDS, Composite deployments and policies are automatically synchronized across sites by Data Guard
since they are stored in the database.
Most of the WLS domain configuration that SOACS uses is synced initially across sites using DBFS with
the following considerations:
1. Each SOACS system will maintain the original JDBC URLs used to connect to their local DB
even after the DR set up has completed. Only the schema prefix will be altered so that both
locations point to the same schemas
2. All the configuration under weblogc_domain_name/config is automatically distributed by the WLS
Infrastructure to the other nodes in the same site by the WLS infrastructure.
3. Custom application deployments (workflow task ears, custom ear files, deployment plan updates,
JMS resources, redeployments) and everything residing under the Administration Server WLS
domain directory (except temp data) is synced initially across sites. Data residing in other node’s
or outside the domain directory for the WebLogic Administration Server, Will have to be manually
copied to the secondary location.
The SOACS/MFTCS DR configuration makes failover/switchover transparent to end users or systems
accessing the services by making SOA endpoints agnostic to the site that is being used. For this to
be possible, the front end address of the existing SOA/MFT Cluster is altered to use a virtual hostname
that can be assigned to either DR location’s front end load balancer once the configuration is completed.
Appropriate DNS services (Oracle Cloud DNS, commercial DNS, local DNS server or manual file
hostname resolution) need to be used for the front end url to be mapped to either site. If any composites
are deployed/used before this front end address change is performed, it may be required to alter the
endpoint address to match this new virtual hostname (NOTE: By default end point addresses are
constructed based on the SOA/WLS cluster’s HTTP front end parameter).
It is possible to use the existing system’s front-end host name address (if such exists already) as the
virtual host address for disaster protection. i.e. if the original system was using mysoacs.example.com as
the vanity url for primary, this same virtual hostname can be mapped to the secondary system after a
switchover.
8 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
The initial configuration in the OTD/LbaaS/ OCILBR tier is sufficient to sustain the default routing, virtual
server and pools needed for the SOACS service. Custom configuration in OTD/LbaaS/ OCILBR is not
replicated across sites and any load balancer configuration change applied in one site, needs to be
repeated in the other-
Once Data Guard is configured for the databases used by the primary and secondary SOACS systems,
converting the “physical standby” to “snapshot standby” should be done only without starting the SOA
servers in the standby site or after “draining/truncating” the SOA database. This is required to eliminate
pending messages that could be re-executed.8
The primary and secondary databases need to communicate with each other over their listener port for
redo transport. Middle tiers need to communicate with the remote database for the initial set up also. This
communication can happen over an Internet Gateway (Oracle Net’s traffic is encrypted). Alternatively, if
the PaaS service supports it, private networks and Dynamic Routing Gateways between primary and
standby can be used and is the recommended approach. Depending on the type of access to the backend
database the appropriate ingress rules need to be enabled to allow database-to-database communication.
SOACS/MFTCS DR Deployment
In this document we assume, as a starting point, that the primary site, consisting of a single instance DB System,
SOACS Cluster and Oracle Traffic Director (OTD) is live. A secondary DR configuration, residing in a geographically
remote site, will be created for the existing primary system. Refer to the SOACS Documentation for details about
how to provision the initial set up. If the primary system has not been created yet, refer to Appendix B for details on
how to use an instance names that will allow creating the DR configuration under a single tenancy. If the primary
system exists already and is using a SOACS instance name longer than 8 chars, Appendix B provides also
indications to create the standby under that same tenancy. If the above do not apply (i.e. the primary system exists
and is not using a service name longer than 8 chars) follow these steps.
Since the primary SOACS may already be running production workloads, the DR configuration process is designed
to cause minimum downtime (in fact, only the modification of the front end address requires SOA server restarts). .
These are the summary steps for the set up process:
1. Assign a resolvable virtual hostname as front end for the primary system (or alternatively reuse any
friendly/vanity url or DNS name already used by the existing SOACS system)
2. Update front end address in the primary SOACS cluster
3. Update t3/rmi urls (if used)
4. Setup Secondary Database (configuring Data Guard with OCI Console or manually)
5. Convert the physical standby database to snapshot standby
6. Provision SOACS in secondary location pointing to snapshot DB in a different tenancy
7. Run Disaster Recovery Setup tool (DRS) to configure the SOACS DR.
8. Verify SOACS in secondary site
8 SOA servers will try to process SOA instances and complete pending work (callbacks, async invocation etc.) that may have been processed already
on the primary site. If the snapshot database has not been drained, only the administration server should be brought up in the standby domain for
systems already in production.
9 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
9. Convert snapshot database to physical standby
10. Optionally, switchover to standby to verify a complete DR scenario
The following sections describe these steps in detail.
Set up Details
Before creating the SOACS system in standby, it is recommended that the middle tier and the database in the
primary system contain the latest patches and PSUs. More precisely a DB System Data Guard configuration across
different DB domains requires a fix for bug 22611167. Make sure the pertaining patch for your precise database
version is applied in the system.
Make sure you have reviewed the requirements in previous section Configuration Requirements and specially note
down the instance name on the primary instance (which is assumed to exist already). Additionally, make sure that
the SOACS version and patch level to be provisioned in the secondary location matches the one running in the
primary site. 9
Once the DB System and the SOACS instance in the primary location are patched and functioning, follow these
steps:
1. Assign virtual hostname for the frontend OTD/LBaaS/OCILBR and update frontend address in
primary site
By default SOACS’s provisioning sets the value of the SOA cluster’s front end address to the IP of
OTD’s/LBaaS/OCILBR listen address. This IP needs to be replaced with a virtual hostname and this hostname
needs to be resolvable by both the external clients and by the WLS servers that OTD/LBaaS/OCILBR is routing
requests to. To resolve the virtual hostname externally, local hosts file or any formal DNS resolution may be
used. To resolve the virtual hostname in the scope of the SOACS instance, either local hosts file resolution or
Oracle DNS Cloud Service can be used. In this last case the resolv.conf files for the SOA nodes needs to be
updated with the new DNS server hosting the record for the virtual hostname.
A. To determine the IP matching the virtual hostname for the SOACS Cluster, follow these steps:
Login to the WebLogic Server Administration Console for your SOACS instance.
In the left pane, choose Environment in the Domain Structure window and then choose
Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster.
Select HTTP
Note down the IP used in the Frontend Host filed.
NOTE: Alternatively you can get this IP also from the SOACS instance screen information.
This IP would be listed as the PUBLIC IP for the Load Balancer:
9 Note that if primary SOACS instance was created long time ago, the underlying OS image version may be older that the OS image provisioned in the
new secondary SOACS instance. Contact My Oracle Support if that is the case.
10 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
B. Add the IP that maps to the virtual hostname to the DNS resolution in the nodes. In a command
shell prompt for your SOACS nodes, sudo to the root user and edit /etc/hosts to map this IP to a
virtual hostname (if a DNS is going to be used for the external resolution of this IP, this hostname
will have to be used here. If you are going to use file-based (/etc/hosts) hostname resolutions,
this would be your own choice). For example:
[oracle@soacsdroci-wls-1 ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.11 drDb2a.sub10171440110.soacsvcnash.oraclevcn.com drDb2a
129.213.81.253 soacsdroci.paasmaaoracle.com soacsdroci
Repeat this step in ALL the other SOACS nodes that are hosting members of the
SOACS Cluster
To make the virtual hostname available to the external clients either update your DNS
server or modify the hosts file for those clients as above. For example in your on-
premises windows client, edit your C:\Windows\System32\drivers\etc\hosts file as
above
NOTE: This configuration for external clients will work if direct connections from the internet are used. Connections
from custom intranets will need to account for this hostname to be added to the required proxy server used by the
browsers or http clients
As a plausible alternative, Oracle DNS Zone management can be used to maintain the front end
hostname mapping. In this example, the front end address hostname would be a record more in the
appropriate zone. Client would need to update their /etc/resolv.conf nameserver entry to use the
appropriate DNS server.
11 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
C. Once the appropriate hostname resolution is in place per the steps above, modify the front end
address in the SOACS Cluster:
Login to the WebLogic Server Administration Console for your SOACS instance.
In the left pane, choose Environment in the Domain Structure window and then choose
Clusters. The Summary of Clusters page appears. Select the SOA_Cluster cluster.
Select HTTP
Set the value for the Frontend Host=virtual_hostname_created, in the example above
this would be soacsdroci.paasmaaoracle.com
Click Save.
To activate the changes, click Activate Changes in the Change Center section of the
Administration Console.
Restart the SOACS cluster using the WebLogic Administration Console for the front
end change to be effective.
2. Update t3/RMI urls (if used)
The urls used for RMI invocations in the SOACS cluster need to be agnostic to the IPs or hostnames used by each
site in the SOACS/MFTCS DR configuration. Instead of using the typical host:port,host:port JNDI urls, change them
to use the cluster syntax10. The cluster syntax is as follows: cluster:t3://cluster_name. For example, to modify the
JMS adapter factory properties to use this syntax, follow these steps:
A. Log into your Oracle WebLogic Server Administration Console for your SOACS instance
B. Click Deployments in the left pane for Domain Structure.
C. Click JmsAdapter under Summary of Deployments on the right pane.
D. Click the Configuration tab.
10 Obviously this is feasible only for intra-domain invocations. T3/rmi clients that are external to the SOA domain will not be able to use this approach
and will have to use the appropriate DNS mapping of the host:port list when switching to the secondary site. A TCP load balancer can be used for the
JNDI InitialContext retrieval, but subsequent requests from JMS clients will connect to the host:port directly, so the DNS mapping to secondary site ips
is required also in this case.
12 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
E. Click the Outbound Connection Pools tab and expand
oracle.tip.adapter.jms.IJmsConnectionFactory to see the configured connection factories.
F. Click the specific instance you are using (for example, eis/wls/Queue). The Outbound
Connection Properties for the connection factory opens.
G. Click Lock & Edit.
H. In the FactoryProperties field (click on the corresponding cell under Property value), alter the
java.naming.provider.url filed to use the cluster syntax (leave the rest of the fields as they were):
java.naming.provider.url= cluster:t3://cluster_name;
I. Click Save after you update the properties. The Save Deployment Plan page appears.
J. Enter a location for the deployment plan.
K. Copy the deployment plan from your SOACS node1 to your SOACS node2 in the exact same
directory/location or use the default DBFS mount point present in SOACS system as the location
to host these deployment plans (all nodes in the SOACS cluster can access
/u01/soacs/dbfs/share)
L. Click Save and Activate.
M. Click Deployments.
N. Click Lock & Edit
O. Select the JMS Adapter.
P. Click Update.
Q. Select Update this application in place with new deployment plan changes (A deployment plan
must be specified for this option.) and select the deployment plan saved in a shared storage
location; all servers in the cluster must be able to access the plan.
R. Click Finish.
S. Activate the changes.
Similarly, any other custom JNDI urls used in the SOACS system should also be updated so that when a
switchover/failover occurs in the SOACSDR system, the urls are valid also in the secondary site.
3. Setup Secondary Database
The secondary SOACS system will be provisioned using the normal procedure. However, as explained in previous
sections, it needs to use the SOACS same instance name as primary, hence it cannot be created in the same
tenancy (unless the primary system is also created from scratch with flexibility in the instance name that it will use. In
this case refer to Appendix B – Using a Single Tenancy to create a DR system). The SOACS Service is created
pointing to a database that will be configured as the Data Guard standby of the database that is being used for the
primary system. In order to use the standby database as the container for the secondary SOACS provisioning, this
physical standby needs to be converted to a snapshot standby.
Hence, before provisioning the secondary SOACS system, the secondary database must be setup. The
secondary database is created as a Data Guard physical standby of the primary database. There are 2 options to do
this: one is to use the OCI Console to enable Data Guard (referred in this document as “automated Data Guard”),
13 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
and the other option is to manually configure the standby database with dgmgrl commands, using scripts provided in
this whitepaper (referred in this document as “manual Data Guard”).
The recommended approach is to configure the Data Guard using the OCI Console (option 1). This approach is
possible when the same tenancy is used for primary and secondary systems (refer to Appendix B – Using a Single
Tenancy to create a DR system). Configuring Data Guard the OCI Console User Interface allows you to use the
Console to manage Oracle Data Guard associations in your DB system. It also provides out of the box configuration
for backups in the Data Guard. Follow the point Option 1) Configuring the Data Guard using the OCI Console to
enable the Data Guard using the OCI Console.
If for some reason the feature to enable Data Guard is not available for your case, for example, when using 2
different tenancies for primary and secondary (please refer to the DB System documentation for checking the
availability of the Data Guard across regions feature in each DB Systems flavor/edition), you can still configure the
Data Guard manually using scripts provided in this whitepaper. Follow steps described in Option 2) Configuring the
Data Guard manually for this.
NOTE: the SOACS DR Setup procedure described in this paper has not been certified with RAC database
3.1 Option 1) Configuring the Data Guard using the OCI Console
When enabling Data Guard using the OCI Console, the secondary DB system is automatically provisioned and
configured as physical standby when you click on Enable Data Guard in the primary DB System. There are
some requirements for this, for example: both DB systems will be in the same tenancy and compartment, both DB
Systems will be the same shape type, if the DB Systems will be in different regions, they must be connected via
remote VCN peering, etc. See Using Oracle Data Guard in Oracle Cloud Infrastructure Documentation for more
details about these requirements.
To enable Data Guard to the primary database, login to OCI Console and navigate to the primary DB System and
click in the primary database. You can enable Data Guard in the section “Data Guard Associations”. Most of the
configuration properties of the secondary DB System (like version, DB name, etc.) are predefined because they are
inherited from primary, but you need to provide some configuration properties. The following table provides
examples and requirements for these properties:
Cloud Account
Configuration Property
Existing Primary DB
System/example
Secondary Db System being
created/example
Oracle Cloud Tenancy XXXX/paasmaa XXXX/paasmaa (must be the same
when configuring Data Guard using the
OCI Console)
Oracle Cloud Account User XXXX/[email protected] XXXX/[email protected] (same)
Oracle Cloud Account Password XXXX/Acme1234# XXXX/ Acme1234# (same)
DB System Configuration
Property
Existing Primary DB
System/example
Secondary Db System being
created/example
Compartment XXXX/soacsdr XXXX/soacsdr (must be the same)
Region XXXX /Ashburn YYYY/Phoenix (can be different,
different region recommended for DR)
Availability Domain XXXX/efEXT:US-ASBURN-AD1 YYYY/ efXT:PHX-AD-3 (should be
different from primary)/
14 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Peer DB System Name XXXX/soacsdrDBa YYYY/soacsdrDBb (can be different
from primary)
Shape XXXX/VM.Standard2.1 XXXX VM.Standard2.1 (same as
primary)
Virtual Cloud Network XXXX/soacsdrASH YYYY/soacsdrPH (systems are
expected to be remote, hence different
from primary. Needs to include a public
Internet Gateway)11
Client Subnet XXXX/Public Subnet efXT:US-ASBURN-
AD1
YYYY/ Public Subnet efXT:PHX-AD3
(systems are expected to be remote,
hence different from primary)
Hostname Prefix XXXX/soacsdrASH YYYY/soacsdrPH (can be different from
primary)
Database Admin Password XXXX/Acme1234# XXXX/Acme1234# (same as primary)
3.2 Option 2) Configuring the Data Guard manually
Note: In case that the Data Guard has been enabled using the OCI Console as explained in 3.1, these steps must
be skipped and you can continue with section 4 Provision the secondary SOACS System
It is required to manually configure Data Guard manually when different tenancy is used for primary and standby or
when the automated Data Guard option provided by OCI is not enabled for the DB flavor and/or locations involved in
the DR configuration.
In this case, the secondary DB System must be provisioned as a regular DB System, and then it will be configured
as the standby by executing some setup scripts provided in this document. Follow the steps described in Appendix
A - Setting up Data Guard manually to provision secondary DB system and to configure Oracle Data Guard
manually.
4. Provision the secondary SOACS System
The secondary SOACS system will be provisioned using the normal procedure. However, as explained in previous
sections, it needs to use the same instance name as primary, hence it cannot be created in the same tenancy
(unless the primary system is also created from scratch with flexibility in the instance name that it will use. In this
case refer to Appendix B). The following are key considerations for this set up:
The SOACS Service is created pointing to a database that will be configured as the Data Guard standby of
the database that is being used for the primary system. In order to use the standby database as the
container for the secondary SOACS provisioning, this physical standby needs to be converted to a
snapshot standby.
Oracle SOACS provisions SOA schemas using a prefix that is specific to each cloud services/instance.
This means that in the initial provisioning the secondary location SOACS servers will point to the same
Database but will use different schemas. This is critical for systems that are already running because this
will prevent the execution of composites/flows by the initial SOACS domain in the secondary location. It is
needed that only one site has active SOA servers pointing to an available database at any point in time.
Otherwise message and callback duplications could occur leading the SOA system to inconsistencies.
11 Remote peering can also be used for DG traffic across regions
15 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
IMPORTANT: Once the secondary location JDBC strings are updated to point to the same schemas as
production, the SOA servers in the secondary location will see the same data that the production ones were
seeing when the snapshot conversion occurred. If any SOA flows, callbacks etc. are pending, the servers in the
secondary location will try to complete those. Thus, it is important that instances are drained and completed on
the primary site before converting the standby database to snapshot or duplications could occur. Alternatively
the SOA servers in the primary location may be stopped and the database can be switched over to the
secondary location (incurs in higher downtime).
Follow these steps to provision the secondary SOACS system:
A. Convert the physical standby database to snapshot standby. As oracle user in the primary
Database node, execute the following (replace your_sys_password and
secondary_db_unqname with the appropriate values for your case)
[oracle@soacsdrDBa ~]$ dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> CONVERT DATABASE secondary_db_unqname to SNAPSHOT STANDBY;
Converting database "secondary_db_unqname " to a Snapshot Standby database, please
wait...
Database "secondary_db_unqname" converted successfully
B. Provision the secondary SOACS instance using the SOACS provisioning wizard:
Follow the steps in the SOACS documentation to create the secondary site SOACS system pointing to the
secondary DB System converted to snapshot in the previous step. Use the EXACT same name for the SOACS
service that you are using in your primary location. The following table summarizes the provisioning wizard
options for the set up:
Cloud Account
Configuration Property
Existing Primary SOACS
System/example
Secondary SOACS System
being created/example
Oracle Cloud Tenancy XXXX/paasmaa YYYY/paasmaa2 (should be different
from primary)
Oracle Cloud Account User XXXX/[email protected] YYYY/[email protected] (user name can
be different, tenancy must be different)
Oracle Cloud Account Password XXXX/Acme1234# XXXX/ Acme1234# (can be different
from primary)
SOACS Configuration Existing Primary SOACS
System/example
Secondary SOACS System being
created/example
SOACS Service Name XXXX/soacsdroci XXXX/soacsdroci (same as in primary)
Region XXXX/us-ashburn-1 YYYY/us-phoenix-1 (systems are expected
to be remote, hence different site from
primary
Availability Domain XXXX/efXT-ASHBURN-AD-1 YYYY/ efXT:PHX-AD-3 (should be different
from primary and the same as the
secondary Db System)
16 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Subnet XXXX/Public Subnet efXT:US-ASBURN-
AD1
YYYY/ Public Subnet efXT:PHX-AD3
(systems are expected to be remote, hence
different from primary)
SSH Public Key XXXX YYYY (can be different from primary)
License Type LI, BYOL/BYOL LI, BYOL/BYOL(can be different from
primary)
Software Release XXXX/12.2.1.3 XXXX/12.2.1.3(same as primary)
Service Type SOA with SB & B2B Cluster or
MFT/SOA with SB & B2B Cluster
SOA with SB & B2B Cluster(same as
primary)
Compute Shape XXXX/VM.Standard2.1 XXXX VM.Standard2.1 (same as primary)
Cluster Size N/2 N/2(same as primary)
WebLogic User Name XXXX/weblogic XXXX/weblogic(same as primary)
WebLogic Password XXXX/Acme1234# XXXX/Acme1234#(same as primary)
Database Type Oracle Cloud Infrastructure Oracle Cloud Infrastructure
Database Compartment Name XXXX/soacsdr XXXX/soacsdrB (can be different from
primary)
Database Name XXXX/ORCL XXXX/ORCL(same as primary)
Database PDB Name XXXX/PDB1 XXXX/PDB1(same as primary)
Database Administrator User name XXXX/SYS XXXX/SYS(same as primary)
Database Admin Password XXXX/Acme1234# XXXX/Acme1234# (same as primary)
Load Balancer Provisioning Yes Yes
Load Balancer Policy Least Connection Count Least Connection Count
Load Balancer Compute Shape XXXX/VM.Standard2.1 XXXX VM.Standard2.1 (same as primary)
Backup and recovery Configuration XXXX YYYY ( different from primary)
Oracle recommends that the exact same capacity and compute configuration is used on both primary and standby
locations for the ideal failover/switchover behavior (otherwise, the required request-throttling in OTD/LBaaS/OCILBR
and sizing of SOACS nodes needs to be done on the secondary location). Once the provisioning process completes
the SOA servers can be sanity verified.
5. Assign virtual hostname for the frontend OTD/LBaaS/OCILBR in the secondary site
Follow the same steps as in section “Assign virtual hostname for frontend OTD/LBaaS/OCILBR and update frontend
address in primary site” above using the public IP of the OTD/LBaaS/OCILBR instance in the secondary site to
map the virtual host. You don’t need to update the front end address for the SOACS cluster as that information will
be copied from the primary WebLogic Domain configuration. Only the update of the /etc/hosts or DNS entry is
required in standby
NOTE: Despite the hostname alias being the same in both sites, the required trust stores/certificates will have to be
recreated in the standby locations and the standby’s OTD/LBaaS/OCILBR certificate will have to be imported in the
appropriate trust store if any SSL invocations are executed from the Cloud servers to the front end LBR. Refer to the
Enterprise Deployment Guide for Oracle SOA Suite for the required steps.
6. Download and run the Disaster Recovery Setup (DRS) tool
The Disaster Recovery Setup tool (DRS) is a framework that orchestrates and runs the configuration steps for the
SOACS disaster recovery setup. The tool can be run from any host (with operating system OEL 7) that has ssh
connectivity to the SOACS and DB hosts where the DR setup is being done.
17 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
DRS tool currently requires the following communication to be allowed:
Secondary midtier hosts to primary DB private IP port 1521. Required when primary and secondary databases connect using private networks, via remote peering and Dynamic Routing Gateway. This is used during the DR setup and also for lifecycle operation like config replication. This is the recommended approach.
Secondary midtier hosts to primary DB public IP port 1521. Required when primary and secondary dababase connect via their public IPs (because no remote peering is used between sites). This is used during the DR setup and also for lifecycle operation like config replication. This is not a recommended approach.
A quick check can be run on all the secondary midtier hosts with user oracle to verify the connectivity to private/public primary database IPs before running DRS, depending on the network scenario:
java -classpath /u01/app/oracle/middleware/wlserver/server/lib/weblogic.jar utils.dbping ORACLE_THIN
system <system_password> <primary_db_private_or_public_ip>:1521/<primary_db_service>
See the table in Appendix E – Summary of networking requirements for DR Setup for more details on network
requirements for DR setup.
Steps to run the DRS tool:
Choose a host to run the DRS tool. This host needs to be able to connect via ssh to all the hosts involved in the DR (midtier and db hosts). The IPs used to connect to the hosts via ssh, along with other input parameters, will be provided to DRS in a yaml property file before running DRS. Normally these are public IPs used for ssh, but in case they are in private networks and there are not public IPs, provide the appropriate IPs that can be used to connect via ssh to the hosts from the host where DRS will run.
TIP: create a compute instance (OEL 7) in your cloud tenancy to run the tool. This compute instance can
be removed later, once the DR configuration is done and DRS is not needed anymore.
Download the DRS tool for SOACS from here and upload it to the host where the tool will run.
Extract the contents of this file using the command ‘tar -xzf drs.tar.gz’ and navigate to the ‘drs’ directory it
creates.
Open README.md and follow instructions. Please make sure you review IN DETAIL the steps and
recommendations in DRS's README.md file. It is critical that you check the config file required to run it
and meet the requirements in it for the appropriate behavior in the set up.
The DRS tool will automatically perform the required steps to configure secondary SOACS as standby SOACS DR
site. These steps include:
Initial checks to verify that the environment is prepared for the DR setup.
Addition of the required host aliases configuration in the /etc/hosts files, in primary and secondary SOA servers. Secondary midtier host names will be added as aliases to primary midtier’s /etc/hosts, and in the other way also: primary midtier host names will be added as aliases to the secondary midtier’s /etc/hosts file.
DBFS wallet recreation for the SOA DBFS mounts and the addition of the required aliases in the tnsnames.ora file in midtier hosts (aliases to remote and local databases will be used for future domain configuration replication). The dbfs mount is remounted in primary admin node and secondary nodes during the DR setup process.
A backup of the secondary domain configuration is done by DRS before modifying it during the DR setup (i.e. /u01/data/domains/soacsdrs_domain_backup_<timestamp>).
18 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Primary domain configuration is copied to secondary site: it copies the domain config from primary to the dbfs mount, and then from the dbfs mount to domain folder in secondary hosts.
The appropriate replacement of the database connection string is done in the secondary config files after copying the domain config from primary to secondary domain: primary database connection string will be replaced by secondary database connection string in the secondary domain configuration. Note that only the db connection string is different between primary and secondary domain, because once DR is configured, the secondary domain will point to the same schema names than primary.
Verification that the secondary domain is correctly configured for DR by starting the secondary managed servers in a rolling manner after the DR configuration, using the database in snapshot mode, and check the connection to the secondary frontend soa-infra url.
During the process, the tool performs some database role conversions in the secondary database (conversion to snapshot standby and back to physical standby). During execution, the framework logs to a log file named "logfile_<date-time-stamp>.log". You can monitor setup progress by monitoring the contents of this file and the output of the process. Once it finishes, it leaves the secondary database in physical standby role and the secondary admin and managed servers stopped.
IMPORTANT: Up to this point, the SOA servers in the secondary location have been pointing to “empty” SOAINFRA
schemas with no composites deployed, no policies and no flows pending of execution. Once the secondary location
JDBC strings have been updated to point to the same schemas as production per the above steps, the SOA servers
in the secondary location will see the same data that the production ones were seeing. If any flows, callbacks, etc.
were pending to be executed; the secondary location servers will try to complete those at this point if started. Thus, it
is important that instances are drained and completed on the primary site before converting to snapshot the standby
database as already indicated above.
19 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
7. OPTIONAL: Verify full switchover process
The system should now be ready for a switchover. It is a best practice to perform a complete and full
switchover test (if primary is already “production” a maintenance window should be scheduled) to confirm
the proper behavior of database and servers. To perform an initial switchover test, follow these steps
1. Switchover Database
Execute these steps as oracle user in the primary Database host:
[oracle@soacsdrDBa ~]$ dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> switchover to secondary_db_unqname
Performing switchover NOW, please wait...
Operation requires a connection to instance "ORCL" on database "primary_db_unqname"
Connecting to instance "ORCL"...
Connected as SYSDBA.
New primary database "secondary_db_unqname" is opening...
Oracle Clusterware is restarting database "orcl" ...
Switchover succeeded, new primary is "secondary_db_unqname "
2. Verify SOACS in secondary site
At this point the SOA servers in both sites are pointing to the same schemas and the database is open in the
secondary location. Start the servers and verify composite deployments and composite instances (if any) in the new
active site
a. Start the Administration and Managed servers in the secondary site
a. Logon to the Enterprise Manager FMW Control Console for the secondary SOACS instance and
verify the soa-infra systems in both servers in the cluster. It is a good practice to perform an
endpoint test to confirm the correct behavior of the system
3. Switchback to original site.
To revert the system back to its original state, switchback the database and restart SOA servers in the primary
location.
20 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
SOACS/MFTCS DR Lifecycle Procedures
Switchover
In a switchover operation an administrator reverts the roles of the two sites in a planned operation. The roles change
from the primary to the standby as well as from standby to primary. Oracle Site Guard can be used to automate the
task required in a switchover and reduce the RTO in such operation. Refer to Appendix D for details. To perform a
manual switchover in a SOACS/MFTCS DR configuration follow these steps:
a. Stop the managed servers in the SOACS primary site.
Use the SOACS documentation to stop the Administration and Managed servers in the primary site
b. If configuration changes have been applied recently to the primary SOA/WebLogic domain,
make sure to propagate those to the standby location (Refer to section “Applying domain
configuration changes to the system” bellow)
c. Perform the required DNS changes to point consumers to the new primary OTD
Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host
resolution to point the front end address of the system to the public IP used by OTD/LBaaS/OCILBR in site2
d. Perform a database switchover using Data Guard Broker
From the primary site’s DB System node and as oracle user:
[oracle@soacsdrDBa ~]$ dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> switchover to secondary_db_unqname
Performing switchover NOW, please wait...
Operation requires a connection to instance "ORCL" on database "secondary_db_unqname"
Connecting to instance "ORCL"...
Connected as SYSDBA.
New primary database "secondary_db_unqname" is opening...
Oracle Clusterware is restarting database "primary_db_unqname " ...
Switchover succeeded, new primary is "secondary_db_unqname"
e. Start Managed servers in the new SOACS primary site
Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site. (the
Administration server and Node Manager can be kept up on standby)
Failover
In a failover operation, the primary site becomes unavailable and an administrator fails over the Database and starts
managed servers in the secondary site. You can role-transition a standby database to a primary database when the
original primary database fails and there is no possibility of recovering the primary database in a timely manner. This
is known as a manual failover. There may or may not be data loss depending upon whether your primary and target
standby databases were transactionally consistent at the time of the primary database failure. Oracle Site Guard can
be used to automate the task required in a failover and reduce the RTO in such operation. Refer to Appendix D for
details to perform a switchover in a SOACS/MFTCS DR configuration follow these steps:
a. Perform the required DNS changes to point consumers to the new primary OTD
21 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Perform the required DNS push in the DNS server hosting the names used by the system or alter the file host
resolution to point the front end address of the system to the public IP used by OTD/LBaaS/OCILBR in Site2
b. Perform a database failover using Data Guard Broker. Start Data Guard broker on the secondary
DB System node. As oracle user execute these steps:
[oracle@soacsdrDBb ~]$ dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> failover to secondary_db_unqname
Performing failover NOW, please wait...
Failover succeeded, new primary is "secondary_db_unqname"
c. Start Managed servers in the new SOACS primary site
Use the SOACS documentation to start the Managed servers in the secondary (now new primary) site
Applying domain configuration changes to the system
The fact that direct file system synchronization across Cloud datacenter virtual machines is not allowed, precludes
WLS domain config changes from being automatically propagated to the secondary site (as it would occur in
standard on-premise Active Passive DR deployments). Two main approaches can be used to maintain the same
configuration (ear deployments, WLS domain configuration, deployment plans etc.) in both locations. The
applicability of each depends on how frequently this “file-system-resident” configuration is modified.
a) For those SOACS cases where the domain configuration is infrequently altered (notice that composite
deployments, domain and WSM policies and MDS updates do not fall into this category as they are
stored in the Database) it is recommended to simply apply the configuration changes manually twice,
once in production and once in standby.
b) For those SOACS cases where the file system configuration is modified regularly, Oracle Database
File System (DBFS) can be used to synchronize the configuration using Data Guard (DBFS provides
a file system view of data that is stored in the Database). Using DBFS for configuration replication has
implications from the setup, disk space allocation and lifecycle perspective and oracle recommends
using it when it is necessary to replicate configuration changes frequently. There are other
alternatives to DBFS such as direct use of rsync across sites, but those present other risks including
lack of transactional control in the copy and possible corruption of the domain structure in the event of
a failure during the copy operations.
Both approaches are described below.
a) Repeating domain configuration changes in both sites
To maintain the file system configuration synchronized by repeating the config change in both sites, follow these
steps:
A. Apply the configuration change normally in the primary site
Use the WLS Administration Console in the primary location to apply the configuration change. Activate the
change, restart the required SOACS servers if needed and verify that the change is working as expected.
B. Convert the standby database to a snapshot standby
Execute these steps as oracle user in the primary Database host:
[oracle@soacsdrDBa ~]$dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> CONVERT DATABASE secondary_db_unqname to SNAPSHOT STANDBY;
22 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Converting database " secondary_db_unqname" to a Snapshot Standby database, please
wait...
Database " secondary_db_unqname" converted successfully
C. Start (if it wasn’t started) the Administration Server on the secondary site
Follow the steps in the Oracle Cloud documentation to start the administration server. It is important that ONLY
the administration server and not the managed servers is started on the secondary location12.
D. Repeat the configuration change in the secondary site
Use the WLS Administration Console in the primary location to apply the configuration change. Activate the
change and verify that the change is working as expected. This change should not alter any of the
configurations in the database, only WLS domain configuration or application deployments. Any modifications
in the database will be overwritten by the primary when the Db is reverted to physical standby in the next step.
E. Revert the database to physical standby Execute this steps as oracle user in the primary Database host:
[oracle@soacsdrDBa ~]$dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> CONVERT DATABASE secondary_db_unqname to PHYSICAL STANDBY;
Converting database " secondary_db_unqname" to a Physical Standby database, please
wait...
Oracle Clusterware is restarting database "orclb" ...
Continuing to convert database " secondary_db_unqname" ...
Database " secondary_db_unqname" converted successfully
b) Using DBFS for configuration propagation
Database File System (DBFS) uses database features to store files and manage relational data and expose them as
a standard file system that can be accessed by any operating system. The Oracle Database File System (DBFS)
creates a standard file system interface on top of files and directories that are stored in database tables. DBFS is
similar to NFS in that it provides a shared network file system that looks like a local file system. Like NFS, both a
server component and a client component are required to run DBFS.
Given the restrictions in file replication across Oracle Cloud data centers, DBFS can be used with Oracle Data
Guard to copy files from a primary site to a secondary site. When the system’s lifecycle involves frequents updates
to the domain file system, DBFS can be used to replicate the WLS domain configuration across sites on a regular
basis. However, the SOACS domain Configuration cannot reside directly on the DBFS mount because that would
make the middle tier dependent on the DBFS infrastructure in order to come up (the dependency is not only on the
database but also on FUSE libraries, mount points etc.). Notice also that the SOACS WebLogic domain
configuration cannot be copied “as is” in this paper’s design since each site in the SOACS/MFTCS DR solution
contains references to the Cloud identity domain and the local DB service in the JDBC connect strings. The
configuration has to be modified after it is copied to each site. In this approach, a directory with a copy of the primary
site’s domain configuration is kept up to date with Data Guard in the standby location. This directory is not available
to the standby site unless the DB is open in read-only mode (when Active Data Guard is used), a conversion to a
snapshot standby is performed or a switchover or failover is executed. The conversion to snapshot will allow
mounting the DBFS file system for writes also, but has the caveat that it will not preclude SOA servers in the
12 Changes to a reduced number of configuration artifacts in SOA and OSB may require the servers to be up in order to be applied, in these cases a
start of the managed servers will be needed. Refer to the specific product documentation to identify these artifacts. In this case and if there are pending
messages in the database those could be re-executed in the standby location. In such scenarios, Oracle recommend draining/truncating the SOA
database schemas in the snapshot database following the SOA standard procedures BEFORE starting the SOA WLS servers
23 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
secondary location from starting (either accidentally, or by a reboot) and may cause duplications and re-executions
of work already processed by the primary Location. Oracle recommends, however that a complete start and test of
the servers in the secondary location is performed on a regular basis to verify that the configuration and
deployments are being replicated properly and that the Cloud identity Domain and JDBC connect strings are being
replaced correctly. This can be done by doing a switchover or by converting the secondary database to snapshot
standby. In this last case, Oracle recommends to “drain” first the Primary Database from any SOA work pending
(this includes pending async invocations and JMS messages) to avoid duplicated executions. Draining may imply
blocking new incoming requests and waiting for all pending messages to be processed/completed. The following
diagram describes the model used to replicate WebLogic Domain configuration changes from the primary to the
secondary location.
For application deployment operations, Oracle recommended using the WebLogic deployment “Upload your files”
option in the WebLogic Administration Console so that the deployed files are placed under the upload directory of
the Administration Server (under domain directory/servers/admin_server_name/upload). That way these files will be
synced to standby by the DBFS copy scripts
In summary the use of DBFS for domain configuration synchronization requires the following steps:
1. Verify the DBFS configuration already created by SOACS instance and the appropriate mount points for
DR.
2. Create domain copy and configuration replacement scripts.
3. Enable domain copy (either by manual execution or by configuring cron jobs)13
The following subsections describe these steps in detail:
13 Changes applied to files in different nodes of the SOACS Cluster that do not reside under the domain/config directory, need to be manually synced to
those nodes. The DBFS approach described in this paper replicate the domain configuration to the Administration Server node only.
24 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Use the DBFS configuration already created by the SOACS instance and verify appropriate mount points for DR
DBFS is already configured out of the box for SOACS systems. A client command-line interface named
dbfs_client runs on each SOACS computer with pre-configured access to the DBFS mount points. dbfs_client
allows users to copy files in and out of the database from any host on the network. DBFS is a part of Oracle
database installation and requires fuse (FileSytem in Userspace) in the nodes that access the mount point. It
implements simple file system commands such as list and copy in a manner that is similar to shell utilities.
dbfs_client and fuse are already installed in SOACS starting 12.2.1.2 VMs. The DBFS mount points and client
configuration created by default in SOACS VMs are used for MFT, B2B and File Adapter cases (to share a common
mount point between the members in the SOA cluster). For performance and administration-isolation purposes the
dbfs mount point used for replicating the domain across sites is configured and mounted separately from the default
DBFS mount points under /u01/soacs). This guarantees that configuration operations will not interfere with the
system’s runtime behavior (for example, thanks to this separation, it will be possible to unmount the DBFS
configuration directory without affecting the processing of files by the file adapter in a cluster). There will be a
“configuration” DBFS mount point under /u01/soacs/dbfs (configured in the steps bellow) and a “runtime” DBFS
mount point under /u01/soacs/dbfs_directio. Both are automatically created/configured with the SOACS instance.
Check the filesystem in DBFS.
As the oracle user in each one of the primary and each one of the standby middle tier nodes, check the existence of
the mount point and the /u01/soacs/dbfs/share/ directory.
[oracle@soacsdroci-wls-1~]$ mount | grep dbfs
dbfs-@ORCLB:/ on /u01/soacs/dbfs type fuse
(rw,nosuid,nodev,max_read=1048576,default_permissions,user=oracle)
dbfs-@ORCLB:/ on /u01/soacs/dbfs_directio type fuse
(rw,nosuid,nodev,max_read=1048576,default_permissions,user=oracle)
[oracle@soacsdroci-wls-1~]$ ls -lart /u01/soacs/dbfs/share/
total 0
drwxr-x---. 4 oracle oracle 0 Nov 21 18:09 ..
drwxr-x---. 2 oracle oracle 0 Nov 23 15:45 .
Create a test file in primary and check that it is readable in standby. In any of the middle tier primary nodes run
[oracle@soacsdroci-wls-1~]$ echo "test" > /u01/soacs/dbfs/share/test.txt
If the database edition is enabled for Active Data Guard (the Extreme Performance Edition required) the standby
database will automatically be open for reads, so changes applied in the dbfs mounts in primary can immediately be
read in standby. If the Database edition has not enabled Active Data Guard, the database in standby will have to be
converted to snapshot in order to be able to read the file created in primary. If your system is not using Active Data
Guard, convert the standby database to snapshot (when using Active Data Guard, the standby file system should
have access to the file created without any steps):
[oracle@soacsdrDBa ~]$dgmgrl sys/your_sys_password@primary_db_unqname
DGMGRL> CONVERT DATABASE secondary_db_unqname to SNAPSHOT STANDBY;
Converting database " secondary_db_unqname" to a Snapshot Standby database, please
wait...
Database " secondary_db_unqname" converted successfully
Check the file on the secondary’s WebLogic Administration node
25 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
[oracle@soacsdroci-wls-1~]$ cat /u01/soacs/dbfs/share/test.txt
test
Once the basic DBFS file system propagation is verified, you can proceed with the customization of the dbfscopy.sh
script that will synchronize the domain directory from primary to standby. Here are some considerations for using
DBFS to replicate configuration:
It is critical that deployments are pushed to the WebLogic domain directory before doing manual
switchovers (or by using the appropriate cron job frequency). Otherwise, there may be windows between
the last update to the standby domain directory and the role switch that could cause data loss. The data
loss window will be determined by the frequency of the cron jobs in primary and standby.
Any data or files that are created OUTSIDE the domain directory in the WebLogic Administration Server
node, are not taken care of by the dbfscopy.sh script and need to be synchronized separately.
Download, customize and run the dbfscopy.sh script. Optionally enable cron jobs
The domain configuration copy and replacement script is intended to copy (using rsync) the WebLogic domain
configuration to the DBFS stage directory in the primary location and to copy (using rsync) the stage directory
contents to the WebLogic domain directory in the secondary/standby system. It also “maps” (replaces) the
Datasources’ connect strings and Cloud identity domain names in each site. The script is used on both sites so that
either one can assume primary role and replicate configuration to the other. It can be executed manually in a
controlled fashion or automatically at regular intervals using cron jobs. Whether the manual or “croned” approach
applies better to each case depends on the frequency and size of the configuration changes applied. The frequency
recommended will vary depending on each system’s lifecycle (how frequently new deployments and configuration
changes are applied). When the configuration copy is “croned” and the changes do not involve large files, then rsync
and Data Guard will provide a quick propagation that will transfer everything in a small lapse of time from the primary
to the secondary location. However if the rsync copy takes a long time (because some files are very large or
because many files are involved) it could occur that only a part of the configuration changes get “applied” to the
primary DBFS stage directory at the point in time where the standby cron moves things to the secondary domain
location. If the primary node crashed at that point in time, the standby domain config could be invalid/inconsistent.
This scenario can be avoided with different approaches:
1)Using the --delay-updates will make the rsync copy operation more “atomic” (this is achieved by creating a
temporary file for each updated file into a holding directory until the end of the transfer, at which time all the files are
renamed into place in rapid succession)
2)Using rsync to copy to another intermediate folder from which a tar file is created, copying this tar file to the DBFS
location and then extracting it in standby. Both approaches incur in additional space requirement.
The applicability of these options will depend on the type of configuration changes and deployment frequency. For
typical change frequencies (a few new deployments per day) and average composite and ear file sizes (less than
500KB) Oracle has found this rsync options unnecessary. The example dbfscopy.sh script provided, however, can
be extended to use also the tar approach.
Additionally, the script requires access from each middle tier site to the remote Database to retrieve status
information. By default the OCI DB System instances used by SOACS need to reside in in public networks and
access to their 1521 port can be enabled from the external world with the appropriate Internet Gateway. If the
access to the DB was configured through a VPN tunnel or a Dynamic Routing Gateway, the primary and standby
26 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
middle tiers will have to be enabled to access these accordingly. Refer to the SOACS and OCI documentation for
details on the network requirements for SOACS on OCI.
To use the DBFS copy scripts follow these steps:
A. Download/copy the script from OTN to the primary WebLogic Administration Server node. The
script is available here.
B. Open the dbfscopy.sh script, read its instructions. Latest version of this script do not need any
variable customization, it just prompts for the sys password.
C. Execute the script manually first in the primary WebLogic Administration Server node before
creating a cron in primary to execute it at regular intervals. Monitor the execution and watch for
any errors. The script will verify the DG status and will copy the domain configuration from the
primary WebLogic domain to the DBFS mount point.
D. Once it completes, download/copy the dbfscopy.sh script to the secondary WebLogic
Administration Server node.
E. Open the script, read its instructions. Latest version of this script do not need any variable
customization, it just prompts for the sys password.
F. Execute the script manually in the secondary middle tier WebLogic Administration Server node
before creating a cron in standby to execute at regular intervals. Monitor the execution and watch
for any errors. The script will verify the DG status, will copy the domain configuration from the
DBFS mount point to the secondary WebLogic domain and will make the required replacement
in the configuration that are required in the standby.
G. Restart the administration server in the secondary location for the changes to be effective (needs
to have the secondary DB in snapshot standby mode or use Active Data Guard)
IMPORTANT: The configuration under domain_home/config is automatically copied over to all other nodes that are
part of the WebLogic domain when the managed servers are restarted and connect to the Administration Server .
Any other configuration residing out of the domain_home/config directory will be copied ONLY to the first node and
will have to be manually replicated to each of the managed servers nodes. This includes any customizations to start
scripts under domain_home/bin domain_home/security etc.
H. Once this initial execution in primary and secondary is complete, the scripts can be added to the
cron list in the system so that they get executed regularly.
IMPORTANT: Notice that “croning” the copy script automates synchronization but also has the following
implications:
1) Synchronization may incur in latency as high as the frequency of the cron jobs in both locations added up. i.e. if
the cron jobs are set to execute every 30 minutes each, the changes may take 60 minutes to be available if the
window in primary overlaps with the one on the secondary location. Before performing a switchover, make sure that
this amount of time has passed by after the last configuration change. Otherwise, you could switchover before the
change is present on standby and overwrite the changes originally applied with the role switch
27 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
2) The cron frequency should be set at minimum to the largest amount of time a deployment or configuration change
may take to be copied from the domain directory to the dbfs stage directory. Otherwise, copy jobs may overlap
Scaling the SOACSDR system
Scale out operations may not be performed in a SOACSDR service in the current Cloud version.
How to recreate the dbfs wallet
The dbfs wallet, tnsnames.ora and dbfsMount.sh in the <domain_home>/dbfs folder of the midtier hosts are updated during the DR setup14. In case you need to update the wallet because the password of the SchemaPrefix_DBFS user has been changed, you can follow same steps described in Update the DBFS Wallet Password in SOA Cloud Service documentation, but taking into account that the alias used to mount dbfs is the PDB name (instead of default “ORCL” alias). In the commands to generate the alias you should use the PDB name:
$middleware_home/oracle_common/bin/mkstore -wrl /u01/data/domains/domain_name/dbfs/wallet
-create < /var/tmp/dbfsp
$middleware_home/oracle_common/bin/mkstore -wrl /u01/data/domains/domain_name/dbfs/wallet
-createCredential <PDB_NAME> SchemaPrefix_DBFS < /var/tmp/dbfsp
Make sure that the dbfsMount.sh script in <domain_home>/dbfs/ folder is using the <PDB_NAME> alias in the dbfs client mounts commands. Example:
$ORACLE_HOME/bin/dbfs_client -o wallet /@<PDB_NAME> -o direct_io $MOUNT_PATH_DIRECTIO
&>dbfs.log &
$ORACLE_HOME/bin/dbfs_client -o wallet /@<PDB_NAME> $MOUNT_PATH &>dbfs.log &
Make sure also that the alias <PDB_NAME> exists in the <domain_home>/dbfs/tnsnames.ora pointing to the local PDB. In order to mount dbfs, it is recommended that you use the script <domain_home>/dbfs/dbfsMount.sh instead of using dbfs_client commands directly. In case you use dbfs_client commands, make sure you use the correct alias. Note that the wallet recreation after a password change in SchemaPrefix_DBFS user is required to be done both in primary and standby host midtiers, because the folder <domain_home>/dbfs/ is specific to each domain (and it should not be replicated from primary to standby).
NOTE: in order to update the rest of the schema passwords (SOAINFRA, STB, etc) , you can simply follow the
steps described in Change the Database Schema Password Manually in primary domain, and then use dbfscopy.sh
to replicate changes to secondary domain. The password change in the datasources and other files under the
domain configuration will be replicated to secondary.
14 Note that these updates are performed during DR setup in the primary admin node host and in all the secondary midtier hosts.The rests of the
primary midtier hosts keep the original dbfs wallet, tnsnames.ora and dbfsMount.sh in <domain_home>/dbfs folder, because only primary admin node is
used to copy the primary configuration to dbfs mount. Once the DR setup is done, you can homogenize this by copying these files from primary admin
node host to the rests of the primary managed server hosts.
28 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Conclusion
Disaster recovery in a SOA Cloud Service configuration consists of a production database and a standby database
synchronized by Oracle Data Guard. Two separate middle tier configurations, each pointing to their local database,
are created to minimize the file synchronization needs across data centers. With this Disaster Recovery solution,
Oracle Cloud eliminates the costs and complexity of owning and managing a standby hardware, software, and
remote data center – while achieving the best Recovery Time Objective and Recovery Point Objective.
The use of Oracle Data Guard for disaster recovery provides better RTO and RPO than restoring a remote backup;
production is quickly failed over to an already running and synchronized copy of your production database on the
Oracle Cloud. The standby database in the cloud not only provides disaster recovery, it can also be used to seed
clone databases for development and test.
The use of middle tiers with a streamlined configuration replication facilitates maintenance and reduces the
overhead caused by continuous configuration approaches. However, an appropriate methodology and regular
standby verifications are need to guarantee a consistent recovery. Depending on each system’s lifecycle, different
synchronization approaches may be used for optimum behavior.
29 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Appendix A - Setting up Data Guard manually
It is required to configure Data Guard manually when it is not possible to use a single tenancy for primary and
standby or when the automated Data Guard option provided by OCI is not enabled for the DB flavor and/or locations
involved in the DR configuration. The manual process described in this section assumes that the primary’s site
Database is already provisioned. This paper provides tools and examples to create a Data Guard configuration for a
single instance DB.
NOTE: the SOACS DR Setup procedure described in this paper has not been certified with RAC database
Proceed with the following steps.
1. The secondary database needs to be created with specific database name, shape and version to
match primary. To obtain the required data for your OCI Database click on your Database System in
the OICI Console
Note down the following information:
SHAPE: for example VM.Standard2.1
SOFTWARE EDITION: For example Enterprise Edition Extreme Performance
AVAILBALE DATA STORAGE: For example 256GB
DB SYSTEM VERSION: For example 12.2.0.1.180417
DATABASE NAME: For example ORCLTEST
PORT: 1521
Verify also the pdb name that is used in primary. This can be determined with the following SQL query in the
primary DB node
30 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
SQL> SELECT NAME, CON_ID FROM V$CONTAINERS;
NAME CON_ID
-------------------- ----------
CDB$ROOT 1
PDB$SEED 2
PDBTEST 3
2. Provision secondary DB System. For this, change the region in the OCI- Console to the appropriate
location where the standby will reside. Click the launch DB System button to create the secondary
DB. Make sure that the Database Name, Release, Patch level and PDB Names used in the
secondary DB instance are the same ones that were used when creating the primary one. This may
require patching the primary system (especially if it has been running for a long time) before creating
the standby. Oracle recommends using the same Compute Shape and Storage Size that were used
for primary also. Follow the steps in the SOA CS documentation to provision the required Database
for the standby datacenter. The following table provides examples and requirements for the properties
that need to be used in the standby DB System creation process
The following table describes the parameters that should be used in the secondary system’s
provisioning screen using examples:
Cloud Account
Configuration Property
Existing Primary DB
System/example
Secondary Db System being
created/example
Oracle Cloud Tenancy XXXX/paasmaa YYYY/paasmaa2 (can be different or
same from primary. See Appendix B)
Oracle Cloud Account User XXXX/[email protected] YYYY/[email protected] (can be different
or same from primary. See Appendix B)
Oracle Cloud Account Password XXXX/Acme1234# XXXX/ Acme1234# (can be different or
same from primary. See Appendix B)
DB Sytem Configuration Existing Primary Db System
System/example
Secondary DB System System
being created/example
Display Name XXXX/ soacsdrDBa YYYY/soacsdrDBb (can be different from
primary)
Availability Domain XXXX/efEXT:US-ASBURN-AD1 YYYY/ efXT:PHX-AD-3 (systems are
expected to be remote, hence different from
primary but needs to be the same where the
secondary SOACS service will be created)
Shape XXXX/Virtual Machine VM Standard 2.1 XXXX/Virtual Machine VM Standard 2.1
(same as primary)
Total Node Count N/1 N/1 (same as primary)
Oracle Database Edition Enterprise Edition High Performance or
Enterprise Edition Extreme Performance
/Enterprise Edition Extreme
Performance
XXXX/ Enterprise Edition Extreme
Performance (same as primary)
Available Storage Size XXX/256 XXX/256 (same as primary)
SSH Public Key XXXX YYYY (can be different from primary)
License Type LI, BYOL/BYOL LI, BYOL/BYOL(can be different from
primary)
SSH Public Key XXXX YYYY (can be different from primary)
31 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Virtual Cloud Network XXXX/ soacsdrASH YYYY/soacsdrPH (systems are expected to
be remote, hence different from primary)15
Subnet XXXX/Public Subnet efXT:US-ASBURN-
AD1
YYYY/ Public Subnet efXT:PHX-AD3
(systems are expected to be remote, hence
different from primary)
Hostname prefix XXXX/soacsASH YYYY/soacsPH (can be different from
primary)
Database Name XXXX/ORCL XXXX/ORCL(same as primary)
Database Version XXXX/12.2.0.1.180417 XXX/12.2.0.1.180417 (check the “Display
All available versions” and make sure you
select the same one as in primary)
PDB Name XXXX/PDBTEST XXXX/PDBTEST (same as primary)
Database Admin Password XXXX/Acme1234# XXXX/Acme1234#(same as primary)
Database Workload ON-LINE TRANSACTION
PROCESSING
ON-LINE TRANSACTION PROCESSING
Automatic Backup X/Checked X/Unchecked (disable backup in standby)
NOTE: The default database instance created on the secondary site will be deleted later as it cannot be used as a
Data Guard standby database. It is created with the same name as primary to get the required lifecycle scripts
seeded in the system with the same configuration as the primary DB
Once the secondary DB is created make sure to apply the required patches to the DB in both
locations (primary and secondary) so that to both are at the same patch level. More precisely a Data
Guard configuration across different DB domains requires a fix for bug 22611167 in database
12c versions. Verify if the patch for this bug is applied in both the primary and secondary DB
systems by checking the opatch output, and apply it in case it is not. Make sure the pertaining patch
for your precise database version is applied in both the primary and standby systems. Latest OCI
12cR2 DB systems have the patch for this bug pre-installed. This patch is not required in 18c
onwards.
3. Before running the following steps to configure the Data Guard, verify that the appropriate ingress
rules are defined in the OCI console to allow connections between the primary and secondary
databases. It is also needed that each database can reach its own “public” IP16 on the appropriate
listener port.
4. Download/copy the dataguardit_primary.sh script from OTN to the primary Database node. The
script is available here.
5. Open the script as oracle user, read its instructions and customize the variables explained in its initial
section.
15 SQLNet connectivity is required between the primary and standby DB. This connectivity can be achieved using the public internet since the DG traffic
is encrypted. Other options like Dynamic Routing Gateways are possible and recommended as long as the SOACS system using the DB system
supports them. By default the DB System instances used by SOACS need to reside in in public networks and access to their 1521 port can be enabled
from the external world with the appropriate Internet Gateway. If the access to the DB was configured through a VPN tunnel or a Dynamic Routing
Gateway, the primary and standby middle tiers will have to be enabled to access these accordingly. Refer to the SOACS and OCI documentation for
details on the network requirements for SOACS on OCI
16 This IP is the one that will be used for redo transport and will be provided as input in the dataguard configuration scripts. It can be the public one
(when primary and standby databases communicate using an Internet Gateway), or the internal one (when primary and standby databases
communicate using a Dynamic Routing Gateway, this is recommended).
32 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
6. Once customized, execute the script as oracle user. As explained in the script, it’s execution will
create a tar file that needs to be copied from primary to standby. The location where the tar is created
can be customized with a script variable.
7. Copy the output tar file to the standby database node. As the opc user in the primary db node execute
the following:
[oracle@soacsdrDBa ~]$ scp -i cloud_private_sshkey.ppk /tmp/ORCL.GZ opc@standby_DB
System_public_ip:/tmp/
For example
[oracle@soacsdrDBa ~]$ scp -i /u02/install/fca-toshare.ssh.ppk /tmp/ORCL.GZ
[email protected]:/tmp/
8. As opc user in standby, change the rights on the file to make it readable by the oracle user
[opc@soacsdrDB2 ~]$chmod o+r /tmp/ORCL.GZ
9. Download/copy the dataguardit_standby_root.sh script from OTN to the secondary Database node.
The script is available here.
10. Open the script as root user, read its instructions and customize the variables explained in its initial
section.
NOTE: As documented in the Oracle Public Cloud documentation, you can gain root access to a
provisioned cloud instance by connecting to the opc user and then using the sudo command:
[opc@soacsdb ~]$ sudo su -
11. Once customized, execute the script as root user. The script will delete the existing database instance
and will create a new one duplicating the primary. It will also set up de database listener and
configuration required for Oracle Data Guard broker. Monitor de execution of the script and check for
any errors. The script will create a log file (/tmp/dataguardit_date.log ) Check this log for
troubleshooting information. The script can be executed again in case of failure (it will do the cleanup
of any previous failed executions).
12. After the script completes, enter the Data Guard Broker CLI from the primary system to check the
configuration (redo apply may take some time to catch up)
DGMGRL> show configuration verbose
Configuration - ORCL_ORCLB_12:55:19-22-11-18
Protection Mode: MaxPerformance
Members:
orcl - Primary database
orclb - Physical standby database
Properties:
33 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
FastStartFailoverThreshold = '30'
OperationTimeout = '30'
TraceLevel = 'USER'
FastStartFailoverLagLimit = '30'
CommunicationTimeout = '180'
ObserverReconnect = '0'
FastStartFailoverAutoReinstate = 'TRUE'
FastStartFailoverPmyShutdown = 'TRUE'
BystandersFollowRoleChange = 'ALL'
ObserverOverride = 'FALSE'
ExternalDestination1 = ''
ExternalDestination2 = ''
PrimaryLostWriteAction = 'CONTINUE'
ConfigurationWideServiceName = 'ORCL_CFG'
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
13. OPTIONAL: Set TCP Socket Maximum Sizes recommended for DG in primary and secondary
[root@soacsdb ~]$ sysctl -w net.core.rmem_max=10485760
[root@soacsdb ~]$ sysctl -w net.core.wmem_max=10485760
The /etc/sysctl.conf file should also be edited to reflect the changes so that they would be preserved during an
instance reboot.
NOTE: the dataguardit_standby_root.sh script performs a default sizing of the Online Redo Logs The redo logs
created by default in the Database Cloud Service were 1GB in size and these were sufficient for our testing. The
online redo logs should be sized by this formula, but should not be less than 1GB in size:
peak redo rate per minute x 20
Redo rates can be extracted from AWR reports during peak workload periods such as batch processing, quarter or
year end processing. It is very important to use peak workload and not averages (averages can obscure peak redo
rates and lead to provisioning redo logs that are too small).
The MAA best practice for Standby Redo Logs (SRLs) is to create the same number as there are groups of Online
Redo Logs (ORLs) plus 1. On RAC this must be done for each thread. Use the following query to discover how
many redo log groups you have in each thread.
select thread#, count(group#) from v$log group by thread#;
34 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
SRLs should be created the same size as the largest of the ORLs. The group numbers are shared with the ORLs
and so SRLs are created with higher group numbers. To gather the size of the largest ORLs and the current highest
group number execute the following query:
select max(bytes), max(group#) from v$log;
The MAA best practice is that SRLs are not duplexed.
35 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Appendix B – Using a Single Tenancy to create a DR system
As explained in in the Configuration Requirements section in this paper, it is possible to create a SOACS/MFTCS
DR configuration using a single tenancy. The WebLogic domain name and WebLogic server names used in a
SOACS/MFTCS system are based on the SOACS/MFTCS cloud instance name provided during the provisioning
process. This way, if soacsdroci (for example) is used as the SOACS cloud instance service name, the provisioning
tools will create a WebLogic domain named soacsdro_domain, with a cluster named soacsdro_cluster and two
servers named socsdro_server_1 and soacsdro_server_2. To keep these WebLogic names consistent in primary
and standby under a unique tenancy, it is required that the primary cloud service instance uses a name with at least
eight characters (which is the limit used for the WebLogic domain, cluster and server names). That way we can
create a secondary instance name (that is called soacsdrocistandby for example) in that same tenancy and get the
exact same WebLogic artifacts’ names (soacsdro_domain, soacsdro_cluster, soacsdro_server_1 and
soacsdro_server_2). With this, since there are no conflicts at the Cloud Service instance name, a single tenancy can
be used to create primary and standby. In summary, primary and standby SOACS/MFTCS instance will have to
share the first 8 characters in their name (for example soacsdroci/soacsdrocistandby,
soacsdmycompany/socsdrmycompanystandby, mycompanyprimary/mycompanysecondary etc.)
Once a single tenancy has been used use the same steps and procedures defined in the “SOACS/MFTCS DR
Deployment” section in this white paper with the following considerations:
1. When using a single tenancy, the configuration option available in the OCI Console for DB Systems
can be used to configure Data Guard for the primary Database (refer to the Database Cloud Service
Documentation to set up Data Guard) i.e. you do not need to follow the process to configure Data
Guard manually as described in Appendix A. Keep in mind, however, the following considerations
a. SOACS and MFTCS need support such a database “flavor” (Database version, RAC,
deployment platform etc.)
b. The DG configuration option needs to be available in the precise two locations that you want
to use for your SOACS DR system.
c. It will be needed to access the Database Nodes to manage the Data Guard configuration as
described in different set up sections in this white paper (converting the physical standby to
physical snapshot, reverting to physical standby etc.)
d. Even when the Data Guard configuration is created automatically, make sure that the latest
patches are applied to the Db software (Data Guard configuration across different DB
domains requires a fix for bug 22611167. Make sure the pertaining patch for your precise
database version is applied in both the primary and standby system)
2. For the SOACS/MFTCS set up process using the automated Data Guard configuration, follow the
same steps described in the “Set up Details” section in this document. Notice however that the
process to create a database service and configure Data Guard for it are replaced directly with the
automated Data Guard process in the Cloud Console. Refer to the Database Cloud service
documentation for details on this.
36 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
2. Once the Data Guard configuration is ready and the appropriate patches have been applied,
convert the physical standby database to snapshot standby logging to the secondary Database
node just like described in previous sections for the tow tenancies case.
3. Per the above, use the appropriate nine characters (at least) service name when creating the
SOACS secondary service where the 8 first characters are common with the primary service
name.
4. The execution of the DRS tool is required as described in previous sections for the two tenancies
case.
5. The lifecycle procedures are the same whether the DG system was created manually or using
the Cloud Console.
37 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Appendix C – Cloud backups in DB Systems
Backing up the DB System is a key aspect of any Oracle database environment. Oracle Cloud offer various approaches: you can store backups in local or cloud storage; the backup can be automatic, custom using rman, or dbcli. In a DR scenario, there are some special considerations because the databases are configured with Data Guard. When the Data Guard is configured manually (with steps explained in the previous Appendix A), the backup needs to be configured manually in order to get the optimal configuration in a Data Guard environment. You can perform the backups in one of the databases (primary or standby) and control the archivelog growth in the other one. To configure manual backups in the primary DB System:
If the automatic backup was enabled in OCI Console for this system, the backup module should be already configured by the automatic backups. In that case, disable automatic backup so you can customize it. If automatic backup have never been enabled before, you can follow the steps described in Backing Up a Database to Object Storage Using RMAN to install and configure the backup module in the Primary DB.
Configure rman settings as recommended in the link. In addition to that, ensure that you also include the archivelog deletion policy recommended for Data Guards:
RMAN> CONFIGURE ARCHIVELOG DELETION POLICY TO BACKED UP 1 TIMES TO 'SBT_TAPE'
APPLIED ON ALL STANDBY;
Create your rman backup scripts as per your backup requirements and include it in the crontab. This is just an example to run a full backup:
# Run RMAN
export ORACLE_HOME=/u01/app/oracle/product/12.2.0.1/dbhome_1
export ORACLE_SID=ORCL
$ORACLE_HOME/bin/rman <<RMAN
connect target /
SET ENCRYPTION ON;
BACKUP DATABASE PLUS ARCHIVELOG TAG "FULL_BACKUP";
exit;
RMAN
echo "Completed full backup for" $ORACLE_SID
To control the archivelog growth in the standby:
Disable automatic backup if it was enabled for this system, and configure the proper archivelog deletion policy so archivelog are not deleted if they are not yet applied to standby (archivelog deletion policy to “applied on all standby”).
Although setting the correct archivelog deletion policy should be enough to control the archivelog growth in the FRA, you can also create a cleanup script to delete old archive logs. This is an example to clean old archive logs that uses a archivelog deletion policy to prevent undesired archivelog deletion:
######################################################
# Use this script to clean old archive logs from disk
# when the database is in STANDBY role and no backups are performed
# Run RMAN
export ORACLE_HOME=/u01/app/oracle/product/12.2.0/dbhome_1
export ORACLE_SID=ORCL
$ORACLE_HOME/bin/rman <<RMAN
connect target /
38 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
# To prevent undesired archivelog deletion if this DB takes primary role
CONFIGURE ARCHIVELOG DELETION POLICY TO APPLIED ON ALL STANDBY;
# Delete archivelog older than 20 days
delete noprompt archivelog all completed before 'SYSDATE-20';
exit;
RMAN
echo "deleted applied old archivelogs on $ORACLE_SID"
######################################################
When the Data Guard is configured using Cloud Console UI, you can enable automatic backups in the primary database and this is a good approach. The default rman configuration in those cases uses the recommended archivelog deletion policy for the Data Guard scenario. However, you will have to control the archivelog growth in the secondary database as well as explained before.
39 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Appendix D – Using Enterprise Manager Site Guard to manage the SOACS DR switchovers
Oracle Site Guard is a disaster-recovery solution that enables administrators to automate complete site switchover or failover. It orchestrates the coordinated failover of Oracle Fusion Middleware, Oracle Fusion Applications, and Oracle Databases. Oracle Site Guards offers the following benefits:
Fully automate disaster recovery operations and launch them with a single click
Minimizes disaster-recovery time
Reduces human errors
Flexible and customizable
Eliminates the need for special skills
Use a single pane of glass to manage disaster recovery
Assure disaster recovery readiness using on-demand or scheduled disaster recovery drills glass to
manage disaster recovery
Oracle Site Guard can be used to coordinate the SOA Cloud Service Disaster Recovery switchover as described in this whitepaper. Follow the steps in the whitepaper “Using Oracle Site Guard to manage Disaster Recovery for OCI PaaS Systems” to configure Oracle Site Guard for the SOA Cloud Service Disaster Recovery environment and use it to perform the switchovers and failovers.
Appendix E – Summary of networking requirements for DR Setup
Specific network requirements for SOACS DR are listed in the following table:
ACTION SSH SQLNET (1521) HTTPS
DR setup
(with DRS)
From the host running DRS to all
db and midtier hosts, to the IPs
set in yaml config file.
(Normally public IPs, but they
could be set to private ips when
DRS can connect though a
bastion or through internal
subnets to the nodes).
From all secondary midtier hosts to
primary DB private IP (when
primary and secondary regions
communicate via Dynamic Routing
Gateway).
or
From all secondary midtier hosts to
primary DB public IP (when primary
and secondary regions
communicate via Internet).17
From the host running DRS to
the primary frontend IP.
From the host running DRS to
the secondary frontend IP.
Configuration Replication
(dbfscopy.sh)
The user running the script
needs SSH access from his
location to execute the scripts
directly in the SOA admin nodes
From each midtier admin host to
remote DB private IP (when
primary and secondary regions
communicated via Dynamic
Routing Gateway).
or
From each midtier admin host to
remote DB public IP (when primary
and secondary regions
communicated via Internet).17
Normal runtime Between primary and secondary
databases (this is a requirement for
Data Guard).
17 Discouraged in latest versions, since OCI allows Dynamic Routing Gateway for private traffic between VCN networks located in different regions.
2 | SOA CLOUD SERVICES DISASTER RECOVERY ON OCI
Oracle Corporation, World Headquarters Worldwide Inquiries
500 Oracle Parkway Phone: +1.650.506.7000
Redwood Shores, CA 94065, USA Fax: +1.650.506.7200
Copyright © 2017 Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only, and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document, and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark of The Open Group. 0615 White Paper Title: SOA Cloud Service Disaster Recovery - Production and DR in the Cloud June 2020
C O N N E C T W I T H U S
blogs.oracle.com/oracle
facebook.com/oracle
twitter.com/oracle
oracle.com