oracle webcenter portal 12c - nloug.nln_garcÍa.… · performance tools: em overview •oracle...

63
Oracle WebCenter Portal 12c Enhance your Enterprise Portal Performance Daniel Merchán García

Upload: doanthuy

Post on 15-Apr-2018

253 views

Category:

Documents


3 download

TRANSCRIPT

Page 1: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Oracle WebCenter Portal 12cEnhance your Enterprise Portal Performance

Daniel Merchán García

Page 2: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Agenda

• Introduction

• Overview of Performance Issues

• Oracle WebCenter Portal• Performance Checking Tools

• Configuration, tuning and tips.

• BUGs and Patches in 12.2.1.2

• Oracle WebCenter Content• Performance Checking

• Components.

• Configuration Variables

• Oracle WebLogic Application Server• Security Providers

• Data Sources

• Policy and Identity Store

• Java JVM• Memory

• Garbage Collectors

• O.S Entropy

• Contact Details

• Ask / Share your Experience with WebCenter Portal

Page 3: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

IntroductionWhy Oracle WebCenter Portal can be slow?

Page 4: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

History

Oracle WebCenter Portal performance is a common topic in Oracle WebCenter community.

Happily now… We are in 12cR2 synchronized with the latest release of Oracle ADF.

• 12c: The TPS is around 38% much better than 11.1.1.9

• 12c: The ART is around 26% much better than 11.1.1.9

But, many customers still run their platforms on 11gR1!

Reference: http://www.oracle.com/technetwork/middleware/webcenter/portal/learnmore/wcportal-performance-benchmarks1221-2859701.pdf

Page 5: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Why is it slow then?

• Is it Oracle ADF really slow? -> NO.

• Unexperienced developers can create easily bottlenecks by using Oracle ADF.

• 12cR2 performance is much better compared to 11gR1.

• Oracle ADF can be slower than Front-End technologies such as Oracle JET. Do not promise a Front-End technology Average Response Time.

• Is it Oracle WebCenter Portal slow? -> NO.

• Default installation configurations are not suitable for Production Environments.

• Custom Developments -> Non usage of Caching Mechanism or Lazy Loading.

• Non-Patching / Upgrading.• Abusive number of components in a single page.

Page 6: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

OverviewMy system is slow, but where… why?

Page 7: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Where and why!

It is important that you identify what and where is specifically slow before of checking all the components within the Architecture.

For example:

• Is it the Login slow?• Maybe there is a bottleneck with the LDAP or the settings are not tuned properly.

• Is it the Home Page of your Portal slow? • Do not overload the Home Page with million of components!. • Enable the “perfDebug” mode to quick understand which components are loading slow.

• Is it in General everything slow?• If OOTB Portals, Administration Portal and everything in general seems to be very slow then we need to have a look into HW,

OS, Database, Java and other under layer components.

• The documents and data files served from Oracle WebCenter Content are slow• WebCenter Content has not been tuned properly.• Database queries are performing slow due to fragmentation on Database indexes.

Page 8: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Where are the Bottlenecks? The Pyramid Load

Balancer

Oracle HTTP Server

Custom Developments

Oracle WebCenter Portal

Oracle WebCenter Content

WebLogic

Java / JVM

Database

Networking and NFS

O.S

Hardware

Page 9: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

What we will discuss in this presentation

• Performance check, Configuration, Tuning and tips for Oracle WebCenter Portal.

• Performance check, Configuration, Tuning and tips for Oracle WebCenter Content.

• Configuration, Tuning and tips in Oracle WebLogic Application Server.

• Some tuning an tips for the Java JVM

• You will not find biggest infrastructure tweaks here or tutorials! .

Page 10: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Oracle WebCenter PortalPerformance Check, Configuration, Tuning and Tips

Page 11: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: EM Overview• Oracle WebCenter Portal 12c as part of

the Oracle Fusion Middleware is tightly integrated with Oracle Diagnostic Tools and Oracle Enterprise Manager.

• It allows to quickly check the following metrics:

• % of CPU and Memory usage.

• Health of WebLogic Server (STUCK Threads?)

• WebCenter Pages Perfomance.

• Portlet Performance.

• Monitor LDAP server performance.• Monitor Portal Tools & Services.

Page 12: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: EM Metrics I

General Metrics

• Security Metrics: It will shows the LDAP statistics. Cache hits and the Average Response Time in the lookups. This metric is useful to identify if there is any bottleneck against the LDAP server.

• WebCenter Portal Tools & Services: It displays metrics and statistics relative to Portal Services such as Announcements, Discussions, Content Repositories.

Page 13: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: EM Metrics II

General Metrics

• WebLogic Health Check: JVM, Garbage Collector, JDBC, CPU and RAM consumption.

Page 14: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: EM Tips

• Use the Enterprise Manager Console for a quick overview of the Health status of Oracle WebCenter Portal. Do not make it the only tool you will use to check the performance status .

• Once you have identified the areas with issues then focus on them: LDAP, Networking, WebCenterContent….

• I would recommend you to use Oracle WebLogic Administration Console for a quick Health check for Threads Status. In this console is easier to generate and check a Thread Dump in case of existing STUCK Threads.

• Help your JVM Performance analysis JConsole or Java Mission Control if necessary.

• Page Performance Metric is a little bit tricky. The default Page Response Threshold is 10 seconds. Personally, I consider this value is too high. How many seconds is your Page Response Threshold acceptance?. So, it means that maybe some indicators can be shown as “Green”, but for the Customer is “Red” .

Page 15: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: WCP Performance Pack I

• Oracle WebCenter Performance Pack is a new performance agent / tool that can be installed and enabled in an Oracle WebCenter Portal Platform. The information looks similar to Java Mission Control as it focus is to identify bottlenecks on Code Execution, Threads etc…

Page 16: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: WCP Performance Pack II

• The best tools offered by WCP Performance Pack are:• Event List: Track a Request and check all the method calls latency.

• Method Statistics: To identify which methods are being executed extremely slowly.

Page 17: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: WCP Performance Pack Tips• Use WCP Performance Pack for

identify Methods that are executed very slow. It can give a more specific vision where is the bottleneck.

• Run it ONLY when checking Performance. The Agent should be STOPPED in Production!

• WCP Performance Pack does not replace Java Mission Control (The legacy JRockit Mission Control). Java Mission Control is a very powerful tool for checking JVM issues . However, WCP Performance Pack will provide statistics and relevant information pre-configured.

Page 18: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: PerfDebug Mode

• WebCenter Portal can be configured to display the Response Time of the components embedded in the Page.

• Enable it by changing webcenter-config.xml file (property <webcenter:perfdebug-enabled>true</webcenter:perfdebug-enabled>)

Page 19: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Performance Tools: PerfDebug Mode Tips

• Do not forget to switch off the perfDebug mode for Production / Live environments.

• The time displayed is how long takes to execute and display the first activity of each ADF Component. If you have implemented any Lazy Loading strategy then perfDebug will display only good latency responses .

• Use it to identify what are the components which are consuming excessive time to execute.• Out-of-the-Box components: Which services consume? Any clue why are they so slow?.

• Custom Components: Bad design which ends in bad performance.

Note: Oracle ADF Bounded Task Flows are executed sequentially and synchronously. If one of the Task Flows has a very bad performance then it will affect the rendering of the Page.

Page 20: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WCP: Oracle ADF Tuning Tips I

• Always try to follow the Oracle ADF Best Practices for developing. For example:• Tune your ADF BC Application Modules to avoid excessive Passivation / Activation.

• Tune your View Objects accordingly to the rows Fetch Size expected in the UI.

• [extend.spaces.webapp.war]. Do not forget to switch OFF what you have turn ON in your web.xml during the development stage.

Parameter Description Default value

org.apache.myfaces.trinidad.COMPRESS_VIEW_STATE Controls whether or not the page state is compressed

True*. In 11g there was a bug that keep this parameter in False

org.apache.myfaces.trinidad.resource.DEBUG To disable client for caching resources

False

oracle.adf.view.rich.CHECK_FILE_MODIFICATION Controls whether ADF faces check for modification date of JSP pages and discard any saved state if the file is changed

False

org.apache.myfaces.trinidad.DISABLE_CONTENT_COMPRESSION

Disables the Skin CSS compression for helping development of Skins

False

org.apache.myfaces.trinidad.DEBUG_JAVASCRIPT Debug JavaScript False

Page 21: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WCP: Oracle ADF Tuning Tips II

• Pages with multiple Task Flows can delay the Page rendering. Follow one of the following tips to reduce the Page loading time:

• Lazy Loading: An Oracle WebCenter Portal Page is rendered when all the ADF Bounded Task Flows in the Page has executed. Use an ADF Task Flow Template for adding a “Fake” initial activity which will trigger a navigation to the first view activity of the Task Flow.

• Panel Tabbed: If you have ADF Bounded Task Flows in separate tabs within a Panel Tabbed. Add an activation condition to each Task Flow. It will only execute the Task Flow that is rendered currently (instead of executing all).

• Re-design: Consider to re-design your navigation and pages. It is better to have a Navigation / Buttons to access information quick than wait multiple seconds to display everything in a single page.

• Try to do not place more than 8 ADF Task Flows in a single Oracle WebCenter Portal Page.

• Consider the possibility of developing also ADF Portlets instead of ADF Bounded Task Flows. They are executed in a separate JVM.

Page 22: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Portlets and MDS Cache Size Tips

• Remember that for JSR-286 Portlets there are two mechanisms for caching:• Expiry-based caching: This option is more suitable for Portlets

displaying static data.• Validation-based caching: This option is more suitable for dynamic

data. It is responsibility of the developer to check if the view must be re-generated.

• In Oracle WebCenter Portal 12c, JSR-286 Cached Responses are stored in Coherence. This configuration can be override if needed .

• In earlier versions of Oracle WebCenter Portal tuning the MDS was a mandatory task.Since 11.1.1.8 version of Oracle WebCenter Portal MDS is tuned with the following settings:• auto-purge is enabled by default with a TTL of 3600 seconds.• Cache Size: 100Mb

Tunning Tips: • Increase the MDS Cache Size to a higher value if you are

planning to have multiple Portals with multiple pages.• Do not discard the usage of Portlets . I find always the

WC_Portlet JVM lonely wasting resources!

Page 23: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WebCenter Portal – Content Overview

• In Oracle WebCenter Portal 12c it has been simplified the integration between Portal – Content.

• Now it as been reduced to two main pillars.

Content Manager Task Flow Content Presenter

Page 24: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WCP – WCC: Common Bottlenecks• Forget to tune Oracle WebCenter Portal – Oracle WebCenter

Content connection. (Focused in this section)

• Not using Site Studio / Content Presenter Integration properly:

• Bad design and development of Content Presenter Templates.

• Do not enabling Coherence for Caching.• Do not enabling Sweeper Mode ON.• Do not using Image Renditions.• Do not use Browser Caching for the Images served in

Content Presenter.

• Oracle WebCenter Content default settings are not suitable to production environment. Default settings easily can produce (Later in the WebCenter Content chapter):

• An insane amount of SQL Statements producing a huge bottleneck against the Database or just killint it .

• An overhead against LDAP consulting user information.

• Enabling WebCenter Content Components that are not in use and execute heavy filters on behind . (Later in the WebCenter Content chapter)

• Cluster of Oracle WebCenter Content using a very poor NFS (Network File System) .

Page 25: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WCP – WCC: Connection Settings• Configure Cache Details:

• Cache Invalidation Interval is used by WebCenter Portal for polling and checking if any of the Cached Content has changed to invalidate it.

• Maximum Cached Document in Bytes: The maximum cacheable size for the binary documents. Larger documents are not cached.

Tuning Tips: A Cache Invalidation interval of 2 to 5 minutes is more than enough. Adjust the maximum binary size depending on the average size of Data Files, Binary Documents that you will serve accordingly to the JVM memory assigned .

Page 26: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Content Presenter: How it works

• Content Presenter is the unique mechanism for exposing structured Web Content Data.

• Uses PortalVCR Component which exposes VCR_... IDC Services for consuming the information from the Content Repository. .

• In a Un-cached / Non-tuned first request the following calls happens:

Content Presenter

WebCenter Content (Portal VCR)

Get list of available Content Types: VCR_GET_CONTENT_TYPES1

Calculate Content Type: VCR_GET_CONTENT_TYPE2

Calculate Content Type: VCR_GET_CONTENT_TYPEi

i+1 Data File Request: VCR_GET_DOCUMENT_BY_NAME

i+2 Data File Request: VCR_GET_DOCUMENT_BY_NAME

Page 27: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Content Presenter: Enable Coherence

• Always enable Coherence for Content Presenter .

• Default Values of the sample-content-coherence-cache-config.xml are not suitable for a Production Environment.

Tunning Tips:

• ContentBinaryCaches Coherence Cache Schema: • Only stores the binary data with less size than the configured in WCP-WCC connection will be cached.

• Define how much binary data components we will store. Consumed Memory = [Size of Binary in WCC Connection] * [Number of Elements defined in Coherence Config

• IT DOES NOT STORE SITE STUDIO DATA FILES!!!

• ContentNodeCaches Coherence Cache Schema: • Setup this cache schema to use an expiry-delay of 0. It will enable the mechanism of Sweeper ON for Data Files.

• Setup the number of nodes to an 75-80% of the number of Data Files you expect to have.

• IT AVOID UNNECESSARY CALLS TO VCR_GET_DOCUMENT_BY_NAME service

• ContentTypesCaches: Stores the Content Types (Profiles + Region Definition Content Types). Consider to increase the expiry-delay or make it un-expirable.

• IT AVOID UNNECESSARY CALLS TO VCR_GET_CONTENT_TYPES / VCR_GET_CONTENT_TYPE calls.

• If your license of Coherence accepts it… DISTRIBUTE this cache over your cluster .

Page 28: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Content Presenter: Content Types

Tuning Tips:

• VCR_GET_CONTENT_TYPES + VCR_GET_CONTENT_TYPE are execute the following number of calls:

Number of VCR calls = 1 + [Number of Profiles] + [Number of Region Definition]

For example: 35 Profiles + 10 Region Definition = 45 VCR_GET_CONTENT_TYPE calls!!!

What can we do? Cry is not an option!!! . Let’s be strong and improve it!

• Enable Content Type Lazy Loading by adding to WC_Portal managed servers the following JVM Parameter.

-DContentTypeLazyCaching=true

• Really consider to setup the ContentTypesCaches to do not expire Content Types or put a longer expiry-delay.

Warning:

In 12c we have been identified a BUG which makes VCR calls to take crazy times (VCR_GET_CONTENT_TYPES can be > 30 seconds)!. In BUGs section you will find the Patch associated .

Page 29: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Content Presenter: Image Rendering I

• Image Rendering in Content Presenter is one of the most common bottlenecks… Avoid using Native Rendition of huge sizes and re-sizing its height and width with CSS.

• Always configure Inbound Refinery to produce Image Renditions of different sizes / formats to be used in your Custom Content Presenter Templates.

• Use the Thumbnail rendition for small renderings.

• Use the Web rendition for bigger renderings.

• Create your own renditions to full fill your Template needs different sizes / formats.

• Use EL Expressions to access to the specific rendition:• Image Node:

#{node.renditionsMap[‘nameOfRendition:url']

• Image Element Definition within a Region Definition:

#{node.propertyMap['regionDefinitionName:elementName/Rendition:renditionName'].asTextHtml}

Tip:• Avoid to use URLs using the /wccproxy servlet for rendering images. It download the Native File!.

Page 30: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Content Presenter: Image Rendering II

• Out-of-the-box the URLs generated for the images are an IDC Service called GET_FILE.Images served by GET_FILE cannot be cached by the browser!.

For example: /cs/idcplg?IdcService=GET_FILE&dDocName=…&rendition….

• In 11.1.1.5.4 was introduced a configuration file called linkCaching.properties in WebCenter Portal for supress GET_FILE and turn the URLs into WebLayout URLs that can be cacheable.

• oracle.webcenter.content.integration.spi.ucm.linkConversion=true

• oracle.webcenter.content.integration.spi.ucm.suppressGetFileUrl=on

For example: /cs/groups/public/documents/document/mtu2/otm1/~edisp/ucm_00001.jpg

• However since 11.1.1.8.8 all “/cs” responses contains a Cache Control: no-cache, no-store header due to a security issue with SSL caching.

Page 31: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Content Presenter: Image Rendering III

Tip:

Override certain WebLayout path responses in OHS to make the Cache-Control / Pragma privateUse a specific security group only for images to differentiate the WebLayout root URL.

<Location /cs/groups/myimages/>Header set Pragma privateHeader unset Last-ModifiedHeader unset ExpiresHeader set Cache-Control "max-age=2592000, public“…. [ Deflate Options] ….

</Location>

Page 32: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WebCenter Portal + OHS DEFLATE

Configure the DEFLATE module of Oracle HTTP Server to compress the size of the HTTP responses.

<IfModule mod_deflate.c>SetOutputFilter DEFLATEAddOutputFilterByType DEFLATE text/plainAddOutputFilterByType DEFLATE text/xmlAddOutputFilterByType DEFLATE application/xhtml+xmlAddOutputFilterByType DEFLATE text/cssAddOutputFilterByType DEFLATE application/xmlAddOutputFilterByType DEFLATE image/svg+xmlAddOutputFilterByType DEFLATE application/rss+xmlAddOutputFilterByType DEFLATE application/atom+xmlAddOutputFilterByType DEFLATE application/x-javascriptAddOutputFilterByType DEFLATE text/htmlSetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-gzip dont-varySetEnvIfNoCase Request_URI \.(?:exe|t?gz|zip|bz2|sit|rar)$ no-gzip dont-varySetEnvIfNoCase Request_URI \.(?:pdf|doc?x|ppt?x|xls?x)$ no-gzip dont-varySetEnvIfNoCase Request_URI \.avi$ no-gzip dont-varySetEnvIfNoCase Request_URI \.mov$ no-gzip dont-varySetEnvIfNoCase Request_URI \.mp3$ no-gzip dont-varySetEnvIfNoCase Request_URI \.mp4$ no-gzip dont-vary</IfModule>

Page 33: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

CM Task Flow: Enhance Inline Preview I• Content Manager Task Flow is the

new component that has merged all the legacy 11g Task Flows for interacting with documentation.

• Content Manager Task Flow is just a wrapper of the known and existing Oracle WebCenter Content ADF UI.

• The more common issue reported with Content Manager Task Flow is: Slow Inline Preview of Docuemnts.

Page 34: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

CM Task Flow: Enhance Inline Preview II• When a document is uploaded it is

converted to PDF if Inbound Refinery is setup.

• When the document is requested. There is a /imaging Servlet which will end in a call to DYNAMIC CONVERTER for generating the HTML rendition for the preview.

• So it generates and cache the HTML rendition on demand!.

WebCenter Portal

Upload Document or Media File

WebCenter Content

Upload Inbound Refinery

Converted

Convert

Convert to different Formats

Renditions

Document ConvertedReleased

OIT (Oracle Outside In

Technology)

Tra

nsf

ormDisplay Content

HT

ML

Re

nd

itio

ns

HTML Rendition Generated, if Dynamic Converter Enabled

Page 35: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

CM Task Flow: Enhance Inline Preview III

• Dynamic Converter can be configured for producing the HTML rendition when the document is Checked-In / indexed.

• It also offers other options such as Re-evaluation during re-indexing or if a change on a metadata should fire a conversion as well.

Tip:

• Configure Dynamic Converter to produce the HTML renditions of the desired document formats when Check-In.

• Dynamic Converter / OIT takes time to produce the conversions… The client must be patient!

• Check – Uncheck Re-evaluate conversions if you are planning for a full Re-index build. It can delay your indexing process.

Page 36: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Recommended Patches:

Relevant Patches that affects Performance in Oracle WebCenter Portal.

• Bug 25528107 - Slow performance from VCR_GET_CONTENT_TYPES when IPM components are enabled.

• Bug 24743966 : stuck thread at org.eclipse.persistence.indirection.IndirectMap.get

Tip:

Oracle My Support has a specialized page for Oracle WebCenter Portal for highlighting the most important patches which affects Oracle WebCenter Portal. There are important patches for Oracle WebLogic and Oracle ADF which affects Oracle WebCenter Portal Peformance.

Page 37: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Oracle WebCenter ContentConfiguration, Components and tips

Page 38: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Overview

• Default configuration of Oracle WebCenter Content is not optimized for production environment.

• There are many things to take into consideration when tuning an Oracle WebCenter Content platform.• Components enabled / disabled. Some components are known to degrade the performance.

Are you really using them? Are them mandatory because of Oracle Documentation?Did you challenge Oracle Documentation to see if they are really mandatory?

• Miss configuration of Components in config.cfg which affects the performance of certain components.This misconfiguration can lead easily in:

• LDAP Bottlenecks

• Database Bottlenecks

• Slow Searching.

• WebCenter Content Indexes must be optimized periodically.

• WebCenter Content in Cluster should run in an optimize NFS.

Page 39: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

How to check bottleneck: Core Trace levels

• Oracle WebCenter Content traces gives a very good traceability of the Components and actions happening on behind.

• Tip: Externalize your intradoc logs to write in the local storage of the machine (not in the NFS ).

• The best common traces to enable for checking bottlenecks are

Trace Description What can you find

requestaudit Time expend on execution of WebCenter Content Services It can give an idea if certain services are executing slow

systemdatabase Database SQL operations and execution times fired by WebCenterContent services

It can give an idea if there is any slow SQL Query or bottleneck in the database

fileaccess To track which files are being locked and used It can give an idea about problems with the File System

services Traceability of filters executed by each WebCenter Content Service It can give an idea of all the filters call by a specific service. May it can execute excessive due to multiple Components has been activated in the Content Repository

searchcache If the Search Cache is enabled. Track what is served from the Cache

*We will talk about UseSearchCache separately

userstorage, jps Operations against WebLogic Server to retrieve user / security information

It can give an idea if there is an overhead or bottleneck with the LDAP

Page 40: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Bottleneck: Database

• We have identified slow SQL Queries or bad performance against the database using systemdatabase trace.What now?

• Ask your DBA to produce an AWR report to confirm that there are slow queries.

• Use SQL Developer for executing “Explain Plan” to analyse in details the slow queris.

• Check your WeCenter Content index fragmentation by calling ctx_report.index_stats PL/SQL function

• Optimize and defragment your WebCenter Content Indexes by calling: ctx_ddl.optimize_index PL/SQL function

• Tips: • Mark custom metadata as “Searchable” to be used as index. It will improve certain search operations

• Configure a PL/SQL job to optimize indexes each week during out of business hours.

Page 41: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Bottleneck: NFS and Bonnie++• Database responses OK… suspicious… NFS issue?!.

• fileaccess traces shows issues against the UCM instance folder where the WebLayout is located.

What to do in this case?

• Perform a proper NFS testing using powerful tools such as Bonnie++.Bonnie++ will perform a synchronous / asynchronous and all kind of write / read possibilities against your NFS producing a very good report.

• If your O.S accepts it try to mount it using versions of NFSv4 etc…

DD Linux Command for checking Read / Write Speed

Bonnie++ doing a complete NFS testing

vs

Tips:• Perform your testing by using different mount options• Reduce the rsize and wsize if your bandwidth is not too

fast.

Page 42: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Components

The following components are suitable for creating performance problems. Take care with them!.

Component Description Warning

AutoSuggest* As Content Manager Task Flow is a wrapper of WCC ADF UI.It provides type ahead suggestion in the CM Task Flow UI.

The AutoSuggest indexing produces heavy load on the DB and also impacts on the Automatic Update Cycle.

LinkManager The Link Manager component evaluates, filters, and parses the URL links of indexed content items before extracting them for storage in a database table. Documentation says that is mandatory for enabling Site Studio Integration.

You can disable it if you face indexing performance issues. It will not affect on how Content Presenter works (as far as I know!).

ContentTracker Provides statistics and reports about system / content usage Disable it if you are not planning to use it .

*: AutoSuggest has more slides! .

Page 43: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Config Variables: AutoSuggest IVariable Description Default

ValueSuggestion

AutoSuggesterPerGramTermsProcessedLimit AutoSuggesterPerGramTermsProcessedLimit can be used to set a max limit on the number of terms a gram can process to avoid scenarios which result in processing terms for the entire index when no matches are found for a query

2000 Around 500 should not affect the quality of the results

AutoSuggestIndexInterval This sets the indexing interval in seconds for auto suggest index which adds / removes entries from the index using the data sources setup. By default it is set to 60.(seconds)Modifying this setting will delay the indexing of information found in the type ahead fields

60 (1 minute) 1 minute will destroy your DB REDO log in just seconds.Consider to increase it

AutoSuggesterThreadCount Number of threads used for getting suggestions 2 Increase if you have enough CPU.

AutoSuggestIndexShortcuts Indicates if Shortcuts needs to be indexed 1 (true) No relevant for performance

AutoSuggestMinGramLength Minimum size of the gram length to index 2 Consider to setup at the same as the Maximum (3) to manage the size of the CacheStore table.

AutoSuggestMaxGramLength Maximum size of the gram length to index 3

AutoSuggestProximityCutOff Filter out the terms based on their proximity score 0.1 No relevant for performance

EnableAutoSuggest It’s the way to turn ON / OFF to index also in suggested index

1 (enabled) * (next slide an amazing recommendation)

Page 44: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Config Variables: AutoSuggest II

AutoSuggest component really generates an overhead traffic to Database and can fill your REDOLog

Use the following recommendations to setup it properly

• Consider to increase the AutoSuggestIndexInterval to bigger values.

• Tune AutoSuggestMinGramLength, AutoSuggestMaxGramLength as suggested to reduce the CacheStore Table size

Tips:

• Tunning AutoSuggest component requires multiple rounds to be tuned properly .

• Enable autosuggest, idccache and systemdatabase trace level to check the AutoSuggest activity.

• Disable AutoSuggest if you are not planning to use it

Page 45: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Config Variables: SearchVariable Description Default

ValueSuggestion

OracleTextDisableSearchSnippet WebCenter Content can retrieve document snippets as part of search results to show the occurrence of search terms in context of their usage

false If you are not planning to search over terms that can be inside of the native document then you can disable it. It will boost the Search Performance.

MaxSearchConnections Maximum simultaneous Searches hold by Oracle WebCenter Content

5 Consider to increase if you have enough resources

MaxResults If a Search does not specify how many content to retrieve then MaxResults count is retrieved

200 Contingency about the amount of data gathered.Setup to a proper value where the customer is satisfied with.E.g.: If you make an empty search, it will return the latest 200 updated content.

CachedResultRowCount Size of the search cache 40000 No need to tune

SearchCacheCleanerInterval TTL between cache cleanup attempts 120 (seconds) No need to tune

SearchCacheMaxItemAgeInMinutes Set the maximum age of an item in cache 240 (minutes) 4 hours… -> Reduce them to 10 minutes maybe?

UseSearchCache* Enable / Disable WebCenter Content Search Cache What to suggest here?* Depending on the WCP release version this is BUGGED or NOT. Content Presenter CMIS Queries returns inconsistent data if this cache is enabled.

*Discuss about UseSearchCache with the amazing Audience

Page 46: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Config Variables: LDAP and User InformationVariable Description Default

ValueSuggestion

UserCacheTimeout For how long user information will be cached in Oracle WebCenter Content.

60000 (milliseonds)

Only holds for 1 minute the user information. Increase this value if you are not expecting to update user information quite often.3600000 (1 hour) is a nice value

DoNotQueryLDAPForEmail Avoid overhead of LDAPs Lookups in case of the user email in case of it is not present in the User Cache or User table.

False True in case of no need to refresh or get Email address from LDAP.In case of False, make sure your LDAP Provider Query is optimised and users has an email address associated.

Page 47: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Config Variables: Core + RenditionsVariable Description Default

ValueSuggestion

DisableSharedCacheChecking Disable the poller of Oracle WebCenter Content for checking updates on its componets.

True From 12c it is disable by default which is the recommended value for production environments.

RenditionListExportedForStaticAccess By default, renditions are compressed in zip and stored in the WebLayout. To avoid unzip operations when accessing to the different image renditions by the applications then setup this property

Preview|Native|Thumbnail|Web

Add to the list all the renditions used by WebCenter Portal Content Presenter

Page 48: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

WebLogicSecurity Providers, Data Sources, Policy Store

Page 49: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Overview

There are three important pillars to tune in WebLogic Application Level:

• Security Providers: The LDAP connections must be setup properly to avoid a bottlenecks and slow LDAP Queries.

• JDBC Data Sources: The default configuration maybe is not suitable for the amount of concurrent users of the platform. In addition, they can be tuned .

• Policy and Identity Store: • Policy Store: Luckily in 12c the Policy Store is Out-of-the-box in Database. In addition, multiple of the most common tunings are

already applied in this version.

• Identity Store: In case of the Identity Store is an Active Directory or OVD or other LDAPs then it can be configure some tuning tweaks in the jps-config.xml file

Page 50: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Security Providers: Connection PoolSetting Description Default

ValueRecommended Value

Connection Pool Size The LDAP connection pool size 6 Adjust to the maximum connections expected. 6 is a very low value

Timeout The maximum time in seconds to wait for the connection to the LDAP server to be established

0 60 (seconds)

Connection Retry Limit Number of re-tries 1 1

Results Time Limit The maximum number of milliseconds for the LDAP server to wait for results before timing out

0 1000 (milliseconds)

Keep Alive Enable Specifies whether to prevent LDAP connections from timing out

False True

Page 51: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Security Providers: User CacheSetting Description Default

ValueRecommended Value

Cache Enable Specifies whether a cache is used with the LDAP server

True True

Cache Size The size of the cache (in kilobytes) that is used with the LDAP server

32 3200

Cache TTL The time-to-live of the cache (in seconds) that is used with the LDAP serve

60 21600

Page 52: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Security Providers: Groups CacheSetting Description Default

ValueRecommended Value

Enable Group Membership Lookup Hierarchy Caching

Boolean value that indicates whether group membership hierarchies found during recursive membership lookup will be cached

True True

Max Group Hierarchies in Cache

The maximum size of the LRU cache for holding group membership hierarchies if caching is enabled

100 1024

Group Hierarchy Cache TTL The time-to-live of the cache (in seconds) that is used with the LDAP serve

60 21600

Page 53: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Security Providers: LDAP LookupSetting Description Recommended

Value

User Base DN The base distinguished name (DN) of the tree in the LDAP directory that contains users

Understand properly your LDAP tree. Maybe you can avoid using root if your users are not under it .

User from Name Filter If the attribute (user object class) is not specified (that is, if the attribute is null or empty), a default search filter is created based on the user schema

People usually use LDAP, E.g. in AD such as (&(sAMAccountName=%u)(objectclass=user))These queries can improved in case of searching over specific folders

Group Membership Searching

Specifies whether group searches into nested groups are unlimited, limited or off

Limited

Max Group Membership Searching

Specifies how many levels of group membership can be searched

Depends on number of nested groups

Page 54: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Policy Store: Tuning Guide

• In 12c version all Policy Store common JVM tuning parameters (such as rolememberwarmup) are now setup by default.

• Anyway double check the following tuning recommendations for Policy Store (PDP settings) which can be made through Enterprise Manager Console or editing jps-config.xml file

Setting Description Recommended Value

oracle.security.jps.policystore.rolemember.cache.warmup.enable

This property controls the way the ApplicationRolemembership cache is created. If set to True, the cache is created at server startup; otherwise, it is created on demand (lazy loading).

By default True in 12c.If you face slow login performance double check that it is Enabled by default

oracle.security.jps.policystore.policy.cache.size

The size of the permission cache. If you cache all policies, then you can set this value to the total number of grants.

The Oracle recommendation is to 5 times of expected number of Portals. The default values is 1000which is high enugh

Tip:Tuning JVM Parameters related to Policy Store are already configured in 12c. Have a look into the Managed Server JVM Startup Parameters

Page 55: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Identity Store: Tuning Guide

Depending on the LDAP configured as Identity Store. There are some tuning that can be added in the jps-config.xml depending on the LDAP configured.

For example for Active Directory:

<serviceInstance provider="idstore.ldap.provider" name="idstore.ldap">

<property value="oracle.security.jps.wls.internal.idstore.WlsLdapIdStoreConfigProvider" name="idstore.config.provider"/>

<property value="oracle.security.idm.providers.stdldap.JNDIPool" name="CONNECTION_POOL_CLASS"/>

<property name="PROPERTY_ATTRIBUTE_MAPPING" value="WIRELESS_ACCT_NUMBER=mobile:MIDDLE_NAME=middlename:MAIDEN_NAME=sn:DATE_OF_HIRE=pwdLastSet:NAME_SUFFIX=generationqualifier:DATE_OF_BIRTH=pwdLastSet:DEFAULT_GROUP=primaryGroupID" />

<property value="sAMAccountName" name="username.attr"/>

<property value="sAMAccountName" name="user.login.attr"/>

</serviceInstance>

Tip:Check Oracle Documentation to find your specific LDAP – Identity Store tuning

Page 56: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

JDBC Data Source Tuning Guide

Oracle WebCenter Portal Tuning guide recommends:

• Increase the Initial Capacity of the pool size at least till 10. it will maintain always a pool of 10 open without losing time on opening the first connections.

• Increase the Max Capacity of the pool to 50 or more. Depending on the number of concurrent users.

• Shrink value to 0 can increase the performance in production. By disabling Shrink frequency you will not allow waits before shrinking a connection pool that has incrementally increased to meet demand.

Out of Oracle WebCenter Portal recommendations also take a look into the following properties:

• Pinned-to-Thread vs Shrink: Pinned-to-Thread is another option (with lot of care and WARNINGs) for increasing the performance. However is not supported for Oracle RAC.Note: Shrink setting is ignored if Pinned-to-Thread is checked.

• The Statement Cache Size attribute determines the total number of prepared and callable statements to cache for each connection in each instance of the data source. Initially is set to 10. This consume lot of memory, be careful and check Oracle Documentation to properly setup a size .

Page 57: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Java and JVMEntropy, JVM Memory and Garbage Collector

Page 58: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

About Java JVM: Overview

• Tuning the JVM always has been a very HOT topic of discussion. • How big has to be the Heap Size?

• Setting up the Minimum = Maximum Heap Size is a performance boost?

• Which Garbage Collector should I use?

• What is the Entropy in Linux? Why it affects to Java?

• Adjusting the proper JVM parameters is an exercise of setting up, load testing, check metrics with jvisualvm, JConsole, Java Mission Control or your favourite diagnostic tool.

Page 59: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

About Java JVM: Entropy

• Being a real Full Stack Developer gives you the power to understand that sometimes the performance can be degrade by external factors to Java.

• What is Entropy in Linux? Entropy is the measure of the random numbers available from /dev/urandom, and if you run out, you can’t make SSL connections.

• How can I know that I am running a low Entropy?• Is your WebLogic Servers starting very slow ?.

• By executing cat /proc/sys/kernel/random/entropy_avail

• If you have an output like 100-200 you are in trouble!

• Install rng-tools or use other tuning mechanisms to improve the Entropy of your Linux! .

More information: http://danielmerchanoracle.blogspot.nl/2015/11/wcp-1221-installation-ii-oracle-jdk-8.html

Page 60: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

About Java JVM: Memory

• Java 7: Perm = Memory in the JVM• Assign the –Xms and –Xmx memory accordingly to the real consumption of memory of your application

• Assign enough space for –XX:PermSize and –XX:MaxPermSize to do not find the lovely OutOfMemory

• A typical recommendation by Oracle was to setup –Xms and –Xmx to the same value.“In production environments, set the minimum heap size and the maximum heap size to the same value to prevent wasting VM resources used to constantly grow and shrink the heap”

• Java 8: Good bye PermSize (OutOfMemory). Hello Metaspace and hello OutofRAM

• Assign the –Xms and –Xmx memory accordingly to the real consumption of your application.

• Always limit the Metaspace by using –XX:MaxMetaspaceSize

• There are discussions if still makes sense to setup –Xms and –Xmx to the same value.

• Java 9: Experimenting with new possibilities like the memory will be adjusted automatically depending on the memory and resource available .

Tip:• Personal tip. Since Java 8 I use to setup –Xmx to x2 of the –Xms value

Page 61: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

About Java JVM: Garbage Collector

Serial vs Paralell vs CMS vs G1

• Java 7 and Java 8 uses ParallelGC by default.

• Depending on the nature of the Application, the Heap Size and CPU resources choose for one of the Garbage Collectors.

• Check Garbage Collector Pause Times by activating the GC log file –XX:loggc=/tmp/gc.log

• If you find multiple Full GC taking long time probably your will have performance issues .

• G1GC has been introduced in latest versions of Java 7 in experimental mode.

• G1GC works very well as it focus on limiting GC pauses.

Page 62: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

Questions?

Share your experience

Page 63: Oracle WebCenter Portal 12c - nloug.nln_garcÍa.… · Performance Tools: EM Overview •Oracle WebCenter Portal 12c as part of the Oracle Fusion Middleware is tightly integrated

ContactDaniel Merchán GarcíaFounder / CEO / Architect / Developer / DreamerMagic Pigeon Ltd

Website: https://magicpigeon.com

Blog: http://danielmerchanoracle.blogspot.co.uk

Email: [email protected]

Twitter: https://twitter.com/dmerchang

LinkedIn: https://www.linkedin.com/in/danielmerchangarcia/

Co-Author of Beginning Oracle WebCenter Portal 12c Book