content transfer timeout configurations for wdk - dell emc · content transfer timeout...

EMC CONFIDENTIAL (AVAILABLE TO EMPLOYEES AND PARTNERS ONLY)

EMC IIG

Abstract

This whitepaper is intended to provide information about the configuration of timeout values, which are used to release content transfer-related hung and stuck threads on the application server.

December 2011

CONTENT TRANSFER TIMEOUT CONFIGURATIONS FOR WDK-BASED APPLICATIONS Troubleshooting Guide

Content Transfer Timeout Configurations for WDK Based Applications Troubleshooting Guide

2

Copyright © 2011 EMC Corporation. All rights reserved.

October, 2011

EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice.

The information in this publication is provided “as is”. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

VMware is a registered trademark of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners.

Content Transfer Timeout Configurations for WDK-based Applications

Troubleshooting Guide

Part Number H9527



3

Contents

Chapter 1 Configuration of request input timeout values for DFC and UCF events ..................................................................................... 7

Problem .......................................................................................................... 7

Sample snippet of stuck or hung thread that is waiting for DFC Event from the thread dumps ................................................................................................................... 8

Sample snippet of a thread that is waiting for a UCF Event when generating a thread dump ..................................................................................................................... 9

Stuck or hung thread information in the weblogic app server log file .............. 9

Resolution – Configuring the time out values for DFC and UCF events ........... 10

Resolution Result - Releasing the stuck threads which are waiting for DFC and UCF events .................................................................................................... 11

Sample snippet of the stack trace in the application server logs when releasing stuck threads for UCF events ................................................................................ 12

Sample snippet of the stack trace in the application server logs when releasing stuck threads for DFC events ................................................................................ 13

Chapter 2 Configuration of timeout values for SessionSync threads ..... 14

Problem ........................................................................................................ 14

Sample snippet of a thread waiting in the SessionSync component in a thread dump ............................................................................................................................ 14

Resolution – Releasing the stuck or hung threads that are waiting for SessionSync ................................................................................................. 16

Sample snippet of the stack trace in the application server logs when releasing stuck or hung threads for the SessionSync component ........................................ 16

Resolution Result – Releasing the SessionSync related stuck or hung threads after configured time out value ..................................................................... 18

Chapter 3 Configuring the timeout value for ProgressBarUpdater threads ..................................................................................... 19

Problem ........................................................................................................ 19

Sample snippet of a ProgressBarUpdater thread waiting for an update ................ 19

Resolution – Releasing the stuck or hung threads waiting in the ProgressBarUpdater ...................................................................................... 21


4

Resolution Result – Releasing the ProgressBarUpdater-related hung threads after the configured timeout value ................................................................ 22

Sample snippet of the stack trace in the application server logs when releasing stuck or hung threads for the ProgressBarUpdater thread .................................... 22

Chapter 4 Supporting large content transfer operations in a managed server environment .................................................................................... 24

Chapter 5 Conclusion ......................................................................... 25

Content Transfer Timeout Configurations for WDK-based Applications Troubleshooting Guide

5

Preface

As part of an effort to improve and enhance the performance and capabilities of its product line, EMC from time to time releases revisions of its hardware and software. Therefore, some functions described in this guide may not be supported by all revisions of the software or hardware currently in use. For the most up-to-date information on product features, refer to your product release notes.

If a product does not function properly or does not function as described in this document, please contact your EMC representative.

Note This document was accurate as of the time of publication. However, as information is added, new versions of this document may be released to the EMC online support website. Check the website to ensure that you are using the latest version of this document.

Purpose

This document provides information about how you can configure and use content transfer timeout options for WDK-based applications.

Executive summary

This white paper is intended to explain the purpose of the WDK-based SessionSync, JobAdapter, and ProgressBarUpdater Listener threads.

This white paper explains the different configuration steps required to ensure the release of hung or stuck content transfer threads on the application server.

Introduction

This whitepaper explains the end-to-end configuration steps for releasing the content transfer-related hung threads from the application server.

Audience

The audience for this whitepaper consists of those responsible for the configuration and administration of the application server production environment with regard to WDK-based client applications. This document is intended for internal EMC personnel, partners, and customers.



7

Chapter 1 Configuration of request input timeout values for DFC and UCF events

Problem At the time of performing content transfer operations, the end user will need to provide inputs for some Documentum Foundation Classes (DFC ) and Unified Content Transfer (UCF ) events. For example, when you check in a file, an identical file name may exist already in the same folder from an export operation. In this case, if the end user has not provided any input for these events, then the Webtop-based threads will be stuck or hung on the application server.

If the application does not release these threads, then these threads will build up on the application server, and application server may become non-responsive after some time. To avoid such situations, you will require a mechanism to release these stuck or hung threads from the application server.

By default, the stuck threads will be in a hung state in the application server up to the session timeout configuration value, which is configured in the <app-root>/WEB-INF/web.xml file.

Here, <app-root> indicates the location where you have deployed the WDK-based application (for example, Webtop).

• If you have configured the session timeout value in the web.xml file to 30 minutes, then the request input threads for DFC and UCF events will be stuck for 30 minutes.

<session-config>

<session-timeout>30</session-timeout>

</session-config>


8

• If you have configured the session time out value in the web.xml file as 2 hours, then the request input threads for DFC and UCF events will be stuck in the application server up to 2 hours.

<session-config>

<session-timeout>120</session-timeout>

</session-config>

The following is a sample snippet of the Webtop-based stuck or hung threads for DFC and UCF events, which are waiting for the end user input.

Sample snippet of stuck or hung thread that is waiting for DFC Event from the thread dumps

WebContainer : 170" daemon prio=3 tid=0x056da688 nid=0xd64 in Object.wait() [0x9e8fe000..0x9e8ffbf0] at java.lang.Object.wait(Native Method) at com.documentum.job.Job.requestInput(Job.java:146) at com.documentum.web.contentxfer.JobAdapter.onDFCEvent(JobAdapter.java:285) at com.documentum.web.contentxfer.DfOperationMonitorSupport.reportError(DfOperationMonitorSupport.java:51) at com.documentum.operations.DfOperation.reportError(DfOperation.java:456) at com.documentum.operations.nodeactions.DfNodeAction.reportError(DfNodeAction.java:54) at com.documentum.operations.nodeactions.inbound.DfCheckinObject.checkin(DfCheckinObject.java:363) at com.documentum.operations.nodeactions.inbound.DfCheckinObject.execute(DfCheckinObject.java:39) at com.documentum.operations.steps.impl.OperationStep.executeStep(OperationStep.java:163) at com.documentum.operations.steps.impl.OperationStep.execute(OperationStep.java:41) at com.documentum.operations.impl.OperationExecutionEngine.execute(OperationExecutionEngine.java:51) at com.documentum.operations.DfOperation.execute(DfOperation.java:379) at com.documentum.operations.inbound.impl.InboundOperation.execute(InboundOperation.java:104) at com.documentum.web.contentxfer.DFCOperationSupport.execute(DFCOperationSupport.java:61) at com.documentum.web.contentxfer.ContentTransferService.execute(ContentTransferService.java:377) at com.documentum.web.contentxfer.JobAdapter.execute(JobAdapter.java:108) at com.documentum.job.async.AsyncJobManager$AsyncTimerJob.executeJob(AsyncJobManager.java:236) at com.documentum.job.async.AsyncJobManager$AsyncTimerJob.run(AsyncJobManager.java:215) at java.util.TimerThread.mainLoop(Timer.java:527) at java.util.TimerThread.run(Timer.java:477)


9

Sample snippet of a thread that is waiting for a UCF Event when generating a thread dump

WebContainer : 170" daemon prio=3 tid=0x056da688 nid=0xd64 in Object.wait() [0x9e8fe000..0x9e8ffbf0] at java.lang.Object.wait(Native Method) com.documentum.job.Job.requestInput(Job.java:142) com.documentum.web.contentxfer.JobAdapter.onUCFEvent(JobAdapter.java:278) com.documentum.web.contentxfer.ucf.NotificationMonitorSupport.notifyEvent(NotificationMonitorSupport.java:186) com.documentum.ucf.server.notification.impl.NotificationServlet.doPost(NotificationServlet.java:109) javax.servlet.http.HttpServlet.service(HttpServlet.java:727) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:108) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.env.WDKController.processRequest(WDKController.java:95) com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173) >

Stuck or hung thread information in the weblogic app server log file In the case of weblogic, the hung threads can be seen in the application server log file. By default, the weblogic application server has the mechanism to display threads that are in a running state for longer that a predefined threshold. When the threshold has been exceeded, the server log files display the information.

The following is a sample snippet of the stuck thread information that will be displayed in the weblogic server log file for the UCF event. A similar stack trace will be observed for the DFC events related stuck threads.

ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26d1c9a503b0d316:-19403be4:12f8d86e866:-8000-0000000000000018> <1304407732798> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,510" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@1680a31[ POST /webtop/com/documentum/ucf/server/notification/impl/Notification HTTP/1.1 Cookie: JSESSIONID=bL57N1ndL0G7Gvfgl87tqhyJZ6h0KL1Jk4Sf6J2Mn4xbF1TPfT0G!-660706996 Content-Type: application/x-www-form-urlencoded;charset=utf-8 User-Agent: Java/1.6.0_13 Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Connection: keep-alive Content-Length: 141


10

]", which is more than the configured time (StuckThreadMaxTime) of "1,500" seconds. Stack trace: java.lang.Object.wait(Native Method) com.documentum.job.Job.requestInput(Job.java:142) com.documentum.web.contentxfer.JobAdapter.onUCFEvent(JobAdapter.java:278) com.documentum.web.contentxfer.ucf.NotificationMonitorSupport.notifyEvent(NotificationMonitorSupport.java:186) com.documentum.ucf.server.notification.impl.NotificationServlet.doPost(NotificationServlet.java:109) javax.servlet.http.HttpServlet.service(HttpServlet.java:727) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:108) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.env.WDKController.processRequest(WDKController.java:95) com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173) >

Resolution – Configuring the time out values for DFC and UCF events The default timeout configuration for hung threads is the session timeout value, which means that these hung threads will be released after the session time out.

Instead of the session timeout value, we can explicitly configure the timeout value to release the hung threads in the application server.

WDK-based applications contain a configuration element <requestinput-timeout>, located in the <app-root>/wdk/app.xml file, which accepts a request input timeout value in seconds. This is the default value set for DFC and UCF events related to request input threads.

11

The following is a sample snippet of tags in the <app-root>/wdk/app.xml file required to set the request the input timeout value.

<contentxfer> ………………………. ………………………  <requestinput-timeout>0</requestinput-timeout> </contentxfer>

If the value is 0, then the request input timeout uses the session timeout value configured in the web.xml file. By default, this request input timeout value is configured to 0. You can explicitly configure it to a different value depending on the requirement.

The above configuration is available from Webtop version 6.6 and above. To achieve the same in earlier versions of Webtop, apply the hotfix for defect number WEBTOP-9439.

After applying the hotfix to earlier versions, configure the proper time out value in the <app-root>/wdk/app.xml file using the <requestinput-timeout> tag.

Resolution Result - Releasing the stuck threads which are waiting for DFC and UCF events

After configuring the appropriate request input timeout value, the UCF and DFC events related hung threads will be released on the application server on reaching the configured timeout value.

At the time of releasing these hung threads from the application server, error or exception messages will be displayed. This error occurs when you have been requested to provide the input, but did not respond within the stipulated time.

The release of hung threads is performed to ensure that the socket connection (UCF layer) is not blocked for more than the configured time (a pre-configured value) waiting for the user input or confirmation. This is non-fatal exception and can be ignored. These messages are an indication that the waiting threads have timed out and have been released back to the execution queue at this point in time.


12

Sample snippet of the stack trace in the application server logs when releasing stuck threads for UCF events

<[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26d1c9a503b0d316:-19403be4:12f8d86e866:-8000-000000000000001a> <1304408022175> <BEA-101020> <[weblogic.servlet.internal.WebAppServletContext@261ac7 - appName: 'webtop', name: 'webtop', context-path: '/webtop', spec-version: '2.5'] Servlet failed with Exception java.lang.IllegalStateException: Timed out requesting input at com.documentum.job.Job.requestInput(Job.java:146) at com.documentum.web.contentxfer.JobAdapter.onUCFEvent(JobAdapter.java:278) at com.documentum.web.contentxfer.ucf.NotificationMonitorSupport.notifyEvent(NotificationMonitorSupport.java:186) at com.documentum.ucf.server.notification.impl.NotificationServlet.doPost(NotificationServlet.java:109) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:108) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at com.documentum.web.env.WDKController.processRequest(WDKController.java:95) at com.documentum.web.env.WDKController.doFilter(WDKController.java:83) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(Unknown Source) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) at weblogic.work.ExecuteThread.run(ExecuteThread.java:173) >


13

Sample snippet of the stack trace in the application server logs when releasing stuck threads for DFC events

<[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26d1c9a503b0d316:-19403be4:12f8d86e866:-8000-000000000000001a> <1304408022175> <BEA-101020> <[weblogic.servlet.internal.WebAppServletContext@261ac7 - appName: 'webtop', name: 'webtop', context-path: '/webtop', spec-version: '2.5'] Servlet failed with Exception java.lang.IllegalStateException: Timed out requesting input at com.documentum.job.Job.requestInput(Job.java:146) at com.documentum.job.Job.requestInput(Job.java:146) at com.documentum.web.contentxfer.JobAdapter.onDFCEvent(JobAdapter.java:285) at com.documentum.web.contentxfer.DfOperationMonitorSupport.reportError(DfOperationMonitorSupport.java:51) at com.documentum.operations.DfOperation.reportError(DfOperation.java:456) at com.documentum.operations.nodeactions.DfNodeAction.reportError(DfNodeAction.java:54) at com.documentum.operations.nodeactions.inbound.DfCheckinObject.checkin(DfCheckinObject.java:363) at com.documentum.operations.nodeactions.inbound.DfCheckinObject.execute(DfCheckinObject.java:39) at com.documentum.operations.steps.impl.OperationStep.executeStep(OperationStep.java:163) at com.documentum.operations.steps.impl.OperationStep.execute(OperationStep.java:41) at com.documentum.operations.impl.OperationExecutionEngine.execute(OperationExecutionEngine.java:51) at com.documentum.operations.DfOperation.execute(DfOperation.java:379) at com.documentum.operations.inbound.impl.InboundOperation.execute(InboundOperation.java:104) at com.documentum.web.contentxfer.DFCOperationSupport.execute(DFCOperationSupport.java:61) at com.documentum.web.contentxfer.ContentTransferService.execute(ContentTransferService.java:377) at com.documentum.web.contentxfer.JobAdapter.execute(JobAdapter.java:108) at com.documentum.job.async.AsyncJobManager$AsyncTimerJob.executeJob(AsyncJobManager.java:236) at com.documentum.job.async.AsyncJobManager$AsyncTimerJob.run(AsyncJobManager.java:215) at java.util.TimerThread.mainLoop(Timer.java:527) at java.util.TimerThread.run(Timer.java:477)


14

Chapter 2 Configuration of timeout values for SessionSync threads

Problem Webtop acquires a lock on the SessionSync component before performing any operations in WDK-based applications, which would include UCF content transfer. After completing the operation, Webtop releases the SessionSync lock. In some situations, for example, when UCF operations fail without providing any notification to Webtop, or a user has not provided the relevant input when prompted by the content transfer operation, and the lock has not been released on this component, the SessionSync threads will be stuck or hung on the application server.

Sample snippet of a thread waiting in the SessionSync component in a thread dump

at java.lang.Object.wait(Native Method) - waiting on <0x0000000192b9bcd8> (a com.documentum.web.common.SessionSync) at com.documentum.web.common.SessionSync.lock0(SessionSync.java:167) - locked <0x0000000192b9bcd8> (a com.documentum.web.common.SessionSync) at com.documentum.web.common.SessionSync.lock(SessionSync.java:68) at com.documentum.web.form.FormProcessor.lockSession(FormProcessor.java:2026) com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1632) com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1487) com.documentum.web.form.FormProcessor.fireActionEvent(FormProcessor.java:1303) com.documentum.web.form.RecallOperation.execute(RecallOperation.java:101) com.documentum.web.form.FormProcessor.processAction(FormProcessor.java:113) com.documentum.web.form.FormAction.processAction(FormAction.java:107) com.documentum.web.env.WDKController.doStartRequest(WDKController.java:191) com.documentum.web.env.WDKController.processRequest(WDKController.java:92) com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173)>

In the case of weblogic, the hung threads can be seen in the application server log file. By default, the weblogic application server has the mechanism to display threads that are in a running state for longer that a predefined threshold. When the threshold has been exceeded, the server log files display the information.


15

The following is a sample snippet of the stuck thread information that will be displayed in the weblogic server log file for the SessionSync Component.

<[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26d1c9a503b0d316:-19403be4:12f8d86e866:-8000-000000000000004a> <1305536265853> <BEA-000337> <[STUCK] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,522" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@2b5139[ POST /webtop/wdk/system/ucf/invoker.jsp HTTP/1.1 Accept: */* Accept-Language: en-us Referer: http://172.22.253.60:7007/webtop/component/residentucfinvoker?Reload=1305534638603&__dmfFrameId=MainEx_residentucfinvoker_0&__dmfClientId=1305534318070 Content-Type: application/x-www-form-urlencoded UA-CPU: x86 Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Content-Length: 858 Connection: Keep-Alive Cache-Control: no-cache ]", which is more than the configured time (StuckThreadMaxTime) of "1,500" seconds. Stack trace: java.lang.Object.wait(Native Method) com.documentum.web.common.SessionSync.lock0(SessionSync.java:167) com.documentum.web.common.SessionSync.lock(SessionSync.java:68) com.documentum.web.form.FormProcessor.lockSession(FormProcessor.java:2026) com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1632) com.documentum.web.form.FormProcessor.invokeMethod(FormProcessor.java:1487) com.documentum.web.form.FormProcessor.fireActionEvent(FormProcessor.java:1303) com.documentum.web.form.RecallOperation.execute(RecallOperation.java:101) com.documentum.web.form.FormProcessor.processAction(FormProcessor.java:113) com.documentum.web.form.FormAction.processAction(FormAction.java:107) com.documentum.web.env.WDKController.doStartRequest(WDKController.java:191) com.documentum.web.env.WDKController.processRequest(WDKController.java:92) com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173) >


16

Resolution – Releasing the stuck or hung threads that are waiting for SessionSync

To release the SessionSync related threads in the application server, a timeout mechanism is implemented. Based on the timeout configuration, the SessionSync stuck threads will be released. If the application does not release these threads, then these threads will build up on the application server, and application server may become non-responsive after some time. To avoid such situations, you will require a mechanism to release these stuck or hung threads from the application server.

At the time of releasing these hung threads from the application server, error or exception messages will be displayed. These are non-fatal exceptions and can be ignored. These messages will provide information about the hung threads that are being released at this point in time.

Sample snippet of the stack trace in the application server logs when releasing stuck or hung threads for the SessionSync component

<[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26d1c9a503b0d316:173c93e3:12f248e175b:-8000-0000000000000025> <1303045176993> <BEA-101020> <[weblogic.servlet.internal.WebAppServletContext@13a494d - appName: 'webtop', name: 'webtop', context-path: '/webtop', spec-version: '2.5'] Servlet failed with Exception java.lang.IllegalStateException: SessionSynch@13b5302[lockCount=169,lockOwner=Thread[[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads]]: Timed out waiting in lock() at com.documentum.web.common.SessionSync.lock0(SessionSync.java:181) at com.documentum.web.common.SessionSync.lock(SessionSync.java:68) at com.documentum.web.formext.component.ComponentDispatcher.mapRequestToComponent(ComponentDispatcher.java:434) at com.documentum.web.formext.component.ComponentDispatcher.doGet(ComponentDispatcher.java:326) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at com.documentum.web.formext.component.ComponentDispatcher.doService(ComponentDispatcher.java:300) at com.documentum.web.formext.component.ComponentDispatcher.serviceAsNonController(ComponentDispatcher.java:138) at com.documentum.web.formext.component.ComponentDispatcher.service(ComponentDispatcher.java:119) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)


17

at com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:84) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at com.documentum.web.env.WDKController.processRequest(WDKController.java:95) at com.documentum.web.env.WDKController.doFilter(WDKController.java:83) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(Unknown Source) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) at weblogic.work.ExecuteThread.run(ExecuteThread.java:173) >

By default, the SessionSync related threads will be released on the application server after 30 minutes, if not released under normal conditions. The timeout value is configured in the SessionSync.properties file as a “maxWaitTime” property located at: <app-root>/ WEB-INF/classes/com/documentum/web/common. # Max wait time to obtain the lock after which warning messages will be logged: maxWaitWarningTime=900000 # Max wait time to obtain the lock after which the thread gives up: maxWaitTime=1800000 In the case “maxWaitTime=1800000”, the stuck or hung SessionSync threads will timeout or be released after 30 minutes. You can explicitly configure the “maxWaitTime” property to release the SessionSync related stuck or hung threads in the application server according to your business needs. Prior to the Webtop version 6.5 SP2, the maxWaitTime parameter was commented (#maxWaitTime=10000) and as a result these SessionSync related threads could indefinitely be stuck or hung on the application server. Hence, you will need to remove the "#"at the beginning of the “#maxWaitTime“ property to set the timeout value. Restarting the application server will allow the change to take effect.


18

Resolution Result – Releasing the SessionSync related stuck or hung threads after configured time out value

After configuring the appropriate maxWaitTime value for SessionSync threads, the SessionSync related threads will be released when the configured timeout value is reached. This occurs in the case where the release has not been performed during normal processing.


19

Chapter 3 Configuring the timeout value for ProgressBarUpdater threads

Problem Webtop will acquire the lock on the ProgressbarUpdater Listener class to display the content transfer status in the user interface. On completion of the content transfer operation, Webtop will release the lock on the ProgressBarUpdater Listener class. In some exceptional circumstances (for example, UCF operations are not sending notifications back to Webtop about the status of content transfer), if the lock is not released on the Listener class, the ProgressBarUpdater threads will be stuck or hung on the application server.

If the hung threads are not released, then these threads will build up on the application server, and the application server may become non-responsive after some time. To avoid such situations, you will require a mechanism to release these stuck or hung threads from the application server.

Sample snippet of a ProgressBarUpdater thread waiting for an update

WebContainer : 170" daemon prio=3 tid=0x056da688 nid=0xd64 in Object.wait() [0x9e8fe000..0x9e8ffbf0] at java.lang.Object.wait(Native Method) at com.documentum.web.form.control.ProgressBarUpdaterTag.waitForListenerUpdate(ProgressBarUpdaterTag.java:311) - locked <0xce5ea438> (a com.documentum.web.form.control.ProgressBarUpdater$ProgressListener) com.documentum.web.form.control.ProgressBarUpdaterTag.renderEnd(ProgressBarUpdaterTag.java: 216) com.documentum.web.form.ControlTag.doEndTag(ControlTag.java:873) jsp_servlet._webcomponent._library._progress.__realtimeprogress._jsp__tag13(__realtimeprogress.java:676) jsp_servlet._webcomponent._library._progress.__realtimeprogress._jspService(__realtimeprogress.java:299) weblogic.servlet.jsp.JspBase.service(JspBase.java:34) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:108) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.env.WDKController.processRequest(WDKController.java:95)


20

com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173)

In the case of weblogic, the hung threads can be seen in the application server log file. By default, the weblogic application server has the mechanism to display threads that are in a running state for longer that a predefined threshold. When the threshold has been exceeded, the server log files display the information.

The following is a sample snippet of the stuck thread information that will be displayed in the weblogic server log file for the ProgressBarUpdater related hung or stuck threads.

<[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <26d1c9a503b0d316:-19403be4:12f8d86e866:-8000-000000000000004a> <1305536265854> <BEA-000337> <[STUCK] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "1,577" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@616670[ GET /webtop/webcomponent/library/progress/realtimeprogress.jsp?__dmfRequestId=__client9~~8&Reload=1305534670901&__dmfClientId=1305534318070 HTTP/1.1 Accept: */* Referer: http://172.22.253.60:7007/webtop/component/jobprogressmonitor?jobId=1305534669431&__dmfRequestId=__client9~~7&__dmfJumpType=nested&Reload=1305534670792&__dmfClientId=1305534318070 Accept-Language: en-us UA-CPU: x86 Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 1.1.4322; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729) Connection: Keep-Alive Cookie: ]", which is more than the configured time (StuckThreadMaxTime) of "1,500" seconds. Stack trace: java.lang.Object.wait(Native Method) com.documentum.web.form.control.ProgressBarUpdaterTag.waitForListenerUpdate(ProgressBarUpdaterTag.java:339) com.documentum.web.form.control.ProgressBarUpdaterTag.renderEnd(ProgressBarUpdaterTag.java:216) com.documentum.web.form.ControlTag.doEndTag(ControlTag.java:873) jsp_servlet._webcomponent._library._progress.__realtimeprogress._jsp__tag13(__realtimeprogress.java:676) jsp_servlet._webcomponent._library._progress.__realtimeprogress._jspService(__realtimeprogress.java:299) weblogic.servlet.jsp.JspBase.service(JspBase.java:34) weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42)


21

com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:108) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.env.WDKController.processRequest(WDKController.java:95) com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180) weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173) >

Resolution – Releasing the stuck or hung threads waiting in the ProgressBarUpdater

To release the ProgressBarUpdater-related hung or stuck threads in the application server, a timeout mechanism is implemented. Based on the time out configuration mechanism, the ProgressBarUpdater threads will be released on the application server.

If the application does not release the ProgressBarUpdater threads, then these threads will build up on the application server, and application server can be non-responsive after some time.

By default, the ProgressBarUpdater-related hung threads will be released after 30 minutes. This timeout value is configured in the “maxProgressUpdateWait” property in the ProgressBarUpdater.properties file located at: <app-root>/ WEB-INF/classes/com/documentum/web/form/control. # Max wait time is used to release the lock on ProgressBarUpdater Listener: maxProgressUpdateWait=1800000

In the case of “maxProgressUpdateWait =1800000”, the idle ProgressBarUpdater thread should timeout or be released after 30 minutes. You can explicitly configure the “maxProgressUpdateWait“ property to release the ProgressBarUpdater-related stuck/hung threads in the application server according to your business needs.

Prior to Webtop version 6.5 SP2, this timeout setting configuration was not available.

Note: This is the idle timeout value for ProgressBarUpdater threads, which indicates that if Webtop does not receive any notification from UCF within this configured value, then these threads will be timed out or released. Normally, UCF will continue to send notifications to Webtop about the status of the content transfer. The ProgressBarUpdater thread is alive in the application server as long as the UCF notifies Webtop within the configured time.


22

This ProgressBarUpdater thread will be released or timed out from the application server in the following scenarios:

• On successful completion of content transfer. UCF sends a 100% completion

status notification back to Webtop.

• If the notification (content transfer status) is not sent by the UCF within the configured time.

Resolution Result – Releasing the ProgressBarUpdater-related hung threads after the configured timeout value

At the time of releasing these stuck or hung threads from the application server, error or exception messages are displayed. These are non-fatal exceptions and can be ignored. These messages will inform you about the stuck or hung threads, which are being released at that point in time.

Sample snippet of the stack trace in the application server logs when releasing stuck or hung threads for the ProgressBarUpdater thread

ERROR [http-8160-47] com.documentum.web.common.Trace - Reached max progress wait timeout (1800000 ms), exiting from progress update ProgressBarUpdaterTag@1d37157[m_percentComplete=14,m_stepPercentComplete=0,m_lastPercentComplete=14,m_lastStepPercentComplete=0,m_lastUpdate=1297887291381] java.lang.IllegalStateException: Reached max progress wait timeout (1800000 ms), exiting from progress update ProgressBarUpdaterTag@1d37157[m_percentComplete=14,m_stepPercentComplete=0,m_lastPercentComplete=14,m_lastStepPercentComplete=0,m_lastUpdate=1297887291381] at com.documentum.web.form.control.ProgressBarUpdaterTag.renderEnd(ProgressBarUpdaterTag.java:211) at com.documentum.web.form.ControlTag.doEndTag(ControlTag.java:873) at org.apache.jsp.webcomponent.library.progress.realtimeprogress_jsp._jspx_meth_dmf_005fprogressbarupdater_005f0(realtimeprogress_jsp.java:560) at org.apache.jsp.webcomponent.library.progress.realtimeprogress_jsp._jspService(realtimeprogress_jsp.java:264) com.documentum.web.form.control.ProgressBarUpdaterTag.renderEnd(ProgressBarUpdaterTag.java:216) weblogic.servlet.jsp.JspBase.service(JspBase.java:34)weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227) weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125) weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:292) weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.ResponseHeaderControlFilter.doFilter(ResponseHeaderControlFilter.java:317) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.servlet.CompressionFilter.doFilter(CompressionFilter.java:108) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) com.documentum.web.env.WDKController.processRequest(WDKController.java:95) com.documentum.web.env.WDKController.doFilter(WDKController.java:83) weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3496) weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) weblogic.security.service.SecurityManager.runAs(Unknown Source) weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2180)


23

weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2086) weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1406) weblogic.work.ExecuteThread.execute(ExecuteThread.java:201) weblogic.work.ExecuteThread.run(ExecuteThread.java:173)


24

Chapter 4 Supporting large content transfer operations in a managed server environment

If you are deploying a Weblogic Managed Server environment and you use UCF to perform large content operations, set the WLIOTimeoutSecs parameter for the Web Server plug-in to a large value. UCF requires a sticky session for a single operation. For additional details, refer to BEA’s documentation on Web Server plug-in parameters.


25

Chapter 5 Conclusion

This white paper provides information about how you can configure various timeout values for WDK-based content transfer operations. This white paper also provided an explanation about the root cause of WDK-based JobAdapter, SessionSync, and ProgressbarUpdater stuck or hung threads and the timeout configuration settings required to release these hung threads on the application server.

For all WDK-based client applications, you will need to ensure that all Webtop-related stuck or hung threads (SessionSync, JobAdapter, ProgressBarUpdater and so on) have a timeout value configured. If these hung threads are not timed out, then the application server may become non-responsive.

content transfer timeout configurations for wdk - dell emc · content transfer timeout...

Documents