troubleshooting emc documentum webtop kerberos · pdf filetroubleshooting emc documentum...

White Paper

Abstract

This white paper provides information about a systematic approach to solving problems in EMC® Documentum® Webtop Kerberos SSO environments. March 2011

TROUBLESHOOTING EMC DOCUMENTUM WEBTOP KERBEROS SSO ENVIRONMENTS

2 Troubleshooting EMC Documentum Webtop Kerberos SSO Environments

Copyright © 2011 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice. The information in this publication is provided “as is”. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All other trademarks used herein are the property of their respective owners. Part Number h8209


Table of Contents

Executive summary.................................................................................................. 5

Audience ............................................................................................................................ 5

Terminology ....................................................................................................................... 5

Setting up the Kerberos environment machines........................................................ 6

Limitations .............................................................................................................. 6

Debugging tools ...................................................................................................... 6

Working model ........................................................................................................ 6

A quick decision tree ............................................................................................... 8

Client machine cannot obtain an initial TGT .............................................................. 8

Incorrect encryption types .................................................................................................. 8

Solution ......................................................................................................................... 8

Clock skew ....................................................................................................................... 10

Solution ....................................................................................................................... 10

Incorrect hostname/DNS configuration............................................................................. 11

Solution ....................................................................................................................... 11

Kerberos error codes ........................................................................................................ 11

Client has a valid TGT but is unable to obtain the Kerberos service ticket ................ 12

Duplicate SPNs ................................................................................................................. 13

Solution ....................................................................................................................... 13

Incorrect domain/realm mapping ..................................................................................... 13

Solution ....................................................................................................................... 14

Missing browser configurations ........................................................................................ 15

Kerberos error codes ........................................................................................................ 15

Kerberos service ticket authentication fails ............................................................ 15

Missing JAAS configurations ............................................................................................. 15

Solution ....................................................................................................................... 16

Wrong keytab file ............................................................................................................. 16

Solution ....................................................................................................................... 17

Problem with the service ticket ......................................................................................... 17

Solution ....................................................................................................................... 18

GSS exceptions ................................................................................................................ 18

Problem with delegation ........................................................................................ 18

Missing configurations ..................................................................................................... 19

Duplicate SPN and domain/realm mapping ...................................................................... 19

Exceptions ....................................................................................................................... 19

Authentication fails at the Content Server machine ................................................ 20


Failure in initializing the Kerberos plug-in ......................................................................... 20

Solution ....................................................................................................................... 20

Failure in authenticating the service ticket ........................................................................ 21

Conclusion ............................................................................................................ 21

References ............................................................................................................ 21


Executive summary The Kerberos environment is complex and created from various configurations, and Kerberos error messages are not clear or user-friendly. This white paper aims to reduce the time and effort required to resolve Kerberos installation issues. It provides a systematic approach to solving problems in EMC® Documentum® Kerberos environments.

The References section can provide additional information.

Audience

This white paper is intended for customers, partners, and consultants who may face problems with the installation of Kerberos on Documentum Webtop.

Terminology Authentication Service (AS): A Kerberos service that provides the Ticket Granting

Ticket (TGT) to the requesting client.

Kerberos Realm: A realm is where the Kerberos database is stored. It is always located in the domain controller machine where the Active Directory is also configured.

Key Distribution Center (KDC): The KDC is a network service that supplies session tickets and temporary session keys to users and computers within an Active Directory domain. The KDC runs on each domain controller as part of the Active Directory Domain Services (ADDS). In Kerberos context, virtually the KDC is split into an AS and TGS.

Keytab File: A keytab file contains the Service Principal Name (SPN) and the encryption key for that service, which can be used to decrypt the service ticket for any client.

Service Principal Name (SPN): An SPN corresponds to a Windows service and is registered against the Active Directory user using the ktpass command.

Service Ticket (ST): The service ticket is specific to a particular service of a server. The ST contains the session key and the client’s information. It can be decrypted only using the encryption key in the keytab file.

Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO): This is a GSSAPI “pseudo mechanism.” In this mechanism, a SPNEGO token will be used to carry the service ticket from the client to the server.

Ticket Granting Service (TGS): This is a Kerberos service that provides the service ticket for a service registered in the Active Directory as Service Principal Name. The service ticket will be provided to the client only after accepting the valid Ticket Granting Ticket (TGT) from the client.

Ticket Granting Ticket (TGT): This ticket is used to obtain the service ticket for a server service from the Ticket Granting Server.


Setting up the Kerberos environment machines The EMC Documentum Kerberos SSO Authentication—A Detailed Review white paper explains how to set up Documentum Kerberos environments.

Limitations The following are limitations of a Kerberos environment:

Documentum supports Kerberos only on Windows client machines.

Hostnames and domain names cannot have underscore characters.

The version of the Active Directory machine should be above Windows Server 2003.

DES support should be enabled explicitly in the Windows 7 and Windows Server 2008 servers.

Debugging tools The following are debugging tools that can be used:

Charles proxy debugging tool—This web proxy tool allows you to monitor HTTP traffic between the client and application server machines.

Windows Toolkits—This package contains various tools such as Kinit and Kerbtray, which help you view the details of the TGT obtained from the KDC.

Wireshark—This tool helps you monitor network traffic between any two machines.

WDK Tracing—This tool enables the SESSIONENABLEDBYDEFAULT and SESSION tracing flags in the following location: <web-app-root>/WEB-INF/classes/com/documentum/debug/TraceProp.properties

The WDK trace logs will be created in the following location: <web-app-root>/WEB-INF/classes/log

Working model The basic Kerberos authentication and ticket exchange proceed as shown in the following figure, with explanations of each step following.


Figure 1. Kerberos authentication and ticket exchange process

1) The Windows domain user provides the user credentials and requests for a TGT from Authentication Service.

2) Authentication Service verifies the user credentials against the Active Directory. On successful verification, Active Directory provides the TGT and an encrypted session key, which can be decrypted using the client’s password hash code. This session key can be used for future communication between the client and KDC.

3) For any service in the network that is registered in the Active Directory, the user requests for a service ticket for that service from TGS by providing the TGT to the TGS. Here, the client requests the TGS for a service ticket for Webtop service.

4) TGS verifies the validity of the TGT and provides the service ticket and session key to the client, which can be used for communication between the client and server. Here, TGS verifies the client request and provides the service ticket for Webtop service.

5) Select the appropriate repository on the repository login page displayed. Provide the service ticket details (containing client information and session key) to Webtop. Webtop decrypts the service ticket using the keytab file in the application server and authenticates the client. On successful authentication, a session key is shared between the application server and client machine. The client credentials are now delegated to obtain the service ticket for the repository service.

6) Webtop uses the delegated user credentials to request the service ticket for the repository service from TGS. This is performed through the JAAS framework.


7) TGS verifies the credentials and provides the service ticket for the repository service to Webtop.

8) Webtop sends the service ticket to the Content Server for authentication. Upon successful authentication, the client will be authenticated against the repository and logged in.

A quick decision tree In case of problems in Webtop Kerberos installation, identify the root cause of the problems and categorize them as follows:

Client machine cannot obtain an initial TGT

Client has a valid TGT but is unable to get the Kerberos service ticket from KDC

Webtop gets the valid service ticket wrapped in an SPNEGO token from the client machine but an error is thrown while authenticating the service ticket against the keytab file

Service ticket has been accepted and authenticated using the keytab file but an error is thrown upon delegation

Credentials have been successfully delegated but authentication fails at the Content Server

Client machine cannot obtain an initial TGT Several errors can occur when a client attempts to obtain a TGT from the Kerberos KDC. This step may fail due to the following reasons.

Incorrect encryption types

There is a possibility that the KDC does not find the appropriate encryption type to encrypt the response. When a Kerberos 5 client contacts a KDC through the AS exchange for an initial TGT, the client sends a list of encryption types that it can decrypt. If the KDC cannot find the secret key associated with one of the encryption types included in the request, it will return an error.

Solution

In Wireshark trace, you can view the encryption types supported by the client machine in the Authentication Service Request (AS-REQ) and by KDC in the response (see Figure 2 and Figure 3). If the login user is a domain user, verify if the encryption type supported by the client is configured or not.


Figure 2. AS-REQ


Figure 3. AS-REP

Clock skew

The AS request identifies the client to the KDC in plain text. If pre-authentication is enabled, a timestamp will be encrypted using the user’s password hash as the encryption key. If the KDC reads a valid time when using the user’s password hash (stored in the Active Directory) to decrypt the timestamp, the KDC will decipher that the request is not a replay of a previous request. The pre-authentication feature may be disabled for specific users in order to support some applications that do not support the security feature. Access the user account from the Active Directory. From the Account Options window, select the “Do not require Kerberos” pre-authentication option.

Solution

Kerberos requires that all computers in the environment have system times within 5 minutes of one another. Verify if each computer in the environment is within 5 minutes of all the others. Note that an environment where the client is 3 minutes slower than the Kerberos server, and the application server is 3 minutes faster than the Kerberos server, represents a time sync problem, as the total skew is 6 minutes.


Incorrect hostname/DNS configuration

The first requirement to set up the Kerberos environment is that the client, application server, and Content Server machines be in the same domain. Although cross-domain is supported in Kerberos, it is not supported in Documentum Webtop. If the client machine is not in the same domain, then obviously the machine cannot connect to the KDC or Active Directory and Kerberos will fail.

Solution

Verify the domain configuration by right-clicking My Computer and selecting Computer Name and then selecting Local Area Network > Properties > Internet Protocol as shown in Figure 4.

Figure 4. Domain name configuration

Kerberos error codes

The following are the Kerberos error codes:

KRB5KDC_ERR_ETYPE_NOSUPP (14): This error may occur due to an encryption type mismatch.

KRB5_KDCREP_SKEW: This error may occur due to the time clock skew.

KRB5_ERR_BAD_HOSTNAME: This error may occur due to the wrong hostname and DNS configuration.

KDC_ERR_C_PRINCIPAL_UNKNOWN: This error may occur as user details are not available in the Active Directory or the user account has expired.


Client has a valid TGT but is unable to obtain the Kerberos service ticket When a user logs in to the client machine, the TGT is obtained by default. The next step would be to obtain the service ticket for the Webtop application when the user logs in to Webtop using a browser. The Kerberos-configured Webtop application returns a 401 response when the user logs in (see Figure 5).

Figure 5. 401 response from Webtop

Upon receiving the 401 response, the client browser will receive a service ticket for Webtop service from the KDC. Verify if a successful TGT has been obtained from the KDC (see Figure 6). Usually, you will receive an NTLM ticket if Kerberos fails. This can be viewed again in the authentication header of the request in the Charles log. You can also search for the “Token type” string in the Documentum log, to verify if you have received the Kerberos ticket or not. The following are some of the reasons for this step to fail.

Figure 6. klist command to view the tickets


Duplicate SPNs

When the same SPN is registered multiple times in the Active Directory to the same or different users, the KDC will be unable to obtain the service ticket for the SPN.

Solution

Run the following command in the Active Directory machine, which will display the number of SPN entries registered in the Active Directory. This number should not be greater than one.

ldifde -r (servicePrincipalName=HTTP/egwtwebtop.egwtiig.egwebtop.com) -v -f somefile.txt

If you find more than one entry for that command, then retain one entry and remove the other entries by running the following command:

setspn –d spn_name user_name

Incorrect domain/realm mapping

DNS problems are often encountered only during a service ticket request, after a successful TGT request transaction. If a client can successfully authenticate initially, but is then unable to acquire a service ticket or access services, then it indicates that DNS problems are the likely cause. The error “Server not found in Kerberos database” is common and can be misleading because it often appears when the service principal is not missing. The error can be caused by domain/realm mapping problems or it can be the result of a DNS problem where the Service Principal Name is not being built correctly. In Wireshark trace, you can view the realm and service name for which the service ticket has been requested from the KDC (see Figure 7).


Figure 7. Kerberos service ticket request

Solution

Kerberos is case-sensitive. Problems can occur in an environment that uses hostnames in mixed case. In the world of Kerberos, appserver1.EXAMPLE.COM and appserver1.example.com are not the same. Note that DNS resolves hostnames with consistent case.

Verify if the Kerberos machines are able to ping each other using the fully qualified domain name. Also run the nslookup command and verify if you can see the result as in Figure 8.

Figure 8. Result of the nslookup command


If you see an unknown server in the result, it indicates that the Reverse Lookup Zone is not configured properly.

You might have configured DNS appropriately, which will provide an IP address to a machine for its Domain Name Space name. But for Kerberos to work, the reverse should also be possible, that is, you should be able to obtain the name of the machine from its IP address and this could be done by Reverse Lookup Zone. Configure Reverse Lookup Zone in the domain controller machine.

Missing browser configurations

The browser may receive the NTLM ticket from KDC instead of a Kerberos ticket, in spite of the following conditions:

There is no error in Wireshark.

SPN duplication has been eliminated.

nslookup has been verified.

The root cause for this will probably be the missing configurations in the browser. Hence, ensure that the browser configuration is accurate (refer to the white paper EMC Documentum Kerberos SSO Authentication—A Detailed Review ). Additional security configurations will have to be disabled in the latest Windows OS, such as Windows 7 and Windows Vista.

Kerberos error codes

The following are the Kerberos error codes:

KDC_ERR_PRINCIPAL_NOT_UNIQUE: This error may occur due to a duplicate SPN.

KDC_ERR_S_PRINCIPAL_UNKNOWN: This error may occur due to a duplicate SPN or wrong domain/realm mapping or both. This can also occur if the SPN itself has not been registered.

KDC_ERR_C_PRINCIPAL_UNKNOWN: This error may occur as the user details are not available in the Active Directory or the user account has expired.

Kerberos service ticket authentication fails After you receive the service ticket from KDC, Webtop will authenticate that service ticket using Java Authentication and Authorization Service (JAAS). This step may fail due to the following reasons.

Missing JAAS configurations

JAAS uses the following krb5Login.conf.configuration file to create the GSSCredential. The location of this configuration file is given as a Java option in Tomcat: Djava.security.auth.login.config


Using these credentials, the service ticket obtained from the browser will be decrypted and authenticated.

You may see the following exception:

GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Ticket)

This may occur if no valid Kerberos credentials are obtained. In particular, this occurs if you want the underlying mechanism to obtain credentials but you forgot to indicate this by setting the javax.security.auth.useSubjectCredsOnly system property value to FALSE.

Again this setting is done as the following Java option in Tomcat: Djavax.security.auth.useSubjectCredsOnly

Solution

Ensure that the configuration of the application server is accurate (refer to the white paper EMC Documentum Kerberos SSO Authentication—A Detailed Review). The JAAS configuration is case-sensitive; hence, ensure that there are no unnecessary spaces in the configuration parameters.

Wrong keytab file

The application server should have an encryption key that it can use to decrypt the service ticket and authenticate the client. You can obtain the encryption key in the following ways:

Obtaining the password from the user

Communicating with the KDC

Obtaining it from the keytab file

Webtop uses the keytab mechanism. The keytab file can be created using the ktpass command:

ktpass /pass <password> -out <keytab-file> -princ <SPN> -crypto DES-CBC-MD5 +DumpSalt -ptype KRB5_NT_PRINCIPAL +desOnly /mapOp set /mapUser <user-name>

Here, the crypto is mentioned as DES-CBC-MD5, which is the encryption type used to encrypt the service ticket.

The encryption key stored in the keytab file is symmetric, that is, the same key is used for both encryption and decryption.

The location of the keytab file is given in the krb5Login.conf configuration file as a parameter value keyTab. The useKeyTab parameter should be set to TRUE for Webtop to be able to use this keytab file.

When a wrong keytab is copied to the location or when the keytab is corrupted, the authentication will fail.


Solution

Use the klist command that comes with Java to identify the content of the keytab file as given in Figure 9.

Figure 9. klist command for the keytab file

Problem with the service ticket

When the browser requests a service ticket for Webtop, the KDC encrypts the service ticket and sends it to the browser. The encryption key used should be the one stored in the keytab file. If a different key or different encryption type is used, Webtop will not be able to read the service ticket and an integrity failure error will be returned.

There is also a possibility that the service ticket was corrupted while it was being passed through the network. Hence, Wireshark traces will need to be captured simultaneously in the Active Directory, Application Error, and client machines, and you will need to verify if the service ticket encryption part (shown in Figure 10) is the same as what has been sent by the Active Directory.

Figure 10. Service ticket encryption part


Solution

Verify the ktpass command and its output to check if any error has been reported and if any wrong parameters have been passed in the command. Also verify if the “Use DES encryption for this account” option has been enabled.

If there is a problem with the network, then purge all tickets in the client machine using the klist command shown in Figure 11. Log in to Webtop. If the problem persists and is identified as a network issue, then involve the network team to correct the packet corruption.

Figure 11. Purging tickets

GSS exceptions

The following are the GSS exceptions:

Integrity check on decrypted field failed (31): This may be due to the wrong keytab file, a corrupted service ticket, or both.

Failure unspecified at GSS-API level CheckSum Failed: This may due to the wrong keytab file, a corrupted service ticket, or both.

Clock skew too great (37): This is due to the clock skew.

Problem with delegation In a three-tier architecture, the authentication mechanism will end with the authentication of the service ticket with the keytab file. But EMC Documentum Webtop is in a four-tier architecture and the user must be authenticated again with the Content Server machine. So Webtop delegates the client user credentials to obtain the service ticket for the Content Server service. Unlike in the first level of


authentication, here both the TGT and service ticket are acquired using the JAAS framework in Java.

The following is the concept of a subject in JAAS:

To authorize access to resources, applications first need to authenticate the source of the request. The JAAS framework defines the term subject to represent the source of a request. A subject may be any entity, such as a person or a service. Once the subject is authenticated, a javax.security.auth.Subject is populated with the associated identities or principals. A subject may have many principals. For example, a person may have a name principal ("John Doe") and an SSN principal (“123-45-6789”), which distinguishes it from other subjects.

This step may fail due to the following reasons.

Missing configurations

To delegate the user credentials, the “Account is trusted for delegation” option must be enabled in the Webtop SPN user’s account.

The krb5.ini configuration file must be created and its location should be given as a Java option in Tomcat:

Djava.security.krb5.conf

The first step is required so that Webtop will be able to delegate the user credentials obtained from the service ticket and use it again to acquire the service ticket for the repository service. The latter is required so that the JAAS code in Webtop can identify the KDC machine and its realm for acquiring the TGT and service ticket.

Duplicate SPN and domain/realm mapping

As in the second decision tree (Client has a valid TGT but is unable to get the Kerberos service ticket), the problem in acquiring the service ticket for the repository service can occur due to duplicate SPN and a problem in the Domain Name Space.

Follow the procedure provided in that decision tree to resolve these problems.

Exceptions

The following are the exceptions:

KrbException: KDC reply did not match expectations: This may be due to the missing encryption types configured in krb5.ini.

KrbException: KDC has no support for encryption type (14): This may be due to the missing encryption types configured in krb5.ini.

KrbException: Null realm name (601) - default realm not specified: This may be due to the missing default realm in Krb5.ini.

GSSException: No valid credentials provided: This may occur due to the missing JAAS configurations.


Authentication fails at the Content Server machine Webtop passes the service ticket obtained for the repository service SPN to the Content Server. The Content Server authenticates the service ticket using the keytab file copied to the following location: C:\Documentum\dba\auth\kerberos. This keytab file should have been generated previously using the ktpass command and the name of the keytab file should be <docbase_name>.<unique_number>.keytab.

This step may fail due to the following reasons.

Failure in initializing the Kerberos plug-in

When the repository service starts, the Kerberos authentication plug-in will be loaded and initialized. If the plug-in is not initialized, then the authentication will fail.

Solution

The following log files can be seen in the location C:\Documentum\dba\log :

dm_krb_<repository_name>.log

<repository_name>.log

In the first log, we can verify if the Kerberos plug-in has been initialized and also check the success of each user login. View the detailed tracing of the authentication scheme in the latter log. You have to enable the authentication trace in the Content Server by adding the “-otrace_authentication” string before –init_file in the repository service shown in Figure 12.

Figure 12. Service entry of repository

Verify if the correct keytab file has been copied to the location C:\Documentum\dba\auth\kerberos and that the naming convention of the keytab


filename was followed. You may also have to check if there is a bug in the Content Server for that particular Content Server and OS version.

Failure in authenticating the service ticket

Reasons for the failure in this step are similar to that of Kerberos service ticket authentication fails except for JAAS configuration, since JAAS is not used here. So follow the same methods provided in the sections Wrong keytab file and Problem with the service ticket to troubleshoot this problem.

Conclusion This white paper documents various issues that could occur and their causes while setting up a Kerberos environment in Documentum Webtop. A systematic approach to resolve Kerberos issues has also been provided.

References The following websites can provide more information:

“Kerberos Explained” on Microsoft TechNet

“Kerberos and LDAP Troubleshooting Tips” on Microsoft TechNet

“Delegating Credentials to an Application Server” on SPNEGO SourceForge

“Secure Authentication using the Java Authentication and Authorization Service” on the Oracle website

http://technet.microsoft.com/en-us/library/bb742516.aspx

http://technet.microsoft.com/en-us/library/bb463167.aspx

http://spnego.sourceforge.net/credential_delegation.html

http://download.oracle.com/javase/6/docs/technotes/guides/security/jgss/lab/part1.html

troubleshooting emc documentum webtop kerberos · pdf filetroubleshooting emc documentum...

Documents