configuring & troubleshooting xen desktop sites

Ramon Scott – Lead Escalation Engineer

Configuring & Troubleshooting XenDesktop Sites

August 29, 2013

Citrix Support Secrets Webinar Series

Student Guide - Official Training - Confidential Documents - Internal TrainingCisco - Citrix - Oracle - VMware - Microsoft - IBM - HP - NetAppvspheredatabaseunified.wordpress.com or http://ouo.io/8u1RP

© 2012 Citrix | Confidential – Do Not Distribute3

Presenter Bio: Ramon Scott

Over 17 Years of Experience in IT

Joined Citrix in April 2010

Started directly into the Escalation Team – primary focus on XenApp

Assigned as the Dedicated Engineer for a Major Strategic Account from Q4-2010

Moved to XenDesktop team in July 2011

Additional details • Bachelor’s Degree in Information Technology with a specialization in Network

Administration

• Certifications: CCA, CCNA, CCDA, MCSE and MCITP-EA

© 2012 Citrix | Confidential – Do Not Distribute

Presentation Goals

4

Provide an Understanding of the

Architecture

Instruct on How to Configure

Provide Proven Troubleshooting

Methodologies and Resources

High-Level XenDesktop Database And Services Architecture


Database


• Supported Databases:• SQL Server 2008 SP1 / 2008R2

(including Express)

• Database Schema• Full Relational Schema

• Tables, Views, Stored Procedures• Single Database (for core product)• Multiple SQL ‘Schemas’ in Database

• ‘Schemas’ map onto Windows services running on Broker

XenDesktop 5 Database Overview

Broker

Broker

Database


Setup Process

XD Console

Single Admin

Broker1. Schema

Database

2. Schema

XD Admin 3. Verify

XD Admin credentials used

Separate Admins

XD Console Broker1. Schema

Database3. Schema

XD Admin

4. Verify

SQL Server Console

SQL Admin

2. Schema

“Export”(SQL script)

SQL Admin credentials used


Database Access

• Security Access Model

ᵒ Network Service Account

“NT AUTHORITY\NETWORK SERVICE”

ᵒ Computer Account

“DOMAIN\MACHINE$”

• SQL Login per Broker

• Restricted permission setᵒ Brokers do not have rights to change schema

Controller

DatabaseController

Broker Service

Controller

DatabaseController

Broker Service

Database


• Broker is critically dependant on Database• Existing connections not impacted• Creating new connections and reconnecting to desktops

impacted

• Database Failure = Broker Failure

• Supported Database H/A Options: (expected popularity order)

1. SQL Mirror2. Virtual Machine H/A3. SQL Cluster

Database High-Availability

Citrix Confidential - Do Not Distribute


Database Schema Roles and Permissions

XenDesktop Service Database Role

AD Identity Service (Acct) ADIdentitySchema_ROLE

Broker Service (Broker) chr_Broker

chr_Controller

Central Configuration Service (Config) ConfigurationSchema_ROLE

Machine Creation Service (PvsVM) DesktopUpdateManagerSchema_ROLE

Hosting Management Service (Hyp) HostingUnitServiceSchema_ROLE

Machine Identity Service (Prov) MachinePersonalitySchema_ROLE


Health Checks: XDDBDiag

• Provided consistency data check on the data

• Provides connectivity verification

It also provides the following:ᵒ Virtual Desktop Agent Informationᵒ Hypervisor Connections Informationᵒ Policy Informationᵒ Controller Informationᵒ Desktop Groups Informationᵒ SQL Informationᵒ Current Connections / Connection Log

XDDBDIAG


Services


Machine CreationServices

BrokerService

InfrastructureServices

XenDesktop 5 Services Architecture

14

Controller

Broker Service

Machine CreationService

AD Identity Service

Machine Identity Service

Host Service

ConfigurationService

Virtual Desktop

Agent (VDA)

WinRM 2.0[5985/5986]

Desktop Studio

WCF [80]

PowerShell Desktop Director

WCF [80]

PowerShell

SQL Server

Windows Communication Foundation (WCF)


Service Status

XenDesktop Service PowerShell Cmdlet

AD Identity Service (Acct) Get-AcctServiceStatus

Broker Service (Broker) Get-BrokerServiceStatus

Central Configuration Service (Config) Get-ConfigServiceStatus

Machine Creation Service (Prov) Use Get-ProvServiceStatus

Hosting Management Service(Hyp) Get-HypServiceStatus

Machine Identity Service (PvsVM) Get-PvsvmServiceStatus


Machine Creation


Desktop Catalog models

• Existing

• Dedicated

• Pooled

• Pooled with personal vDisk

• Streamed

• Streamed with personal vDisk

Base Image

App

Profile

App

Profile

App

Profile

PvD

PvD

PvD

Image

Image

Image

Profile

Profile

Profile

Profile

Profile

Profile

Base Image with Apps

Base Image with Apps

Streamed Base Image

with Apps

Streamed Base Image

App

Profile

App

Profile

App

Profile

PvD

PvD

PvD

*Image Streamed from Citrix Provisioning Server

(PVS)

*Image created with Machine Creation Services

(MCS)

*Image created outside of XenDesktop


Desktop Catalog models

PVSStreame

dVirtua

l

Physical

Streamed with

PvDVirtual Only

MCSPooled

Random

Static

Pooled with PvD*

Dedicated

PreAssigned

First Use

* Behaves like pooled-static


MCS – ID Disk, Difference Disk, Base VM

Virtual Desktop 1Diff Disk ID DiskVHD Chain

Windows 7 Master

This is what the user sees as Drive C:\

This is hidden from the users view

Virtual Desktop 2Diff Disk ID DiskVHD Chain

Virtual Desktop xDiff Disk ID DiskVHD Chain

Storage Subsystem


MCS with PvD – ID Disk, Diff Disk, Base VM, PVDisk

Virtual Desktop 1

VHD Chain

Windows 7 Master

Diff Disk ID Disk

Personal vDisk• This part is hidden from user• Merged with the Diff Disk• Seen by user as Drive C:\• E.g. Installed apps

• Seen by the user as Drive P:\• USERDATA e.g. My Documents• Free space is the split allocation

• PVDisk auto-created during catalog creation by copying PvD template from Base VM

• 10GB by default with 50 / 50 split for App Data / User Data


PVS – Streamed vDisk, Cache, Base VM

Virtual Desktop 1StreamedvDisk

Write Cache

PVS Stream

Windows 7 Master

This is what the user sees as Drive C:\

Visible file on another disk, typically D:\

Virtual Desktop 2StreamedvDisk

Write Cache

PVS Stream

Virtual Desktop xStreamedvDisk

Write Cache

PVS Stream

Storage Subsystem


PVS with PvD–Streamed vDisk, Cache, Base VM, PvDisk

Virtual Desktop 1

PVS Stream

Windows 7 Master

Streamed vDisk

Write Cache

Personal vDisk• This part is hidden from user• Seen by user as Drive C:\• E.g. Installed apps

• Seen by the user as Drive P:\• USERDATA e.g. My Documents• Free space is the split allocation

• PvDisk auto-created during catalog creation by copying PvD template from Base VM

• 10GB by default with 50 / 50 split for App Data / User Data


Where are some of the common Issue ?

• Hypervisor communication• Domain permissions• Previously failed attempts still present

in database• Host Connection configured with

incorrect storage• Naming convention on the host


What logs do we need for this issue ?

24

Machine CreationServices

BrokerService

InfrastructureServices

Broker

Broker Service

Machine CreationService

AD Identity Service

Machine Identity Service

Host Service

ConfigurationService

Desktop Studio

WCF [80]

PoSH

SQL Server


Troubleshooting Methodology

25

• Understand issue history• Verify configuration, error logs and alerts

• Gather and review log data of issues• Compare data to working environment


Enabling Log from the Command Line

Service –LogFile <Location>Citrix.ADIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\AD.log

Citrix.MachineCreation.SdkWcfEndpoint.exe -Logfile “c:\xdlogs\MCS-PVSvm.log”

Citrix.MachineIdentity.SdkWcfEndpoint.exe -LogFile c:\xdlogs\mi.log

Case Study 1Machine Creation Services


Case Study Walk Through

Background:

• New Deployment

• Latest Hotfixes

• Full Administrator account used

• Worked before they rebuilt environment

Case Study 1: MCS Fails after wizard


Log Analysis: Desktop Studio LogsCase Study 1: Machine Creation Service fail after wizard

24/04/13 02:37:10.7603 : DesktopStudio: [6] Script SetActionMetaData(402): [RES] Value: Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.

Search Terms: [Time of Issue]Fail | Error | Exception | Denied



Case Study 1: Machine Creation Service fail after wizardLog Analysis: Machine Creation Service Logs

Failed to copy disk. Reason : SR_HAS_NO_PBDS

ManagedMachineException: Failed to copy disk. Reason : SR_HAS_NO_PBDS

Concluding job d5ea54c6-b7f1-4d45-ac08-2e2abae39e48 with state DiskConsolidationFailed.

WorkflowAddMetadata(, Citrix_DesktopStudio_ExtraWarnings, Failed to copy all master images to all of the Hosts. No machines have been added to the Catalog.)

30


Root Cause analysis: Misconfiguration

31

• Failed to copy disk Reason : SR_HAS_NO_PBDS

• Hypervisor Connection’s did not include correct storage for the Master Image

• Target device disk could not be copied due to this Hypervisor -Storage misconfiguration

*Definitions: SR - Storage RepositoriesPBD - Physical Block Devices

VDA Startup and Registration


Controller

DDC

Broker Service

VDA Registration

VDA

Desktop Service

VDA

Active Directory Controller

WCF

LDAP

Database

Registered


Troubleshooting VDA Startup and Registration

• XDPing Log• Basic Checks• Logs:ᵒ Workstation Agent Logsᵒ Broker Logs

• Network TraceController

Broker Service

VDA

Desktop Service

1011011010 SSL 1011011010 SSL 101101


XDPING

• Can be run on both the DDC and VDA

• Used to collect data related to basic components

• Will verify if the components are working correctlyᵒ Verify Domain Membershipᵒ Network Interfacesᵒ WCF Endpointsᵒ Servicesᵒ DNS lookupᵒ Time difference between machine and

Domain Controller

XDPING


Basic Checks

• Check the Network: Ping , Telnet and NetStat, Firewall

• Ensure Services started without errors

• Listening on the correct port

• Check time

• Check configured list of DDCs in registry

Case Study 2Startup and Registration


Case Study Walk Through Background:

• Locked down environment

• Special configuration needed to manually enable needed services

• Worked in the Proof of Conference Lab but failed in production

Case Study 2: New Catalog Fail to Register


Log Analysis: Workstation Agent Service Logs

Failed to register with http://FTLRSCOTT2RHONE.lab.net:80/Citrix/CdsController/IRegistrar.

WCF Fault with detail CallbackCommunicationError, message 'Fail worker callback using SPN host/RS2-SynPool01.lab.net and IP address 10.19.196.945'

Register FAILURE: HighAvailabilityActive = False, InHighAvailabilityMode = False, _firstRegistrationAttemptTime = 05/18/2013 13:54:31, HighAvailabilityRegistrationTimout = 00:05:00

Message following Error pattern

Could not register with any controllers. Waiting to try again in 9407 ms39




Log Analysis: Broker Service Logs

Broker:TestWorkerComms failed for worker S-1-5-21-1123877020-465626563-

3648135752-1267 caught exception:

System.ServiceModel.Security.SecurityNegotiationException: The caller was not

authenticated by the service. ---> System.ServiceModel.FaultException: The

request for security token could not be satisfied because authentication failed.

40




Root Cause analysis: Misconfiguration

41

• The DDC was not authorized the initiate a connection to the VDA

• “Access To Computer From The Network” Computer Policy did not have an entry for the Controlled and the default everyone was removed in production.

Resolution: Customer added explicit entry to a Group that included all the Brokers as members


• PVD maintains logs in the base of the volume attached to the VM◦ (alongside the VHD containing the PVD user-installed applications)

• These logs contain a wealth of information that should be captured and provided to support/engineering if you experience problems

• Most frequently seen PVD support cases …◦ Failure of PVD to start virtualization (PVD can’t locate volume/VHD, etc.. …)◦ Customers trying to install unsupported apps ◦ Customers trying to move PVDs between VMs

Troubleshooting and Support

© 2012 Citrix | Confidential – Do Not Distribute 43

• Desktop Director has helpdesk-facing PVD metrics and support◦ % of application area in use / total size◦ % of user profile area in use / total size◦ PVD reset

• PVD reset allows the helpdesk to reset the application area while leaving the user’s data intact◦ Aka “revert to factory default”◦ Useful to reset PVDs that become wedged due to users installing broken applications

Troubleshooting and Support (cont’d)

VDA Launch


VDA Launch

Controller #1

BrokerService

DDC

VDA

Desktop Service

VDA

WI

Idle

Launch Request

SQL

WCF

XML broker queries DB for a ready worker

Broker signals worker to Prepare

for a SessionUser Clicks to launch session

ICA Service

PolicySettings

Preparing New Session


VDA Launch (cont’d)

Controller #1

BrokerService

DDC

VDA

Desktop Service

VDA

WI

SQL

WCF

Work State: ActiveWork State: Connected

Request to Validate Ticket sent Controller

ICA file is sent to Endpoint

ICA Service

PolicySettings

1. Validates Ticket 2. Validates License3. Policies

Ticket is ValidAuthNTicket

Connected

Portica gets License

Active


What Happened ?


Troubleshooting VDA Launch

• Event Logs (Web Interface, Controller, Storefront)

• Desktop Studio

• Broker Logs

• Workstation Agent

• Portica Logs

• Network Packet tracing

Case Study 3VDA Launch


Case Study Walk Through Background:

• They recently converted all images to a Citrix PVS image

• The original image worked

• All streamed images including the golden image failed to launch

Case Study 3: Launch Failure 1030


Search: Prepare


Troubleshooting :VDA Launch• Search Strings:

Checkpoint|connectionaccept|WaitforincomingConnection|sessionicaconnect


Root Cause analysis: MFAphook Module Failed to Load

54

• Conversion via provisioning server had changes the long name format of the drive

• mfaphook failed to load and this is needed for interaction with the OS.

Resolution: Add back short name to system see CTX133773 for more information


XD Tools

• HDX Monitor• CDF Control• Citrix Scout• Site Checker• Desktop Director


HDX Monitor

• Thinwire (Graphics)

• Direct 3D (Graphics)

• Media Stream (aka RAVE)

• Flash

• Audio

• USB Devices


HDX Monitor

• Mapped Client Drives (CDM)

• Branch Repeater

• Printer

• Client

• Smart Card

• Scanner

• System


Citrix Scout / XD Collector (CTX130147)

59

• Push button easy data collection system

• Makes data collection and upload push button easy

• Integrates data collected by Scout with the Citrix Tools as a Service

(TaaS) backend

• Simplifies data collection & analysis

© 2012 Citrix | Confidential – Do Not Distribute#CitrixSummit

CDF Control: CTX111961

Tip:

• Use this tool to remotely

enable and collect CDF

traces when system are non

persistent

60


Site Checker Tool: CTX133767

• Enumerate Environment• Checks Services Status • Checks service instances registration status

• Reset Controllers Services instances into Database

© 2012 Citrix | Confidential – Do Not Distribute62

Desktop Director

• Web Based

• Unified view of apps and desktops

• End-user details empower the help desk

• Includes HDX Monitor

• Access to personal vDisk tasks

Resources discussed


Optimal deployment recommendations• CTX124087 - XenDesktop Modular Reference Architecture

• CTX127939 - XenDesktop 5 Database Sizing and Mirroring Best Practices

• CTX123244 - High Availability for Desktop Virtualization - Reference

Architecture

• CTX120760 - XenDesktop - Design Handbook

• CTX128700 - XenDesktop Planning Guide - XenDesktop Scalability

• Whitepaper - Benchmarking Citrix XenDesktop using Login Consultants VSI

64

http://support.citrix.com/article/CTX124087





http://support.citrix.com/servlet/KbServlet/download/24130-102-647344/XenDesktop%20with%20Login%20Consultants%20VSI%20-%20Guide.pdf


For More Information

• CTX132536 - Worker Unregisters at Session Launch

• CTX130147 - Citrix Scout

• CTX111961 - CDFControl

• CTX127492 - How to enable Controller Service Logging in XenDesktop 5

• CTX128075 - XDDBDiag: XenDesktop 5 Database Diagnostics

• CTX128909 - XenDesktop 5 Logon Process and Communication Flow

65








For More Information

• Vmware – Using VMware with XenDesktop

• SCVMM Using Microsoft SCVMM 2008 with XenDesktop

• CTX127538: How to Reconfigure a XenDesktop Site to Use a Mirrored

Database

• CTX127998 : Database Access and Permission Model for XenDesktop 5CTX133160 - LSQuery - License Server Data Collection Tool CTX127314 - How to Collect Data for Troubleshooting Licensing Issues

66

http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-vmware-rho.html

http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-scvmm-rho.html





Takeaways


Presentation Goals Recap

68

Provide an understanding of the

architecture

Instruct On How To Configure

Provide Troubleshooting Resources


AboutCitrix Services

Citrix Services make sureyou succeed with yourvirtualization programs.

How we can help

Citrix Education – The fastest, most efficient way toget your team the virtualization skills they need. Online,on-site or in class.citrix.com/training

Citrix Consulting – Intensive engagements forcomplex, critical or just plain massive projects.citrix.com/consulting

Citrix Support – Always-on support services thatleverage everything we know about best-practicedeployment and maintenance.citrix.com/support

Educate | Guide | Support | Succeed

http://citrix.com/training

http://citrix.com/training

http://www.citrix.com/support/consulting.html

http://www.citrix.com/support/consulting.html

http://citrix.com/support

http://citrix.com/support


• 40 insider troubleshooting tips

• Covering XenDesktop, XenServer, XenApp and NetScaler

• Citrix Support top engineers

• FREE eBook

• Citrix Auto Support

• Now available!

Secrets of the Citrix Support Ninjas

http://deliver.citrix.com/WWEL1212SUPTNINJASEBOOK


Premier Support Calculator

Check it out

http://www.citrix.com/pscalculator


Next Webinar: September• Title: Troubleshooting a XenDesktop environment using the PowerShell SDK

• Description: The Citrix XenDesktop PowerShell SDK is the foundation for all interactions with a XenDesktop database and is the same SDK used by Desktop Studio.

• This deep dive session will include a behind-the-scenes look at several tools used by Citrix Technical Support that utilize the PowerShell SDK, including common configuration cmdlets and scripts.

• When: Sept 26th

Registration Now!

https://www1.gotomeeting.com/register/528989824

Work better. Live better.Work better. Live better.