Using XPERF to troubleshoot slow logons

Installing XPERF to capture a slow boot or logon trace

1. Install XPERF from the Windows SDK for Windows 7 and .NET Framework on the slow boot or logon computer.
   Hint 1: It is possible to install only the Windows Performance Toolkit from the Windows SDK.
   Hint 2: I suggest installing the WPT in an X:\XPERF directory rather than the default directory recommended by setup. It’s easier to access and copy files in and out of, and change paths, to the short-labeled directory.
   Hint 3: Once installed on a computer, the XPERF installation directory can be copied to other computers that you want to capture ETL traces from or view ETL traces on. There are no external files, DLL registration or registry changes required to make or view a capture. Make a copy of the X:\XPERF directory and copy at will.
2. If taking a network trace on a 64-bit computer, enable the following registry key and reboot before capturing ETL data. This prevents kernel mode data from being paged out of memory.

Registry Path	HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
Setting	DisablePagingExecutive
Data Type:	REG_DWORD
Value:	1

Using XBOOTMGR to capture slow boots, or slow logons caused by slow boots

1. Logon as an Administrator of the computer you want to trace (either a local Administrator or Domain Admin account that is a member of the local machine’s Administrators group).
2. Open an elevated command prompt.
3. Run the following command in the WPT directory (default path is C:\Program Files\Microsoft Windows Performance Toolkit). This syntax is useful to capture slow boots as well as slow logons thought to be caused by a delay in OS startup:

   xbootmgr -trace boot -traceflags base+latency+dispatcher -stackwalk profile+cswitch+readythread -notraceflagsinfilename -postbootdelay 10       

   This command will:

• Reboot the local computer
• Capture ETL tracing during the boot and logon operation (you provide user name, domain name, and password for the slow logon account)
• Stop tracing at 10 seconds after disk and CPU utilization fall below a certain threshold after user logon. Increase the value for “-postbootdelay” as required to troubleshoot user desktops that are unresponsive to mouse and keyboard input post boot.

Using XPERF to capture slow logons

1. Logon as an Administrator of the computer you want to trace (either a local Administrator or Domain Admin account that is a member of the local machine’s Administrators group).
2. Open an elevated command prompt and run this command from WPT Install directory (default path is C:\Program Files\Microsoft Windows Performance Toolkit.                                                                                                                                                                                                                                                                                                  xperf -on base+latency+dispatcher+NetworkTrace+Registry+FileIO -stackWalk CSwitch+ReadyThread+ThreadCreate+Profile -BufferSize 128 -start UserTrace -on “Microsoft-Windows-Shell-Core+Microsoft-Windows-Wininit+Microsoft-Windows-Folder Redirection+Microsoft-Windows-User Profiles Service+Microsoft-Windows-GroupPolicy+Microsoft-Windows-Winlogon+Microsoft-Windows-Security-Kerberos+Microsoft-Windows-User Profiles General+e5ba83f6-07d0-46b1-8bc7-7e669a1d31dc+63b530f8-29c9-4880-a5b4-b8179096e7b8+2f07e2ee-15db-40f1-90ef-9d7ba282188a”  -BufferSize 1024 -MinBuffers 64 -MaxBuffers 128 -MaxFile 1024
3. Note: This syntax works on Windows Vista (Windows Server 2008) and Windows 7 (Windows Server 2008 R2) computers
4. Press CTRL+ALT+DEL and then Switch User.
5. Logon with the user account experiencing the slow user logon to reproduce the issue.
6. Stop the trace. While logged on with the slow user account, open an elevated CMD prompt and type:

xperf -stop -stop UserTrace -d merged.etl

Close the slow logon user session and the admin logon session opened in step 2 as required.

IMPORTANT: The double “-stop” call in step 5 is not a typo but is required. The first “-stop” terminates kernel tracing. The second “-stop” terminates user mode tracing.

Note: You can also stop the trace by using Switch User to return to the admin user logon established in step #2 and running the same XPERF stop command in the elevated command prompt used to start the trace. This results in a larger, longer trace and requires that you discern the different logons encapsulated in the trace.

1. Send the MERGED.ETL file to Microsoft or an Independent Solution Vendor (ISV) for analysis, or review it yourself.By default, the MERGED.ETL file will exist in the XPERF installation directory, which by default is %systemdrive%\program files\microsoft windows performance toolkit directory or if you followed our recommendations early in the doc, the c:\XPERF directory (that is, the XPERF installation directory).

Double Click the merged.etl file to open it in Windows Performance Analyzer ( Ensure that you have the Windows Performance Toolkit Installed on the computer that you are opening the file from)

I am now going to talk about one of our customer environments where that had a Citrix XenApp 6.5 deployment and the user logons used to be around 90 sec. I agree that this isn’t such a bad situation to be as I have seen even worser logon times.

The customer environment consists of the below

• Citrix XenApp 6.5 with HRP 02
• EdgeSight Server with agent version 5.4 loaded on the servers
• StoreFront 1.2
• Group Policies applied with login scripts and policy preferences
• ESET NOD 32 Antivirus

The users complain that the screen will sit at “Welcome” screen for approx 30 sec after which the logon moves faster and towards the end , there is a further delay after “Preparing the Desktop”

First things first, during these situations AV will be the first to be blamed. Decided to stop the AV service and turned off Real Time scanning to rule that out. Tried to login with a test account and it is the same 90 +secs

Tools at my disposal ( the customer has got a platinum XenApp License, so they own a lot of good stuff) 🙂

• EdgeSight Reports – showed no problems whatsoever with the Group Policy processing, logon scripts or printer mapping scripts. The issue seems to be well before the GP processing and scripts are run. What else is there to look at?
• Citrix UPM logs – Citrix User Profile Manager has logs of its own and i decided to take a look at it, there was nothing obvious with it other than the little hint it could give towards the end of the logon process
• Autoruns – This gave me a hint at what could be going wrong towards the end of the logon process. It was an edocs print pro process. Turned that off and tried a logon. Bingo, the final 15 seconds of desktop preparation has gone away.
• XPerf – Analyzing the XPerf logs also pointed out issue with the “edocs Print Pro” software printer. I just can’t find the screenshots of my XPerf analysis anymore as this happened a while ago and I just couldn’t write about it on time.

 

XPerf is a super cool tool to troubleshoot slow logon/boot up times alongside Citrix UPM logs. Autoruns does help to selectively stop all or some logon processes and thus should not be overlooked when troubleshooting long logon times. I will try to include more XPerf tracing and analysis in my future blog posts. I will talk to you in my next post.