XenApp & XenDesktop 7.x – Logon delay of 20 seconds at ” Please wait for Citrix User Profile Manager”


If you ever have a situation where your Citrix logons sits on “Please wait for Citrix User Profile Manager” for around 20 – 25 sec, you have come to the right post. I have had this strange one where users started to see their logons taking longer than usual. Below is my setup and your results may vary depending on the UPM version that you are running

  • XenApp 7.5 site
  • Server 2012 R2 VDAs
  • Citrix UPM 5.1
  • McAfee AV 8.8

Issue Manifestation – Delay of 20-25 seconds on ” Please wait for Citrix User Profile Manager” and this was consistent.

Remediation – Turn ON Citrix Profile Streaming

I am partially to blame for this issue as I had turned OFF Profile Streaming when I had my users complaining that the logoff was taking a little while. Now i need to figure out what is causing the logoffs to take roughly 10 seconds to complete. Already looked at the AV side of things and i have the required Exclusions. Hunt continues and will update with the findings..

So that’s a quick one there for you guys and hopefully someone will find it useful. If you find it working/not working for you, let me know

Citrix XenApp – Long logon times and potential fixes


Long login times are something that we have been hearing from time to time working with Citrix XenApp /XenDesktop environments. I have had a similar issue recently for one of my deployments with XenApp 7.5 on Windows Server 2012 R2 workloads. My logon times averaged around 30 seconds which is not bad at all. I still wanted to make it better and my target was to bring it under 15 seconds 🙂

Below are a few things that you can do to reduce the logon times. Please note that this is not a comprehensive list so feel free to comment below with your findings on this post so that i can update it and make the list better.

If you have Citrix Director in your environment, that would be the first place to look. It gives you in-depth details on where the profile load takes longer so that you can focus on those areas first.

I had Citrix Director in the environment and looking at it, Interactive Session seems to take a major chunk of the overall login time.

  1. Anti-Virus – This is one item that is overlooked often so ensure that you have set the required exclusions for your AV product. I would even go head and recommend turning OFF Real-time scanning for MCS/PVS created images as they are only read only. Please ensure that you run Real-time scanning on the network shares that hosts the profiles/home folders and also on the Write Cache location in case of PVS images.
  2. Enable Legacy Graphics Mode –This is a Citrix policy and enabling this is found to increase the logon speed. This is Adaptive Display First Generation which is good on older operating systems like Windows Server 2008 R2. Not recommended to be enabled in Windows Server 2012 R2 as it is found to cause some/all applications to fail consistently or randomly. In short, apply this setting with caution if your workloads are Windows Server 2012 R2 / Windows 8.x
  3. Remove CD-ROM drives from your virtual Citrix servers – May sound silly but having a CD-Drive on the server is found to increase the logon time.
  4. Active Setup – My suggestion is to check the Active Setup on the Citrix servers. I use SysInternals Autoruns tool to disable (not delete) the unwanted Active Setup keys under Installed components for HKLM\SOFTWARE\Microsoft\Active Setup\Installed Components and HKLM\SOFTWARE\Wow6432Node\Microsoft\Active Setup\Installed Components  as well.
  5. Autoruns This is a brilliant tool from SysInternals and throws a lot of light into what runs when a user logs in to a Windows Server. Run this and disable all that is not required for your environment.
  6. Internet Explorer – Uncheck In Internet Explorer Options Advanced -> Security,  disable “Check for publisher´s certificate revocation” and “Check for Server Certificate Revocation
  7. DisableStatus registry – Again apply this fix with caution as this is found to introduce the blue login screen(Windows GINA) when accessing applications which is not ideal. http://support.citrix.com/article/CTX135782 . Some have reported to have reduce the login times by doing this.
  8. Citrix UPM Profile Streaming – Profile Streaming is sometimes found to adversely affect the logon times especially when McAfee is used. Turn OFF UPM Profile Streaming completely to see if it makes any difference.
  9. Themes Key in Active Setup – Remove the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{2C7339CF-2B09-4501-B3F3-F3508C9228ED}. Make sure that the key is removed for the user profile as well under HKCU
  10. Receiver version – Use the latest stable version of Receiver on the client devices. Running the latest version on the server side will help in launching published application quickly.
  11. Drive Mappings – It could either be via logon scripts or via GPPs. Citrix Director can easily show you if this is the case so that you can focus on the right area from the beginning itself.
  12. Printer Mappings – Same as drive mappings. the GPPs should be set to move on if it errors and not wait for it and time out.
  13. Group Policy Processing errors – Look in the EventViewer for any potential policy processing errors and fix them.
  14. Default delay of 5-10 seconds for VDAs based on Windows 8.x and Server 2012 – Microsoft introduced a delay of 5-10 secs for operating systems starting from Windows 8 and hence this does apply to Server 2012 OSes as well. To remove the delay, add the registry value StartupDelayInMSec (REG_DWORD) to 0 in HKEY_CURRENT_USER\Software\Microsoft\Windows \CurrentVersion\Explorer\Serialize   (You can add the key “Serialize” if not present already). This will greatly reduce the “interactive logon” delays
  15. Last but not the least (this should have been higher up in the order), check the size of user profiles and find out what is causing the profile bloat. In most cases, publishing Google Chrome and Firefox is one of the most common causes of large profile sizes. It is recommended to exclude the whole of \AppData\Local\Google\Chrome and just have the per-requisite files/ folder synchronized using UPM policy. I would start with the below synchronization list for Chrome
AppData\Local\Google\Chrome\User Data\First Run
AppData\Local\Google\Chrome\User Data\Local State
AppData\Local\Google\Chrome\User Data\Default\Bookmarks
AppData\Local\Google\Chrome\User Data\Default\Favicons
AppData\Local\Google\Chrome\User Data\Default\History
AppData\Local\Google\Chrome\User Data\Default\Preferences

You can find more about Chrome and Firefox exclusion and synchronization policies here

16. Enable the Microsoft policy “Set maximum wait time for the network if a user has a roaming user profile or remote home directory” and set the value to 0. The policy could be found under Computer Configuration – Policies – Administrative Templates – System – User Profiles

17. Check the Citrix KB here – http://support.citrix.com/article/CTX133595/

18. In case you are using VDA versions below 7.14.1 (7.14 excluded), Check this key out

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Run

Under the Run key, you are most likely to find the below entry

30

If you see this entry, that is why your Interactive Logon times are too high. You will need to delete that key and add the same key under here

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon

Find the Userinit REG_SZ value and append the value as in the screenshot below.

31

Please note that you will need to use a trailing comma at the end of the line, else your applications wouldn’t launch.

In my case it turned out to be the UPM process UpmUserMsg.exe and it made a massive difference to the login times. In newer VDA versions, this issue isn’t there and the UPM process has been renamed to UpmEvent.exe.

19. Citrix Workspace Environment Manager (WEM)This is a must have for any Citrix deployments; Helps speed logon times, provide system optimization, manage user profiles, user personalization and much more. There is absolutely no reason not to deploy this in your environment.

There is another fantastic article out there on XenAppBlog

That’s an interesting read as well. i will continue to update the post as I find out more and please feel free to post your comments below.

Using XPERF to troubleshoot slow logons


Installing XPERF to capture a slow boot or logon trace

  1. Install XPERF from the Windows SDK for Windows 7 and .NET Framework on the slow boot or logon computer.
    Hint 1: It is possible to install only the Windows Performance Toolkit from the Windows SDK.
    Hint 2: I suggest installing the WPT in an X:\XPERF directory rather than the default directory recommended by setup. It’s easier to access and copy files in and out of, and change paths, to the short-labeled directory.
    Hint 3: Once installed on a computer, the XPERF installation directory can be copied to other computers that you want to capture ETL traces from or view ETL traces on. There are no external files, DLL registration or registry changes required to make or view a capture. Make a copy of the X:\XPERF directory and copy at will.
  2. If taking a network trace on a 64-bit computer, enable the following registry key and reboot before capturing ETL data. This prevents kernel mode data from being paged out of memory.
Registry Path HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management
Setting DisablePagingExecutive
Data Type: REG_DWORD
Value: 1

Using XBOOTMGR to capture slow boots, or slow logons caused by slow boots

  1. Logon as an Administrator of the computer you want to trace (either a local Administrator or Domain Admin account that is a member of the local machine’s Administrators group).
  2. Open an elevated command prompt.
  3. Run the following command in the WPT directory (default path is C:\Program Files\Microsoft Windows Performance Toolkit). This syntax is useful to capture slow boots as well as slow logons thought to be caused by a delay in OS startup:
    xbootmgr -trace boot -traceflags base+latency+dispatcher -stackwalk profile+cswitch+readythread -notraceflagsinfilename -postbootdelay 10       

    This command will:

  • Reboot the local computer
  • Capture ETL tracing during the boot and logon operation (you provide user name, domain name, and password for the slow logon account)
  • Stop tracing at 10 seconds after disk and CPU utilization fall below a certain threshold after user logon. Increase the value for “-postbootdelay” as required to troubleshoot user desktops that are unresponsive to mouse and keyboard input post boot.

Using XPERF to capture slow logons

  1. Logon as an Administrator of the computer you want to trace (either a local Administrator or Domain Admin account that is a member of the local machine’s Administrators group).
  2. Open an elevated command prompt and run this command from WPT Install directory (default path is C:\Program Files\Microsoft Windows Performance Toolkit.                                                                                                                                                                                                                                                                                                  xperf -on base+latency+dispatcher+NetworkTrace+Registry+FileIO -stackWalk CSwitch+ReadyThread+ThreadCreate+Profile -BufferSize 128 -start UserTrace -on “Microsoft-Windows-Shell-Core+Microsoft-Windows-Wininit+Microsoft-Windows-Folder Redirection+Microsoft-Windows-User Profiles Service+Microsoft-Windows-GroupPolicy+Microsoft-Windows-Winlogon+Microsoft-Windows-Security-Kerberos+Microsoft-Windows-User Profiles General+e5ba83f6-07d0-46b1-8bc7-7e669a1d31dc+63b530f8-29c9-4880-a5b4-b8179096e7b8+2f07e2ee-15db-40f1-90ef-9d7ba282188a”  -BufferSize 1024 -MinBuffers 64 -MaxBuffers 128 -MaxFile 1024
  3. Note: This syntax works on Windows Vista (Windows Server 2008) and Windows 7 (Windows Server 2008 R2) computers
  4. Press CTRL+ALT+DEL and then Switch User.
  5. Logon with the user account experiencing the slow user logon to reproduce the issue.
  6. Stop the trace. While logged on with the slow user account, open an elevated CMD prompt and type:
xperf -stop -stop UserTrace -d merged.etl

Close the slow logon user session and the admin logon session opened in step 2 as required.

IMPORTANT: The double “-stop” call in step 5 is not a typo but is required. The first “-stop” terminates kernel tracing. The second “-stop” terminates user mode tracing.

Note: You can also stop the trace by using Switch User to return to the admin user logon established in step #2 and running the same XPERF stop command in the elevated command prompt used to start the trace. This results in a larger, longer trace and requires that you discern the different logons encapsulated in the trace.

  1. Send the MERGED.ETL file to Microsoft or an Independent Solution Vendor (ISV) for analysis, or review it yourself.By default, the MERGED.ETL file will exist in the XPERF installation directory, which by default is %systemdrive%\program files\microsoft windows performance toolkit directory or if you followed our recommendations early in the doc, the c:\XPERF directory (that is, the XPERF installation directory).

Double Click the merged.etl file to open it in Windows Performance Analyzer ( Ensure that you have the Windows Performance Toolkit Installed on the computer that you are opening the file from)

I am now going to talk about one of our customer environments where that had a Citrix XenApp 6.5 deployment and the user logons used to be around 90 sec. I agree that this isn’t such a bad situation to be as I have seen even worser logon times.

The customer environment consists of the below

  • Citrix XenApp 6.5 with HRP 02
  • EdgeSight Server with agent version 5.4 loaded on the servers
  • StoreFront 1.2
  • Group Policies applied with login scripts and policy preferences
  • ESET NOD 32 Antivirus

The users complain that the screen will sit at “Welcome” screen for approx 30 sec after which the logon moves faster and towards the end , there is a further delay after “Preparing the Desktop”

First things first, during these situations AV will be the first to be blamed. Decided to stop the AV service and turned off Real Time scanning to rule that out. Tried to login with a test account and it is the same 90 +secs

Tools at my disposal ( the customer has got a platinum XenApp License, so they own a lot of good stuff) 🙂

  • EdgeSight Reports – showed no problems whatsoever with the Group Policy processing, logon scripts or printer mapping scripts. The issue seems to be well before the GP processing and scripts are run. What else is there to look at?
  • Citrix UPM logs – Citrix User Profile Manager has logs of its own and i decided to take a look at it, there was nothing obvious with it other than the little hint it could give towards the end of the logon process
  • Autoruns – This gave me a hint at what could be going wrong towards the end of the logon process. It was an edocs print pro process. Turned that off and tried a logon. Bingo, the final 15 seconds of desktop preparation has gone away.
  • XPerf – Analyzing the XPerf logs also pointed out issue with the “edocs Print Pro” software printer. I just can’t find the screenshots of my XPerf analysis anymore as this happened a while ago and I just couldn’t write about it on time.

 

XPerf is a super cool tool to troubleshoot slow logon/boot up times alongside Citrix UPM logs. Autoruns does help to selectively stop all or some logon processes and thus should not be overlooked when troubleshooting long logon times. I will try to include more XPerf tracing and analysis in my future blog posts. I will talk to you in my next post.

XPERF – All that you would need to troubleshoot slow startups and logon times in Windows


I am going to talk something that I got really excited about as a tool that you must have if you are into IT support.

The tool is a Microsoft built tool named XPERF.

Xperf is an advanced troubleshooting tool that digs deeper into performance issues than Perfmon ever dreamed of. It is part of a product suite called the Windows Performance Toolkit (WPT), which is part of the Microsoft Software Development Kit (SDK). Xperf leverages Event Tracing for Windows (ETW), a built-in component of the server operating system that provides detailed performance and system data.

The Xperf utility excels in troubleshooting areas like responsiveness issues, slow boot times, high CPU and disk usage, application delays and slow transitions. It’s supported across x86, x64 and IA-64 architectures with less than 2.5% CPU overhead while gathering over 20,000 events per second.

In other words, this is one tool you’ll definitely want to add to your toolbox. But before we get into the many features Xperf has to offer, let’s first go through a short tutorial of how to install and use the tool for troubleshooting Windows performance issues.

Installing Xperf
As previously mentioned, the Windows Performance Toolkit is distributed with the Windows Software Development Kit. To install the SDK, however, you must first install Microsoft .NET Framework 4. Unless you want to install the entire 4 GB SDK, you should only select the Windows Performance Toolkit and the Debugging Tools for Windows options, as shown in Figure 1.

You will also want to select Redistributable Packages for the WPT and Debugging Tools options so that individual kits are available for future installations without forcing you to reinstall the SDK. The resulting Windows Performance Toolkit and debugging kits can be found in:

C:\Program Files\Microsoft SDKs\Windows\v7.1\Redist\Windows Performance Toolkit

Figure 1: Installation options for Windows SDK (click to enlarge)
Installation options for Windows SDK

If you run into any problems during the SDK installation, you can refer to this troubleshooting page for assistance with things like clearing the AppData\Local\Temp folder (which I had to do). After the installation, a new program group will be created for the WPT, as shown in Figure 2. The corresponding folder for the tools will be found in:

C:\Program Files\Microsoft Windows Performance Toolkit

Figure 2: Windows Performance Toolkit program group
Windows Performance Toolkit program group

Leveraging and using xPerf
Once you have WPT (Xperf) installed on the troublesome server, the next thing to do is start a collection of event tracing. There are numerous potential events that can be traced depending on the issue you are troubleshooting. A good place to start is with the Xperf CLI command:

Xperf –on DiagEasy

This will start a collection of events with several kernel providers, including PROC_THREAD, LOADER, DISK_IO, HARD_FAULTS, DPC, INTERRUPT, CSWITCH and PERF_COUNTER, which log the data to an event trace log file called kernel.etl. For a list of all kernel providers, issue this command:

Xperf –providers KG

Note: Extensive online help is available that explains each provider in detail.

Once you have started the collection, allow the server to exhibit the behavior you are troubleshooting and then stop the collection with this command:

Xperf –d trace.etl

This will merge the data that was buffered in kernel.etl and create a final trace file called trace.etl. This trace file can be analyzed on the server where the problem is occurring or on a different server where WPT is also installed. To view the resulting graphs and charts, use this command:

Xperf trace.etl

The trace file will be processed in two passes, producing a variety of graphs. The default graphs include:

  • CPU usage by CPU
  • CPU usage by process
  • CPU usage by thread
  • CPU idle states
  • Disk I/O, Disk utilization
  • Disk utilization by process
  • Process lifetimes
  • DPC CPU usage
  • Interrupt CPU usage
  • Hard faults
  • Generic events

By hovering your mouse pointer over a particular line on the graph, additional details will be provided such as the name of the process that is accumulating all the CPU time. For instance, in Figure 3 you can clearly see how the antivirus process Rtvscan.exe is dominating the CPU usage.

Figure 3: CPU Usage by Process graph (click to enlarge)
CPU Usage by Process graph

Another nice benefit of using Xperf is the ability to zoom in on a particular time range to take a closer look at the graph. In Figure 4, you can see the Disk Utilization graph shows a spike in activity after the first 60 seconds. Zoom in on the time period in question by using the mouse pointer to click and drag over that area of the graph, then right-click and choose Zoom To Selection. This will produce another graph for just the time period you specified. To revert to the original graph, right-click the graph and select Unzoom.

Figure 4: Xperf zoom feature (click to enlarge)
Xperf zoom feature

To get even more detailed, you can right-click a graph and select Summary Table. This will produce a chart that documents all the corresponding counters and data associated with the graph. For instance, Figure 5 illustrates the Disk Summary Table for the Disk I/O graph. Line 1 of the chart clearly shows how the Rtvscan.exe process is accumulating massive read counts, which would be expected for an antivirus scan.

Figure 5: Disk Summary Table for the Disk I/O graph (click to enlarge)
Disk Summary Table for Disk I/O graph

As you can see, Xperf is a very powerful and intuitive tool that can really help you drill down on performance issues. I have just touched the tip of the iceberg with regards to Xperf’s capabilities, however, as the tool is also great for slow boot investigation, high kernel and interrupt times and hot file analysis. Each of these areas alone deserves its own discussion of the features and benefits.