Don’t let your user-experience be a “Spectre” of itself after “Meltdown”


Bust your ghosts not your user experience

The names Spectre and Meltdown invoke feelings of dread in even the most seasoned IT engineer.  To those uninitiated, let me get you up-to-speed quickly.

Spectre is a vulnerability that takes advantage of “Intel Privilege Escalation and Speculative Execution”, and exposes user memory of an application to another malicious application.  This can expose data such as passwords.

Meltdown is a vulnerability that takes advantage of “Branch prediction and Speculative Execution”, and exposes kernel memory.  A compromised server or client OS running virtualized could gain access to kernel memory of the host exposing all guest data.

Both vulnerabilities take advantage of a 20-year-old method of increasing processor performance.

Server_Protection

As a result, code will need to be updated to address these vulnerabilities at OS and OEM-manufacturer levels, at the expense of system performance.

On their part, Microsoft reluctantly admits that performance will suffer.  “Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance,” wrote Terry Myerson, Executive Vice President for the Windows and Devices group.

According to Geek Wire, these two vulnerabilities which take advantage of a 20-year-old design flaw in modern processors can be “mitigated;” the word we’re apparently using to describe this new world in 2018, in which servers lose roughly 10 to 20% performance for several common workloads.

This affects not only workloads executed against local, on-site resources but even those utilizing services, such as AWS, Google Public Cloud or Azure.

cpu_utilReader submission @ The Register showing CPU before / after patches

We’ve heard from some of our insiders who use Login VSI to validate system performance that they’re seeing a reduction of 5% in user-density after performing Microsoft recommendations. Knowing that the vulnerability wasn’t solved by OS updates alone we, at Login VSI, wanted the ability to test the impending hardware vendor firmware / BIOS changes.

Now is the time to capture your baseline performance

How do you know how much of an impact the fixes for Spectre and Meltdown will be if you don’t have anything to compare it to? Keep in mind that these patches will need to be installed on a number of systems in your solution including server hardware, operating systems, storage subsystems and so on.

Many of our customers perform tests where they compare a known good solution, or a baseline, with changes that have been made. This gives them the ability to accurately assess the performance impact of that change, which in turn allows them to compensate with more hardware, or further tuning of the applications and OS. The patented methods used by Login VSI provide a quantifiable result for determining the impact of a change in virtual desktop and published application environments.

Using Login VSI

If you wish to test the changes before pushing them into your production environment, then use Login VSI to put a load, representative of your production users, on the system. This will objectively show how much more CPU will be used as a result of the Spectre or Meltdown patches. It is expected that the end users will incur increased latency to their applications and desktops as a result of the higher CPU utilization.

Using Login PI

While it is not recommended, if you are planning on pushing the patches into your production environment to “see how it goes”, then install Login PI now to get an accurate representation of performance related to user experience. This will give you the ability to then compare to that same experience after the patches have been installed. We expect that you will see latency to the end user increase as a result of higher CPU utilization. If you already struggle with CPU utilization in your solution, there is a good chance you’ll be also using Login PI to test your availability.

As we complete our testing we will be sharing our findings in a series of articles.

If your computer has a vulnerable processor and runs an unpatched operating system, it is NOT SAFE TO WORK WITH SENSITIVE INFORMATION”. – Security Experts who discovered Meltdown / Spectre 

If sensitive data is part of your business (Such as ours!) patching is not a matter of if, but when.

Ask yourself:

How long can you afford to have your company’s data exposed to malicious intent?  Do you want to be the next Equifax or Target?

In this article series, we will provide some insight from our lab environments. Be aware your results may vary based upon individual workload and configuration.

Microsoft has released a Security Advisory

The vulnerability affects both the client and server OSs of Windows.  This is compounded when dealing with large-scale published application and desktops deployments.  The advisor can be found at the following location:

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180002

The specific details addressed in the security update and Windows KB are outlined in the Common Vulnerabilities and Exposures database.

Included are:

To completely protect yourself there are two phases of patching this vulnerability.

1 – Windows OS updates

2 – OEM device manufacturer firmware updates (not yet available)

Microsoft acknowledges addressing these vulnerabilities from a software perspective is limited, and therefore, without the OEMs providing updates the loop is not closed.

In the interim we can start measuring the impact of the Microsoft fixes.

They offer guidance for both Desktop and Server OSs:

Desktop –  January 2018 Security Update. Security Advisory: Click Here!

Server –  KB405690. Security Advisory: Click Here!

NOTE – Certain AV solutions are not compatible with the security update released by Microsoft. As such, unless an AV vendor has a registry flag, QualityCompat, they will not receive the January Security update and will still be vulnerable

With the upcoming OEM hardware patch releases we expect to be able to produce a variety of interesting and informative results.  Please stay tuned for the next articles!

Reference materials:

https://meltdownattack.com/

https://www.theregister.co.uk/2018/01/09/meltdown_spectre_slowdown/

https://www.geekwire.com/2018/microsoft-admits-meltdown-spectre-patches-will-hit-windows-server-performance/

Advertisements

Issue with Microsoft Outlook 2013 reconfiguring on every launch – XenDesktop / XenApp


In one of the recent XenDesktop 7 deployments, I was told that Outlook 2013 doesn’t launch well on Server 2012 Hosted Shared Catalogs by one of the end users.  Every time Outlook is started, it tries to reconfigure itself as if the Outlook profile isn’t there or it cant find the profile. This caused the Outlook plugins or add ins to fail giving errors. My customer setup was below

  • Outlook 2013 Professional Plus
  • XenDesktop 7.0
  • Server 2012 Hosted Shared Catalogs with Desktops and Apps

Resolution – I came across a Microsoft blog which was talking about how important “Windows Search” Service is for the full functionality of Outlook. I decided to give that a try and started the service and viola, the issue is gone and Outlook launches way faster after that.

 

This might be the same case for older versions of Outlook  and other XenApp versions. i haven’t tested them yet so cant comment.

 

Open File Security Warning – Enable or Disable


Have you seen the below message appearing on your Citrix servers and just wished you could turned that OFF. I did, not once but quite a few times that I have decided to document it so that I can quickly revisit the page when I deploy Citrix next time.

The annoying error looks like the below and it is plain ugly to say the least.

Capture27

 

I tried a few things that Internet had to offer but nothing worked for me until I tried a combination of the below 2 GPOs

Capture28

Capture29

 

The GPOs could be found here

User Configuration - Administrative Templates - Windows Components - Internet Explorer - Internet Control Panel - Security Page
  • Intranet Sites: Include all local (intranet) sites not listed in other zones
  • Intranet Sites: Include all network paths (UNCs)

 

The popup should now go away!

microsoft office can’t find your license for this application. A repair attempt was unsuccessful or was cancelled. microsoft office will now exit – Resolved


Users getting the below error message while working in Office/Outlook. In our case, it was Outlook 2013 client.

microsoft office can’t find your license for this application.  A repair attempt was unsuccessful or was cancelled.  microsoft office will now exit

Capture5

 

Fix is to start/restart the SOFTWARE PROTECTION service on the servers/clients from where Office is run.

In my case, it was a XenDesktop Hosted Shared catalog and faulty servers exhibited the below.

Capture4

 

All I have to do is to restart the Software Protection service on the servers. the executable is sppsvc.exe. I had to open task manager to kill the exe before i could restart the service.

 

Microsoft Office 2013 outlook cannot provide form scripting support. This feature is not available. For more information, contact your system administrator


I have been asked to look into an issue with Outlook published in Citrix when users started getting error when they access files and folders within Outlook 2013 client. I have seen this happening before for older Outlook clients running on Citrix but never bothered to document it. But when I had the issue this time, it look me quite a few hours to figure out the correct GUID for MS Office 2013.

Firstly, I asked Google for the answer and there are tons of article explaining the fix but none of them worked for me as the GUID was incorrect in my case. Most of these articles are written for Outlook 2010 and below.

I went out hunting for the GUID and I figured it out from the C:\MSOCache folder. Note that this is a hidden folder so you need to enable hidden files to be able to view this. Under All Users folder, you can see a deluge of GUIDs from the Office installation and the one that I was looking for contained the below files. Though the issue was specifically for Outlook, the fix should be applied to the Office package itself

Capture3

That’s all I need to run the command that fixes the issue. The command line is below

msiexec /i {90150000-0011-0000-0000-0000000FF1CE} ADDLOCAL=OutlookVBScript /qb

You can omit the -C towards the end of the GUID. The command ran successfully and users never reported the issue with Outlook again