Don’t let your user-experience be a “Spectre” of itself after “Meltdown”


Bust your ghosts not your user experience

The names Spectre and Meltdown invoke feelings of dread in even the most seasoned IT engineer.  To those uninitiated, let me get you up-to-speed quickly.

Spectre is a vulnerability that takes advantage of “Intel Privilege Escalation and Speculative Execution”, and exposes user memory of an application to another malicious application.  This can expose data such as passwords.

Meltdown is a vulnerability that takes advantage of “Branch prediction and Speculative Execution”, and exposes kernel memory.  A compromised server or client OS running virtualized could gain access to kernel memory of the host exposing all guest data.

Both vulnerabilities take advantage of a 20-year-old method of increasing processor performance.

Server_Protection

As a result, code will need to be updated to address these vulnerabilities at OS and OEM-manufacturer levels, at the expense of system performance.

On their part, Microsoft reluctantly admits that performance will suffer.  “Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance,” wrote Terry Myerson, Executive Vice President for the Windows and Devices group.

According to Geek Wire, these two vulnerabilities which take advantage of a 20-year-old design flaw in modern processors can be “mitigated;” the word we’re apparently using to describe this new world in 2018, in which servers lose roughly 10 to 20% performance for several common workloads.

This affects not only workloads executed against local, on-site resources but even those utilizing services, such as AWS, Google Public Cloud or Azure.

cpu_utilReader submission @ The Register showing CPU before / after patches

We’ve heard from some of our insiders who use Login VSI to validate system performance that they’re seeing a reduction of 5% in user-density after performing Microsoft recommendations. Knowing that the vulnerability wasn’t solved by OS updates alone we, at Login VSI, wanted the ability to test the impending hardware vendor firmware / BIOS changes.

Now is the time to capture your baseline performance

How do you know how much of an impact the fixes for Spectre and Meltdown will be if you don’t have anything to compare it to? Keep in mind that these patches will need to be installed on a number of systems in your solution including server hardware, operating systems, storage subsystems and so on.

Many of our customers perform tests where they compare a known good solution, or a baseline, with changes that have been made. This gives them the ability to accurately assess the performance impact of that change, which in turn allows them to compensate with more hardware, or further tuning of the applications and OS. The patented methods used by Login VSI provide a quantifiable result for determining the impact of a change in virtual desktop and published application environments.

Using Login VSI

If you wish to test the changes before pushing them into your production environment, then use Login VSI to put a load, representative of your production users, on the system. This will objectively show how much more CPU will be used as a result of the Spectre or Meltdown patches. It is expected that the end users will incur increased latency to their applications and desktops as a result of the higher CPU utilization.

Using Login PI

While it is not recommended, if you are planning on pushing the patches into your production environment to “see how it goes”, then install Login PI now to get an accurate representation of performance related to user experience. This will give you the ability to then compare to that same experience after the patches have been installed. We expect that you will see latency to the end user increase as a result of higher CPU utilization. If you already struggle with CPU utilization in your solution, there is a good chance you’ll be also using Login PI to test your availability.

As we complete our testing we will be sharing our findings in a series of articles.

If your computer has a vulnerable processor and runs an unpatched operating system, it is NOT SAFE TO WORK WITH SENSITIVE INFORMATION”. – Security Experts who discovered Meltdown / Spectre 

If sensitive data is part of your business (Such as ours!) patching is not a matter of if, but when.

Ask yourself:

How long can you afford to have your company’s data exposed to malicious intent?  Do you want to be the next Equifax or Target?

In this article series, we will provide some insight from our lab environments. Be aware your results may vary based upon individual workload and configuration.

Microsoft has released a Security Advisory

The vulnerability affects both the client and server OSs of Windows.  This is compounded when dealing with large-scale published application and desktops deployments.  The advisor can be found at the following location:

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/ADV180002

The specific details addressed in the security update and Windows KB are outlined in the Common Vulnerabilities and Exposures database.

Included are:

To completely protect yourself there are two phases of patching this vulnerability.

1 – Windows OS updates

2 – OEM device manufacturer firmware updates (not yet available)

Microsoft acknowledges addressing these vulnerabilities from a software perspective is limited, and therefore, without the OEMs providing updates the loop is not closed.

In the interim we can start measuring the impact of the Microsoft fixes.

They offer guidance for both Desktop and Server OSs:

Desktop –  January 2018 Security Update. Security Advisory: Click Here!

Server –  KB405690. Security Advisory: Click Here!

NOTE – Certain AV solutions are not compatible with the security update released by Microsoft. As such, unless an AV vendor has a registry flag, QualityCompat, they will not receive the January Security update and will still be vulnerable

With the upcoming OEM hardware patch releases we expect to be able to produce a variety of interesting and informative results.  Please stay tuned for the next articles!

Reference materials:

https://meltdownattack.com/

https://www.theregister.co.uk/2018/01/09/meltdown_spectre_slowdown/

https://www.geekwire.com/2018/microsoft-admits-meltdown-spectre-patches-will-hit-windows-server-performance/

Advertisements

Citrix Cloud Testing on Amazon EC2 M4


Citrix Cloud on AWS

I was recently afforded the unique opportunity to collaborate on a project to test capacity out of a Citrix XenApp on AWS deployment. The goal of the project was to independently determine the maximum user density for a few different EC2 instance types running XenApp 7.14.

EC2 instances are on-demand and elastic hosted server resources. Which means that they are provisioned dynamically within a pool of available resources, and with an OS you deploy ontop. Amazon provides a variety of templates to easily install Windows, Linux or your other favorite OS. EC2 instances are broken down into a few varieties. They are optimized for storage, memory, compute or graphics. The designation before the name of the instances illustrates their configuration. G3 indicates graphics optimized instance third generation.

The other difference between instance type is the cost. If you are provisioning a 2vCPU 4GB of RAM machine the price per hour would be significantly less than that of a 16vCPU 64GB of RAM machine.

1st

This would allow the customer to match the exact machine size to the purpose of their deployment, and optimize the amount of money they were spending on their hosted application solution.

Utilizing Login VSI’s virtual users I ran a predetermined user count against a Citrix XenApp deployment managed from Citrix Cloud.

For this blog, I will only discuss one data point, and the Citrix Cloud configuration on AWS. We have a significant amount of results, and we will make those available on www.loginvsi.com/blog.

For those of you not familiar Citrix Cloud is providing Citrix capabilities traditionally delivered on premise through a HTML web based user experience therefore installing a receiver is no longer required.

Some of the key components as they move into their cloud forward offerings are StoreFront / Netscaler and Studio.

2nd

StoreFront and NetScaler are completely managed now through a web page. This completely removes the administrator’s responsibilities of configuring hardware / software solutions for Citrix. You simply fire this up, attach it via their “Citrix Cloud Connector” and configure to start deploying your desktops or apps. It works completely flawlessly.

Studio is managed through the connector as well, and provides the Citrix HTML 5 receiver for management access through the Citrix Cloud web portal.

During my time working with it, it proved to be very flexible, easy to configure and reliable for all testing. I would recommend this to any administrator looking at future proofing their Citrix deployments. It is truly ready for market.

Some images below of the management interface:Some images below of the management interface:

There will be a management icon within your Citrix Cloud Dashboard. Select “XenApp and XenDesktop Service” “manage”

3rd

You will then go to the management interface for XenApp / XenDesktop; you have two options Creation and Delivery. Creation – Studio / Delivery – StoreFront / NetScaler:

4th

Management interface for Studio. Notice the Citrix Receiver icon in the middle. Studio is provided through the Citrix HTML 5 receiver. Interesting touch.

5th

Management for Citrix NetScaler / StoreFront:

6th

AWS Configuration for demonstration purposes:

7th

Color coded

8th

Delivery group configuration:

9th

11th

There is only one XenApp host in each delivery group. This is to determine the maximum amount of users for one M4.

2XLarge instance backing the XenApp host. We are delivering Office 2016 applications, and the standard set of VSI Knowledge worker actions.

It is very easy to change the instance type in EC2. You simply select the “Instance” and change the “Instance Type” through the context menu.AWS_Change_Instance_Types

There are a variety of different configuration, which allows you to really get the most out of the testing. If you are aiming for user density numbers you can size it exactly. This allows you to pay for EXACTLY what you need as opposed to over provisioning. This will help drive the cost of VDI / SBC deployments down ultimately, and increase end user experience quality.

If you are sizing your images with Login VSI and backing them up with EC2 AWS instances you are getting an optimal user experience exactly sized right for your needs.

Information on instances:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html

VSI Results

12th

Testing Configuration

For our testing purpose we provisioned a m4.2xlarge machine on EC2. This instance has a machine profile of 8 vCPU and 32 GB of memory. This is either running a XENO E5-2686 or 2676. Mostly a general use machine which is balanced.

Our testing configuration was 50 test users over the course of 48 minutes. We utilized the industry standard Knowledge Workload. This mostly presents a large portion of the VDI / SBC user base. Office application and standard office applications like Adobe Reader.

 

Application start times are all over the place for the most part, but staying for the most part under 12 seconds. Which would be reasonable for the users. Login process takes under 16 seconds even under VSI Max settings.

 

What does the backend look like?

16th

When the CPU is at 100% the VSIMax is being reached within the user session. This means the numbers are indicating the bottleneck to be the CPU provisioned for the M4.2Xlarge instance which is approximately.

Wrap-up

Seeing is believing and after testing it I can confirm that Amazon EC2 is ready for the prime time. We were able to support 42 concurrent users on a M4.2Xlarge and we were able to have a continuous level of excellent user experience while doing so.

Amazon is ready to supplement your traditional on premise solutions with readily available and quickly scalable resources in the cloud. Using Citrix Cloud services you can very easily scale your delivery out to support your user base as it dynamically changes.

Using VSI you can validate your configurations with support your users and put a check box next to user experience.

Using these three solutions you can future proof your company, and deliver on a promise of value & experience

Finally, if you are looking for some testing for your deployment please reach out to me here or b.martynowicz@loginvsi.com.

As always stay tuned for more results.

AUTOMATION OF A XENDESKTOP/XENAPP DEPLOYMENT – PART 3


This article will cover how to install the Citrix Studio component.  For those of you who are unfamiliar this will act as your centralized point of control for your Citrix environment.  You can also interact with it as you’ve seen in previous examples through command line operations as well.

In order to install the Studio for management the process is straight forward.  Run the executable with a few switches –

[C:\Downloads – “Your download location”]\x64\XenDesktop Setup\XenDesktopServerSetup.exe /PASSIVE /NOREBOOT /CONFIGURE_FIREWALL /COMPONENTS DESKTOPSTUDIO

Once your Studio component is installed, you can validate the installed components by reviewing their accessibility form your server where the command was executed.

Studio

Next up would be installing your StoreFront component.  This will provide front end access to your delivered applications and desktops.

StoreFront

The process is the same.  Utilizing your Citrix installed, you will call the following executable with the following switches:

[C:\Downloads – “Your download location”]\x64\StoreFront\CitrixStoreFront-x64.exe -silent -logfile “%am_logpath%\StoreFrontSetup.log”

For adding additional Storefront servers to a load balanced group, you can utilize a variety of different scripts to determine if the server is already part of a group, and if not join it.

Creates a script to review if the Citrix services are running on the server where the scripts are executing, and if so join the new StoreFront sever to a group. If not create a group of its own.

Set a conditional statement to evaluate the value of this being returned, and execute one of the following scripts.

For Creating:

Set-DSInitialConfiguration -hostBaseUrl [Hostname]-farmName [FarmName] -port [ListeningPort] -transportType [TransferType] -sslRelayPort [RelayPortforSSL] -servers [ServerToAdd] -loadBalance [LoadBalancer] -farmType [FarmType] -clusterId [ID]

 

Or, For Joining an Already Existing:

Obtain the passcode and store value –

Start-DSClusterJoinService

$Passcode = (Get-DSClusterJoinServicePasscode).Response.Passcode

and finally join,

Start-DSClusterJoinService

Start-DSClusterMemberJoin -authorizerHostName $ExistingNode -authorizerPasscode $JoinPass

 

The final portion would really only be interesting if you were automating the same process over and over again.  This is where a product like Login AM which organizes your scripts, and variables would become handy.  If you are interested in any information about Login AM or automation of Citrix Deployments please reach out and let me know or drop and comment I would like to have meaningful conversations about automation, and increase my understanding of the products as well.

Automation of a XenDesktop/XenApp deployment – Part 2


As I discussed during my previous blog there are many parts necessary for a fully functional Citrix XenDesktop / XenApp environment.  In this part of the series we are going to cover the Citrix Licensing server.

The blog will be shorter than the previous blog due to the fact that there is only one component to the Citrix Licensing server. However, we will also cover how to deal with a known issue during automation which is related to automation leaving the license location in the License server empty.

The command line switches for automation of the license component is again very straight forward.

XenDesktop/XenApp Licensing Server

Example – C:\{Location}\x64\XenDesktop Setup\XenDesktopServerSetup.exe /COMPONENTS LICENSESERVER /NOREBOOT /QUIET /CONFIGURE_FIREWALL

This will install the License Server portion of a XenDesktop/XenApp deployment silently and defer the reboot.  You can run this string as many times as you would like and the end result will be the same.

Based upon my experience with automation I have discovered a known issue which is that within the License Server configuration the location of the specified license file will be blank.  This can lead to issues with functionality.

Know Issue with unattended installation

We can automate address this issue through PowerShell.  Below I will outline how to do so.  The company that I work for make a software solution for managing your PowerShell solutions.  This will provide you with a centralized location for management of your scripts, and assist with WHEN the scripts will be executed.  Additionally, you can design your solution 1 time and utilizes the solution anywhere when deploying XenDesktop / XenApp.

  1. Stop the Citrix License Server service
    1. This is done through a net stop command
  2. Parse through the license server configuration xml file located at
    1. C:\Program Files (x86)\Citrix\Licensing\LS\conf\server.xml
    2. This can be done by piping the contents of the XML file into a variable
    3. $serverxml = [xml] (Get-Content -path “C:\Program Files (x86)\Citrix\Licensing\LS\conf\server.xml” )
    4. You have now captured the contents of the XML file
  3. Locate within the XML file where the license file is specified, under the following value and assign it to a variable
    1. $element = $serverxml.configuration.licenseServer.vendorDaemons.daemon
      1. Where-Object {$_.executable -eq “CITRIX”}
    2. Write the value of your licensing file into the XML file
      1. $element.license = “The location of your license file”
    3. Save the server.xml file
      1. $serverxml.Save
    4. Start the Citrix License Server service

 

Bonus points – You could also utilize this level of automation to quickly replace the licensing file within your deployment in an automated method vs. manually going through the License Server web interface.  IE – Whenever it would be time to change out your license file simply replace the file and run the script.

In my next article in the series we will be outlining the process for deployment of the Citrix Studio portion of your XenDesktop / XenApp.  If you have any tips or tricks that could be helpful.  Please share I would love to share ideas, and share any information you are aware with the rest of my readers.

Automation of a XenDesktop/XenApp deployment


There are many pieces involved in deploying Citrix XenDesktop/XenApp.  For simplification purposes while discussing automation, let’s focus on a single feature of a Citrix deployment- the Delivery Controller.

The Delivery Controller is responsible for delivery of either applications or desktops to end users.

The components of a Delivery Controller are:

  • Database – SQL – Pre-requisite
  • Application – Citrix Delivery Controller

Installing each of these components individually is straight forward.  The applications are packaged in such a way that you can utilize a few switches to install the software.

SQL

Example – C:\{Location}\setup.exe /QUIET /IACCEPTSQLSERVERLICENSETERMS /ACTION=install /FEATURES=SQL,Tools /INSTANCENAME=MSSQLSERVER /SQLSVCACCOUNT=”NT Authority\Network Service” /SQLSYSADMINACCOUNTS=domain.local\Administrator /AGTSVCACCOUNT=”NT Authority\Network Service”

This will install SQL on a Windows Server OS silently, and create a database named MSSQLSERVER.

The same can be done with the Delivery Controller.

XenDesktop/XenApp Delivery Controller

Example – C:\{Location}\x64\XenDesktop Setup\XenDesktopServerSetup.exe /PASSIVE /NOREBOOT /CONFIGURE_FIREWALL /COMPONENTS CONTROLLER /NOSQL

This will install the Delivery Controller portion of a XenDesktop/XenApp silently, deferring the reboot and not installing SQL. You can run this string as many times as you would like and the end result will be the same.

Now, let’s get fancy.

In order for the Delivery Controller to function a “Site” must be present.  A site contains all of the data necessary for a Delivery Controller to function, and is stored in your SQL database; this includes configuration and logging information.

A site is the first step necessary to deliver resources with Citrix for your end users.  With PowerShell you can configure this without user interaction, thus enabling you to automate the deployment process.

You will need 3 databases:

  • “Site”
  • “Configuration”
  • “Logging”

 

You can utilize PowerShell to create these with the “New-XDDatabase” command.  This is a function of the “Citrix.XenDesktop.Admin.V1” PowerShell snap-in.  This will enable you to create the three database necessary for a “Site” to function properly.  Once the databases are created, you can create your “Site”.

You can utilize PowerShell to create your “Site” with the “New-XDSite” command.  This is a function of the “Citrix.XenDesktop.Admin.V1” PowerShell snap-in as well.

XDDatabase

Combining packages from Microsoft, Citrix and PowerShell, you are able to automate the process of creating your first “Site” for a critical component of your Citrix deployment, the Delivery Controller.

There are many platforms for managing, and executing your collection of automation frameworks.  Some of the popular ones are Chef, Puppet and Login AM.  It creates a logical organization structure for doing so, variables necessary and provides an interface for management.

If you are interested in finding out more about this, please get in touch with me.

Also, if you have any neat tricks with PowerShell please share them in the comment section.  Happy automating.

I will be following this post up with articles about the remaining components of Citrix.

Article 2 in the series is now live! See it here – Automation of a XenDesktop/XenApp deployment – Part 2

Article 3 online – Automation of a XenDesktop / XenApp Deployment – Part 3

Stay-Tuned-button-1024x192