Ever wondered about securing your Citrix ADC (formerly NetScaler) or Gateway implementation further with all the DDoS news going around of late. If you already have a NetScaler/ADC implementation, you can easily leverage it and configure Rate Limiting feature which is a fantastic weapon to stop such threats and keep the malicious actors at bay. This could be even be implemented on NetScaler/ADC Standard edition so there is no excuse. If you are interested in improving the security posture of your Citrix ADCs/Gateways/SD-WANs/SDXs or CPXs, then read on.
Common use-cases for Rate limiting
Limit the number of requests per second from a URL.
Drop a connection based on cookies received in request from from a particular host if the request exceeds the rate limit.
Limit the number of HTTP requests that arrive from the same host (with a particular subnet mask) and that have the same destination IP address.
Create Rate Limiting Policies
We are going to utilize the Responder feature to complete the configuration. So when you are ready to get started, logon to the NetScaler console with root privileges (nsroot preferably) and follow the below steps
You will need to then navigate to AppExpert node in the management portal. This is where you will find the Rate Limiting policies.
Expand AppExpert and select Rate Limiting
Expand Rate Limiting and click on Selectors
Click Add and enter a name for the Selector
Select the expressions as follows ( Note that there is a DOT after REQ for Expression 2)
Now, how do you test this? It isn’t easy with the existing threshold values. So to make the testing easier, I would adjust the threshold numbers to something that we could easily achieve. For eg, if we reduce the threshold value to 10 and time slice to 10000 msec, that literally means we only need to perform 10 requests in a matter of 10 secs. You can also bump up the timer to 20 sec(20000 msec) if you think 10 secs are still harder. For that you will need to go to the limit identifier that you set up earlier and adjust the value as below.
Now, open a browser page and navigate to the URL in question. Refresh the page a few times and after 10th successful attempt, 11th attempt will be dropped/reset by the NetScaler. End user may see something like the below
Of course, there are other selectors that you can target instead of the Client IP and HTTP URL that I have used for my example. Below are some useful links to get started if you want more literature to read on.
The Network Policy Server (NPS) extension for Azure MFA adds cloud-based MFA capabilities to your authentication infrastructure using your existing servers. With the NPS extension, you can add phone call, text message, or phone app verification to your existing authentication flow without having to install, configure, and maintain new servers.
This extension was created for organizations that want to protect VPN connections without deploying the Azure MFA Server. The NPS extension acts as an adapter between RADIUS and cloud-based Azure MFA to provide a second factor of authentication for federated or synced users.
When using the NPS extension for Azure MFA, the authentication flow includes the following components:
NetScaler receives requests from VPN clients or Citrix ICA Proxy users and converts them into RADIUS requests to NPS servers.
NPS Server connects to Active Directory to perform the primary authentication for the RADIUS requests and, upon success, passes the request to any installed extensions.
NPS Extension triggers a request to Azure MFA for the secondary authentication. Once the extension receives the response, and if the MFA challenge succeeds, it completes the authentication request by providing the NPS server with security tokens that include an MFA claim, issued by Azure STS.
Azure MFA communicates with Azure Active Directory to retrieve the user’s details and performs the secondary authentication using a verification method configured to the user.
There are some requirements that are needed to be met for deploying this solution.
The NPS Extension for Azure MFA is available to customers with licenses for Azure Multi-Factor Authentication (included with Azure AD Premium, EMS, or an MFA stand-alone license). Consumption-based licenses for Azure MFA such as per user or per authentication licenses are not compatible with the NPS extension.
These libraries are installed automatically with the extension.
The Microsoft Azure Active Directory Module for Windows PowerShell is installed, if it is not already present, through a configuration script you run as part of the setup process. There is no need to install this module ahead of time if it is not already installed.
Azure Active Directory
Everyone using the NPS extension must be synced to Azure Active Directory using Azure AD Connect, and must be registered for MFA.
When you install the extension, you need the directory ID and admin credentials for your Azure AD tenant. You can find your directory ID in the Azure portal. Sign in as an administrator. Search for and select the Azure Active Directory, then select Properties. Copy the GUID in the Directory ID box and save it. You use this GUID as the tenant ID when you install the NPS extension.
The NPS server needs to be able to communicate with the following URLs over ports 80 and 443.
Verify that your sync status is Enabled and that your last sync was less than an hour ago.
Determine which authentication methods your users can use
There are two factors that affect which authentication methods are available with an NPS extension deployment:
The password encryption algorithm used between the RADIUS client (VPN, Netscaler server, or other) and the NPS servers.
PAP supports all the authentication methods of Azure MFA in the cloud: phone call, one-way text message, mobile app notification, OATH hardware tokens, and mobile app verification code.
CHAPV2 and EAP support phone call and mobile app notification.
The input methods that the client application such as VPN, Netscaler, or others can handle. For example, does the VPN client have some means to allow the user to type in a verification code from a text or mobile app?
Register users for MFA
Before you deploy and use the NPS extension, users that are required to perform two-step verification need to be registered for MFA. More immediately, to test the extension as you deploy it, you need at least one test account that is fully registered for Multi-Factor Authentication.
Install the Network Policy Server role in your environment. You can choose to install this on any domain joined Server OS machine in the network.
Ideally, you would want to sit close to your Active Directory server just to make it quicker to send traffic for Authentication and Authorization. Or Just install this straight on your AD server, it’s totally up to you.
Installing the NPS role is dead easy. Just fire up your Server Manager and go to Manage – Add Roles and Features. Select Network Policy and Access Services
It will ask you to install Remote Server Administration Tools. Say Add Features.
Click Next (3 times) until you reach the Confirmation page. Click Install
Once installed, you will need to register the server in Active Directory.
Open the NPS console as below and right click the NPS node and click Register Server in Active Directory
Now it’s time to install the NPS extension for Azure.
Installing and Configuring NPS Extension for Azure MFA
Once downloaded, run the NpsExtnForAzureMfaInstaller.exe as an Administrator. If you want to change the install location, Click Options and choose a different location.
if not, just Click Install
The setup is quick. Click Close, once finishes.
Open PowerShell as Administrator. You have to have your Azure Portal admin credentials handy before this step.
Navigate to the install location for NPS Extension C:\Program Files\Microsoft\AzureMfa\Config using PowerShell.
Run the Powershell script in that directory AzureMfaNpsExtnConfigSetup.ps1 as below
PowerShell will begin the installation of NuGet provider assemblies including MSOnline cmdlets
It’s gonna tell you that you are installing this from an untrusted repository. Just say A for Yes to All and continue.
Now, PowerShell will take you to portal.azure.com where you will need your Azure AD admin credentials to login.
Login with your Azure credentials
At this stage, it will ask for the Tenant ID. Copy the Directory ID and paste it in the PS window
It does a few things as below
## It creates a Self-Signed certificate
## It grants private key access to NETWORK SERVICE
## Restarts the NPS Policy Service
You may now exit out of PowerShell as it is time to configure NPS.
Configure RADIUS Clients
Open the NPS console and navigate to RADIUS Clients and Server Folder
Expand the folder and Right Click on RADIUS Clients
Configure the settings as below
Give it a Friendly Name
Enter the IP address of the NetScaler (NSIP)
Enter a Shared Secret Key (Save this key as we will need this later)
Add all the RADIUS clients following the steps above. If you set this up on a NetScaler HA configuration, you will have 2 NetScaler NSIPs to add. You should something similar as follows.
Configure Remote RADIUS Servers
Select the node – Remote RADIUS Server Groups
Right- click and select New
Give a Group Name
Type the IP address or name of the Active Directory Domain Controller Server in there and Click OK.
You can choose to add the FQDN of the domain controller or just use the IP address. You can multiple DCs in here for redundancy.
Click on the Authentication/Accounting tab. Configure it as below
Click on the Load Balancing tab now and supply the weightage to the servers if you are adding multiple AD servers.
You can also configure the Timeout settings in here.
Notice that I have increased the timeout values to 60. This is important when using phone calls and SMS based authentication because they take more time. Even when using the Microsoft Authenticator app, default values are a little too less, so adjust it according to your environment.
Add all the servers that you intend to use as Domain Controllers in here.
Configure Connection Request Policies
It is time now to create a Connection Request Policy. We need a couple of them for this deployment. There are a few things to keep in mind as follows before we proceed to create the policies.
The default built-in connection request policy uses NPS as a RADIUS server and processes all authentication requests locally.
To configure a server running NPS to act as a RADIUS proxy and forward connection requests to other NPS or RADIUS servers, you must configure a remote RADIUS server group in addition to adding a new connection request policy that specifies conditions and settings that the connection requests must match.
If you do not want the NPS to act as a RADIUS server and process connection requests locally, you can delete the default connection request policy.
If you want the NPS to act as both a RADIUS server, processing connection requests locally, and as a RADIUS proxy, forwarding some connection requests to a remote RADIUS server group, add a new policy using the following procedure and then verify that the default connection request policy is the last policy processed by placing it last in the list of policies. This is the approach we are using for NetScaler deployment.
Create a Connection Request Policy for No Forward
Open the NPS server console and expand Policies node
Right Click Connection Request Policies and choose New
Give the policy a Name
Select Client IPv4 Address
Click Add again
Specify the Client IP v4 Addresses – This will be the NetScaler NSIP if RADIUS isnt load balanced. If load balanced, you must use the Subnet IP of the NetScaler (SNIP)
Configure Authentication as below
Configure the Authentication exactly as below
Click Next a couple of times until the Summary page is reached.
Create the second Connection Request Policy for Forwarding
Right Click Connection Request Policies and choose New
Give the policy a Name
Select NAS Identifier
Click Add again
Enter the name of the NAS Identifier – MFA
Configure the Authentication as below – MS-CHAP-v2
If you are on the Summary page, click Finish
The two connection request policies should be moved up in the policy priority order and should look like the below.
Create the Network Policy
Go to the Network Policies node
Right Click and select New
Give the policy a Name
Select NAS Identifier
Enter MFA in there
Click Add again
Configure the Authentication methods as below
a few more extra clicks will get you to the Summary page.
Click Finish on the Summary page.
Make the policy that we just created higher up in the order.
Disable the existing or built-in Network policies.
Disable the existing Network Policies (Default)
Move the new Network Policy to the top and assign it priority 1
You can now proceed to create your vServer in NetScaler. It could be a NetScaler Gateway or a VPN vServer. In this post, i will not be showing how to create a NetScaler vServer. It is fairly straightforward and there are tons of blog posts on it on the internet. You will just need to set eveything up just like how you would setup a single factor Gateway portal in NetScaler.
You will need to make sure that ports 1812 and 1813 are open from the NetScaler to the backend NPS server (bi-directional)
If you have multiple subnet IPs on the NetScaler, use a Net profile to isolate traffic to a particular source IP address.
If you aren’t load balancing NetScaler, NSIPs are the source IP address. Otherwise SNIPs will need to be used. (The client IPv4 address entries that you made in the previous step will change accordingly)
Create RADIUS Policies and Profiles
Go to NetScaler Gateway node – Policies – Authentication – RADIUS
Go to Servers tab and click Add
Give a name to the Server profile
Enter the IP address of the NPS server
Port is 1812
Enter the Shared Secret Key
Change the time out to 60 seconds if you intend to use phone calls, SMS or phone app auth.
Test the connection and ensure that you get all green
Enter the NAS ID here – MFA
Password encoding as mschapv2
Similarly, create additional RADIUS servers using the same steps above.
Create RADIUS policies now to attach the RADIUS server profiles so that it could be bound to vServers.
Create a RADIUS policy and attach the profile as below
Once, your vServer is ready, the RADIUS policy could be attached to the vServer as a primary authentication. Doing this will still perform Active Directory LDAP authentication after which the NPS extension will check the second factor authentication.
OR you can create an Authentication profile which is attached it to a non-addressable Auth vServer. Only advanced policies are supposed from NS 13.1 onwards so it is a better idea to go with the Authentication Profile method to future proof the solution.
Give the Authentication Virtual Server a Name
Click on No Authentication Policy
Click Add and specify the details below
The Auth vServer will be marked as Down. Don’t worry about that. If you are using a Standard edition licenses, there is no way you can attach a certificate to the Auth vServer via the GUI.
For Enterprise editions and above you can go to Security – AAA Traffic node and attach a certificate to the vServer if you don’t like the idea of vServer being down or you don’t want SNMP traps to trigger unwanted alerts.
Bind the authentication profile to the Gateway vServer
You can now test with an account that is MFA enabled. If everything is setup correctly, MFA will work fine and prompt with a second factor.
Always check the Authentication server status of RADIUS server in NetScaler. It should be green when the traffic is allowed. if it is not, check why? Work with your NW team to figure out why the traffic doesn’t reach the NPS backend or being returned back. I have also seen instances where the Dashboard shows red but things work just fine. Citrix GUI based RADIUS testing is flawed in my opinion and should be never be solely relied upon. Use other methods such as setting up a RADIUS monitor to test RADIUS reachability.
Add a DNS A record entry for the Remote URL for Citrix access
If the NetScaler IPs (NSIP) don’t work, try the Subnet IP as RADIUS clients. If you make a change, ensure that the change is reflected in the Network Policies too. SNIPs are used when you load-balance RADIUS services, otherwise use the NSIPs as RADIUS clients.
On NetScalers where multiple subnet IPs are used, isolate the traffic using NET Profiles.
Check aaad.debug logs on NetScaler.
if you get the below, it is mosty likely an issue with the RADIUS client IPs. It is just that the wrong IP is being used.
Look out for Routing issues. If your NPS servers are sitting in a different subnet as compared to NetScaler IPs, looking at the Route table could shed some light. If routes are missing, add them. But please remember not to break existing traffic. If unsure, ask the network guy for assistance.
Check the Dial-In tab in AD properties for the user. Ensure that the user is allowed access. Or You can configure NPS to override the AD settings by setting the below (look for the red dot below)
Event Logging – Ensure that NPS logs are turned ON. Log files will be found at C:\Windows\System32\LogFiles. Make sure that the logs are set to DTS compliant. Event Viewer is also a reliable source.
If you don’t want to limit non-MFA users from accessing the portal, you can add the below registry keys to the NPS servers. This will allow users who aren’t registered in Azure MFA to continue to authenticate using LDAP authentication. This is vital during migration phase. However, this setting must be removed before you move into production.
Commands that can be used to see the configuration and shared secret key netsh nps show config netsh nps show client
Fix for the error message “NPS Extension for Azure MFA: CID: e9fef35b-b365-4dde-b347-357c008b38e6 : Request Discard for user [email protected] with Azure MFA response: BecAccessDenied and message: MSODS Bec call returned access denied,BecAccessDenied,SAS.Shared.Exceptions.BecWebServiceException: The BEC web service failed to successfully respond to a call after 0 retries —> System.ServiceModel.FaultException`1[Microsoft.Online.Administration.WebService.AccessDeniedException]: Access is denied for the specified domain.”
Verify that the user logon name in on-prem AD is the same as the Azure AD login. if email addresses are used to login to Azure AD, then modify a registry key on the NPS servers to specify an alternate logon attribute as below HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\AzureMfa Reg_SZ=LDAP_ALTERNATE_LOGINID_ATTRIBUTE value=mail
I came across this by accident while setting up NetScaler GSLB for a Citrix solution for one of my customers. The service groups in NetScaler were giving an error message for the monitoring probes – https and CITRIX-XD-DDC (both secure). The NetScaler was VPX running 184.108.40.206nc firmware.
Last response: failure – Time out during SSL handshake stage
How do you troubleshoot such issues? There are a couple of options.
Wireshark can be a good tool or you could use the built-in nstrace utility in NetScalers. Additionally, you will also need to make sure that the required ports are open between your NetScaler and the backend servers/services.
If you run a trace and look at it, you can see the below
The certificate that was installed on the DDCs and Storefront servers were created with a key size of 4096 and that was the issue.
Fix was to generate a fresh certificate with 2048 key size and the issue will be gone. Also, note that this issue is only prevalent in VPX versions of NetScaler. NetScaler MPXs will NOT exhibit this issue due to their built-in SSL chips.
The RDP Proxy functionality is provided as part of the Citrix Gateway and currently is available to all NetScaler Enterprise and Platinum customers.
The following RDP Proxy features provide access to a remote desktop farm or an RDSH session host server through Citrix Gateway:
Secure RDP traffic through CVPN or ICAProxy mode (without Full Tunnel).
Single sign-on (SSO) to RDP servers through Citrix Gateway. Also provides an option to disable SSO if needed).
Enforcement (SmartAccess) feature, where Citrix ADC administrators can disable certain RDP capabilities through Citrix Gateway configuration.
Single/Stateless(Dual) Gateway solution for all needs (VPN/ICA/RDP/Citrix Endpoint Management).
Compatibility with native Windows MSTSC client for RDP without the need for any custom clients.
Use of existing Microsoft-provided RDP client on MACOSX, iOS, and Android.
RDP proxy requires port 3389 to be opened from the internet. You could also choose to use other port numbers if you don’t want to use the 3389 port. In a nutshell, just opening 443 port isn’t enough to get this to work.
Now to get started, we will need to enable RDP proxy feature if it isn’t turned ON. For that, navigate to System – Settings – Configure Advanced Features and ensure that RDP proxy is turned ON. if not, tick the box to Turn ON RDP proxy feature. You will need NetScaler Enterprise and above for this feature to work.
Create LDAP Profile and Policy
Create an LDAP profile for authentication. Navigate to NetScaler Gateway – Policies– Authentication – LDAP
Now create the LDAP policy. Click on the Policies tab, click Add. Enter the entries as shown in the picture below. Ensure that the correct LDAP profile is selected.
Create the RDP Client Profile
Navigate to NetScaler Gateway – Policies – RDP Profiles and Connections – Client Profiles
Give it a name such as RDProxy_Profile and leave the rest of the values default if you would like. I changed the RDP Cookie Validity from 60 sec to 120 seconds
Create an RDP Server Profile
Create an RDP Server Profile. Click on the first tab that says Server Profile
Click Add and enter a name for the server profile. Enter the IP address (this is the IP address of the RDP Proxy Virtual Server that you will configure under the NetScaler Gateway). Enter the port number – You can choose to go with the default RDP port if you wish to or choose another one
Create a Session Profile
Now, go to NetScaler Gateway – Policies – Session – Session Profiles. Click Add
Give the profile a Name
No changes under the Network Configuration tab. Leave everything as default there
Under Client Experience tab, change Clientless Access to ON and tick Single Sign-on to Web Applications and Credential Index to Primary. the last setting is turning ON Single Sign-on with Windows
Under the Security Tab, select Default Authorization to ALLOW and Secure Browse to ENABLED
Under Published Applications, set ICA PROXY to OFF
Under the Remote Desktop tab, pick the RDP Client profile that was created in the previous step
Create a Session Policy
Now create a Session Policy that will be bound to the NetScaler Virtual Server. Remember that we haven’t created the virtual server yet.
Switch to Session Policies tab and click Add. Give the session policy a Name and pick the session profile that we just created in the previous step.
Create a Bookmark
Now create a Bookmark and this is what will appear to the users in the form of an application icon to click on.
Give a Name to the bookmark and enter the name of the string that you want to be displayed in the portal. Enter the Bookmark link in the format rdp://IPaddressOfTheBackendRDSServer
Create the Gateway Virtual Server
Let’s create the Gateway Virtual server next. Navigate to NetScaler Gateway node, expand that and under Virtual Servers, click Add
Under Basic Settings, configure the below items
Name – RDPProxy_rdpproxy.fqdn.co.nz
IP Address type – IP Address
IP Address – X.X.X.X
Port – 443
Pick the RDP Server Profile – RDP Server Profile
Ensure that Enable Authentication, AppFlow Logging and State is turned ON
Disable ICA Only
Attach a Server Certificate. The certificate can be a wild card cert or you could choose to get a named certificate that matches the external RDP proxy FQDN
Now bind the Primary authentication policy. We are going to use LDAP and hence I will use LDAP policy that we created in the steps above.
Under SSL Parameters, ensure that only TLS1.2 is turned ON for enhancing the security of client connections.
You can choose to go with the default SSL ciphers or modify the ciphers according to the company requirements.
Under Portal theme, I went with RfWebUI which I think is one of the cleanest UIs. You could choose to create a custom one and use that instead.
Under Published Applications, choose the URL Name and select RDP Link (this is the bookmark link that was created)
Under Policies, attach the Session Policy named RDP Session Policy
Testing the Setup
Selecting the RfWebUI gives the below logon page and users could simply use their domain user name and password to log in. They don’t need to enter the domain name.
Upon login, you will be shown the Favorites page where you could add links for quick access. This is very similar to the subscriptions in Storefront.
Click on the Desktops tab and you will be able to see all the published Bookmarks there. I have one in there, you can choose to have any number of bookmarks.
Click on the RDP link to launch the application. It will first download the app.rdp file which could be used to launch the application. You will just need to give the users access to the servers locally by adding them to the Remote Desktop Users group or you could choose to do this via AD domain groups to manage it centrally.
Ever had issues with your Storefront load balancing setup not working after painstakingly setting up NetScaler load balancing?
Now, let’s discuss a few reasons why cookie based persistence are better (in some scenarios) compared to Source-IP based persistence.
In some circumstances, using persistence based on source IP address can overload your servers. All requests to a single Web site or application are routed through the single gateway to the NetScaler appliance, even though they are then redirected to multiple locations. In multiple proxy environments, client requests frequently have different source IP addresses even when they are sent from the same client, resulting in rapid multiplication of persistence sessions where a single session should be created. This issue is called the “Mega Proxy problem.” You can use HTTP cookie-based persistence instead of Source IP-based persistence to prevent this from happening.
If all incoming traffic comes from behind a Network Address Translation (NAT) device or proxy, the traffic appears to the NetScaler appliance to come from a single source IP address. This prevents Source IP persistence from functioning properly. Where this is the case, you must select a different persistence type.
HTTP Cookie Persistence
When HTTP cookie persistence is configured, the NetScaler appliance sets a cookie in the HTTP headers of the initial client request. The cookie contains the IP address and port of the service selected by the load balancing algorithm. As with any HTTP connection, the client then includes that cookie with any subsequent requests.
When the NetScaler appliance detects the cookie, it forwards the request to the service IP and port in the cookie, maintaining persistence for the connection. You can use this type of persistence with virtual servers of type HTTP or HTTPS. This persistence type does not consume any appliance resources and therefore can accommodate an unlimited number of persistent clients.
By default, the NetScaler appliance sets HTTP version 0 cookies for maximum compatibility with client browsers. (Only certain HTTP proxies understand version 1 cookies; most commonly used browsers do not.) You can configure the appliance to set HTTP version 1 cookies, for compliance with RFC2109. For HTTP version 0 cookies, the appliance inserts the cookie expiration date and time as an absolute Coordinated Universal Time (GMT). It calculates this value as the sum of the current GMT time on the appliance and the time-out value. For HTTP version 1 cookies, the appliance inserts a relative expiration time by setting the “Max-Age” attribute of the HTTP cookie. In this case, the client’s browser calculates the actual expiration time.
So some of the benefits of using cookie based persistence are
Easy to configure
Load balancer does all the work without involving the back end services (mostly a web server)
No connection table to maintain
Activity based – Time out is based on idle connection and not the total connection time
caters to most of the load balancing scenarios
if proxies are involved in the setup where source IPs are masked, this makes it an easy choice
Cons of Cookie based persistence
can throw off the load balancing where timeouts are set to 0 or when time outs are long lived
when Type 0 cookie is used, time of the cookie value is absolute(GMT time). When type 1 is used, relative time is used (local time)
hard to troubleshoot
Testing COOKIEINSERT persistence
Use Putty or some other SSH clients to login to the NetScaler.
sh lb vservervservername
If you take a closer look at the snippet above, the letters in yellow shows that COOKIEINSERT persistence is turned ON with a time out value of 0 mins. This means that the cookies doesn’t have an expiry time set by the NetScaler appliance. The backup persistence is set to SourceIP with a time out value of 60mins. You can also see the cookie values at the bottom of the results staring with NSC_xxxxxxx
You will now need to get into Shell with an account that has permissions. To get to shell, type shell and hit enter
Now type the following
nsconmsg -i vservername -s ConLb=1 -d oldconmsg
In the snippet above, the highlighted text in yellow shows that persistence is set to Cookieinsert and the number of hits for each service. You will also get values such as response times, CPU and memory utilization.
VIP value shows the total number of hits to the vServer hits Each service has a line starting with S (IP Address) and its status. Hits (x, x/sec) shows the initial (method) hits to the service for a new connection. P (x, x/sec) shows hits to the service that are served using a persistence cookie
Also, check the current persistent session via the commands below
sh persistentsessions sh persistentsessions -summary