I came across this by accident while setting up NetScaler GSLB for a Citrix solution for one of my customers. The service groups in NetScaler were giving an error message for the monitoring probes – https and CITRIX-XD-DDC (both secure). The NetScaler was VPX running 18.104.22.168nc firmware.
Last response: failure – Time out during SSL handshake stage
How do you troubleshoot such issues? There are a couple of options.
Wireshark can be a good tool or you could use the built-in nstrace utility in NetScalers. Additionally, you will also need to make sure that the required ports are open between your NetScaler and the backend servers/services.
If you run a trace and look at it, you can see the below
TLSv1 Record Layer: Alert (Level: Fatal, Description: Unsupported Certificate)
Content Type: Alert (21)
The certificate that was installed on the DDCs and Storefront servers were created with a key size of 4096 and that was the issue.
Fix was to generate a fresh certificate with 2048 key size and the issue will be gone. Also, note that this issue is only prevalent in VPX versions of NetScaler. NetScaler MPXs will NOT exhibit this issue due to their built-in SSL chips.
Hope, this helps somebody out there.