Looking Inside Netscaler and CAG Ent SSL Payload

At some point, you may need to see what is happening inside the SSL posted to your Citrix Access Gateway Enterprise or Netscaler appliance. I had that need today and the web was not much help. There was one Citrix Knowledge Center article on capturing traffic, but looking at SSL handshakes isn't very useful in my world.

I put on my ninja hat and went to town, as they say.
The first step is to determine what certificate is being used by your appliance. You may have more than one certificate if you have multiple v-servers. Login to the appliance and go t SSL, under configuration. You will need to know the names of your certificate file and key file.



In the screen above note the certificate file is /nsconfig/ssl/bpcag.cert and the key file is /nsconfig/ssl/bpcag.key

1. SSH into the Netscaler. Putty will work
This is done via the management IP as nsroot. You will need to type shell and hit enter to get a useful shell.

2. change to the /nsconfig/ssl directory

3. Execute the following commands

4. openssl pkcs12 -export -inkey bpcag.key -in bpcag.cert -out bpcagout.pfx
If you already have a pfx from a certificate vendor, you can skip this step and reference the vendor pfx in the command on line 6.

5. You may be asked to enter the key decrypt password. If you do not have this, you are out of luck and will need to get a new cert and keep the password. You will be asked to create an export password. Make sure to remember this.

6. openssl pkcs12 -in bpcagout.pfx -out clearpemout.txt -nodes

7. You will be asked for the password that you created with the command in line 4. The resulting file clearpemout.txt is your certificate and RSA private key in clear form. If this is a production key, make sure to keep a close watch on this file. Your SSL is only as secure as the private key.

8. Use Filezilla or WinSCP to get the file from the netscaler location /nsconfig/ssl, and copy it to your workstation. I like Filezilla better...

9. Create a text file containing only the private key section of clearpemout.txt to be read by Wireshark.
This text is an unencrypted copy of the private key, so handle the file with caution
The key looks like this.
---BEGIN RSA PRIVATE KEY---
MIICXQIBAAKBgQDQVLGU+2OWUkD03PGkHnR1LrnMqKchtEXnNX49+8wrvTBlkisi
+JSnf6XhDoJRQggrC/IIS/K6ptDWaTqDC1hX4kQnRLa42ogSHVZHRSXrmRYjKCR4
R7N+E/sxuhjaf8zmcwf8UxuM4Y1b4UB4/3L48yXDdVBW
---END RSA PRIVATE KEY---

10. Open Wireshark and go to Edit > Preferences.

11. Find SSL under Protocols

12. Make sure the check boxes are checked.

13. In the field RSA keys list, enter your key and server data. A typical string looks like this
70.103.221.144,443,ssl,c:\cert.key;10.0.0.194,443,ssl,c:\cagx.txt;10.0.0.193,443,ssl,c:\cagx.txt;10.0.10.4,443,ssl,c:\cagx.txt

It must reference the IP address of the V-server you are sniffing, the port, the protocol (SSL), and the full path to the decrypted key file. In the example about, each separate entry is delimited by a semi-colon. If you are using alternate ports for SSL make sure to pay attention to the string. Each combination of protocol, IP, port and key file must have its own entry.

Now you are ready to read some data!!!

To sniff the traffic from the CAG/Netscaler, you must SSH in and use the command line

nstrace.sh -sz 1514 -tcpdump 1

Don't use the native tcpdump, as its fought with peril.

The command will create numbered capture files, rolling to the next every 3600 seconds. The files will ve dumped into /var/nstrace

Start your trace, generate your traffic, then stop the trace ctrl+c, then copy the trace files to your machine and open them in wireshark. To view the SSL payload in wireshark, you must find the initial SSL handshake, right click on one of the handshake packets and select Follow SSL Stream. If you don't find the handshake, you may have started your trace after the handshake occurred.

To filter out some of the pesky extraneous data (there will be a lot), you can use the filter bar. A common filter would be (tcp.port eq 443). This only shows traffic on tcp port 443. You may want to constrain it further, if you have many active v-servers with (ip.addr eq 10.0.10.4), using the appropriate v-server IP. You can combine the filters (tcp.port eq 443) && (ip.addr eq 10.0.10.4)

Enjoy...