How I Learned to Stop Hating Java and Start Loving s_client

Got SSL problems? Ask me how s_client can help!

Have you ever had one of those days when a customer tells you that they can connect to your server using SSL, but not to the F5 load balancer in front of it? Here's my tale.

I provide LDAP over SSL access, to a group of my domain controllers, by way of an F5 load balancer. None of the apps that use this are AD aware, so we just have one virtual IP (VIP) on an F5 on port 636. I don't want to add Subject Alternate Names (SANs) to my DC certificates, so we terminate SSL at the F5 and then create a new SSL session from F5 to the appropriate DCs.

As the PKI that issues our SSL/TLS certificates is internal, it is common to have to help our customers provision their local CA trust stores to trust our Root CA. Nearly all of the systems that use the F5 are Java based. Since setting up the F5, we've noticed that the systems required our customers to install all three tiers of our PKI in to their local trust store. This seemed odd, as my experience has always been that importing the root alone was good enough to allow proper chaining. We always chalked it up to weak chaining code in the Java stack. How foolish of me/us!

Last week we had a fresh windows machine try and use the F5 and they got the same result. That is, he failed to connect to the load balancer, getting a failed trust error. The customer had only installed our Root CA in his trusted roots store. Instead of just adding all the tiers of our PKI, the customer decided to point directly at one of our DCs. To my surprise, he connected with no chaining failures. Apparently the Root CA was good enough when connecting to the DC, but no for the F5.

At that point, I stopped blaming Java and demanded answers. Enter my new trusty friend openssl s_client. s_client establishes an SSL/TLS session with a server and then waits for you to send commands. Think of it as getting a socket, like telnet, only after establishing an SSL session first. But wait, there's more... s_client can also tell you all about the SSL session in question.

openssl s_client -connect server:port -showcerts will also show you what is being negotiated by your server and client.

When connecting to my DC directly I got:

CONNECTED(00000730)

---

Certificate chain

0 s:/CN=DCserver.domain.org

i:/DC=org/DC=domain/CN=Fun USA Issuer CA 02

-----BEGIN CERTIFICATE-----

MIIF5TCCBM2gAwIBAgIKS5ibh

-----END CERTIFICATE-----

1 s:/DC=org/DC=domain/CN=Fun USA Issuer CA 02

i:/CN=Fun USA Intermediate CA 01

-----BEGIN CERTIFICATE-----

MIIF/zCCBOegAwIBA

-----END CERTIFICATE-----

2 s:/CN=Fun USA Intermediate CA 01

i:/CN=Fun USA Root CA

-----BEGIN CERTIFICATE-----

MIIF5jCCBM6g

-----END CERTIFICATE-----

---

Server certificate

subject=/CN=DCserver.domain.org

issuer=/DC=org/DC=domain/CN=Fun USA Issuer CA 02

---

Acceptable client certificate CA names

/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority - G2/OU=(c) 1998 VeriSign, Inc. - For authorized use only/OU=VeriSign Trust Network

/C=US/O=First Data Digital Certificates Inc./CN=First Data Digital Certificates Inc. Certification Authority

/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting/OU=Certification Services Division/CN=Thawte Personal Basic CA/emailAddress=personal-basic@thawte.com

/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority

/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority - G2/OU=(c) 1998 VeriSign, Inc. - For authorized use only/OU=VeriSign Trust Network

/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Root

/OU=Copyright (c) 1997 Microsoft Corp./OU=Microsoft Corporation/CN=Microsoft Root Authority

/DC=com/DC=microsoft/CN=Microsoft Root Certificate Authority

---

SSL handshake has read 7832 bytes and written 318 bytes

---

New, TLSv1/SSLv3, Cipher is RC4-MD5

Server public key is 1024 bit

Compression: NONE

Expansion: NONE

SSL-Session:

Protocol : TLSv1

Cipher : RC4-MD5

Session-ID: 80050000AB7EA7AC223A31DAE93EFE553CFFEB093D224D59C6C3DD169AFEA1D3

Session-ID-ctx:

Master-Key: 382218AB63FFC1A874DAD739E48630D8410F05FFE24BAEF54DF7E88578DE2720AF111FF4E7C052A42A357AB476C205B5

Key-Arg : None

Start Time: 1266967263

Timeout : 300 (sec)

Verify return code: 20 (unable to get local issuer certificate)

---

You can see that the server and client negotiated a TLSv1 session using RC4-MD5. More importantly, for me, we see what certificates the server offered up for the SSL connection. I've truncated the output to remove most of the base64 encoded certificate data, as you don't need it. It can be useful if you know what to do with it, but that is out of scope. You can see that my DC will allow client certificates, but is not requiring them. You will see that the DCserver cert was offered up, along with the Issuer 02, the Intermediate, and the Root. This gives a client all the hierarchy to allow chaining to the Root, which was trusted.

When I ran the s_client connect against the F5, I got this:

CONNECTED(00000730)

---

Certificate chain

0 s:/C=US/ST=WA/L=Renton/O=Fun USA, Inc/OU=Business Systems/CN=f5.domain.org

i:/DC=org/DC=domain/CN=Fun USA Issuer CA 02

-----BEGIN CERTIFICATE-----

MIIGATCCBOmgAwIB

-----END CERTIFICATE-----

---

Server certificate

subject=/C=US/ST=WA/L=Renton/O=Fun USA, Inc/OU=Business Systems/CN=f5.domain.org

issuer=/DC=org/DC=domain/CN=Fun USA Issuer CA 02

---

No client certificate CA names sent

---

SSL handshake has read 1687 bytes and written 306 bytes

---

New, TLSv1/SSLv3, Cipher is RC4-MD5

Server public key is 1024 bit

Compression: NONE

Expansion: NONE

SSL-Session:

Protocol : TLSv1

Cipher : RC4-MD5

Session-ID: B53C2347A4947080D27A65929C3E2ADCFE6DCCFC70F57456BA8155E036347B4E

Session-ID-ctx:

Master-Key: 24E43E3324F07EDCA3C2F0DC790FFE0134B5B0FC027E08B46988A02A26A402780D548C1DAC8C8F3F1EFA0F071A098F8A

Key-Arg : None

Start Time: 1266967301

Timeout : 300 (sec)

Verify return code: 21 (unable to verify the first certificate)

---

As you can see, this is much shorter. The F5 was offering up only the server certificate and none of the chain. As the client only has the Root trusted, it cannot succeed in processing the chain. The only way a session like this can be trusted is if the client explicitly trusts the server certificate or the CA that issued the certificate.

I love s_client...

As it turns out, the F5 team forgot to load up the chain. Some systems won't even offer up SSL without a valid chain, but F5 is way better!