Thursday, March 15, 2012

Proper PKI Configuration for SSL/TLS Servers

Don’t Teach Bad Habits

 A large number of servers and applications are built with SSL configured incorrectly.  If your SSL is misconfigured, it is as good as no SSL.  In reality, it may be worse than no SSL, as it gives the feeling of safety where none exists.  This is most common when developers and administrators leverage a framework that they don’t fully understand. 

Any time a user sees one of the following errors or warnings, the server admin has failed.  A user, no matter how technical, should NEVER be forced to answer these questions.  Every time a user is trained to hit Yes, or OK to one of these errors, the user is being trained to fail.

Figure 1 IE SSL Error

Figure 2 Firefox SSL Error

Figure 3 Opera SSL Error

Figure 4 Chrome SSL Error




Different browsers may have slightly different text for different failures, but most users will think of them as the same thing, “The place where I have to do an extra click to get to my site”.   


Who is Protecting Whom

This document only covers standard, run of the mill SSL, where the server has a certificate and the client does not.  Two way or Mutual TLS is not commonly used and not covered here.  One of the things that confuses people about SSL is that it is designed to allow the client to protect itself and provides on protection or authentication of the client for the benefit of the server.   The server operator or application owner is supposed to be giving the client everything it needs to make sure the connection is secure.  Presumably the user is going to submit some data that he or she does not want disclosed.  If you are running an application where the liability is not on the client, standard one way SSL is not enough for you.  You are assuming a risk that you have no control over and you need additional controls.  This is the case with most online banking in the US.   If the customer loses his password to an attacker, and the attacker drains the account, in many cases, the bank is liable.  There have been a few cases recently where banks have been able to successfully put the burden on the customer.

It is safe to assume that if your application is misconfigured such that the client cannot accurately verify that the connection is secure, your end user will be able to make a successful case as to why the risk should be shifted to you, despite any Terms of Service or other contract that may state otherwise. 


Help Them Help Themselves

There are many criteria that the client can use to determine if it should trust a connection to a server.  While some clients may chose to ignore some of the criteria, it is wise to do everything right, and avoid any issues.

Subject Common Name

The Subject of the certificate is a description of the entity that offers it to a client.  To those familiar with LDAP, it may look like and x.500 Distinguished Name (DN), and it is in fact referred to as a DN.  For example:

subject=/C=US/ST=California/L=Mountain View/O=Google Inc/

However, most of the DN is arbitrary and with the inclusion of E=, for email address, it has no hierarchical convention like an LDAP DN.   While some lament this inclusion, unless there really is an LDAP to look at, who cares if the data is actually hierarchical… 

There are two exceptions to the “arbitrary” nature of the subject.  First, parties may choose to check the subject for a specific value.  Cisco VPN concentrators, for instance, can check that a certain OU or L is included.

By far the most important, for SSL/TLS is the CN, or common name.  Most clients check to make sure that the CN matches the name that the client is attempting to connect to.   This may be the most common configuration error.  The reason for this is that as a web developer, you can’t control what the user enters in their address bar.  Maybe the user will type or maybe they will type, or they might try and access some named service like   They may even type in an IP address.  Often two or more of the sites may be hosted on the same server at the same IP address.  To keep things nice and consistent, many web developers may redirect the connection to the pretty named site.  This is how google does it if you enter into the address bar:

HTTP/1.1 301 Moved Permanently



While this works great with non-SSL sites, this will fail with HTTPS unless you plan correctly.  The reason that this fails is that the SSL negotiation happens BEFORE HTTP data can be passed.  You can’t have an HTTP redirect if the SSL negotiation fails.  This is the reason that many think that you cannot use host header based virtual sites with SSL.  They are wrong, you just have to know what you are doing and get the right certificate.

We have two different options, if we want a certificate to work with multiple names.


Wild Cards Certificates

One answer to the problem of name matching is to just get one certificate that can answer for all hosts in your domain.  RFC 5425 defines a wildcard that can match any host in a domain, so the subject common name * would work for,, and  While some clients seem to allow the wildcard to match domains as well as hosts, the RFC forbids this, so avoid it.  For example, * should not work for, as only the leftmost name is the host portion.  Everything to the right is domain, and the * is not supposed to figure into domain matching.  The domain here is


Subject Alternate Name

The other option, especially if you are using host headers for sites that do not share the same domain name, is to use the subject alternate name or SAN.  The SAN is a list of names that can also be checked by a client.  One certificate could then be used for and   While clients are supposed to be able to parse both the subject common name and the SAN list, some do this poorly, therefore it is wise to add the subject common name to the list of SANs as well.


Picking the Right Name

As stated above, the client will match the name it thinks it is connecting to to the names on the certificate.  In the case of a web browser, this is the name in the address or URL bar.  This means that if your site is behind a load balance, then the common name must match the DNS name that resolves to the load balancer.  The client has no idea there may be a cluster and load balancer.


Figure 5 Typical Web Cluster



In Figure 5, the proper common name for the certificate would be

In some corporate environments, users may bypass the load balancer in certain circumstances.  In this case, the certificate may need SANs or to use a wild card.


Time Validity

Certificates contain a Valid From and Valid To date.  The client will check to verify that the date of the connection attempt is between the two dates.  Don’t let your certificates expire.


Certificate Revocation List Distribution Points

Generally it is thought that only PKI operators need to be concerned with CRLs and CDPs (CRL Distribution Points).  This is not the case.  A CRL is a list of certificates that the issuing CA has determined are not to be trusted.  Not all certificates list a CRL, obviously if one does not exist, you don’t care about them.  If the client determines that the certificate it is inspecting is listed on the CRL, it should reject the connection.  The reason you should be concerned with CDPs, as an application owner, is that your clients will attempt to read them.  Different clients react differently when they fail to access the CRL.  Some clients will fail open and allow the connection and some will deny it.  If you know your clients are behind highly restrictive firewalls, you may need to make sure that they have access not just to your site, but also to the CDP.  This is a common place to fail in a B2B or corporate environment.


Basic Constraints, Key Usage, and Enhanced Key Usage

A good PKI book or web site would never lump these all together, but this is neither of those.  Each of these is a field that can describe how the certificate or the associated private key can be used.  Some clients may choose to make decisions based on these.  IE for instance will allow and SSL connection to a server with no EKUs listed, but if the EKU section is used and the server authentication usages is not listed, it will deny the connection.

If the Basic Constrains listed in the certificate of the CA that signed your SSL certificate doesn’t list it as a CA, the client should abort the connection.  This keeps someone from using a certificate and private key as if it were a CA.  In the past, vendors forgot to check this, and anyone who bought a legitimate SSL certificate could use it to sign other certificates that were otherwise cryptographically sound.

Key Usage should list Digital Signature and Key Encipherment; if Key Usage is defined without Key Encipherment the client should abort the connection.  If Key Usage is not listed, all uses are allowed.


The Chain

The server’s SSL certificate does not exist in a vacuum.  It is at the bottom of a PKI hierarchy.  Without the context of the PKI the SSL certificate is useless.  The beauty of using PKI to secure (authenticate, negotiate keys and provide integrity checks) an SSL connection is that this can be done with a relatively small number of trusted third parties.  One of the big reasons to leverage PKI is to avoid the risks and overhead of increasing the number of parties you, or your clients, have to trust. 

A quick reminder about key management; there is NO WAY to exchange key material securely, over an un-trusted network, without an out of band trust being established.  Diffie–Hellman does not provide authentication, so there is no way to know which party you are negotiating with.  RSA key pairs are useless as well, as the public key must be distributed securely in order for it to be trusted.  This is why OSs, Java, web browsers, and such have keystores as part of their distributions.  Ponder this; did the browser or java package you downloaded come over via SSL or could an attacker have poisoned the keystore in the package?

The root CA is the most important element in a PKI.  The trust it conveys is significant and makes it easy to extend trust to a huge number of parties.  In order to extend this trust, subordinate CAs can propagate that trust.  It is common for a well built PKI to be two or three tiers.

Figure 6 Typical 3 Tier PKI




To increase security and usability, one should never ask a client to trust a subordinate CA or SSL certificate explicitly.  Trust should always extend implicitly from the Root CA.   Any time you ask an end user to make the decision to trust a certificate, YOU HAVE FAILED.

It’s alright to configure dev and test wrong, right? NO!!  None of your testing is valid if you don’t configure your dev and test correctly.  Additionally, dev and test are your chance to make sure you fully understand how to configure your systems correctly.  In the parlance of sports, “You play how you practice”.  So don’t practice failing.   It is well worth the extra 10 minutes to get this right.

This hierarchy is built so that a security failure at a lower level can be eliminated by simply revoking a certificate and effectively pruning an entire branch of the PKI.  Figure 6 shows the chain belonging to, whereas Figure 7 shows the entire PKI.  If the A1 CA, in Figure 7, were compromised, the root could simply revoke the A1’s certificate and A1 and everything below it would no longer be trusted.  This has no negative effect on L1C or L1B or anything below them.

Figure 7 Big PKI




This type of design reduces the overall damage that can be done by a successful attacker.  Usually the root is offline in a vault, so it can’t be hacked.

If the client only trusts the Root, but the SSL certificate is signed by an issuing CA, like L1C in Figure 7, how does the client bridge the gap and get the trust extended to it?  The chain MUST be sent by the server as part of the SSL negotiation. 

This is another area of common misconfiguration.

Figure 8 Wireshark sees 3 certificates




We see in Figure 8, a properly configured server sending the SSL certificate, the Issuer CA certificate, and the intermediate CA certificate.   If we look at the Intermediate CA certificate, it will list its issuer as the Root, Entrust, which the clients hopefully trust.  This is enough data for the client to “build” the chain and cryptographically verify that no monkey business has occurred.

As part of building the chain, the client will perform all the checks above, time validity, not revoked, etc, on each certificate in the chain.  If any check fails, the client should abort the connection.  If the chain successfully builds to the root, then the client can allow the session and security is assured.  


Forcing SSL

Another common mistake is not forcing SSL and simply enabling it.  Some site owners and designers wrongly assume that the only time they need SSL is during authentication.  If the user’s authenticated session has any value at all, then the entire session must be SSL protected.  HTTP sessions are trivial for an attacker to hijack.  Additionally, configurations on a site that mix HTTP and HTTPS are prone to human error induced problems.   Rather than trusting that links include HTTPS or that only certain pages are set to require SSL, instead set the entire site to require SSL.   All the arguments about the extra processor load of SSL haven’t been relevant since 2003 or so. 


Why All the Failures?

It would be nice if there was an easy to identify, universally hated, body like congress that could be blamed for SSL failures.  Vendors meet this requirement well.  SSL certificate vendors point their fingers at the application and OS vendors and the application and OS vendors blame the SSL certificate vendors and the open source community.  The overall problem is that there is a knowledge gap between the vendors and the admins who setup the servers and applications.  Very few admins know what the chain is, how to send it, or how to verify that it is being sent.


Verifying the Chain is Sent

Getting the chain sent, in the first place varies by application and platform, so it is best covered later.  Checking that the chain is sent is easy.  Simply install openSSL and run this command:

openssl s_client -connect host:port


This returns a very handy output full of useful information, including:

Certificate chain

 0 s:/C=US/ST=Texas/L=Frisco/O=T-Mobile USA, Inc./OU=USA/

   i:/C=US/O=Entrust, Inc./ is incorporated by reference/OU=(c) 2009 Entrust, Inc./CN=Entrust Certification Authority - L1C

 1 s:/C=US/O=Entrust, Inc./ is incorporated by reference/OU=(c) 2009 Entrust, Inc./CN=Entrust Certification Authority - L1C

   i:/ incorp. by ref. (limits liab.)/OU=(c) 1999 Limited/ Certification Authority (2048)

 2 s:/ incorp. by ref. (limits liab.)/OU=(c) 1999 Limited/ Certification Authority (2048)

   i:/C=US/ incorp. by ref. (limits liab.)/OU=(c) 1999 Limited/ Secure Server Certification Authority



0 is the SSL certificate and the numbers go up.   The subject and issuer DNs are listed to make an easy visual verification of the chain.  You can see that this application does not send the root. If you wish to verify the entire chain cryptographically, you can use the –CAfile argument and reference your Root CA certificate, in PEM format.

This would include the following line, assuming the chain checked out:

Verify return code: 0 (ok)


Sending the Chain

This section is the real reason this document was written.  There is very little guidance on the web telling admins to send the chain or how to send the chain, and often vendor documents brush over this and or don’t even cover it.


This section may be a bit of a generalization, as many applications can run on windows that use their own SSL code.  If the application uses the OS’s SSL, CAPI or CNG, then the OS will attempt to build and send chain based on what it finds in the machine store, or the store of the account running the server or application.  This will apply to all MS server applications, such as IIS, Exchange, Active Directory, etc.  Unless the developer overrides the default behaviors, this will apply to .NET apps as well.  To make sure that the OS can find the certificates in the chain, make sure that they are all in the same “My” or “Personal” store that holds the SSL certificate.  Windows can find the certificates in the other stores as well, but it is nice to know there they are and be clear which certificates are which.

Figure 9 Windows Machine Stores



No matter which OS your JVM is running on, or the version of JVM you are running, there is one keystore that is part of the build.  The cacerts file found under lib/security holds the list of trusted CAs.  Java keystore files use passwords.  The default password for your cacerts file is ‘changeit’.  This is the list of CAs that you JVM will trust by default.  If a client side Java application will use SSL, this is the store that the trust will be based on, unless the developer made a decision to use a separate store.  The Java keystore format is proprietary, and is sometimes referred to as JKS.  Sun provides the executable ‘keytool’ to create, edit, and inspect keystore files.  Newer versions of keytool and Java seem to make it harder to mess up creating the keystore, but in older versions mistakes are very common and allowed by keytool.  The JKS file format is a password protected wrapper that should contain all the CA certificates in your chain.  Inside that wrapper is another password protected wrapper that holds the private key corresponding to your SSL certificate.   For best results, when creating your keystore file, import the CA certificates starting from the root and working down, installing the signed SSL certificate last. 

The process goes like this:

Italicizes text is entered by you.  Non-italics is command output.


Create the Keystore and Key Pair

From command prompt type the following to create keystore.

 keytool -genkey -alias sslcert -keysize 2048 -keyalg RSA -keystore keystore.jks


The following attributes must be specified.  Remember what the password is since it will be needed when setting up server and importing signed certificates.

Password: password

First and Last Name: (must be the FQDN of the server or the F5)

Organizational Unit: (ex. used Operations in particular example)

Organization: (ex. used Company USA.)

City or Locality: (ex. used Seattle)

State or Province: (ex. used Washington)

Two Letter Country Code: (ex. used US for United States)


Once attributes have been inputted, user is asked to verify information.

Select "y" and hit


The user is then asked for key password for < sslcert >, use same password type above and hit


Keystore has now been created.


Create the Certificate Signing Request

From the command line enter

keytool -certreq -alias sslcert -keystore keystore.jks -file ssl.req


Get the file ssl.req to the CA admin or vendor for signing.


Once the Certificate has been signed, the admin or vendor will return the certificate file.  In the example, this is certnew.cer.


Import the CA Hierarchy Certificates

Before importing the signed certificate into your keystore you will need to establish the trust by importing the CA Certificates.  Always start with the root CA and work your way down.  This example will show a three tier PKI.


keytool -import -trustcacerts -alias MYRoot -keystore keystore.jks -file "Company USA Root CA.cer"

Enter keystore password:  password

Owner: CN=Company USA Root CA

Issuer: CN=Company USA Root CA

Serial number: 311a0088059a3abe4afb6520a6f749d8

Valid from: Thu Jan 04 12:54:35 PST 2007 until: Mon Jan 04 13:00:47 PST 2027

Certificate fingerprints:

         MD5:  82:0D:0C:3C:24:09:D6:9C:84:A7:6D:C3:CA:44:F8:C0

         SHA1: 5F:62:6A:91:5D:0D:26:F6:FB:AF:76:92:8C:2A:F1:C6:CD:D2:66:0F

Trust this certificate? [no]:  yes

Certificate was added to keystore


keytool -import -trustcacerts -alias MYInt -keystore keystore.jks -file "Company USA Intermediate CA 01.cer"

Enter keystore password:  password

Certificate was added to keystore


keytool -import -trustcacerts -alias MYIssuer -keystore keystore.jks -file "Company USA Issuer CA 02.cer"

Enter keystore password:  password

Certificate was added to keystore


The alias creates a friendly name to list the CA in the keystore.  Each alias must be unique, so make up useful, but unique names.


Import the Signed Request File

The signed request file must be imported back into the original file.


keytool -import -trustcacerts -alias sslcert -keystore keystore.jks -file certnew.cer

Enter keystore password:  password

Certificate reply was installed in keystore


You now have a properly built Java keystore.



openSSL is a fantastic tool and set of libraries that is leveraged by many applications.  OpenSSL’s native format for dealing with certificates, certificate signing request, and keys is the PEM format.  OpenSSL is the most versatile tool for dealing with certificates and keys, as it can convert to and from almost any format, except the Java keystore. 

A typical PEM formatted certificate looks like this:









The chain file is just the individual CA certificates in one file, end to end, starting from the issuing CA, moving up to the root, like so:




















Note that the begin and end certificate statements are left in place. 

To create a key pair and certificate signing request do the following:

openssl req -new -newkey rsa:2048 -nodes -out file.csr -keyout file.key

This will output your key files and CSR.  The vendor or PKI admin should sign the CSR and return your SSL certificate as well as the CA certificate in your chain.

With the returned files and openSSL, the sky is the limit.



First off, there are a couple of ways to configure HTTPS on apache.  The main methods are via mod_ssl and Apache-SSL.  Apache-SSL has not been updated in a while and should be avoided.

Mod_ssl uses openSSL libraries, so you will need openSSL to do anything fun.  The native file format for openSSL is called PEM or base64, which are text files, rather than the DER format, that is binary. 

The file that will contain your SSL settings is the httpd.conf file.  There are three settings that you will care about.

SSLCertificateFile  -  This is the actual SSL certificate

SSLCertificateKeyFile  -  this is the associated private key file.  If there is a pass phrase on the key, it must be entered at startup.  This may not be appropriate for some environments.

SSLCertificateChainFile  - This is chain file as described above.

F5 Local Traffic Manager – BigIP

First of all, you probably should not be terminating SSL on your F5.  “Need to know” or “reduced surface area” says that you don’t want the F5 techs to be able to see your decrypted traffic.  There are few good cases for decrypting SSL at the F5.  If you need to inspect host headers or URLs, this may be necessary.

F5 uses openSSL so the section above will help get the needed key and cert material.

In the SSL Client Profile, there is an option to set the Chain file.  Simply copy over the chain file in the openSSL format and apply the setting.


Forcing SSL

SSL should be forced at the web site level, to eliminate human error.


In IIS you can define the SSL settings at the site level, virtual directory level, or the page/object level.  The best place to do this is at the site level.

Figure 10 Requiring SSL in IIS6

Figure 11 Requiring SSL in IIS7



In order to make a better experience, consider changing the default 403.4 HTTP error page to a redirect page, so the user does not get an ugly error.



In the httpd.conf file, use the directive SSLRequireSSL to require SSL all the time.

In order to make the end user experience better, consider implementing a redirect from HTTP to HTTPS, rather than just blocking HTTP.



The LTM does not have extra ports setup by default.  If you do not setup the HTTP port, the user will get a server not found error.  Consider creating a VIP and iRule that redirects the user to your HTTPS site.

The iRule would look like this:


HTTP::redirect "https://[HTTP::host][HTTP::uri]"



This simply redirects the browser, rewriting the URL with and HTTPS replacing the HTTP.


Not Covered

A few things may seem to be missing here.  This document focuses on PKI related configuration.  Different apps and OSs also have other SSL/TLS settings that are not directly related to PKI. 


Cipher Suites

As part of the SSL negotiation, the client and server agree on a ciphersuite that is common to them both, hopefully the strongest common suite.  As attacks and computing power change, the “best” ciphers change.   It is called a suites, as it defines ciphers users in four areas, authentication, key exchange, messaged integrity check (MAC), and the bulk (symmetric) cipher used, including mode, for block ciphers.  There is a wide array of combinations of these parameters.  As the suite is only as strong as its weakest link, most ciphersuites should be eliminated from your configuration.  As of December 2011, you should eliminate any suite that sets any of the four areas to null.  You should eliminate any suite that relies on MD5 for its MAC. You should eliminate any ciphersuite that uses a block cipher in CBC mode.  One should also eliminate suites that utilize a symmetric cipher with a key shorter than 128 bit.

For more on CipherSuites, see



Microsoft allows you to pick the ciphersuites you will allow, as well as prioritize how they will be negotiated.  A good place to start is here .

We aware that XP/2003 and Win7/2008 have different settings and available suites.



The Java Class SSLSocket allows the developer to adjust the suites used.





Apache support for changing your ciphersuite can be found here



Mixed Content

I won't go into how to fix it, or configure your systems, cause you don't pay me...  but, creating a page that uses SSL, but that has elements in the page that are not over SSL is called creating mixed content.  This is common for those who use CDNs, as their images are all served up via HTTP.  



Mixed content gives the user fun symbols that indicate badness, but how bad??  We can't expect users to make the call on this kind of thing.

Figure 12 Google Reader Throwing Mixed Content Error in Chrome 


What's a user supposed to think when they encounter Figure 12?  Can your mom or grandfather get meaning out of this explanation? 



If you want to make the customer experience good, and avoid litigation, make sure to configure SSL on your web servers correctly.  Any failure in your configuration may put the client at risk, but it may be a risk that is assumed by you, due to contracts.

Make sure you:

  • Set the right subject common name
  • Send the chain
  • Have time valid certificates
  • Include the right usage information
  • Force SSL
  • Have verified that the CRLs can be reached

Friday, March 2, 2012

Enrique Salem, president and CEO, Symantec, I WANT MY $2

As my readers know, in May of 2011, I coined and began protecting the terms "Advanced Persistent Protection", "Advance Persistent Controls", and "Advanced Persistent Control Suite".

This week, my brilliant reader Sterling pointed out that I now have one peer who agrees with me, Enrique Salem, president and CEO of Symantec.  In a post for SC Magazine, he makes four bullet points about Advanced Persistent Protection. I'm thinking three is the limit for reasonable use.  If Symantec is going to start pimping my copyrighted phrase, they need to pay!

I am currently in talks with Michael “Let’s Get Ready to Rumble” Buffer's legal team to get my cash out of these guys.

Symantec, I have no desire to litigate as long as you give me what I am due.  I'm glad to take a per use fee or we can negotiate an exclusive long term deal if you'd prefer.  You'd better hurry though, RSA has already started talks. I'd hate for this to sink your upcoming marketing campaign.


Inputting falsified referrals to this site violates the terms of service of this site and is considered unauthorized access (hacking).