Godaddy Asks People NOT TO USE ITS HOSTED EMAIL and May Not Even Use It Themselves

Disclaimer, Godaddy made me angry with a billing issue.  This is what caused me to look into the value I get from them.  While my language may be angry and inflammatory, the facts are not disputable.  I have informed them about their messed up SMTP TLS, but have not heard back.

Try to send a secure mail to Godaddy hosted addresses and they will return this message

Sample server certificate, do not use on production systems!

Maybe they are hosting customers’ mail on non-production systems.   For additional irony, they are hosted in the domain, secureserver.net.

farmtomarketcreations.com. 3600 IN      MX      0      smtp.secureserver.net.

farmtomarketcreations.com. 3600 IN      MX      10    mailstore1.secureserver.net.

Even more irony!!!!  Godaddy doesn’t even use their own hosting for email, they use Microsoft!

                godaddy.com.            3600    IN MX      0     godaddy-com.mail.protection.outlook.com.

OK, so this could be that they just use MSs Cloud Anti-Spam and then relay the spam free mail into their systems, but I am dubious.

A quick word about SMTP and TLS.  It is a great way to keep mail more secure because it does not require an end user to know anything or that it is even there.  It just requires mildly qualified techs to configure their mail servers correctly.  TLS, done right, will protect the message in transit from one mail system to the next. 

Back to the hosting I pay for.  While the MX records do not change for my hosting, the corresponding A records change a bit, and multiple tests against the same IP render different results, in terms of TLS support. It appears they use technologies like global traffic management, round robin DNS, and load balancers, and every host was configured by a different incompetent tech.

Their MX records for both names seem to correspond to the same 4 IPs

smtp.secureserver.net. 300     IN      A       72.167.238.201

smtp.secureserver.net. 300     IN      A       72.167.238.29

smtp.secureserver.net. 300     IN      A       68.178.213.37

smtp.secureserver.net. 300     IN      A       216.69.186.201

mailstore1.secureserver.net. 300 IN     A       68.178.213.37

mailstore1.secureserver.net. 300 IN     A       216.69.186.201

mailstore1.secureserver.net. 300 IN     A       72.167.238.201

mailstore1.secureserver.net. 300 IN     A       72.167.238.29

Here’s what CheckTLS shows me over a decent number of tests. Never a score above 68 and never once a valid SSL certificate.  Of 18 tests, 2/3rds fail to even allow TLS.

So, let’s look at the hosts that do offer up SSL/TLS certificates.   First, they send their Root certificate twice, adding to handshake time and size.  The root and SSL certs are both 1024 bit.  We already covered the clearly stated “Do not use”.   The SSL certificate is good for 10 years?  At least it is not expired.  :-P Crazy… Finally, the subject common name on the SSL certificate doesn’t match any of their server names.   I guess they can’t afford certificates… Wait, isn’t Godaddy an SSL cert provider?

Certificate 1 of 3 in chain:

Certificate:

  Data:

    Version: 3 (0x2)

    Serial Number: 2 (0x2)

    Signature Algorithm: sha1WithRSAEncryption

    Issuer:

      countryName         = US

      organizationName      = Sample, Inc.

      organizationalUnitName  = IT Team

      commonName        = CA

    Validity

      Not Before: Nov 18 14:58:26 2010 GMT

      Not After : Nov 15 14:58:26 2020 GMT

    Subject:

      countryName         = US

      organizationName      = Sample, Inc.

      organizationalUnitName  = IT Team

      commonName        = Server

    Subject Public Key Info:

      Public Key Algorithm: rsaEncryption

        Public-Key: (1024 bit)

        Modulus:

          00:f3:89:dd:43:f0:ad:84:1a:dd:f1:fd:2c:83:bd:

          ae:01:17:d8:ab:4e:02:f4:7f:85:0a:ec:70:5e:8b:

          19:69:78:6c:61:b8:82:5b:dd:e8:ea:48:23:6b:9f:

          68:80:76:67:34:d3:94:e7:a4:54:38:bb:72:c7:ba:

          da:cc:d6:cb:f8:6b:91:53:f2:be:44:61:9c:a0:64:

          d1:02:e8:df:5b:95:7f:ae:e3:82:d1:e7:2a:96:eb:

          53:9e:17:b3:f5:d9:d1:7a:ca:dd:74:1e:97:3a:44:

          54:5d:02:54:8d:f0:7b:85:39:9f:e9:a3:f3:e7:20:

          14:1d:58:c9:f9:0d:63:fc:d3

        Exponent: 65537 (0x10001)

    X509v3 extensions:

      X509v3 Basic Constraints:

        CA:FALSE

      Netscape Comment:        

        Sample server certificate, do not use on production systems!

      Netscape Cert Type:

        SSL Server

      X509v3 Extended Key Usage:

        TLS Web Server Authentication

      X509v3 Key Usage:

        Digital Signature, Key Encipherment

  Signature Algorithm: sha1WithRSAEncryption

    38:d1:85:a8:51:8c:1b:04:a5:95:39:19:7c:6e:38:f6:e8:ef:

    27:23:40:17:11:ba:bc:7a:0c:be:39:ee:f4:2b:8d:5c:5d:dd:

    c4:ea:54:e1:d9:fd:7c:96:b2:a0:9b:67:cd:f9:06:ed:7e:02:

    8a:96:fd:f6:4d:bf:64:22:17:a5:9b:e3:33:15:7e:fe:a7:30:

    53:21:55:ba:20:c5:a6:19:50:f0:d2:44:e9:a9:1c:5a:37:20:

    cb:26:15:da:73:ba:67:29:f3:1d:f2:69:97:31:26:92:04:f9:

    6a:c3:ec:ff:6a:65:60:ef:78:54:44:7f:81:22:24:aa:e8:cd:

    fa:6b

-----BEGIN CERTIFICATE-----

MIICkTCCAfqgAwIBAgIBAjANBgkqhkiG9w0BAQUFADBDMQswCQYDVQQGEwJVUzEV

MBMGA1UEChMMU2FtcGxlLCBJbmMuMRAwDgYDVQQLEwdJVCBUZWFtMQswCQYDVQQD

EwJDQTAeFw0xMDExMTgxNDU4MjZaFw0yMDExMTUxNDU4MjZaMEcxCzAJBgNVBAYT

AlVTMRUwEwYDVQQKEwxTYW1wbGUsIEluYy4xEDAOBgNVBAsTB0lUIFRlYW0xDzAN

BgNVBAMTBlNlcnZlcjCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA84ndQ/Ct

hBrd8f0sg72uARfYq04C9H+FCuxwXosZaXhsYbiCW93o6kgja59ogHZnNNOU56RU

OLtyx7razNbL+GuRU/K+RGGcoGTRAujfW5V/ruOC0ecqlutTnhez9dnResrddB6X

OkRUXQJUjfB7hTmf6aPz5yAUHVjJ+Q1j/NMCAwEAAaOBkDCBjTAJBgNVHRMEAjAA

MEsGCWCGSAGG+EIBDQQ+FjxTYW1wbGUgc2VydmVyIGNlcnRpZmljYXRlLCBkbyBu

b3QgdXNlIG9uIHByb2R1Y3Rpb24gc3lzdGVtcyEwEQYJYIZIAYb4QgEBBAQDAgZA

MBMGA1UdJQQMMAoGCCsGAQUFBwMBMAsGA1UdDwQEAwIFoDANBgkqhkiG9w0BAQUF

AAOBgQA40YWoUYwbBKWVORl8bjj26O8nI0AXEbq8egy+Oe70K41cXd3E6lTh2f18

lrKgm2fN+QbtfgKKlv32Tb9kIhelm+MzFX7+pzBTIVW6IMWmGVDw0kTpqRxaNyDL

JhXac7pnKfMd8mmXMSaSBPlqw+z/amVg73hURH+BIiSq6M36aw==

-----END CERTIFICATE-----                                                                                                                                                                                                                                      

[003.724]                             

Certificate 2 of 3 in chain:

Certificate:

  Data:

    Version: 3 (0x2)

    Serial Number:

      92:69:45:90:f7:aa:ec:38

    Signature Algorithm: sha1WithRSAEncryption

    Issuer:

      countryName         = US

      organizationName      = Sample, Inc.

      organizationalUnitName  = IT Team

      commonName        = CA

    Validity

      Not Before: Nov 18 14:58:26 2010 GMT

      Not After : Nov 15 14:58:26 2020 GMT

    Subject:

      countryName         = US

      organizationName      = Sample, Inc.

      organizationalUnitName  = IT Team

      commonName        = CA

    Subject Public Key Info:

      Public Key Algorithm: rsaEncryption

        Public-Key: (1024 bit)

        Modulus:

          00:b8:ed:8e:6f:80:6c:10:56:21:16:b9:53:a9:00:

          95:b9:60:18:48:76:fd:a2:e4:b9:4c:21:03:d9:17:

          0f:f7:09:ae:31:f6:1b:ee:3f:d9:d6:fe:53:70:84:

          5b:df:63:7b:f4:a2:9d:34:4f:0e:55:33:e6:ee:a7:

          4c:b9:43:0f:70:51:71:bc:ee:50:6c:fd:4e:41:f2:

          4d:cf:9d:9c:94:a4:40:85:e9:27:74:08:78:fc:f6:

          2e:e4:a9:d5:3e:8b:27:a9:ed:52:06:45:a5:76:a4:

          2a:8d:2a:10:e9:31:6d:5a:6a:75:34:10:4c:85:9a:

          5d:4d:43:3a:24:59:95:29:7f

        Exponent: 65537 (0x10001)

    X509v3 extensions:

      X509v3 Basic Constraints:

        CA:TRUE

  Signature Algorithm: sha1WithRSAEncryption

    a0:04:32:10:92:46:dd:bd:e4:c2:71:0a:b2:d5:5c:9c:1c:8c:

    57:fa:3a:17:ed:aa:d9:28:09:f3:79:3b:18:d0:4d:e0:f8:6c:

    96:a4:b0:52:f9:8a:cd:bc:cf:1c:79:2a:6e:97:4a:89:4e:bb:

    f9:9b:0c:4c:e0:fe:a1:0f:53:7d:6b:04:3b:9b:05:1b:b7:37:

    13:ae:9d:02:58:14:7f:cc:d5:be:26:55:4e:02:15:bb:ec:9f:

    7d:b6:5e:fa:ea:c8:88:b1:b6:57:62:69:ba:c2:b1:d2:2f:a7:

    99:24:90:eb:52:a5:58:20:22:83:33:2c:37:64:84:0e:e9:46:

    90:53

-----BEGIN CERTIFICATE-----

MIICFDCCAX2gAwIBAgIJAJJpRZD3quw4MA0GCSqGSIb3DQEBBQUAMEMxCzAJBgNV

BAYTAlVTMRUwEwYDVQQKEwxTYW1wbGUsIEluYy4xEDAOBgNVBAsTB0lUIFRlYW0x

CzAJBgNVBAMTAkNBMB4XDTEwMTExODE0NTgyNloXDTIwMTExNTE0NTgyNlowQzEL

MAkGA1UEBhMCVVMxFTATBgNVBAoTDFNhbXBsZSwgSW5jLjEQMA4GA1UECxMHSVQg

VGVhbTELMAkGA1UEAxMCQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALjt

jm+AbBBWIRa5U6kAlblgGEh2/aLkuUwhA9kXD/cJrjH2G+4/2db+U3CEW99je/Si

nTRPDlUz5u6nTLlDD3BRcbzuUGz9TkHyTc+dnJSkQIXpJ3QIePz2LuSp1T6LJ6nt

UgZFpXakKo0qEOkxbVpqdTQQTIWaXU1DOiRZlSl/AgMBAAGjEDAOMAwGA1UdEwQF

MAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAoAQyEJJG3b3kwnEKstVcnByMV/o6F+2q

2SgJ83k7GNBN4PhslqSwUvmKzbzPHHkqbpdKiU67+ZsMTOD+oQ9TfWsEO5sFG7c3

E66dAlgUf8zVviZVTgIVu+yffbZe+urIiLG2V2JpusKx0i+nmSSQ61KlWCAigzMs

N2SEDulGkFM=

-----END CERTIFICATE-----                                                                                                                                                                                                                                                                                                                                                                                          

[003.768]                             

Certificate 3 of 3 in chain:

Certificate:

  Data:

    Version: 3 (0x2)

    Serial Number:

      92:69:45:90:f7:aa:ec:38

    Signature Algorithm: sha1WithRSAEncryption

    Issuer:

      countryName         = US

      organizationName      = Sample, Inc.

      organizationalUnitName  = IT Team

      commonName        = CA

    Validity

      Not Before: Nov 18 14:58:26 2010 GMT

      Not After : Nov 15 14:58:26 2020 GMT

    Subject:

      countryName         = US

      organizationName      = Sample, Inc.

      organizationalUnitName  = IT Team

      commonName        = CA

    Subject Public Key Info:

      Public Key Algorithm: rsaEncryption

        Public-Key: (1024 bit)

        Modulus:

          00:b8:ed:8e:6f:80:6c:10:56:21:16:b9:53:a9:00:

          95:b9:60:18:48:76:fd:a2:e4:b9:4c:21:03:d9:17:

          0f:f7:09:ae:31:f6:1b:ee:3f:d9:d6:fe:53:70:84:

          5b:df:63:7b:f4:a2:9d:34:4f:0e:55:33:e6:ee:a7:

          4c:b9:43:0f:70:51:71:bc:ee:50:6c:fd:4e:41:f2:

          4d:cf:9d:9c:94:a4:40:85:e9:27:74:08:78:fc:f6:

          2e:e4:a9:d5:3e:8b:27:a9:ed:52:06:45:a5:76:a4:

          2a:8d:2a:10:e9:31:6d:5a:6a:75:34:10:4c:85:9a:

          5d:4d:43:3a:24:59:95:29:7f

        Exponent: 65537 (0x10001)

    X509v3 extensions:

      X509v3 Basic Constraints:

        CA:TRUE

  Signature Algorithm: sha1WithRSAEncryption

    a0:04:32:10:92:46:dd:bd:e4:c2:71:0a:b2:d5:5c:9c:1c:8c:

    57:fa:3a:17:ed:aa:d9:28:09:f3:79:3b:18:d0:4d:e0:f8:6c:

    96:a4:b0:52:f9:8a:cd:bc:cf:1c:79:2a:6e:97:4a:89:4e:bb:

    f9:9b:0c:4c:e0:fe:a1:0f:53:7d:6b:04:3b:9b:05:1b:b7:37:

    13:ae:9d:02:58:14:7f:cc:d5:be:26:55:4e:02:15:bb:ec:9f:

    7d:b6:5e:fa:ea:c8:88:b1:b6:57:62:69:ba:c2:b1:d2:2f:a7:

    99:24:90:eb:52:a5:58:20:22:83:33:2c:37:64:84:0e:e9:46:

    90:53

-----BEGIN CERTIFICATE-----

MIICFDCCAX2gAwIBAgIJAJJpRZD3quw4MA0GCSqGSIb3DQEBBQUAMEMxCzAJBgNV

BAYTAlVTMRUwEwYDVQQKEwxTYW1wbGUsIEluYy4xEDAOBgNVBAsTB0lUIFRlYW0x

CzAJBgNVBAMTAkNBMB4XDTEwMTExODE0NTgyNloXDTIwMTExNTE0NTgyNlowQzEL

MAkGA1UEBhMCVVMxFTATBgNVBAoTDFNhbXBsZSwgSW5jLjEQMA4GA1UECxMHSVQg

VGVhbTELMAkGA1UEAxMCQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBALjt

jm+AbBBWIRa5U6kAlblgGEh2/aLkuUwhA9kXD/cJrjH2G+4/2db+U3CEW99je/Si

nTRPDlUz5u6nTLlDD3BRcbzuUGz9TkHyTc+dnJSkQIXpJ3QIePz2LuSp1T6LJ6nt

UgZFpXakKo0qEOkxbVpqdTQQTIWaXU1DOiRZlSl/AgMBAAGjEDAOMAwGA1UdEwQF

MAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAoAQyEJJG3b3kwnEKstVcnByMV/o6F+2q

2SgJ83k7GNBN4PhslqSwUvmKzbzPHHkqbpdKiU67+ZsMTOD+oQ9TfWsEO5sFG7c3

E66dAlgUf8zVviZVTgIVu+yffbZe+urIiLG2V2JpusKx0i+nmSSQ61KlWCAigzMs

N2SEDulGkFM=

-----END CERTIFICATE-----  

-->

Demystifying Certificate Requirements in Mutual TLS

Understanding Certificates and SSL/TLS long ago became an IT fundamental.

Somehow, the industry seems to have not noticed.  In my quest to take fewer calls on this stuff, here is my attempt to help demystify all the certificates involved in Client SSL/Mutual TLS.  I seem to be spending 2+ hours a day on the phone talking web and server admins through this stuff.

The reason I am doing this is because Google failed me.  There are a lot of docs that focus on all the other aspects of the SSL/TLS negotiation, but none of them focuses on the certificates and what they do for us. Microsoft has one of the better papers on SSL/TLS, but there is a lot lacking around the certificates and the SSL/TLS handshakes are overly summarized.

Here is the handshake that MS shows us.

While this may be a four message handshake, there are a few different conversations taking place, simultaneously, in the handshake.  The conversations and their certificate requirements are more easily understood when they are pulled out of the stick context of the message format.

I will say that I am doing a bit of summarizing.  SSL/TLS has four major components: authentication, message integrity, key negotiation and encryption.  This post focuses heavily on the authentication aspect.  I am lumping SSL and TLS together as the basic functions of the negotiations are the same, only the implementation details vary a bit.  I am also leaving out a bit about session key generation. 

I break the handshake down into 4 different conversations.

A.      Hellos

a.       The client tells the server that it wants to go secure and offers the suite of ciphers it supports.

b.      The server selects a cipher from the list.

c.       No certificates are involved here.

B.      Server Certificate presentations

a.       The server sends over the SSL certificate AND the chain. 

b.      The server must have the private key for the SSL certificate.

c.       If the chain, (CA hierarchy) is not sent, negotiation MUST fail.

d.      The client SHOULD perform full cert and chain validation against these certs, making sure they chain to a Root CA that is trusted explicitly by the client.

C.      Key Exchanges

a.       The client and server exchange data that allows the negotiation of session encryption keys.

b.      If EDH is used, then the RSA/DSS key is used to block MITM of the key negotiation.

c.       If RSA key negotiation is used, the client encrypts its initial key material with the public key on the server SSL certificates, assuming cert and chain validation succeeded.

D.      Client Certificate  presentations

a.       The server issues a Certificate Request.  This lists the key types allowed on the certificate and a list of Distinguished Names (DNs).  These DNs are used as a hint to help the client determine which of its many possible certificates is the “right” one.

b.      The client selects a certificate, for which it MUST have the private key, and sends the certificate to the server.

c.       The client sends data that is signed using its private key, proving it holds the certificate in question.

d.      The server checks the signature.  If the signature passes, the server may be configured to pass/fail the client based on data in the certificate.  Often the certificate serial number or Subject DN must match a static list, or the certificate must be one that is on file.  The decision to pass/fail the client cert is not specified in any RFC.

 

 

More below

Certificates Needed

	Client	Server
A - Hellos	None	None
B - Server Certificate	Root CA Cert for Trust	SSL Cert and Chain to send
C - Key Exchange	Public key from Server Cert	Private key for Server Cert
D - Client Auth	Client Cert and Private key	Client cert for explicit trust or CA cert to implicit trust.  Cert values MAY be checked.

 

Client and Server Perspective

One often forgotten part of this is that when there are two parties, the role of client and server may change.   For instance, let's say A pushes a full DB copy every night to B, and then B pulls deltas hourly during the day. In this scheme, at night, B is the server, but during the day, B is the client.  This means that both parties need a cert and a server cert with chain, and they need to know how to implicitly or explicitly trust other clients, and they need Root CA certs so they can trust the server cert when they are the client.

Summary

Take the time to understand how your connections are established and it is fairly easy to figure out which certificates are used when. When it comes to figuring out what is wrong, you can usually figure out the issue by where the handshake fails.  The handshake failures usually can only be seen via packet captures.

 

Hello IT Person, Welcome to the Security Organization

This is a post that is long overdue.   The IT industry went through a revolution and most people in IT missed it and are still missing it. 

If you are in any form of IT related job, you are in the information security field.

You may say, “No, I’m just an IT Project Manager (analyst, whatever), security is another team”.  You are wrong and your career is heading towards a cliff.

It only takes a tiny bit of Googling to realize that everyone is getting hacked.  Even the biggies, RSA, Microsoft, Facebook, Symantec, and Apple aren’t immune. There are too many actors with too many motivations.  If it is connected to the web, there is either money in hacking it, or it can be used as a foothold to hack for money.  If it’s connected to the web, there is probably someone with a social or political agenda that makes it a target and if not, it is a platform for the hacktivists to leverage.  On top of the myriad of highly skilled and motivated attackers, there are thousands off wannabe hackers simply looking for low hanging fruit to test their skills or to get a thrill.  Even if you don’t have a penny to your name, your computing power is a commodity if it can be added to a botnet. Hopefully, you already know that it’s a given that everyone is a target.  If you still need persuading on this point--don’t worry, there are plenty of Wal-Marts that need greeters.  Get your app in early.  (That’s “app” as in “application”, which is a paper form you will fill out with a pen. You won’t need LinkedIn for this.—The Editor)

So, why is security your job?  Every day (week if you are a slacker?) you make decisions that impact security.  It doesn’t matter if you specialize in a niche, like UI or UX; or something broad, like program or project management.  If you are working with data in any way, that data has value to your organization.  There will certainly be negative impact if the data is compromised or corrupted.  Even if you run or maintain a static website in which the content is public and can easily be restored if lost, you don’t want your system to be a foothold into your important systems or take part in a DDoS attack.

Your Role

Backup Operators – Don’t lose the data and don’t lose those unencrypted backup files.  This role should be a no brainer.  This role has Confidentiality, Integrity, and Availability components.

Systems Administrators – Once again, this is a no brainer.  SAs have all the access themselves and they configure access to ALL of your data.  Remember, all of your applications sit on a host that an SA “owns”.

DBAs and DBOs – Please, for the love of all that is good, you guys MUST know you have to protect that data. Do you??

Project and Program Managers – I know lumping you guys onto one line will get me all sorts of hate mail.  You guys decide things like what use cases exist, for what users, accessing what data. You decide whether to engage formal security teams for assistance. You decide to cut product or project scope, and everyone knows security is the first to get cut.      

Developers – You guys are the worst.  Yeah, I’m saying it. Inventing your own “cryptography”, passwords in log files, backdoors, assuming users will use your apps the way you want them to, and on and on.  Seriously devs, get your acts together.    

Network Engineers – I think this is the only role that most folks actually think of as a security role.  In fact, this role is the least interesting in terms of security roles. Read those manuals and change those default passwords.  And stop using self-signed certs.  Can you really not remember how a MITM attack works long enough to say to a manager: “Wait, we need a real cert on that appliance admin portal page”?  You know what, if you can’t explain to a manager what a MITM attack is and why your choice of cert matters, you’re part of the problem.

 

Your day to day job may be focused on something else, but it is only a matter of time until IT folks start getting fired for massively bad security lapses.  If it is on the OWASP Top 10 or the SANS Top 25 and you don’t know a bit about it, you may want to pick up that Wal-Mart job app.  No one is asking you to figure everything out about security, but you need to at least understand some basics and keep your eyes out.  If you do run a system, you’d better become a security expert in the context of that system.

 

 

NTLM hasn’t been relevant for like 12 years... and other lies.

A surprising number of foolish Slashdotters have pointed out that my latest work, breaking the NTLM and LM handshakes and phishing for users’ NT hashes, is totally irrelevant and has been for 12ish years.  

As a fan of debate, I’ll start with points that are interesting but have no real bearing on the topic.

• Slashdotters are clearly not qualified to make this assessment.  Their Appeal to Authority fails.
• Microsoft wouldn’t issue an advisory and Fix-It if it weren’t relevant.  My Appeal to Authority is better than theirs.  ;-)
• The existence of a newer protocol, Kerberos, does not make NTLM simply disappear.
• Wikipedia actually doesn’t say NTLM is long dead.  Wikipedia as an appeal to authority is a joke.  I link to it regularly, not for its completeness, but because it is written for a layman audience.  It is a great place to start if you don’t know something.
• I’m not a Linux fanboy looking to disgrace MS.  I’m a long time MCSE and even gave MS some props in my post.  
• Finally, my work is what it is, probably the last nail, of hundreds, in the coffin.  I make no claim to it being inspired by God.

Now to the relevant details...  When MS introduced Active Directory in Windows 2000, they implemented Kerberos 5 as the default authentication protocol FOR DOMAIN ACCOUNTS. This is a pretty important requirement. If a machine is not domain joined, or the account is not a domain account, Kerberos is not an option. The upside here is that when machines are in workgroups, it is much less likely that the accounts will have any sort of value off of the host.  However, this does not stop the SSPI from trying to authenticate using your cached account credentials when accessing resources that are not on the host.  This means a workgroup host could still be vulnerable to my “Send the Hash” attack if not properly configured.

Even for machines that are domain joined, while Kerberos is the default, NTLM is used in several situations:

• If the service is not Kerberos enabled (Kerberized).  Maybe it runs on an NT server?
• The service/server does not have a Service Principal Name (SPN) registered.
• The service/server has duplicate SPNs registered.   
• When accessing the system by IP rather than name.
• Improperly built clusters.
• 3rd party system implemented incorrectly.
• When accessing data across forests, using an older domain type trust.
• When the client can’t access a KDC/DC, such as when it is outside the firewall.
• When the KDC/DC is behind NAT.

Before getting in depth with a couple of these cases, I’ll make a generalization, “Kerberos is very hard to get right, except under simple conditions.”


Outside the Firewall or Behind NAT
MS’s implementation of Kerberos requires that the clients, servers, and KDC/DCs all be on the same routed network with AD integrated DNS or DDNS that allows the DCs to register SRV records. The clients must be able to find and access the KDCs to get Kerberos tickets.  I am not going to cover all the details of Kerberos, but this is a key difference.  With NTLM, the server you want to access does the job of finding a DC and getting the DC to validate the challenge/response after your client has done its handshake.  The resource server passes the challenge and response to the DC over RPC using packet privacy and gets back a pass/fail and a list of group memberships which it uses to build the user’s access token. This is super simple and easy when the client is outside your firewall.  You only need to open one port, the application's port.  

If you intend to make Windows Kerberos work across NAT or behind a firewall, prepare for pain.  Each Windows client has a component called the dcLocator.  Its exact operations vary slightly from version to version of Windows.  You might think you just need to open up TCP88 to a KDC and you’re set.  You might get a pony in the mail too.  

I’ll blog on the exact details at some point, but the dcLocator first needs to find the KDC DNS SRV records in the _msdcs.domainname.org zone.  Right off the bat, this means that you need split DNS, as the answers inside your firewall will not be the same IP as outside. Once you have your external DNS zone and main DC records, the client will ping all the DCs and select the fastest to respond. If ICMP is blocked, nothing proceeds. The client sends a CLDAP query to the fastest DC. This is connectionless LDAP over 389 UDP.  This query is to ask which AD site the client is a member of.  This query is not answered in a traditional way, based on the filter.  Instead, AD uses the source IP to map the IP to an AD subnet which maps to an AD site, which the LDAP search response will contain.  If your client is behind NAT, then the source IP will likely be a SNAT address.  From this response, the dcLocator then does a second DNS SRV query to get the DCs that are in the AD site returned from the CLDAP query.  The dcLocator then pings each of those DCs and the first to respond is queried and if the response is satisfactory, then this becomes the default DC for a period of time.  This time can vary by OS version.  Now we are ready to do Kerberos. Some versions of Windows try UDP88 first and then when they get back the “response too large” they try TCP88 route. If UDP is blocked, these versions may not try TCP88 even if it is open. I will not be swearing to this in court as it has been over a year since I configured this type of scenario and I am writing this without a net.  

This means, that for Kerberos to work outside the firewall or behind NAT, you need to:

• Setup Split DNS
• Create at least one domain level SRV pointing to the external IP address
• Create a site DNS SRV record for EVERY DC in the default site, pointing to the external IP address.
• Open port 389 UDP
• Open port 88 UDP
• Open Port 88 TCP
• Open ICMP

Kerberos Diagram

NTLM is a lot easier to use in both NAT and outside the firewall scenarios.

NTLM Diagram


Messed up SPN Scenarios

A Service Principal Name (SPN) maps an instance of a service running on a server to the account that it runs under.  Kerberos uses shared secrets, between each party and the KDC/DC, to allow for authentication and key exchange.  When a client wants to use a service, it asks the KDC for a ticket to the specific service/server combination.  The ticket is encrypted by the KDC so that only the service can open it.  If there is no SPN registered for the service, a ticket can’t be granted.  If two accounts have the SPN registered, the ticket cannot be granted.  If the service is running under the context of a different account, the ticket cannot be decrypted.  

Clusters

Many web admins just allow their appPools to run as local system or network service.  This means they are running under the machine account, which is a domain account and has an SPN or at least can have one.  In order to make your cluster work with Kerberos, all appPools must run under the same domain account and usually an SPN must be manually created by an admin. MS has made some changes to IIS to make this easier, but in many cases it is out of the frying pan...

SQL

SQL server tries to register its SPN whenever it comes online.  If you are following best practices, and using a domain account to run SQL, then SQL will try to register the SPN attribute on that account.  This is a proper configuration, however most accounts so not have the SELF write SPN permission (ACE), and fail.  This means that Kerberos won't work and NTLM is negotiated.

Connection by IP Address

This fails as there is not an SPN set that uses the IP, such as HTTP/10.0.0.5. Rather the SPN is HTTP/www.domain.com.  This can be overcome by registering the IP SPN, but does not scale well.  I’ve thought about building a simple tool to do this, but my list of to-dos is long.

In the end, the threat and attack models are very different for the enterprise and stand alone users.

Enterprise Exposure

The enterprise is likely to block many outgoing ports, making it less likely that “send the hash” attacks will succeed.  This however doesn’t mean there is no risk from NTLMv1.  With any type of foothold, the attack is VERY effective.  All it takes is one compromised host, one badge not checked, a network port activated outside of your secure areas, etc.

Enterprise users spend time in hotels, airports, hospitals, etc. These types of places are ripe for the picking. 

Workgroup Exposure

For workgroup members, the threats vary considerably.  The user may spend much of their time behind NAT, which may make it hard for an attacker who steals a hash to use it.  The user may spend time in coffee shops, etc, making them ripe for attack.

Summary

While Kerberos is more secure than NTLMv2, it is not really fair to say it is better.  They both have pros and cons.  NTLM is VERY commonly used today both by design and due to it being a fallback for failed Kerberos.  Depending on your systems’ settings, you may be sending LM and NTLMv1.  ALL VERSIONS of Windows still accept LM and NTLM by default, so you may still be allowing the issue. You may not be initiating dirty handshakes, but you will accept them if offered.

Finally, the fix is very simple. There is almost no downside to making the change. If due to some crazy twist of fate you have a system that only works with NTLMv1 or LM, that system will break.

-->

NTLM Challenge Response is 100% Broken (Yes, this is still relevant)

Updated again: CloudCracker has dropped off the web. A new successor has arisen. Meet Crack.sh. 

Updated as it seems a lot of folks don't read. The reason this attack is interesting is that I steal or phish the hash in the attack WITHOUT physical or administrative access. Yes, I am aware of the 30 amazing tools that "already did that" using local admin access. Also here for why this is still matters.

First off, I can’t say I broke the NTLM handshake; the march of time did it.  Apparently, I am just the one who bothered to put it together.

There have been numerous whitepapers, hacker conference sessions, and blog posts dedicated to the weaknesses of NTLM (and LM) authentication.  However, the weaknesses described in previously published works were theoretical, or required stealing hashes using admin rights. This means the host was already compromised, thus the exploits themselves are a bit boring and redundant.  One couldn’t just phish for the hashes or MITM the hashes; due to the challenge response mechanism.   The best case for getting to the hash or password from outside the host was to do a MITM attack (or a phish) and substitute a chosen challenge.  This only worked if the victim was willing to negotiate NTLM without the Session Security Flag. This would then allow an attacker to build rainbow tables to get the hash or password. Rather, the attacker probably already had tables built for the chosen challenge.  This scenario is a pretty high bar to reach.

To frame the conversation, there are actually 4 handshakes in the NTLM suite that move up in security and complexity. They are LM, NTLM, NTLM with session security, and NTLMv2.  As of now, only NTLMv2 stands as secure.  There is no way, other than encrypting your link with say IPSec, to secure the 3 weak handshakes.  The good news is, all Microsoft OSs already have a registry key that can control the handshakes options.  MS will release a KB and Advisory on 1/8 on this and the information can be found below.

Now, thanks to Moxie Marlinspike’s Cloudcracker, an attacker can skip the pre-chosen challenge and brute force the challenge response to get the NTLM hash.  If the victim is running XP, the situation is even worse, as Cloudcracker will return the LM hash which can always be broken overnight to derive the user’s password .

Why Does this Matter Now?

Before I get to the exploit details, let me clarify how relevant these technologies still are in today’s world. You might think that with all the papers and presentations, no one would be using NTLM...or, God forbid, LM. NTLMv2 has been around for quite some time. Surely, everyone is using it. Right?

Wrong!

According to the last data from the W3 Schools, 21% of computers are running XP, while NetMarketShare claims it is 39%.  Unless someone has hardened these machines (no MS patches do this), these machines are sending LM and NTLM responses!  While these lists leave out server OSs, 2003 Server still sends NTLM responses by default.  Yes, every MS OS since NT 4.0 SP4 has supported NTLMv2, but NTLM and LM were not excluded by default until Vista.  

But wait, there’s more! It is also very common for companies that have heterogeneous environments to use Active Directory Group Policy to keep the settings weak, usually out of fear of breaking Samba connectivity.  Sure, Samba has supported NTLMv2 for a long time, but most IT folks tend to think “Why beef up security if you might break something? No one is claiming to have broken NTLM.”

Well, here it is: I’VE BROKEN NTLM.

Now, get to fixin’.

More on fixin’ later.  (I can’t take credit for breaking NTLM. I’m no math whiz. I just happen to specialize in applied crypto, and I looked in the right place at the right time.)

The Attack

When I read a summary of Moxie’s MS-CHAPv2 crack, I saw that the big deal was not that the implementation had some crazy flaw, it was that Moxie had affordably built a system that can brute force the DES keys that make up the heart of the challenge response mechanism.  In less than 24 hours, given a known 64 bit plaintext (challenge) and a ciphertext (response), Cloudcracker can return the key to you.  

This made me wonder what else was broken, given affordable DES brute forcing now exists.  

I didn’t dig a lot deeper into the attack at the time, as I was researching NTLM so I could write a blog post on Pass the Hash Attacks.  I know this is well covered territory, but I never found a paper that covered all my questions, so I figured I’d do it myself.  Much of my research was done by reading the protocol details on Eric Glass’s exhaustive page on the topic. I’ve found no better source for understanding the protocols. 

That’s until I stumbled across this: (Bold added for emphasis.)

The NTLM response is calculated as follows:

The MD4 message-digest algorithm (described in RFC 1320) is applied to the Unicode mixed-case password. This results in a 16-byte value - the NTLM hash.

The 16-byte NTLM hash is null-padded to 21 bytes.

This value is split into three 7-byte thirds.

These values are used to create three DES keys (one from each 7-byte third).

Each of these keys is used to DES-encrypt the challenge from the Type 2 message (resulting in three 8-byte ciphertext values).

These three ciphertext values are concatenated to form a 24-byte value. This is the NTLM response.

Note that only the calculation of the hash value differs from the LM scheme; the response calculation is the same.

That’s right, to reverse the challenge response, I just need to brute force two DES iterations that have 56 bit keys and one that only uses 2 bytes of the 56 bit key space for the last crack.  Moxie shows it like this, for those who are more visual.

I immediately went to Moxie’s posts on the topic to figure out how to use Cloudcracker to brute force out my hashes.  That’s when I discovered that MS-CHAPv2 uses the EXACT same math as the LM and NTLM challenge response.  Moxie was kind enough to point me to the line in his code that demonstrates how one can submit the challenge and response to Cloudcracker to get back the LM or NTLM hash.

print "CloudCracker Submission = $99$%s" % base64.b64encode("%s%s%s%s" % (plaintext, c1, c2, k3[0:2]))

Just concatenate your challenge, and the first two thirds of your response, and k3 as base64 and Bob’s your uncle.  The only thing that threw me was, why is k3 only 2 bytes?  To save on processing power, I assume, you must brute force the final key before sending it to Cloudcraker, which just appends it to the first two recovered keys and sends it back.  As the last key is the one that has 5 bytes that are always all 00, this is easy.

Using Eric Glasses example, I sent in $99$ASNFZ4mrze8lqYwcMegYR0ZrKbLfRoDz2Ag=  to Cloudcracker.  This is the challenge 0123456789ABCDEF, low response 25A98C1C31E81847, mid response 466B29B2DF4680F3, and D808, the first two bytes of the final third of the hash, which I brute forced locally.  If you decide to follow along with me, make sure to look at Glass’s parity adjusting of the keys.  DES keys are really 64 bit, not 56, but as the right bit of each byte is a parity bit, it can’t count toward the total entropy or key space.  Or, you can just use his provided Java code.

Then I just had to wait until I got an email with this:

CloudCracker has successfully completed its attack against your CHAPv2 handshake. The NT hash for the handshake is included below, and can be plugged back into the 'chapcrack' tool to decrypt a packet capture, or to authenticate to the server:

cd06ca7c7e10c99b1d33b7485a2ed808                           

This run took 68799 seconds. Thank you for using cloudcracker.com, this concludes your job.

For those who missed it, I just got the NT hash from the challenge and response, which are easily observed on the wire. I just got the hash WITHOUT compromising the host first.  To use this in an attack I just need the right MITM foothold or a phishing email.

To: All Employees

From: HR Communications

Subject: Updates to the Employee Handbook

Body: Human Resources has completed a significant rewrite and update to the Employee Handbook.  While some of the changes are minor, it is worth a look for all employees.  Employees with aging parents will likely be excited to see the increase in paid time off for emergency care of elder dependents.  The guidelines for company events where alcoholic beverages are provided have also been updated. 

Finally, with the passing of Washington Initiative 502, we are publishing the new guidelines for Marijuana in the work place.     

The handbook can be found here:                                                                          

\\hrFiles.ru\HRFiles\EmployeeManualv3.docx 

Best Regards,

Human Resources

I call this attack “Request the Hash”.  Sorry for the pun, but who wouldn’t immediately click the link to find out their companies new pot policy???

The Good News

First of all, I commend MS for changing the default to NLTMv2, as of Vista, and for leaving us the option if we just can’t live without LM or NTLM.  At this point though, I think it is incumbent on MS to push the issue at the local security policy level, letting domain administrators override common sense as needed.

There might be some testing and some interoperability failures for enterprises, but the good news is the settings for LM, NTLM, and NTLMv2 have been around a long time. The reg key HKLM\SYSTEM\CurrentControlSet\Control\Lsa\LmCompatibilityLevel allows you to choose the level of fallback your system will allow.  Set this to 3 or greater for workstations and you are probably set. This situation is a bit more complex as almost all windows computers are not just clients, but also servers, offering you the admin$ share and c$, etc.  The same setting has slightly different bearing on which versions the machine will accept when acting as a server, say when doing remote administration of your desktops. This great article, by Jesper Johansson, covers both the client and server aspect. 

Personally, I am setting all my systems to 5.  If you allow systems to be set below 5, you may be masking the fact that clients are still performing NTLM or LM handshakes. 

Level	Group Policy Name	Sends	Accepts	Prohibits Sending
0	Send LM and NTLM Responses	LM, NTLM, NTLMv2 Session Security is negotiated	LM, NTLM, NTLMv2	NTLMv2 Session Security (on Windows 2000 below SRP1, Windows NT 4.0, and Windows 9x)
1	Send LM and NTLM—use NTLMv2 session security if negotiated	LM, NTLM NTLMv2 Session Security is negotiated	LM, NTLM, NTLMv2	NTLMv2
2	Send NTLM response only	NTLM, NTLMv2 Session Security is negotiated	LM, NTLM, NTLMv2	LM and NTLMv2
3	Send NTLMv2 response only	NTLMv2 Session Security is always used	LM, NTLM, NTLMv2	LM and NTLM
4	Send NTLMv2 response only/refuse LM	NTLMv2 Session Security	NTLM, NTLMv2	LM
5	Send NTLMv2 response only/refuse LM and NTLM	NTLMv2, Session Security	NTLMv2	LM and NTLM

You may ask, “Who cares if my server accepts LM or NTLM?”  If not all your clients are managed, your server could be unwittingly used to compromise an account used by a client with weak settings.

I did a fresh Ubuntu install and verified that its SMB/CIFS client only sent NTLMv2 response, even when I attempted to downgrade using Cain.  I have not yet had a chance to test web browsers that support NTLM auth via SPNEGO.   That’s right; most web browsers perform NTLM auth in HTTP Headers when a trusted site requests it. 

A Reminder About the Downside of Doing Nothing

While web browsers only perform the NTLM auth for trusted sites by default, generally, Windows OSs seem to make no distinction between trusted and un-trusted zones for other protocols, such as CIFS/SMB, SQL/TDS, and RPC.  This means that a phishing attack with a file://server//share link could yield great results for an attacker.  Based on the rapid succession of tries and retries, it is safe to assume that the first 2 – 5 attempts by a machine are the OS, via SSPI, trying to auth with the victim’s cached creds.  This means the attacker does not need to see a successful auth to trust that the credentials are valid. Maybe you took my advice on Pass the Hash, and blocked all the protocols that use NTLM, at your perimeter.  This doesn’t stop one of your clients, say a laptop at home outside your perimeter from trying to do something that won’t try an NTLM handshake. There are just too many ways to fail, if you allow LM or NTLM in any context. OK, so the attacker has the hash, but they are outside the firewall… You’re safe, right?  No!  At this point, the attacker has a valid hash and is ready to use the account to get your data.  Now the attacker has two options:

·         If your system is sending LM responses, getting that hash makes for super light work in cracking the password.  If you are sending the better NTLM hash, you are still in hot water.  Cloudcracker also has a huge set of rainbow tables.  For another $20 I can submit the hash and likely get the password of an average user. There are surely plenty of ways inside your perimeter with just username and password. Moxie, if you are reading this, how about a one step operation (from the customer perspective) that pipes the recovered hash into the rainbow tables after it’s been cracked. Maybe a price a tad under $40 for the two cracks? 

·         BUT WAIT!!  Attackers don’t need your stinking password! If I have the hash, it is password equivalent! The first step of every auth involving NTLM and NTLMv2 is to convert the password to the NT hash.  Your weak settings just saved the attacker a step.  Once the hash has been stolen, the only ways to render it useless are to change the password or remove all of the channels by which it can be passed. (The latter feels infeasible). 

Protecting Yourself and Your Company

If you are a lone user, make sure all of your systems are set to LmCompatibilityLevel 3 or higher via local security policy, group policy, or via the registry key.

If you are a corporate type, make sure the GPO is set to force 3 or higher.  You may notice that levels 4 and 5 talk about domain controllers refusing different levels.  This is not a great protection and may even mask a problem.  While the DC blocking the LM auth passed by a client won’t let the user access the system, the damage is already done: The LM response was sent, and the attacker may have it.  Sure, it is nice for the attacker to see the successful auth so they know the crack is worth the time, but the rapid fails and retries will tell the attacker that the client OS had the creds cached and the hash that was used is almost certainly good, at a minimum on that client machine, if not the domain.

If you are running a Samba client, enter this line ‘client ntlmv2 auth’ into your smb.conf file.

Final Paranoid Thought

If you have any of the weak settings, CHANGE YOUR PASSWORD after fixing them!

I made this discovery when I realized DES was brute-forcible on a budget.  I immediately started thinking of a list of all of the processes that broke along with DES.  I’m sure this is exactly the list that every nations’ spy services started making when they acquired the computing power to brute force DES.  I assume that it has been affordable for most of them for 5 – 10 years now. 

Sorry, spies, if this closes a door for you.  I read a book that says another one will open.  ;-)

Appendix A: Proof of Concept Code

In order to validate my work, I built a simple tool proof the crypto and create the CloudCracker token.  First off, I am not a great or professional coder, so don’t laugh when you look at the code.  Second, Eric Glass’s code was a great help.  His is in Java, while mine is c#.  Last, I used Bouncy Castle’s c# crypto library, for two reasons.  The .NET libraries do not support MD4, because for hashing it is VERY weak and the .NET libraries block the use of weak DES keys.  NTLM does not account for weak keys.

My code is a fair bit redundant, checking and rechecking itself.  This is because each CloudCracker summation costs $20 and I didn’t want to have to send/spend several failed submissions during debugging.

The code can be found here.

The challenge and response data should be in hex, while the password should be ASCII.