If I Can't Reach Active Directory, it's Down

Unless it's not.

I recently had a customer tell me that my AD servers were broken. They were unable to set SPNs via Setspn.

They were able to run AD queries and were able to do other "AD Stuff". As always, I demanded a packet capture.

In very short order, the issue was clear. Setspn, for reasons I cannot guess, uses RPCs to the domain controller to set SPNs. I have not clue why it doesn't just use LDAP. LDAP is better, it only requires one port, that we know will be open.

RPCs are a pain, they require TCP 135, the end point mapper, then some random high port, named at the time of connection.

Below, we see that the customer hit the EPM in Frame 873 and was assigned a new connection on port 1028. We the SYN to 1028 in 874, then retries in 966 and 1146.

Firewalls and windows RPCs don't mix. Click here for larger.

RPC OPEN ALL THE PORTS!!

Living off the land with Kerberos and netsh interface portproxy

Have you ever been in the situation where you need to do some remote PowerShell on a machine, but you can’t find a layer 3 path to the server?

Did you find out that you could get remote PowerShell on the machine next to it, but you don’t want to pass your credentials to that machine, to double hop?  You know, ‘cause you aren’t insane…

Did you ever say, why can’t I use that machine and protect my credentials?

Well, for only $9.99, admin access on the relay machine, and the write access to the target machine’s computer object in AD, you too can do the needful.

Recently, I needed to shutdown two hosts that I had admin rights on, but I only had layer 3 access to one. That one had access to the other.

I want to Enter-PSSession on MAGICHOST01, but can only get to MAGICHOST02.

PS C:\Windows\system32> ping MAGICHOST01.domain.net   

Pinging MAGICHOST01.domain.net [192.168.72.152] with 32 bytes of data:

Request timed out.

Request timed out.

Request timed out.

Request timed out.

Ping statistics for 192.168.72.152:

    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

PS C:\Windows\system32>

PS C:\Windows\system32> ping MAGICHOST02.domain.net 

Pinging MAGICHOST02.domain.net [192.168.72.149] with 32 bytes of data:

Reply from 192.168.72.149: bytes=32 time=107ms TTL=48

Reply from 192.168.72.149: bytes=32 time=107ms TTL=48

Ping statistics for 192.168.72.149:

    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 107ms, Maximum = 107ms, Average = 107ms

Control-C

Port proxying is time honored method for connecting through a host to another, and now can be done with no 3rd party tools!

Enter-PSSession MAGICHOST02.domain.net

[MAGICHOST02.domain.net]: PS C:\Users\adm_magamach\Documents> netsh interface portproxy add v4tov4 listenaddress=192.168.72.149 listenport=44444 connectaddress=192.168.72.152 co

nnectport=5985

[MAGICHOST02.domain.net]: PS C:\Users\adm_magamach\Documents> netstat -ano | Select-String -Pattern 44444

  TCP    192.168.72.149:44444     0.0.0.0:0              LISTENING       248

[MAGICHOST02.domain.net]: PS C:\Users\adm_magamach\Documents>

The port is proxied, now let’s connect!

PS C:\Windows\system32> Enter-PSSession MAGICHOST02.domain.net -port 44444

Enter-PSSession : Connecting to remote server MAGICHOST02.domain.net failed with the following error message : WinRM cannot process the request. The following error with

Error code 0x80090322 occurred while using Kerberos authentication: An unknown security error occurred.

Possible causes are:

  -The user name or password specified are invalid.

  -Kerberos is used when no authentication method and no user name are specified.

  -Kerberos accepts domain user names, but not local user names.

  -The Service Principal Name (SPN) for the remote computer name and port does not exist.

  -The client and remote computers are in different domains and there is no trust between the two domains.

After checking for the above issues, try the following:

  -Check the Event Viewer for events related to authentication.

  -Change the authentication method; add the destination computer to the WinRM TrustedHosts configuration setting or use HTTPS transport.

Note that computers in the TrustedHosts list might not be authenticated.

   -For more information about WinRM configuration, run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic

At line:1 char:1

+ Enter-PSSession MAGICHOST02.domain.net -port 44444

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidArgument: (MAGICHOST02.domain.net:String) [Enter-PSSession], PSRemotingTransportException

    + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

Oh NO! Lots of kerberos fail words.  Oh yeah, I am probably using the ticket meant for MAGICHOST02. Let’s force NTLM using IP.

PS C:\Windows\system32> Enter-PSSession 192.168.72.149 -port 44444

Enter-PSSession : Connecting to remote server 192.168.72.149 failed with the following error message : The WinRM client cannot process the request. Default authentication

may be used with an IP address under the following conditions: the transport is HTTPS or the destination is in the TrustedHosts list, and explicit credentials are

provided. Use winrm.cmd to configure TrustedHosts. Note that computers in the TrustedHosts list might not be authenticated. For more information on how to set

TrustedHosts run the following command: winrm help config. For more information, see the about_Remote_Troubleshooting Help topic.

At line:1 char:1

+ Enter-PSSession 192.168.72.149 -port 44444

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidArgument: (192.168.72.149:String) [Enter-PSSession], PSRemotingTransportException

    + FullyQualifiedErrorId : CreateRemoteRunspaceFailed

This is scary, I don’ think I want to over trust.  What are the implications?  I have no idea? Let’s look at how Kerberos works…

PS C:\Windows\system32> klist | Select-String -Pattern MAGIC

    Server: HTTP/MAGICHOST02.domain.net @ DOMAIN.NET

Sure enough, we have a ticket for 02 and thus we are sending it when called by name. Let’s call it by a new, unique name.  A principal, in this case, computer, can have many Kerberos names (SPNs).  We simply add the new name to the real computer object of MAGICHOST01 . MAGICHOST33 is free.

PS C:\Windows\system32> setspn -s HTTP/MAGICHOST33.domain.net MAGICHOST01

Checking domain DC=domain,DC=net

Registering ServicePrincipalNames for CN=MAGICHOST01,OU=PRODUCTION,DC=domain,DC=net

        HTTP/MAGICHOST33.domain.net

Now we just need to direct traffic bound for MAGICHOST33.domain.net to MAGICHOST02, via the hosts file.  

192.168.72.149  MAGICHOST33.domain.net

Now we connect via the alternate port and alternate name. PowerShell lies about the name, in yellow, but the host knows who it is, in green!

PS C:\Windows\system32> Enter-PSSession MAGICHOST33.domain.net -port 44444

[MAGICHOST33.domain.net]: PS C:\Users\adm_magamach\Documents> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : MAGICHOST01

   Primary Dns Suffix  . . . . . . . : DOMAIN.NET

   Node Type . . . . . . . . . . . . : Hybrid

   IP Routing Enabled. . . . . . . . : No

   WINS Proxy Enabled. . . . . . . . : No

   DNS Suffix Search List. . . . . . : DOMAIN.NET

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :

   Description . . . . . . . . . . . : HP NC362i Integrated DP Gigabit Server Adapter

   Physical Address. . . . . . . . . : B4-B5-2F-68-FE-A8

   DHCP Enabled. . . . . . . . . . . : No

   Autoconfiguration Enabled . . . . : Yes

   IPv4 Address. . . . . . . . . . . : 192.168.72.152(Preferred)

   Subnet Mask . . . . . . . . . . . : 255.255.255.240

   Default Gateway . . . . . . . . . : 192.168.72.148

Enjoy!

Keep an Eye on Your Index Fund Dollars. You May be Surprised.

There has been a lot of talk about index funds in the last six months.

Warren Buffett recently said he recommends you, and his wife, don’t pick your own stocks, but simply “buy an S&P 500 low-cost index fund”. 

I recently heard on the news that at least 60% of the money in the market is in index funds.

Some Wall Street types are referring to Index fund investors as “dumb money” and worrying that no one is looking a company stats any more.

I am not going to talk about these things. They are for more nuanced investors. I am just a fairly smart guy who is a bit risk averse and likes things that make some sense.

We know we should not pick our own stocks unless we really understand the business sector that a company is in and the performance metrics of the business. Very savvy investors even look at management teams and may even compare them to the management teams of rival companies. Most of us to not have time to do that for more than a company or two.  We can’t even look at one whole market sector, let alone the whole market. It really does make sense from this perspective to look at index funds, but wait…

We are also told to diversify our portfolios.  All the more reason to use an index fund. Only…

We are also told to not put all our eggs in one basket. We should have a good speared of U.S. and foreign stocks, we should have a lot of different business sectors. There are international index funds, so we are covered there too.

BUT WAIT

You and I probably thought that if we bought an S&P 500 Index fund that we’d be getting a good mix of the S&P 500. WRONG!!  All your eggs are in one basket! If we look at just the top 25 holdings, 12.5% of your portfolio in in Tech stocks! That’s right, the index is weighted, usually by market cap or some sort of adjusted market cap. If you go down the list, the top is dominated by Tech, finance, pharmaceuticals, and finance

Breaking down your ownership by sector in the top 50 of the S&P 500.

Sector	%
Aerospace	0.64
Consumer Goods	2.77
Finance	6.13
Insurance	0.87
Oil	3.02
Pharma	4.28
Retail	0.56
Soda	1.61
Technology	16.4
Telecom	2.96
Tobacco	1.44
(blank)	6.8
Total	47.48

 

That’s right, just under 50% of your investment in 500 companies is in only 50 companies, and a very unbalanced 16% is in the tech sector. If we look at the bottom 50 companies, you only have 1.4% in them.

As always, these are just things to consider.  I have no useful money advice.

 

Detecting Attackers in a Windows Active Directory Network

TL;DR If you know Pass the hash, Mimikatz, and BloodHound, jump down to the detection section.

The Pain

Windows and the Active Directory environment are pervasive in the corporate world. Due to high adoption and lax patching and misconfiguration issues on many fronts, the “AD network” is often the first thing that an attacker or red team looks at.

Since time immemorial, attackers of Windows have known that any authenticated user can enumerate the list of local admins on a Windows machine. Slightly less known, is that authenticated users can also enumerate session info on a Windows machine, that is, who is logged in. The rise of Mimikatz and Invoke-Mimikatz has made life hard on defenders. Mimikatz steals tokens, hashes, and depending on settings, plaintext passwords, of any user interactively logged into a machine.

Using the admin and session enumeration tools allows an attacker with the tiniest foothold to leverage Mimikatz, extend the foothold, and collect more credential. If you don’t know pass the hash and cred theft there are only two sources you need to get up to speed. The important thing to know is that the attacker moves from machine to machine collecting an ever-growing set of credentials.  As the set of credentials grows, they have a larger set of machines they can exploit. This is often exponential. The attacker’s Mount Everest is getting access to Domain Administrator credentials.  Once the attacker has these, they can get to all the data on all the machines in the domain, and can setup unbeatable persistence.     

Enter BloodHound

As a defender, BloodHound is your worst enemy. It is the leading tool, by miles, in mapping trust relationships in an Active Directory domain. Watch this video and feel your world crumble. Unleash the tool in your environment and you won’t sleep for weeks. But, I digress…

BloodHound uses lists of admins on hosts, AD group memberships, AD ACLs, logged in sessions, and a bit more, to map where an attacker can use Mimikatz next, what credentials they will get there and what new access they will gain with the new creds. The BloodHund team refers to the extension of the attack and the escalation chain as derived admin rights. In the past, finding these was trial and error and went one step at a time.  That is, the attacker gets some access and tests everywhere to see if the cred works to gain admin rights. The attacker gets a little more access and tests everything again and so on. This is slow and noisy. BloodHound assesses all this with regular user access before any exploit tool is ever run. A typical attack graph looks like this.

There are even plugins that have been written to automate the attack path, once found, so that the exploitation can occur quickly before defenders can cut off access.

Prior Detections

Unless the defender has the tools and understands their expected net flow, the discovery done by tools like BloodHound blends into standard Windows traffic.  Keep in mind that almost every Windows admin has, at some point, been told that 40% of LAN traffic is Windows’ super chatty browser and NetBIOS traffic.

One of the reasons that plugins are needed to exploit an attack path quickly is that most detections are based on AV or anti-malware tools. They are likely to draw actual attention. Once the attacker has gotten this far, the horse is out of the barn and it is often a race to try to detect and cut off all of their persistence mechanisms and find all the other compromised hosts.  Many shops never even detect any of this.  *sad emoji*

Detecting the Foothold

As stated, the attacker only needs a tiny foothold to start enumerating your Windows environment. They don’t even need local admin rights on their initial compromised host. They only need a domain user account to perform the enumeration. At some point, they will need admin rights to steal creds, or they can phish them, or … so many paths to destruction.

Enumeration of local admins and user sessions generates traffic that we can log. No one seems to log it though, probably because the settings seem useless and probably noisy, but in practice, the noise is easily filtered. If you use a lot of SMB file sharing, you will get many useless logs, but they are easily scrubbed.

In both enumeration cases, the attacker connects to the \\machine\IPC$ share of the host it enumerates.

Since Windows 7 and 2008 R2, we have had the option to “Audit File Share”.  In your Group Policy, this is under “Object Access” in “Advanced Audit Policy Configuration”.

This setting is little used, probably because warnings say it can be chatty and even the file server teams at Microsoft don’t suggest turning it on. I’ll cover it more later, but the good news is, you don’t need the setting on busy file servers or Domain Controllers, as long as you have the setting on a lot of other machines.

The event we are looking for is in the Security event log and is event ID 5140. The Share Name of \\*\IPC$ is the key here. You can toss out any 5140 that does not have this value. The Subject Account Name and Domain will be that of the comprised account that is running the scan. The Source Address is the IP address of the host running the scan. You can filter out the source IP of 127.0.0.1 to reduce noise. The computer that logs this, of course, is the target of the attack.

Don’t call security just yet.  There are other legitimate reasons that something might access your IPC$, but they are far and few between. One of my monitoring systems generates a ton of these so I filter its IP address out. Going back to what I said earlier about not having to put this on your DCs and file servers, so you don’t get buried in logs… the key with this detection is to collect the logs on a large set of machines. You can do a simple group by source IP and count to see that some IPs are clearly doing something bad. Attacker detected!  *happy jumping guy emoji*

In two weeks, with very few hosts logging this event, I have had great success. I have found two blue teams scanning our user network. In our prod network, I have detected two red teams, numerous legitimate vulnerability scanners, and one other curiosity that I am investigating. *slightly smug nerdy guy emoji*

Looking at the Logs

One of the places I collect my logs, that is VERY handy for this situation, is Microsoft Operations Management Suite Online (OMS). I can setup email alerts every time we see an IPC$ event.  The basic query look like this.

(Type=SecurityEvent) EventID=5140 \\\\*\\IPC$

I have some known good destinations that I filter out:

(Type=SecurityEvent) EventID=5140 "\\\\*\\IPC$" NOT (Computer="server.domain.com" OR Computer="otherone.domain.com")

I can pipe that to a SQLish query and see who the top talkers are.  This is key to finding the attacker and not just a little legitimate noise.

Based on the size of your environment you will want to scope the number of hits from a single IP and a time range.  That said, even very few hits, from one IP to several sources, is likely an attack.

One easy and low cost way to get these events at scale might be Windows Event Forwarding. Without correlating the scans across many hosts, you are likely to chase many false positives.

Call to Arms

Stop giving out local admin rights like candy.

As a defender, map your environment with BloodHound. You may want to get the Xanax ready first.

Monitor those IPC$ connections.

Read my next piece on blinding enumeration tools.

Keep an Eye on Those Bond Investment Fees

While re-balancing my portfolio stock/bond/cash ratios, I discovered something I had not seen before…

While I talk about my Fidelity account, it is almost certain that this issue will be found with most brokerages. 

If you use the Brokerage link option to manage your 401K investments, and if you have other Fidelity investment accounts, it is worth taking a look at your “core position”.  Depending on the account type, your “un-invested” money is not in cash. When you sell a stock or bond, if you don’t re-invest, it sits in a core position. 

If you aren’t leaving your money in the core for long, it doesn’t matter much.  If you decide to use it to “pull some money out of the scary market”, this is probably a bad place for it to sit. Your core position is probably mostly bonds, so it may just be your default bond allocation. 

If the position is a non-cash position, like in the Brokerage link account or the one you get with an ESPP, then the money is in a super low risk investment. The key here is that they are in a managed investment with fees! Everyone knows how to invest for low risk and low returns, so this should be free or near free.  The fees are low for mutual and bond funds in general, but for some default positions, surprisingly high for the lack of value the fund manager has to deliver.

I find 3 possible core positions in my plans:

·         SPAXX 0.42%

·         FZFXX 0.35%

·         FDRXX 0.35%

I don’t like to give investment advice, just things to ponder, but I will point out that there are many Vanguard funds with much lower expense ratios, like 0.07% .  Many of these are in super low risk government bonds. Most consider those bonds less risky than cash due to inflation.  All three above have about 70% of your money in cash, 25% in government bonds and 5% misc. As far as I can tell, you are paying a higher fee to have your money as cash.

Before you say, “All this is sub 1% stuff, so what”, take a look at the cost of a long term over pay. Take a look at a comparison. 

FWIW, if you are using Index Funds for you Stock investments, the expense ratios should be crazy low as well.  How hard is it to read the S&P 500 and buy a bunch for your investors??? 

As always, trust no one…

Copying the NTAuth Enterprise store certificates from one Forest to another

The enterprise NTAuth store is a key Active Directory configuration item. It is key to allowing user to login with smartcards. When using PKI cross forest, we usually use the PKISync.ps1 script to lihnk the two forests PKI configurations. This script is designed to allow cross forest certificate enrollment, wich it does well.  It does not cover the NTAuth config for smartcards.  This seems to get missed a lot. 

Below is a two liner to copy the NTAuth in one forest to another, assuming the NTAuth object exists and just needs to be populated. 



$caLIst = (Get-ADObject -SearchBase "CN=NTAuthCertificates,CN=Public Key Services,CN=Services,CN=Configuration,DC=domain1,DC=com" -SearchScope Base -Filter * -Properties * -Server domain1.com).cACertificate

foreach($ca in $caLIst) {Set-ADObject "CN=NTAuthCertificates,CN=Public Key Services,CN=Services,CN=Configuration,DC=domain2,DC=com" -add @{cacertificate=$ca}}

I Just Saved $121!!

I'm not one to pimp products unless they are really good and I understand them, but today is an exception.

I have NO IDEA how GoodRX works, but I do know it saved me $121.11 on a single prescription.  Usually, when you install an app that get's you something for free, you are the commodity.  Most apps want all sorts of crazy access to your phone.  GoodRX wanted pretty basic access and gave me s super coupon.

Drug cost was $221.  My insurance only covered $48.  I was going to pay $173.  Enter the GoodRX coupon and I saved $121.  I only paid $52!!!!