IIS Healthchecks Spam on Win Server 2016 & 2019

I’ve written before about how IIS healthchecks can spam application servers because IIS web farms are not directly linked to application pools. IIS is designed that any web application (application pool) can route a request to any web farm defined on the system. This means that all the application pools on a system read in all web farm configurations, and then believe that they are supposed to monitor the healthchecks on the web farms. Because of that, the more application pools on a server the more healthchecks that occur.

• How to Crash Exchange Using IIS Healthchecks
• Lowering Database Impact of IIS Healthcheck Spam

The issue was introduced in the first version of Web Farm Framework 1.1, but continues within the latest release of Application Request Routing 3.0 (Latest Release 1/25/2018, Web PI Version 3.0.1988).

I first discovered the problem when working on a Windows 2008 server, and was able to reproduce it on a Windows 2012R2 after that. I then filed a Microsoft Support ticket (#116110914917188) in which the product team did try to create a hotfix. But, the fix didn’t work and the ticket was closed stating that the issue would be fixed in a future release of IIS.

I’ve recently re-tested the feature in Windows Server 2016 and 2019. Both of which have the same problematic behavior. There’s a good chance that Application Request Routing is the subsystem which implements web farms, so it might not be an IIS specific issue. It might be that Application Request Routing needs to be updated.

Is SQL Server looking to Dockerize on Windows?

Microsoft is a very large company and there is never going to be a single statement which encapsulates the exact direction that every division in the company is moving. The leaders of the company try to point in a wise direction and they hope like hell that the organization see their wisdom and starts to move toward that direction.

One of those big directional statements was Microsoft throwing its support behind Docker. MS runs a very large cloud provider in Azure and looking over the statistics they have (and I would assume internal feedback) they most likely are seeing a large shift towards the use of docker within their systems. It would only be reasonable to think that they should provide a platform that will support the systems their customers are using.

But, what are the challenges that they face with Docker …

• Docker Containers are not Domain Joined
  
  Microsoft’s security methodology for many years has been based around Kerberos/AD and domain credentials. In order to provide least privilege access for your applications you create an AD account for that application and then you setup permissions for that account based upon it’s needs. Those credentials are authenticated against Domain Controllers and then a Kerberos token is passed everywhere to authn/authz the account on all services within the domain (SQL Server, Disk Access, LDAP).
  
  So, if the docker instance isn’t domain joined, running an application under a domain account becomes a difficulty. How do they deal with this?
  
  
• Docker Hosts and gMSA accounts
  
  Group Managed Service Accounts (gMSA) is a concept that was introduced into Active Directory prior to Docker. The idea behind the accounts are that they would be more locked down/secure than a normal AD user account. These accounts would be registered within AD to only be usable on a particular set of machines within the domain, and the accounts would need to pre-register themselves on those machines before they could be used.
  
  Microsoft architects looked at this and thought, if we already have these accounts registered on the Docker Host machine and the Docker container can interact with the Host machine, maybe we can find a way to slide the authenticated Kerberos credentials into the Docker Container for use?
  
  Which they did. But, their are a number of ‘gotchas’ along the way to make gMSA accounts work with Containers:
  
• Container hostname must match the gMSA name for Windows Server 2016 and Windows 10, versions 1709 and 1803
  
• You can't use gMSAs with Hyper-V isolated containers on Windows 10 versions 1703, 1709, and 1803
  
• Container initialization will hang or fail when you try to use a gMSA with a Hyper-V isolated container on Windows 10 and Windows Server versions 1703, 1709, and 1803.

• Using a gMSA with more than one container simultaneously leads to intermittent failures on Windows Server 2016 and Windows 10, version 1709 and 1803.

So, with the all the issues listed above here’s the Use Case:

Can you use Microsoft SQL Server within a Docker Container?

I think the answer is “I guess … but it feels like the SQL Server team (or the MS Docker Team) is focusing on supporting SQL Server in Linux Containers more than on Windows Containers.”

• Microsoft SQL Server Images for Windows Are Not Being Updated
  
  The last version of MSSQL on Windows Servers was built in Feb. 2018 against Windows 10 1709 / Windows Server 2017-GA / Windows Server Core 2017. So, there hasn’t been an update for Windows 10 1803/1809 or Windows Server 2019.
  
  So, why is that? I don’t know the answer, but maybe it’s because their isn’t a lot of usage of SQL Server in Containers due to licensing costs. Or that setting up a SQL Server instance to run under a gMSA account doesn’t necessarily mean it’s going to be able to authenticate Kerberos tokens/SSPI from clients (I never got to a place where I could test this). Or, maybe Azure usage statistics show that people aren’t using MSSQL in Windows Containers.
  
  Either way, I’m not sure the MSSQL Team is really sold on investing their time into that platform. Only they know.
  
  
• Microsoft SQL Server Images for Linux are Working Great!
  
  However, they are keeping up to date on the SQL Server for Linux images. Using Linux simplifies things as it breaks out of the constraints of using Kerberos/SSPI for authentication and will only need to support the SQL Login authentication model.
  
  Potentially, that’s a good enough reason on it’s own to make supporting a container easier for the MSSQL team. But, I wonder if they have statistics from Azure that show the market is strongly preferring this configuration when using containers?

So, Is SQL Server looking to Dockerize on Windows?

It just doesn’t feel like it.

Weak Password used in Ransomware Attack

Recently, it’s been reported that there have been a number of ransomware attacks against Github and Bitbucket. The attacks have a similar attack pattern of using a brute force/dictionary attack to gain access to accounts with weak passwords. At least, that’s the theory at this time. (Updated 5/18/2019: Github has posted on their blog that credentials were gained through account leakage.)

We are currently in the process of thinking about moving a large amount of source control into Github and wanted to see what sort of mitigations we have in place for access to source control through a browser.

1. Secondary SSO System
   
   We are going to use a Github Enterprise account, which will be connected to an on-premise SSO solution.
   
   To log into our GitHub repositories you will need to log in through Github username/password system and then you will have to log in a second time through our on-premise SSO solution.
   
2. Strong Password Requirements in On-Premise SSO
   
   Logging in twice doesn’t prevent a weak password from being vulnerable. So, that needs to be a way to enforce strong password strength.
   
   I don’t know if it’s possible to enforce strong password policies within Github (ie. 15 characters, a variety of character types, etc.). But, I know that we do have strong password enforcement within our on-premise SSO. So, that should solve the “weak password” problem.
   
3. Required MFA in Organizational Github Enterprise Account
   
   Another quick thing that should alleviate weak passwords is MFA. Within Github Organizations you can require that 2FA be turned on for your organization. Github offers a number of options for 2FA, but the preferred option should be a TOTP compatible device (like Google Authenticator). Text messages have proven to be a bit insecure.
   
4. Require MFA on On-Premise SSO
   
   So, this ones a bit trickier to work out. I haven’t seen an option in Github which would allow it’s third-party integration system (the one that interacts with the On-Premise Authentication System) to support checking for a field within the returned attributes which states that the third-party/on-premise authentication system used MFA to authenticate the end user. If it was possible to do that, then it would be useful to require that the source control users setup MFA within the On-Premise SSO system as well.

Lowering Database Impact of IIS Healthcheck Spam

I’ve mentioned before that IIS 8.5 has an issue with spamming healthchecks because it’s web farms aren’t directly associated with application pools. Instead, every running application pool believes it needs to monitor the health of all the web farms, which causes redundant healthcheck calls.

• How to Crash Exchange Using IIS Healthchecks
• IIS Healthcheck & Ninject Memory Leak

I’m also a proponent of using Healthchecks to test connectivity to databases, external web APIs, and core functionality: Healthchecks Should Not Be Pings.

Because I prefer that healthchecks check that the database is available, and that domain account can log into the database, the healthcheck spam can have a very minimal effect on database performance and connection thresholds. It’s never really been a problem that has been seen, but it’s theoretically something to be concerned about. Soo … here are two techniques that can lower the impact that the healthcheck spam can have on databases:

Cache the Healthcheck Response for a Little While

There are a number of output caching systems built into ASP.NET which can easily provide this functionality:

• ASP.NET MVC
• ASP.NET Core MVC

But! Nothing was every made for ASP.NET WebApi 2. It was really surprising that nothing was ever built in for this. Luckily, there is a great community package the provides the functionality while following the standards that Microsoft was using:

• ASP.NET WebApi 2: Strathweb.CacheOutput.WebApi2

Use Database.Exists (instead of a real call) to help prevent locking

To test if your application is going to be able to communicate with a database successfully, the easiest (and maybe best) way is to use a real call. Just use a function that is a part of the application. This not only tests connectivity, but it also checks if permissions are setup correctly. However, this can drive some concern around table locking if the storm of healthcheck calls aren’t prevented (by the caching mentioned above). Another way to prevent table locking is to use Entity Frameworks Database.Exists method.