Azure/Elastic Kubernetes Service and gMSA

I’ve written previously about how Docker Containers Are Not Domain Joined and all of the difficulties that it creates.

This post simply adds to that previous article with a little more links and information.

When I first heard of Docker, I imagined a system where you would throw a container at a service and it would figure out everything that was needed to run the container and just make it happen. Obviously that’s extremely difficult to do and as I learnt more about Docker the larger and more engrossing the problem became. My current understanding is no where near complete but here’s some more info on the problem.

In 2018, around the time I looked at AWS’ ALB prices, I looked into a price comparison of a Dockerized Web Farm vs an a IIS EC2 Web Farm. When developing out the system architecture for the Dockerized Web Farm I ran into two major issues:

• Theoretically, it looks like, Windows containers use an absolute limit (search for “CPU limit is enforced as an absolute limit”) when allocating CPU utilization to the container.
  
  NOTE: I have not gotten to the point where I can prove or disprove the above statement; and OLDER Docker documentation doesn’t seem to indicate that Windows has this problem.
  
  What this means is that if you have a 2 CPU Host system, and you were to allocate .5 CPU to a Windows Container, then the Windows container would be given that .5 CPU for it’s sole usage. No other container could use the .5 CPU and the allocating container would be hard-capped at .5 CPU.
  
  In Linux containers this is not an issue. You can allocate dozens of containers on a single host to use .5 CPU and they would (a) all share the full 100% CPU resources available, (b) never be hard-capped, and (c) only use the .5 CPU hard cap once the CPU reached 100% utilization and it needed to share the CPU between two containers that were fighting over the CPUs time.
  
• The gMSA issue that was brought up in previous Is SQL Server looking to Dockerize on Windows? post.

Even with those issues, I was curious about what AWS was doing with containers in hopes that they had the same idea that I did: We should be able to give a container image to a service and the service just figures out everything needed to run it and maked it happen. And they did: AWS Fargate.

But!! …

They were also frustrated with the permissions and gMSA security issues that the Windows OS introduced into the equation. And, as such, they don’t support Windows Containers on Fargate. They don’t directly say that they don’t support it because of the gMSA/permissions issues, but when you look at what needs to be done to support gMSA it becomes an easily rationalized conclusion. Here’s what it looks like to use a gMSA account on a Windows Container (with all the secret/password storage and management removed):

1. Create a gMSA account in Active Directory.
2. Select the Docker Host that will host the new container instance.
3. Update Active Directory to register the gMSA to be usable on that Docker Host.
4. Register the gMSA on the Docker Host (checks with Active Directory to validate the request).
5. Start the container, and you’re now able use the gMSA account within the container.
6. You’ll need to reapply the registrations (steps 2-4) for each Docker Host that the container will run on.

With a fully automated provisioning process, that’s not that difficult. It’s really doable in fact. However, here’s the list of difficult specifics that a fully managed Kubernetes infrastructure (like Fargate) would have to deal with:

1. Where is the Active Directory located?
2. Are the networking routes open for the ECS/Fargate infrastructure to it?
3. Are there other security requirements?
4. What versions of Active Directory are supported?
5. etc, etc, etc …

I don’t know at what bullet point you just *facepalm* and say “We’re not supporting this.”

But!! …

Figuring out all the details of this should be in the wheel house of Azure, right? It’s the Microsoft OS and platform, they are probably working on this problem.

So, here’s what the landscape looks like today with AKS:

• AKS does not support gMSA accounts (along with a number of other limitations).
• However it does have some interesting integrations with Azure Active Directory.
  
  The caveat to using Azure Active Directory with gMSA accounts is that not all resources used in an application will be running in Azure. For many applications, you need the gMSA to be in the companies on-premise Active Directory and replicated into the Azure AD. And, I don’t know if registering a gMSA account with a Host name within the on-premise Active Directory flows that registration into the Azure AD.
  
  And then, does that lead back to the same set of “Are the networking routes open for the AKS infrastructure to the on-premise AD” questions?
• There seems to be something going on with Open Service Broker API (OSBA) that is trying to tackle these these permissions problems, but I don’t fully understand it yet.
• I don’t see direct documentation on the CPU limiting issue, so maybe they have that solved?

So there’s the update.

Is SQL Server looking to Dockerize on Windows?

Microsoft is a very large company and there is never going to be a single statement which encapsulates the exact direction that every division in the company is moving. The leaders of the company try to point in a wise direction and they hope like hell that the organization see their wisdom and starts to move toward that direction.

One of those big directional statements was Microsoft throwing its support behind Docker. MS runs a very large cloud provider in Azure and looking over the statistics they have (and I would assume internal feedback) they most likely are seeing a large shift towards the use of docker within their systems. It would only be reasonable to think that they should provide a platform that will support the systems their customers are using.

But, what are the challenges that they face with Docker …

• Docker Containers are not Domain Joined
  
  Microsoft’s security methodology for many years has been based around Kerberos/AD and domain credentials. In order to provide least privilege access for your applications you create an AD account for that application and then you setup permissions for that account based upon it’s needs. Those credentials are authenticated against Domain Controllers and then a Kerberos token is passed everywhere to authn/authz the account on all services within the domain (SQL Server, Disk Access, LDAP).
  
  So, if the docker instance isn’t domain joined, running an application under a domain account becomes a difficulty. How do they deal with this?
  
  
• Docker Hosts and gMSA accounts
  
  Group Managed Service Accounts (gMSA) is a concept that was introduced into Active Directory prior to Docker. The idea behind the accounts are that they would be more locked down/secure than a normal AD user account. These accounts would be registered within AD to only be usable on a particular set of machines within the domain, and the accounts would need to pre-register themselves on those machines before they could be used.
  
  Microsoft architects looked at this and thought, if we already have these accounts registered on the Docker Host machine and the Docker container can interact with the Host machine, maybe we can find a way to slide the authenticated Kerberos credentials into the Docker Container for use?
  
  Which they did. But, their are a number of ‘gotchas’ along the way to make gMSA accounts work with Containers:
  
• Container hostname must match the gMSA name for Windows Server 2016 and Windows 10, versions 1709 and 1803
  
• You can't use gMSAs with Hyper-V isolated containers on Windows 10 versions 1703, 1709, and 1803
  
• Container initialization will hang or fail when you try to use a gMSA with a Hyper-V isolated container on Windows 10 and Windows Server versions 1703, 1709, and 1803.

• Using a gMSA with more than one container simultaneously leads to intermittent failures on Windows Server 2016 and Windows 10, version 1709 and 1803.

So, with the all the issues listed above here’s the Use Case:

Can you use Microsoft SQL Server within a Docker Container?

I think the answer is “I guess … but it feels like the SQL Server team (or the MS Docker Team) is focusing on supporting SQL Server in Linux Containers more than on Windows Containers.”

• Microsoft SQL Server Images for Windows Are Not Being Updated
  
  The last version of MSSQL on Windows Servers was built in Feb. 2018 against Windows 10 1709 / Windows Server 2017-GA / Windows Server Core 2017. So, there hasn’t been an update for Windows 10 1803/1809 or Windows Server 2019.
  
  So, why is that? I don’t know the answer, but maybe it’s because their isn’t a lot of usage of SQL Server in Containers due to licensing costs. Or that setting up a SQL Server instance to run under a gMSA account doesn’t necessarily mean it’s going to be able to authenticate Kerberos tokens/SSPI from clients (I never got to a place where I could test this). Or, maybe Azure usage statistics show that people aren’t using MSSQL in Windows Containers.
  
  Either way, I’m not sure the MSSQL Team is really sold on investing their time into that platform. Only they know.
  
  
• Microsoft SQL Server Images for Linux are Working Great!
  
  However, they are keeping up to date on the SQL Server for Linux images. Using Linux simplifies things as it breaks out of the constraints of using Kerberos/SSPI for authentication and will only need to support the SQL Login authentication model.
  
  Potentially, that’s a good enough reason on it’s own to make supporting a container easier for the MSSQL team. But, I wonder if they have statistics from Azure that show the market is strongly preferring this configuration when using containers?

So, Is SQL Server looking to Dockerize on Windows?

It just doesn’t feel like it.

Tyk in Docker on Windows 10

I’m very new to all this technology so, please take this with a grain of salt. The reason I’m writing it is because I couldn’t find another guide that had end-to-end setup on Tyk in Docker on Windows 10.

Tyk is an API Gateway product that can be used to help manage a centralized location of many services/micro services. It is a product which is built on top of the nginx web server. And, nginx is really only supported as a “server” product on *nix based systems. Their Windows build is considered a beta.

So, there are already some good guides for each of the next steps, I’m just gonna pull them all together, and add one extra piece at the end.

Install Docker

There are a couple ways to get around the limitation of nginx only being “production ready on *nix”, but I choose to try out Tyk on Docker. Docker is the multiplatform container host that has created a lot of buzz within the cloud space. But, it also seems pretty awesome at setting up small containers on your local machine too.

Note: At this time, 2016-10-16, if you download a Docker for Windows installer, use the Beta Channel. The stable channel has a bug when trying to mount volumes into containers.

The docker installation wizard is pretty straight forward, so no worries there. Once, installed right-click on the Docker systray icon and select Open Kitematic …

A pop-up window should come up containing instructions on how to download and install Kitematic. It was amazingly simple and gave a nice GUI interface over the command line.

Follow Tyk’s Installer Instructions

Tyk provides instructions to setup the API gateway & dashboard with Docker on their website. I would suggest getting an account at Docker Hub. I don’t remember when in the process I created one, but I needed it to access … something.

In Step 2. Get the quick start compose files you’ll need to git clone the files to an folder under you C:\Users\XXXX folder. For me, Docker had a permissions restriction that only allowed containers to mount volumes from folders under my user folder. (So, that could be interesting if you run a container on a server under a service account.)

The silver lining about this set of containers is that they only need to use config files from your local drive. So, it’s not like your C:\Users folder is going to store a database.

In Step 4. Bootstrap your dashboard and portal, if you have bash available to you I would suggest trying it when you run ./setup.sh. I haven’t installed Win10 Anniversary Update, Git Bash, or Cygwin so I didn’t have bash available to run setup.sh.

However, I do feel somewhat comfortable in powershell, and the setup.sh script didn’t look too long. Below is the powershell conversion, which you should be saved in the same directory as setup.sh, and you should run .\setup.ps1 from the PowerShell ISE with the arguments that you want.

After that, I had a running Tyk API Gateway.

Other Thoughts

Since this was all new technology I ran into a lot of errors and read through a lot of issue/forum posts. Which makes me think this might not be the best idea for a production setup. If you’re able to make linux servers within your production environment, I would strongly suggest that.

Because I made so many mistakes I got used to these three commands which really helped recreate the environment whenever I messed things up. I hope this helps.