Azure/Elastic Kubernetes Service and gMSA

I’ve written previously about how Docker Containers Are Not Domain Joined and all of the difficulties that it creates.

This post simply adds to that previous article with a little more links and information.

When I first heard of Docker, I imagined a system where you would throw a container at a service and it would figure out everything that was needed to run the container and just make it happen. Obviously that’s extremely difficult to do and as I learnt more about Docker the larger and more engrossing the problem became. My current understanding is no where near complete but here’s some more info on the problem.

In 2018, around the time I looked at AWS’ ALB prices, I looked into a price comparison of a Dockerized Web Farm vs an a IIS EC2 Web Farm. When developing out the system architecture for the Dockerized Web Farm I ran into two major issues:

• Theoretically, it looks like, Windows containers use an absolute limit (search for “CPU limit is enforced as an absolute limit”) when allocating CPU utilization to the container.
  
  NOTE: I have not gotten to the point where I can prove or disprove the above statement; and OLDER Docker documentation doesn’t seem to indicate that Windows has this problem.
  
  What this means is that if you have a 2 CPU Host system, and you were to allocate .5 CPU to a Windows Container, then the Windows container would be given that .5 CPU for it’s sole usage. No other container could use the .5 CPU and the allocating container would be hard-capped at .5 CPU.
  
  In Linux containers this is not an issue. You can allocate dozens of containers on a single host to use .5 CPU and they would (a) all share the full 100% CPU resources available, (b) never be hard-capped, and (c) only use the .5 CPU hard cap once the CPU reached 100% utilization and it needed to share the CPU between two containers that were fighting over the CPUs time.
  
• The gMSA issue that was brought up in previous Is SQL Server looking to Dockerize on Windows? post.

Even with those issues, I was curious about what AWS was doing with containers in hopes that they had the same idea that I did: We should be able to give a container image to a service and the service just figures out everything needed to run it and maked it happen. And they did: AWS Fargate.

But!! …

They were also frustrated with the permissions and gMSA security issues that the Windows OS introduced into the equation. And, as such, they don’t support Windows Containers on Fargate. They don’t directly say that they don’t support it because of the gMSA/permissions issues, but when you look at what needs to be done to support gMSA it becomes an easily rationalized conclusion. Here’s what it looks like to use a gMSA account on a Windows Container (with all the secret/password storage and management removed):

1. Create a gMSA account in Active Directory.
2. Select the Docker Host that will host the new container instance.
3. Update Active Directory to register the gMSA to be usable on that Docker Host.
4. Register the gMSA on the Docker Host (checks with Active Directory to validate the request).
5. Start the container, and you’re now able use the gMSA account within the container.
6. You’ll need to reapply the registrations (steps 2-4) for each Docker Host that the container will run on.

With a fully automated provisioning process, that’s not that difficult. It’s really doable in fact. However, here’s the list of difficult specifics that a fully managed Kubernetes infrastructure (like Fargate) would have to deal with:

1. Where is the Active Directory located?
2. Are the networking routes open for the ECS/Fargate infrastructure to it?
3. Are there other security requirements?
4. What versions of Active Directory are supported?
5. etc, etc, etc …

I don’t know at what bullet point you just *facepalm* and say “We’re not supporting this.”

But!! …

Figuring out all the details of this should be in the wheel house of Azure, right? It’s the Microsoft OS and platform, they are probably working on this problem.

So, here’s what the landscape looks like today with AKS:

• AKS does not support gMSA accounts (along with a number of other limitations).
• However it does have some interesting integrations with Azure Active Directory.
  
  The caveat to using Azure Active Directory with gMSA accounts is that not all resources used in an application will be running in Azure. For many applications, you need the gMSA to be in the companies on-premise Active Directory and replicated into the Azure AD. And, I don’t know if registering a gMSA account with a Host name within the on-premise Active Directory flows that registration into the Azure AD.
  
  And then, does that lead back to the same set of “Are the networking routes open for the AKS infrastructure to the on-premise AD” questions?
• There seems to be something going on with Open Service Broker API (OSBA) that is trying to tackle these these permissions problems, but I don’t fully understand it yet.
• I don’t see direct documentation on the CPU limiting issue, so maybe they have that solved?

So there’s the update.