IIS Healthcheck & Ninject Memory Leak

This post is definitely not related to the new Healthcheck system introduced in ASP.NET Core.

I’ve written before about a problem with IIS where the Web Farm healthcheck subsystem creates “healthcheck spamming”. The memory leak described in this post is exacerbated by the healthcheck spamming problem and can create application pools with excessive memory use.

The Ninject Memory leak problem is not a problem with Ninject itself, but a problem with the implementation that I had prescribed within our architecture. I just want to be clear that Ninject doesn’t have a memory leak problem, but that I created an implementation which used Ninject improperly and resulted in a memory leak.

Originally, in our Top Level applications (ie. Company.Dealership.Web) the NinjectWebCommon loading system would configure a series of XxxHealthcheckClient and XxxHealthcheckProxy objects to be loaded In Transient Scope. Which is normally fine, as long as you use the created object within a using block; which would ensure the object is disposed. However, the way I was writing the code did not use a using block.

This meant that when a request came into the Company.Dealership.Web.Healtchcheck Controller an instance of Company.Car.HealtcheckClient and Company.Car.HealthcheckProxy would be created and loaded into memory. The request would then progress through Company.Dealership.Web and result in the Proxy making a call to a second website, Company.Car.Web.Healtcheck(Controller). The problem was that once all the calls had completed, the Client and Proxy objects within the Company.Dealership.Web website would not be disposed (the memory leak).

For a low utilization website/webapi this could go unnoticed as IIS’ built-in application pool recycle time of 29 hours could clean up a small memory leak before anyone notices. But, when you compound this memory leak issue with IIS’ healthcheck spamming the issue can become apparent very quickly. In my testing, a website with a single healthcheck client/proxy pair could consume about 100 MB of memory every hour when there are ~200 application pools on the proxy server. (200 appPools x 1 healthcheck every 30 seconds per appPool = 24,000 healthchecks per hour).

The guidance from Ninject’s Web development side is to change your configurations to no longer use Transient Scope when handling web requests. Instead, configurations should scope instances to In Singleton Scope or In Request Scope. I did some testing and In Singleton Scope consistently proved to remove the memory leak issue every time, which In Request Scope didn’t. I tested In Request Scope a few times and one of the times, the memory leak issue reoccurred. Unfortunately, I could not determine why it was leaking and it truly made no sense to me why it happened (it was most likely a misconfigured build). But, either should work.

When using Ninject within a website, any classes which extend Entity Framework’s DBContext should always be configured In Request Scope.

Here is some code which can detect if a class is currently configured to use In Transient Scope (or is not configured at all) and will reconfigure (rebind) the class to In Singleton Scope:

	using System;
	using System.Collections.Generic;
	using System.Linq;
	using System.Reflection;
	using System.Text;
	using Ninject;
	using Ninject.Infrastructure;
	namespace Company.AspNet.WebApi.WebHost
	{
	public static class NinjectWebCommonLoader
	{
	/// <summary>
	/// A standardized way to load assemblies
	///
	/// here's the issue:
	///
	/// The current design pattern for Client classes has
	/// an issue that it never disposes itself or the objects
	/// it uses(Proxy's). This creates a memory leak, as
	/// every requests (especially the thousands of requests
	/// to the healthchecks) will create instances of these
	/// classes and they will never be garbage collected.
	///
	/// okay, so here's the new methodology to fix the problem:
	///
	/// * Load bindings from NinjectModule's
	/// * Search through all the bindings for Client
	/// and Proxy class bindings and ensure they are bound
	/// to either InSingletonScope or InRequestScope.
	/// * If they are not bound to one of those, then bind
	/// it to InSingletonScope.
	///
	///
	/// here's how it does it in detail:
	///
	/// 1. Find all the assemblies made by Ucsb
	/// 2. Split them into the Public and Non-Public assemblies
	/// (Really, this should be done smarter to organize
	/// them by dependency graphs. The lowest level ones
	/// need to be loaded first)
	/// 3. Load the NinjectModules in them.With the Non-Public
	/// first, then load the Public.
	/// 4. Now search through all the assemblies and look for
	/// clases that match* Client, *Proxy and I* Service.
	/// The goal is to use this class/interface list
	/// to find all bindings that have been loaded
	/// for the Client and Proxy list.The I*Service
	/// list is not really needed, but I wanted to cover all
	/// bases.Bindings often look like Bind<I*Service>.To<*Client>.
	/// 5. If we can find the* Service or* Class in Bind<*>.ToSelf() or
	/// Bind<I*Service>.To<*Client> pattern then we need to check if
	/// it was bound to InTransientScope(the default scope).
	/// 6. If it was bound to InTransientScope, or no binding
	/// was found then switch it to InSingletonScope.
	/// </summary>
	/// <param name="kernel">The kernel.</param>
	/// <param name="assemblies">
	/// All the assemblies currently loaded in the application.
	/// Use AppDomain.CurrentDomain.GetAssemblies().
	/// </param>
	/// <param name="callingClassType">
	/// Essentially a this reference for the class that called this method.
	/// Use typeof(ClassName).
	/// </param>
	public static void RegisterServices(IKernel kernel, Assembly[] assemblies, Type callingClassType)
	{
	var topLevelAssembly = callingClassType.Assembly;
	var ucsbAssemblies = assemblies.Where(i => i.FullName.Contains("Company.") && i.IsDynamic == false);
	var notThisOneUcsbAssemblies = ucsbAssemblies.Where(i => i != topLevelAssembly);
	var publicAssemblies = notThisOneUcsbAssemblies.Where(i => i.FullName.Contains("Public"));
	var nonPublicAssemblies = notThisOneUcsbAssemblies.Where(i => i.FullName.Contains("Public") == false);
	foreach (var assembly in nonPublicAssemblies)
	{
	kernel.Load(assembly);
	}
	foreach (var assembly in publicAssemblies)
	{
	kernel.Load(assembly);
	}
	kernel.Load(topLevelAssembly);
	var clientsAndProxies = new List<Type>();
	foreach (var assembly in ucsbAssemblies)
	{
	var found = assembly.GetTypes().Where(i =>
	{
	return i.Name.EndsWith("Client")
	|| i.Name.EndsWith("Proxy");
	});
	clientsAndProxies.AddRange(found);
	}
	foreach (var type in clientsAndProxies)
	{
	var typeMatched = false;
	var typeBindings = kernel.GetBindings(type).ToList();
	if (typeBindings.Count > 0)
	{
	typeMatched = true;
	var binding = typeBindings[0];
	if (binding.ScopeCallback == StandardScopeCallbacks.Transient)
	{
	kernel.Rebind(type).ToSelf().InSingletonScope();
	}
	}
	var interfaces = type.GetInterfaces();
	foreach (var iface in interfaces)
	{
	var ifaceBindings = kernel.GetBindings(iface);
	foreach (var binding in ifaceBindings)
	{
	var prototype = GetPrivateVariable2(binding.ProviderCallback.Target, "prototype");
	if (prototype != null)
	{
	if ((Type)prototype == type)
	{
	typeMatched = true;
	if (binding.ScopeCallback == StandardScopeCallbacks.Transient)
	{
	kernel.Rebind(iface).To(type).InSingletonScope();
	}
	}
	}
	}
	}
	if (typeMatched == false)
	{
	kernel.Bind(type).ToSelf().InSingletonScope();
	}
	}
	}
	/// <summary>
	/// Retrieves a private variables value using reflection.
	/// </summary>
	/// <param name="obj">Object to retrieve the value from.</param>
	/// <param name="variableName">The variable to read.</param>
	/// <returns>The variable's contents as an object.</returns>
	public static object GetPrivateVariable2(
	this object obj,
	string variableName
	)
	{
	// TODO: Move into Enterprise.Utility.Core
	var currentType = obj.GetType();
	FieldInfo info;
	const BindingFlags bFlags = BindingFlags.GetField | BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance;
	do
	{
	info = currentType.GetField(
	variableName,
	bFlags
	);
	currentType = currentType.BaseType;
	} while (info == null && currentType != null);
	if (currentType == null) return null;
	return info.GetValue(obj);
	}
	}
	}

view raw NinjectWebCommonLoader.cs hosted with ❤ by GitHub