Preload a WCF svc before adding to a WebFarm

WCF services hosted in IIS have a first request processing overhead. This is usually just a couple hundred milliseconds, but that can be enough to cause dangerous lag spikes and failed requests when the service is being added to a web farm already under load.

It would seem like using WCF’s serviceAutoStartProvider functionality would prevent the ‘first request’ overhead, but it only lessens it. To truly prevent the overhead from occurring you need to have a request fully processed by the service. Application Initialization is an option, but I haven’t used it in conjunction with a serviceAutoStartProvider.

Instead, I used PowerShell to create a function which would send a request to the service and verify the response. If the response was incorrect or in error, then an exception would be thrown by the function. This has the added benefits of:

• It can be reused by Operations to check the health and status of production systems
• It can be used by Developers and Operations to inspect and test individual application servers when issues are noticed
• It can be used to test the health of a system before being added back into a web farm, in order to ensure that a bad deployment isn’t put into production

To do this I used the built in PowerShell function New-WebServiceProxy. In a script similar to this one:

Function Test-BrokerService {
Param(
    [Parameter(Mandatory=$true)]
    [string]$Environment,
    [Parameter(Mandatory=$true)]
    [string]$BrokerName,
    [string]$DnsName = ""
)
Process {
    # Get environment info
    $envInfo = Get-BrokerEnvironmentVariables $Environment;

    # Setup dnsName
    if($DnsName -eq "") { $DnsName = $envInfo.dnsName; }

    $uri = "https://" + $DnsName + "/broker.svc";
    $proxy = New-WebServiceProxy -Uri $uri

    Write-Warning "Testing $BrokerName on $uri ..."
    switch($BrokerName.ToLower()) {
        "firstBroker" {
            # ...
        }
        default {
            throw ("Broker " + $BrokerName + " is unknown. No test could be performed on " + $uri + ".");
        }
    }

    Write-Host "Successfully tested $BrokerName on $uri"
    $proxy.Dispose();

	return $true;
}
}

I then setup a test system which had:

• A 2008 R2 proxy server using WFF/ARR/UrlRewrite (Proxy)
• Two 2008 R2 application servers to host the WCF services (AppServer1 & 2)

The load test uses:

• 20 concurrent users
• 0 delay between requests
• 2 minute run time
• A deployment script which
  
  • Takes AppServer1 out of the farm
  • Does a code deployment
  • Places AppServer1 back in the farm, without running an initial request
  • Repeats the actions for AppServer2

The small spikes that occur at 1:05 and 1:25 are when AppServer 1 & 2 are added back into the farm. In this particular test no timeouts occurred from it; but it has happened in previous tests.

After changing the deployment script to run an initial request before the server is added back into the web farm, the response timings smoothed out.