Use PowerShell to Process Dump an IIS w3wp Process

on Monday, August 27, 2018

Sometimes processes go wild and you would like to collect information on them before killing or restarting the process. And the collection process is generally:

  • Your custom made logging
  • Open source logging: Elmah, log4Net, etc
  • Built in logging on the platform (like AppInsights)
  • Event Viewer Logs
  • Log aggregators Splunk, New Relic, etc
  • and, almost always last on the list, a Process Dump

Process dumps are old enough that they are very well documented, but obscure enough that very few people know how or when to use them. I certainly don’t! But, when you’re really confused about why an issue is occurring a process dump may be the only way to really figure out what was going on inside of a system.

Unfortunately, they are so rarely used that it’s often difficult to re-learn how to get a process dump when an actual problem is occurring. Windows tried to make things easier by adding Create dump file as an option in the Task Manager.

image

But, logging onto a server to debug a problem is becoming a less frequent occurrence. With Cloud systems the first debugging technique is to just delete the VM/Container/App Service and create a new instance. And, On-Premise web farms are often interacted with through scripting commands.

So here’s another one: New-WebProcDump

This command will take in a ServerName and Url and attempt to take a process dump and put it in a shared location. It does require a number pre-requisites to work:

  • The Powershell command must be in a folder with a subfolder named Resources that contains procdump.exe.
  • Your web servers are using IIS and ASP.NET Full Framework
  • The computer running the command has a D drive
    • The D drive has a Temp folder (D:\Temp)
  • Remote computers (ie. Web Servers) have a C:\IT\Temp folder.
  • You have PowerShell Remoting (ie winrm quickconfig –force) turned on for all the computers in your domain/network.
  • The application pools on the Web Server must have names that match up with the url of the site. For example https://unittest.some.company.com should have an application pool of unittest.some.company.com. A second example would be https://unittest.some.company.com/subsitea/ should have an application pool of unittest.some.company.com_subsitea.
  • Probably a bunch more that I’m forgetting.

So, here are the scripts that make it work:

  • WebAdmin.New-WebProcDump.ps1

    Takes a procdump of the w3wp process associated with a given url (either locally or remote). Transfers the process dump to a communal shared location for retrieval.
  • WebAdmin.Test-WebAppExists.ps1

    Check if the an application pool exists on a remote server.
  • WebAdmin.Test-IsLocalComputerName.ps1

    Tests if the command will need to run locally or remotely.
  • WebAdmin.ConvertTo-UrlBasedAppPoolName.ps1

    The name kind of covers it. For example https://unittest.some.company.com should have an application pool of unittest.some.company.com. A second example would be https://unittest.some.company.com/subsitea/ should have an application pool of unittest.some.company.com_subsitea.


if($global:WebAdmin -eq $null) {
$global:WebAdmin = @{}
}
# http://stackoverflow.com/questions/1183183/path-of-currently-executing-powershell-script
$root = Split-Path $MyInvocation.MyCommand.Path -Parent;
$global:WebAdmin.ProcDumpLocalPath = "$root\Resources\procdump.exe"
<#
.SYNOPSIS
Uses sysinternal procdump to get a proc dump of a w3wp service on a webserver. The file will
be transfered to a shared location for distribution.
.PARAMETER ServerName
The server to pull a proc dump from.
.PARAMETER Url
The url of the website to get a proc dump from
.EXAMPLE
Command:
New-WebProcDumpUcsb -ServerName SA177 -Url my.dev.sa.ucsb.edu/aaa
Output:
#>
Function New-WebProcDump {
[CmdletBinding()]
Param (
[Parameter(Mandatory=$false)]
[string] $ServerName = $env:COMPUTERNAME,
[Parameter(Mandatory=$true)]
[string] $Url
)
# setup variables
$appPoolName = ConvertTo-UrlBasedAppPoolName -Url $Url
$isLocalMachine = Test-IsLocalComputerName -ComputerName $ServerName
if((Test-WebAppExists -ServerName $ServerName -Url $Url) -eq $false) {
throw "IIS $env:COMPUTERNAME - No webapp could be found for url $Url on $ServerName"
}
# ensure procdump exists on the remote server
if((Test-Path $global:WebAdmin.ProcDumpLocalPath) -eq $false) {
throw "IIS $env:COMPUTERNAME - Cannot find local copy of procdump.exe in WebAdministrationUcsb module ($($global:WebAdministrationUcsb.ProcDumpLocalPath)). Ensure it exists before running again."
}
if($isLocalMachine) {
# gonna run procdump locally so the local procdump in the module will be used.
} else {
# gonna run this on a remote server, so ensure that procdump is on the server
$utilRemotePath = "\\{0}\C$\IT\Utilities" -f $ServerName
if((Test-Path $utilRemotePath) -eq $false) {
New-Item -Path $utilRemotePath -ItemType Directory | Out-Null
}
$procdumpRemotePath = "$utilRemotePath\procdump.exe"
if((Test-Path $procdumpRemotePath) -eq $false) {
Copy-Item -Path $global:WebAdministrationUcsb.ProcDumpLocalPath -Destination $utilRemotePath | Out-Null
}
}
# get the process info from the remote server
$processScript = {
if($appPoolName -eq $null) {
$appPoolName = $args[0]
}
Import-Module WebAdministration
$webModule = Get-Module WebAdministration
if(-not $webModule) {
Import-Module WebAdministration
}
$processes = dir "IIS:\AppPools\$appPoolName\WorkerProcesses"
return $processes
}
$params = @($appPoolName)
if($isLocalMachine) {
$w = . $processScript
} else {
$w = Invoke-Command -ComputerName $ServerName -ScriptBlock $processScript -ArgumentList $params
}
if($w -eq $null) {
throw "IIS $env:COMPUTERNAME - No process for appPool $appPoolName on $ServerName could be found."
}
if(@($w).Count -gt 1) {
throw "IIS $env:COMPUTERNAME - Multiple processes for appPool $appPoolName on $ServerName were found. This is weird, contact an administrator. Process Count: $(@($w).Count)"
}
# run the dump remotely
$dumpScript = {
if($processId -eq $null) {
$processId = $args[0]
}
if($procdump -eq $null) {
$procdump = "C:\IT\Utilities\procdump.exe"
}
cd "C:\Users\$($env:USERNAME)\AppData\Local\Temp"
$out = . $procdump -ma -accepteula $processId
$line = $out |? { $_ -match "Dump 1 initiated" }
$ix = $line.IndexOf("ed: ")
$path = $line.Substring($ix + 4)
return $path
}
$processId = $w.processId
$procdump = $global:WebAdmin.ProcDumpLocalPath
if($isLocalMachine) {
$path = . $dumpScript
} else {
$path = Invoke-Command -ComputerName $ServerName -ScriptBlock $dumpScript -ArgumentList $processId
}
# copy dump to local storage
$sharepath = ""
if([string]::IsNullOrWhiteSpace($path) -eq $false) {
$nwPath = $path -replace "C:\\", "\\$ServerName\C$\"
if(Test-Path $nwPath) {
$parent = Split-Path $nwPath -Parent
$leaf = Split-Path $nwPath -Leaf
$null = . robocopy "$parent" "D:\Temp\" /r:1 /w:1 $leaf
$locpath = "D:\Temp\$leaf"
$curnttime = [DateTime]::Now.ToString("yyyyMMddHHmm")
$newfilename = "$ServerName-$appPoolName-w3wp-$curnttime.dmp"
$newpath = "D:\Temp\$newfilename"
mv $locpath $newpath
del $nwPath -Force -ErrorAction SilentlyContinue
$sharepath = "\\$($env:COMPUTERNAME)\d\temp\$newfilename"
}
}
return $sharepath
}
<#
.SYNOPSIS
Tests if a web application exists (this can be a site or application).
This was only written to keep naming conventions consistent. This is the same as
Test-Path IIS:\Sites\$SiteName;
.PARAMETER Url
The url of the match on
.PARAMETER Environment
The environment to apply this to
.EXAMPLE
Test-WebAppExists -Environment Dev -Url "http://unittest.{env}.place.something.com/services"
#>
Function Test-WebAppExists {
Param (
[string] $ServerName = $env:COMPUTERNAME,
[Parameter(Mandatory = $true)]
[string] $Url
)
# setup variables
$appPoolName = ConvertTo-UrlBasedAppPoolName -Url $Url
$scriptBlock = {
if($appPoolName -eq $null) {
$appPoolName = $args[0]
}
Import-Module WebAdministration
$pathToTest = "IIS:\AppPools\{0}" -f $appPoolName
if((Test-Path $pathToTest) -eq $false) { return $false }
$app = Get-Item $pathToTest
$exists = $true
if($app -eq $null) { $exists = $false }
if($app.GetType().Fullname -ne "Microsoft.IIs.PowerShell.Framework.ConfigurationElement") { $exists = $false }
return $exists;
} # end scriptblock
$parameters = @($appPoolName)
if(Test-IsLocalComputerName -ComputerName $ServerName) {
$exists = . $scriptBlock
} else {
$exists = Invoke-Command -ComputerName $ServerName -ScriptBlock $scriptBlock -ArgumentList $parameters
}
return $exists;
}
<#
.SYNOPSIS
Checks if the given ComputerName is for the local computer
.PARAMETER ComputerName
The name of a computer to check.
.EXAMPLE
$session = New-PSSession .
$computerName = $session.ComputerName
if(Test-IsLocalComputerName $computerName) { ... }
#>
Function Test-IsLocalComputerName {
[CmdletBinding()]
[OutputType([bool])]
Param (
[Parameter(Mandatory = $true)]
[string] $ComputerName
)
<# DEBUGGING
$callStack = Get-PSCallStack
if ($callStack.Count -gt 0) {
Write-Host ("$($env:COMPUTERNAME) - Test-IsLocalComputerName - Parent function: {0}" -f $callStack[1].FunctionName)
}
Write-Host "$($env:COMPUTERNAME) - Test-IsLocalComputerName - ComputerName = $ComputerName"
#>
if($ComputerName -eq "localhost") { return $true; }
if($ComputerName -eq $env:COMPUTERNAME) { return $true; }
$address = [System.Net.Dns]::GetHostAddresses($ComputerName).IPAddressToString
if($address.StartsWith("127.")) { return $true; }
$addressesOnThisMachine = [System.Net.Dns]::GetHostAddresses($env:COMPUTERNAME).IPAddressToString
if($addressesOnThisMachine -contains $address) { return $true; }
#Write-Host "$($env:COMPUTERNAME) - Test-IsLocalComputerName - Result = $false"
return $false;
}
<#
.SYNOPSIS
Enforces the formatting standards for application pool names.
This should be used to figure out the application pool name before creating an
new one. The name is also used to create unique ARR rule names.
.PARAMETER Url
The url to parse and convert to our standardized app pool name.
.PARAMETER Environment
If an environment is also passed, the url will be run through Get-WebEnvironmentUri
before being parsed/converted.
.LINK
Get-WebEnvironmentUri
.EXAMPLE
$url = "http://unittest.{env}.place.something.com"
$env = "dev"
$appPoolName = ConvertTo-UrlBasedAppPoolName -Url $url -Environment $env
#>
Function ConvertTo-UrlBasedAppPoolName {
[CmdletBinding()]
Param (
[Parameter(Mandatory = $true)]
[string] $Url
)
$parseUrl = $Url
$m = "(https?://)?(.*)"
if($parseUrl -match $m) {
$hostPath = $Matches[2]
}
$hostPath = $hostPath.Replace("/","_")
# this prevents http://aaa.sa.ucsb.edu/ from becoming aaa.sa.ucsb.edu_
$pathLen = $hostPath.Length;
if($pathLen -gt 0) {
if($hostPath[$pathLen - 1] -eq "_") {
$hostPath = $hostPath.Substring(0, $pathLen - 1);
}
}
return $hostPath.ToLower()
}

0 comments:

Post a Comment


Creative Commons License
This site uses Alex Gorbatchev's SyntaxHighlighter, and hosted by herdingcode.com's Jon Galloway.