Sometimes processes go wild and you would like to collect information on them before killing or restarting the process. And the collection process is generally:
- Your custom made logging
- Open source logging: Elmah, log4Net, etc
- Built in logging on the platform (like AppInsights)
- Event Viewer Logs
- Log aggregators Splunk, New Relic, etc
- and, almost always last on the list, a Process Dump
Process dumps are old enough that they are very well documented, but obscure enough that very few people know how or when to use them. I certainly don’t! But, when you’re really confused about why an issue is occurring a process dump may be the only way to really figure out what was going on inside of a system.
Unfortunately, they are so rarely used that it’s often difficult to re-learn how to get a process dump when an actual problem is occurring. Windows tried to make things easier by adding Create dump file as an option in the Task Manager.
But, logging onto a server to debug a problem is becoming a less frequent occurrence. With Cloud systems the first debugging technique is to just delete the VM/Container/App Service and create a new instance. And, On-Premise web farms are often interacted with through scripting commands.
So here’s another one: New-WebProcDump
This command will take in a ServerName and Url and attempt to take a process dump and put it in a shared location. It does require a number pre-requisites to work:
- The Powershell command must be in a folder with a subfolder named Resources that contains procdump.exe.
- Your web servers are using IIS and ASP.NET Full Framework
- The computer running the command has a D drive
- The D drive has a Temp folder (D:\Temp)
- Remote computers (ie. Web Servers) have a C:\IT\Temp folder.
- You have PowerShell Remoting (ie winrm quickconfig –force) turned on for all the computers in your domain/network.
- The application pools on the Web Server must have names that match up with the url of the site. For example https://unittest.some.company.com should have an application pool of unittest.some.company.com. A second example would be https://unittest.some.company.com/subsitea/ should have an application pool of unittest.some.company.com_subsitea.
- Probably a bunch more that I’m forgetting.
So, here are the scripts that make it work:
- WebAdmin.New-WebProcDump.ps1
Takes a procdump of the w3wp process associated with a given url (either locally or remote). Transfers the process dump to a communal shared location for retrieval. - WebAdmin.Test-WebAppExists.ps1
Check if the an application pool exists on a remote server. - WebAdmin.Test-IsLocalComputerName.ps1
Tests if the command will need to run locally or remotely. - WebAdmin.ConvertTo-UrlBasedAppPoolName.ps1
The name kind of covers it. For example https://unittest.some.company.com should have an application pool of unittest.some.company.com. A second example would be https://unittest.some.company.com/subsitea/ should have an application pool of unittest.some.company.com_subsitea.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| if($global:WebAdmin -eq $null) { | |
| $global:WebAdmin = @{} | |
| } | |
| # http://stackoverflow.com/questions/1183183/path-of-currently-executing-powershell-script | |
| $root = Split-Path $MyInvocation.MyCommand.Path -Parent; | |
| $global:WebAdmin.ProcDumpLocalPath = "$root\Resources\procdump.exe" | |
| <# | |
| .SYNOPSIS | |
| Uses sysinternal procdump to get a proc dump of a w3wp service on a webserver. The file will | |
| be transfered to a shared location for distribution. | |
| .PARAMETER ServerName | |
| The server to pull a proc dump from. | |
| .PARAMETER Url | |
| The url of the website to get a proc dump from | |
| .EXAMPLE | |
| Command: | |
| New-WebProcDumpUcsb -ServerName SA177 -Url my.dev.sa.ucsb.edu/aaa | |
| Output: | |
| #> | |
| Function New-WebProcDump { | |
| [CmdletBinding()] | |
| Param ( | |
| [Parameter(Mandatory=$false)] | |
| [string] $ServerName = $env:COMPUTERNAME, | |
| [Parameter(Mandatory=$true)] | |
| [string] $Url | |
| ) | |
| # setup variables | |
| $appPoolName = ConvertTo-UrlBasedAppPoolName -Url $Url | |
| $isLocalMachine = Test-IsLocalComputerName -ComputerName $ServerName | |
| if((Test-WebAppExists -ServerName $ServerName -Url $Url) -eq $false) { | |
| throw "IIS $env:COMPUTERNAME - No webapp could be found for url $Url on $ServerName" | |
| } | |
| # ensure procdump exists on the remote server | |
| if((Test-Path $global:WebAdmin.ProcDumpLocalPath) -eq $false) { | |
| throw "IIS $env:COMPUTERNAME - Cannot find local copy of procdump.exe in WebAdministrationUcsb module ($($global:WebAdministrationUcsb.ProcDumpLocalPath)). Ensure it exists before running again." | |
| } | |
| if($isLocalMachine) { | |
| # gonna run procdump locally so the local procdump in the module will be used. | |
| } else { | |
| # gonna run this on a remote server, so ensure that procdump is on the server | |
| $utilRemotePath = "\\{0}\C$\IT\Utilities" -f $ServerName | |
| if((Test-Path $utilRemotePath) -eq $false) { | |
| New-Item -Path $utilRemotePath -ItemType Directory | Out-Null | |
| } | |
| $procdumpRemotePath = "$utilRemotePath\procdump.exe" | |
| if((Test-Path $procdumpRemotePath) -eq $false) { | |
| Copy-Item -Path $global:WebAdministrationUcsb.ProcDumpLocalPath -Destination $utilRemotePath | Out-Null | |
| } | |
| } | |
| # get the process info from the remote server | |
| $processScript = { | |
| if($appPoolName -eq $null) { | |
| $appPoolName = $args[0] | |
| } | |
| Import-Module WebAdministration | |
| $webModule = Get-Module WebAdministration | |
| if(-not $webModule) { | |
| Import-Module WebAdministration | |
| } | |
| $processes = dir "IIS:\AppPools\$appPoolName\WorkerProcesses" | |
| return $processes | |
| } | |
| $params = @($appPoolName) | |
| if($isLocalMachine) { | |
| $w = . $processScript | |
| } else { | |
| $w = Invoke-Command -ComputerName $ServerName -ScriptBlock $processScript -ArgumentList $params | |
| } | |
| if($w -eq $null) { | |
| throw "IIS $env:COMPUTERNAME - No process for appPool $appPoolName on $ServerName could be found." | |
| } | |
| if(@($w).Count -gt 1) { | |
| throw "IIS $env:COMPUTERNAME - Multiple processes for appPool $appPoolName on $ServerName were found. This is weird, contact an administrator. Process Count: $(@($w).Count)" | |
| } | |
| # run the dump remotely | |
| $dumpScript = { | |
| if($processId -eq $null) { | |
| $processId = $args[0] | |
| } | |
| if($procdump -eq $null) { | |
| $procdump = "C:\IT\Utilities\procdump.exe" | |
| } | |
| cd "C:\Users\$($env:USERNAME)\AppData\Local\Temp" | |
| $out = . $procdump -ma -accepteula $processId | |
| $line = $out |? { $_ -match "Dump 1 initiated" } | |
| $ix = $line.IndexOf("ed: ") | |
| $path = $line.Substring($ix + 4) | |
| return $path | |
| } | |
| $processId = $w.processId | |
| $procdump = $global:WebAdmin.ProcDumpLocalPath | |
| if($isLocalMachine) { | |
| $path = . $dumpScript | |
| } else { | |
| $path = Invoke-Command -ComputerName $ServerName -ScriptBlock $dumpScript -ArgumentList $processId | |
| } | |
| # copy dump to local storage | |
| $sharepath = "" | |
| if([string]::IsNullOrWhiteSpace($path) -eq $false) { | |
| $nwPath = $path -replace "C:\\", "\\$ServerName\C$\" | |
| if(Test-Path $nwPath) { | |
| $parent = Split-Path $nwPath -Parent | |
| $leaf = Split-Path $nwPath -Leaf | |
| $null = . robocopy "$parent" "D:\Temp\" /r:1 /w:1 $leaf | |
| $locpath = "D:\Temp\$leaf" | |
| $curnttime = [DateTime]::Now.ToString("yyyyMMddHHmm") | |
| $newfilename = "$ServerName-$appPoolName-w3wp-$curnttime.dmp" | |
| $newpath = "D:\Temp\$newfilename" | |
| mv $locpath $newpath | |
| del $nwPath -Force -ErrorAction SilentlyContinue | |
| $sharepath = "\\$($env:COMPUTERNAME)\d\temp\$newfilename" | |
| } | |
| } | |
| return $sharepath | |
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <# | |
| .SYNOPSIS | |
| Tests if a web application exists (this can be a site or application). | |
| This was only written to keep naming conventions consistent. This is the same as | |
| Test-Path IIS:\Sites\$SiteName; | |
| .PARAMETER Url | |
| The url of the match on | |
| .PARAMETER Environment | |
| The environment to apply this to | |
| .EXAMPLE | |
| Test-WebAppExists -Environment Dev -Url "http://unittest.{env}.place.something.com/services" | |
| #> | |
| Function Test-WebAppExists { | |
| Param ( | |
| [string] $ServerName = $env:COMPUTERNAME, | |
| [Parameter(Mandatory = $true)] | |
| [string] $Url | |
| ) | |
| # setup variables | |
| $appPoolName = ConvertTo-UrlBasedAppPoolName -Url $Url | |
| $scriptBlock = { | |
| if($appPoolName -eq $null) { | |
| $appPoolName = $args[0] | |
| } | |
| Import-Module WebAdministration | |
| $pathToTest = "IIS:\AppPools\{0}" -f $appPoolName | |
| if((Test-Path $pathToTest) -eq $false) { return $false } | |
| $app = Get-Item $pathToTest | |
| $exists = $true | |
| if($app -eq $null) { $exists = $false } | |
| if($app.GetType().Fullname -ne "Microsoft.IIs.PowerShell.Framework.ConfigurationElement") { $exists = $false } | |
| return $exists; | |
| } # end scriptblock | |
| $parameters = @($appPoolName) | |
| if(Test-IsLocalComputerName -ComputerName $ServerName) { | |
| $exists = . $scriptBlock | |
| } else { | |
| $exists = Invoke-Command -ComputerName $ServerName -ScriptBlock $scriptBlock -ArgumentList $parameters | |
| } | |
| return $exists; | |
| } | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <# | |
| .SYNOPSIS | |
| Checks if the given ComputerName is for the local computer | |
| .PARAMETER ComputerName | |
| The name of a computer to check. | |
| .EXAMPLE | |
| $session = New-PSSession . | |
| $computerName = $session.ComputerName | |
| if(Test-IsLocalComputerName $computerName) { ... } | |
| #> | |
| Function Test-IsLocalComputerName { | |
| [CmdletBinding()] | |
| [OutputType([bool])] | |
| Param ( | |
| [Parameter(Mandatory = $true)] | |
| [string] $ComputerName | |
| ) | |
| <# DEBUGGING | |
| $callStack = Get-PSCallStack | |
| if ($callStack.Count -gt 0) { | |
| Write-Host ("$($env:COMPUTERNAME) - Test-IsLocalComputerName - Parent function: {0}" -f $callStack[1].FunctionName) | |
| } | |
| Write-Host "$($env:COMPUTERNAME) - Test-IsLocalComputerName - ComputerName = $ComputerName" | |
| #> | |
| if($ComputerName -eq "localhost") { return $true; } | |
| if($ComputerName -eq $env:COMPUTERNAME) { return $true; } | |
| $address = [System.Net.Dns]::GetHostAddresses($ComputerName).IPAddressToString | |
| if($address.StartsWith("127.")) { return $true; } | |
| $addressesOnThisMachine = [System.Net.Dns]::GetHostAddresses($env:COMPUTERNAME).IPAddressToString | |
| if($addressesOnThisMachine -contains $address) { return $true; } | |
| #Write-Host "$($env:COMPUTERNAME) - Test-IsLocalComputerName - Result = $false" | |
| return $false; | |
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <# | |
| .SYNOPSIS | |
| Enforces the formatting standards for application pool names. | |
| This should be used to figure out the application pool name before creating an | |
| new one. The name is also used to create unique ARR rule names. | |
| .PARAMETER Url | |
| The url to parse and convert to our standardized app pool name. | |
| .PARAMETER Environment | |
| If an environment is also passed, the url will be run through Get-WebEnvironmentUri | |
| before being parsed/converted. | |
| .LINK | |
| Get-WebEnvironmentUri | |
| .EXAMPLE | |
| $url = "http://unittest.{env}.place.something.com" | |
| $env = "dev" | |
| $appPoolName = ConvertTo-UrlBasedAppPoolName -Url $url -Environment $env | |
| #> | |
| Function ConvertTo-UrlBasedAppPoolName { | |
| [CmdletBinding()] | |
| Param ( | |
| [Parameter(Mandatory = $true)] | |
| [string] $Url | |
| ) | |
| $parseUrl = $Url | |
| $m = "(https?://)?(.*)" | |
| if($parseUrl -match $m) { | |
| $hostPath = $Matches[2] | |
| } | |
| $hostPath = $hostPath.Replace("/","_") | |
| # this prevents http://aaa.sa.ucsb.edu/ from becoming aaa.sa.ucsb.edu_ | |
| $pathLen = $hostPath.Length; | |
| if($pathLen -gt 0) { | |
| if($hostPath[$pathLen - 1] -eq "_") { | |
| $hostPath = $hostPath.Substring(0, $pathLen - 1); | |
| } | |
| } | |
| return $hostPath.ToLower() | |
| } |
