Use PowerShell to Process Dump an IIS w3wp Process

Sometimes processes go wild and you would like to collect information on them before killing or restarting the process. And the collection process is generally:

• Your custom made logging
• Open source logging: Elmah, log4Net, etc
• Built in logging on the platform (like AppInsights)
• Event Viewer Logs
• Log aggregators Splunk, New Relic, etc
• and, almost always last on the list, a Process Dump

Process dumps are old enough that they are very well documented, but obscure enough that very few people know how or when to use them. I certainly don’t! But, when you’re really confused about why an issue is occurring a process dump may be the only way to really figure out what was going on inside of a system.

Unfortunately, they are so rarely used that it’s often difficult to re-learn how to get a process dump when an actual problem is occurring. Windows tried to make things easier by adding Create dump file as an option in the Task Manager.

But, logging onto a server to debug a problem is becoming a less frequent occurrence. With Cloud systems the first debugging technique is to just delete the VM/Container/App Service and create a new instance. And, On-Premise web farms are often interacted with through scripting commands.

So here’s another one: New-WebProcDump

This command will take in a ServerName and Url and attempt to take a process dump and put it in a shared location. It does require a number pre-requisites to work:

• The Powershell command must be in a folder with a subfolder named Resources that contains procdump.exe.
• Your web servers are using IIS and ASP.NET Full Framework
• The computer running the command has a D drive
• The D drive has a Temp folder (D:\Temp)
• Remote computers (ie. Web Servers) have a C:\IT\Temp folder.
• You have PowerShell Remoting (ie winrm quickconfig –force) turned on for all the computers in your domain/network.
• The application pools on the Web Server must have names that match up with the url of the site. For example https://unittest.some.company.com should have an application pool of unittest.some.company.com. A second example would be https://unittest.some.company.com/subsitea/ should have an application pool of unittest.some.company.com_subsitea.
• Probably a bunch more that I’m forgetting.

So, here are the scripts that make it work:

• WebAdmin.New-WebProcDump.ps1
  
  Takes a procdump of the w3wp process associated with a given url (either locally or remote). Transfers the process dump to a communal shared location for retrieval.
  
• WebAdmin.Test-WebAppExists.ps1
  
  Check if the an application pool exists on a remote server.
  
• WebAdmin.Test-IsLocalComputerName.ps1
  
  Tests if the command will need to run locally or remotely.
  
• WebAdmin.ConvertTo-UrlBasedAppPoolName.ps1
  
  The name kind of covers it. For example https://unittest.some.company.com should have an application pool of unittest.some.company.com. A second example would be https://unittest.some.company.com/subsitea/ should have an application pool of unittest.some.company.com_subsitea.

Apigee REST Management API with MFA

Not too long ago Apigee updated their documentation to show that Basic Authentication was going to be deprecated on their Management API. This wasn’t really a big deal and it isn’t very difficult to implement an OAuth 2.0 machine-to-machine (grant_type=password) authentication system. Apigee has documentation on how to use their updated version of curl (ie. acurl) to make the calls. But, if you read through a generic explanation of using OAuth it’s pretty straight forward.

But, what about using MFA One Time Password Token’s (OTP) with OAuth authentication? Apigee supports the usage of Google Authenticator to do OTP tokens when signing in through the portal. And … much to my surprise … they also support the OTP tokens in their Management API OAuth login. They call the parameter, mfa_token.

This will sound crazy, but we wanted to setup MFA on an account that is used by a bot/script. Since the bot is only run from a secure location, and the username/password are already securely stored outside of the bot there is really no reason to add MFA to the account login process. It already meets all the criteria for being securely managed. But, on the other hand, why not see if it’s possible?

The only thing left that needed to be figured out was how to generate the One Time Password used by the mfa_token parameter. And, the internet had already done that! (Thank You James Nelson!) All that was left to do was find the Shared Secret Key that the OTP function needed.

Luckily I work with someone knowledgeable on the subject and they pointed out not only that the OTP algorithm that Google Authenticator uses is available on the internet but that Apigee MFA sign-up screen had the Shared Secret Key available on the page. (Thank You Kevin Wu!)

When setting up Google Authenticator in Apigeee, click on the Unable to Scan Barcode? link

Which reveals the OTP Shared Secret:

From there, you just need a little Powershell to tie it all together:

• Apigee.OTP.ps1: A conversion of James Nelson’s Get-OTP.ps1 script into a function.
• Apigee.Login.ps1: An implementation of Apigee Management API OAuth Login
• Apigee.psm1: A script putting the final pieces in place (like the username/password/sharedsecret).

System.Configuration.ConfigurationManager in Core

The .NET Core (corefx) issue, System.Configuration Namespace in .Net Core, ends with the question:

@weshaggard Can you clarify the expectations here for System.Configuration usage?

I was recently converting a .NET Full Framework library over to a .NET Standard library and ran into the exact problem in that issue, and I also got stuck trying to figure out “When and How are you supposed to use System.Configuration.ConfigurationManager?”

I ended up with the answer:

If at all possible, you shouldn’t use it. It’s a facade/shim that only works with the .NET Full Framework. It’s exact purpose is to allow .NET Standard libraries to compile; but it doesn’t work unless the runtime is .NET Full Framework. In order to properly write code using it in a .NET Standard library you will have to use compiler directives to ensure that it doesn’t get executed in a .NET Core runtime. It’s scope, purpose and usage is very limited.

In a .NET Standard library if you want to use configuration information you need to plan for two different configuration systems.

• .NET Full Framework Configuration
  
  Uses ConfigurationManager from the System.Configuration dll installed with the framework. This uses the familiar Configuration.AppSettings[string] and Configuration.ConnectionStrings[string]. This is a unified model in .NET Full Framework and works across all application types: Web, Console, WPF, etc.
  
  
• .NET Core Configuration
  
  Uses ConfigurationBuilder from Microsoft.Extensions.Configuration. And, really, it expects ConfigurationBuilder to be used in an ASP.NET Core website. And this is the real big issue. The .NET Core team focused almost solely on ASP.NET Core and other target platforms really got pushed to the side. Because of this, it’s expecting configuration to be done through the ASP.NET Configuration system at Startup.

And, for now, I can only see two reasonable ways to implement this:

• A single .NET Standard Library that uses compiler directives to determine when to use ConfigurationManager vs a ConfigurationBuilder tie-in.
  
  This would use the System.Configuration.ConfigurationManager nuget package.
  
  Pros:
  - Single library with a single nuget package
  - Single namespace
  
  Cons:
  - You would need a single “Unified Configuration Manager” class which would have #ifdef statements throughout it to determine which configuration system to use.
  - If you did need to reference either the .NET Full Framework or .NET Core Framework the code base would become much more complicated.
  - Unit tests would also need compiler directives to handle differences of running under different Frameworks.
  
• A common shared project used in two libraries each targeting the different frameworks.
  
  This would not use the System.Configuration.ConfigurationManager nuget package.
  
  This is how the AspNet API Versioning project has handled the situation.
  
  Pros:
  - The two top-level libraries can target the exact framework they are intended to be used with. They would have access to the full API set of each framework and would not need to use any shims/facades.
  - The usage of #ifdef statements would be uniform across the files as it would only need to be used to select the correct namespace and using statements.
  - The code would read better as all framework specific would be abstracted out of the shared code using extension methods.
  
  Cons:
  - You would create multiple libraries and multiple nuget packages. This can create headaches and confusion for downstream developers.
  - Unit tests would (most likely) also use multiple libraries, each targeting the correct framework.
  - Requires slightly more overhead to ensure libraries are versioned together and assembly directives are setup in a shared way.
  - The build system would need to handle creating multiple nuget packages.

Apigee TimeTaken AssignVariable vs JS Policy

Apigee’s API Gateway is built on top of a Java code base. And, all of the policies built into the system are pre-compiled Java policies. So, the built in policies have pretty good performance since they are only reading in some cached configuration information and executing natively in the runtime.

Unfortunately, these policies come with two big draw backs:

• In order to do some common tasks (like if x then do y and z) you usually have to use multiple predefined policies chained together. And, those predefined policies are all configured in verbose and cumbersome xml definitions.
• Also, there’s no way to create predefined policies that cover every possible scenario. So, developers will need a way to do things that the original designers never imagined.

For those reasons, there are Javascript Policies which can do anything that javascript can do.

The big drawback with Javascript policies:

• The system has to instantiate a Javascript engine, populate its environment information, run the javascript file, and return the results back to the runtime. This takes time.

So, I was curious how much more time does it take to use a Javascript Policy vs an Assign Message Policy for a very simple task.

It turns out the difference in timing is relatively significant but overall unimportant.

The test used in the comparison checks if a query string parameter exists, and if it does then write it to a header parameter. If the header parameter existed in the first place, then don’t do any of this.

Here are the pseudo-statistical results:

• Average Time Taken (non-scientific measurements, best described as “its about this long”):
• Javascript Policy: ~330,000 nanoseconds (0.33 milliseconds)
• Assign Message Policy: ~50,000 nanoseconds (0.05 milliseonds)
• What you can take away
• A Javascript Policy is about 650% slower or Javascript has about 280,000 nanoseconds overhead for creation, processing and resolution.
• Both Policies take less that 0.5 ms. While the slower performance is relatively significant; in the larger scheme of things, they are both fast.

Javascript Policy

Javascript Timing Results

Assign Message Policy

Assing Message Timing Results

MyGetUcsb PowerShell Module

MyGet.org makes a great product, as written about many other places (Hanselman, Channel 9). However they don’t have a prebuilt toolset to automate tasks for working with MyGet. It’s a bit ironic that MyGet doesn’t publish their own nuget packages.

So, MyGetUcsb is a small PowerShell wrapper around their REST Management API. It’s not a complete implementation of their API, and it doesn’t work “right out of the box”.

• Not a full implementation
  
• It only implements the functions that were necessary to setup user accounts in the way our department needed them to be setup.
  
• Doesn’t work “right out of the box”
  
• The security system is implemented to use a private instance of Secret Server. You’ll need to replace the code that retrieves the API Key with your implementation.
  
• Unsupported and No Plans to Work On
  
• The implementation was just to solve a problem/need that our department had. There’s no plans to support or improve the project in the future. But, feel free to use it as a reference project to build from.

Surprisingly, the Enterprise package (with limited SSO integration) still requires all team members to have MyGet Accounts. All team members must maintain their passwords for MyGet. It’s important they store their MyGet username/password information securely as it’s needed to use the feeds.

To use the module is pretty straight forward. This is an example that will query AD to pull some user information. Convert the information into user objects and then populate MyGet with the account information.

npm Revoking All Tokens Was A Good Call

A week or so ago an ESLint project maintainer’s npmjs key was compromised. The compromised account was then used to upload alterations into some popular ESLint npm packages. The end goal of the altered code was to capture more credentials from open source package maintainers; presumably to create a truly malicious alteration at a future date.

A few things were interesting about the whole event. One of the first things that struck me was the quick and uniform responsiveness of the community. Looking through npmjs’ timeline and the initial bug report, it looks like the attacker published the malicious package at 3:40 AM PST on 7/12/2018. And the initial bug report came in around 4:17 AM PDT. That is an amazing turn around time of 27 minutes. It seemingly supports the open source mantra that “many eyes make a shallow problem”.

The response of the package maintainers and their community members was also fantastic. By 4:59 AM PDT (42 minutes after the report) they had determined that something was seriously wrong and they needed to (a) unpublish the package and (b) had already communicated with downstream sources that they needed to remove all references to the package. This wasn’t done by one person struggling to grapple with the situation, but everyone that was looking at the issue was seeing if they could inform other groups that a serious problem was occurring and action needed to be taken.

What really struck me was that npm’s administrators determined the correct course of action was to revoke all npmjs tokens created between 7/10/2018 at 5:00 PM PDT and 7/12/2018 at 5:30 AM PDT. npm’s administrators looked at the situation and instead of focusing on just the accounts that they knew to be compromised, they just said we don’t know who to trust right now. They had a reasonable expectation that something very wrong happened in that time frame and they weren’t going to take any chances with it. It was a really big decision, and I’m sure some people were really inconvenienced by it. But it was a well calculated move that took into consideration both security and scope. It was a good call.

Two things still bug me about the whole incident:

1. How did the package maintainer’s credentials become compromised?
2. What if the malicious update didn’t cause a compilation error? How long would it have gone undetected? Can many eyes make a shallow problem if there’s no problem to be seen?

The weird thing about this incident is that it reminds me of a blog post from January 2018, “I’m harvesting credit card numbers and passwords from your site. Here’s how.”. In the (hopefully fictitious) blog post it outlines how a small piece of malicious code could potentially be added into an open source project and get put into production use. If you take many of the reasonable claims that blog author makes and add to them the idea that a package maintainers publishing key is compromised, it would take the first set of eyes out of the equation. And those eyes are arguably the most important. The blog post posits truly cryptic code to get the pull request accepted. But, what if the attacker didn’t need to make the code overly cryptic, because the attacker had the ability to accept and publish?

I don’t know.

The one thing I take away from this is that the community acted very responsibly and swiftly; and that’s something to respect and strive to replicate.

Apigee–API Key from Query String or HTTP Header

Apigees’ API Proxy samples are a great way to get started, but many usage scenario aren’t available as a sample. A somewhat common scenario is wanting to check for an API Key in multiple places. In this exapmle, the proxy will need to be able to check for a API Key that is present as a query string parameter or an HTTP header. This allows the Gateway to adhere to Postel’s Law by allowing the client to determine what’s the best way to transmit the information to the service.

That’s all a fancy way of saying: Let’s make it easy check if the API Key is passed in as a Header or a Query String.

Unfortunately, with Apigee, it’s not so easy to check both of them. In this example, we’ll use a Javascript Policy to check if the API Key exists in the query string. If it does, the value will be copied into a header variable. After that, a normal Verify API Key Policy will be used to check the header value.

Hopefully, in a future post, I can reimplement this using only Apigee Policies and without using Javascript. There is a performance penalty that comes with using Javascript in Apigee and I would like to see just how severe that penalty can be.

Here is a Shared Flow that will do just the two steps mentioned above:

And, now for the Javascript. Which is a combination of a Policy and a .js file. This one does all the heavy lifting:

Finally, use the standard Verify API Key policy to check the header value: