Update: YACLP for Slack/Hubot

This is an update for Yet Another Command Line Parser for Slack/Hubot.

Since the last post, a few minor updates have been made:

• Added a validation set for a required parameter

product-name action {required-param-name=option1|option2|option3}

• Added a validation set for an optional parameter with a default value

product-name action {required-param-name=option1|option2:option2}

• Made the parameter check Case Insensitive
• Added ability to take in an empty string as an input value (“”)
• Added parsing that can handle multiple spaces between parameters
• Added parameters passed in format parameter-name=value format

Here’s the updated version:

IIS Healthcheck & Ninject Memory Leak

This post is definitely not related to the new Healthcheck system introduced in ASP.NET Core.

I’ve written before about a problem with IIS where the Web Farm healthcheck subsystem creates “healthcheck spamming”. The memory leak described in this post is exacerbated by the healthcheck spamming problem and can create application pools with excessive memory use.

The Ninject Memory leak problem is not a problem with Ninject itself, but a problem with the implementation that I had prescribed within our architecture. I just want to be clear that Ninject doesn’t have a memory leak problem, but that I created an implementation which used Ninject improperly and resulted in a memory leak.

Originally, in our Top Level applications (ie. Company.Dealership.Web) the NinjectWebCommon loading system would configure a series of XxxHealthcheckClient and XxxHealthcheckProxy objects to be loaded In Transient Scope. Which is normally fine, as long as you use the created object within a using block; which would ensure the object is disposed. However, the way I was writing the code did not use a using block.

This meant that when a request came into the Company.Dealership.Web.Healtchcheck Controller an instance of Company.Car.HealtcheckClient and Company.Car.HealthcheckProxy would be created and loaded into memory. The request would then progress through Company.Dealership.Web and result in the Proxy making a call to a second website, Company.Car.Web.Healtcheck(Controller). The problem was that once all the calls had completed, the Client and Proxy objects within the Company.Dealership.Web website would not be disposed (the memory leak).

For a low utilization website/webapi this could go unnoticed as IIS’ built-in application pool recycle time of 29 hours could clean up a small memory leak before anyone notices. But, when you compound this memory leak issue with IIS’ healthcheck spamming the issue can become apparent very quickly. In my testing, a website with a single healthcheck client/proxy pair could consume about 100 MB of memory every hour when there are ~200 application pools on the proxy server. (200 appPools x 1 healthcheck every 30 seconds per appPool = 24,000 healthchecks per hour).

The guidance from Ninject’s Web development side is to change your configurations to no longer use Transient Scope when handling web requests. Instead, configurations should scope instances to In Singleton Scope or In Request Scope. I did some testing and In Singleton Scope consistently proved to remove the memory leak issue every time, which In Request Scope didn’t. I tested In Request Scope a few times and one of the times, the memory leak issue reoccurred. Unfortunately, I could not determine why it was leaking and it truly made no sense to me why it happened (it was most likely a misconfigured build). But, either should work.

When using Ninject within a website, any classes which extend Entity Framework’s DBContext should always be configured In Request Scope.

Here is some code which can detect if a class is currently configured to use In Transient Scope (or is not configured at all) and will reconfigure (rebind) the class to In Singleton Scope:

Creating Charts/Graphs with Powershell for Slack - Pt3

Continued from Creating Charts/Graphs with Powershell for Slack – Pt2.

Send the Graph Back Into Slack

Since ChartJs was able to create a .png image of the chart, now we just have to figure out how to get the image into a message and send it back to slack.

In my imagination, the best possible approach would be to use Slack APIs files.upload endpoint to push the image up to slack and simply reference the image using slack’s private urls. However, I could not get this to work. Maybe someday in the future.

The PSSlack module (which is great!) does have an upload feature built into the New-SlackMessageAttachment command. But, in my experimentation I was only able to upload files or large text blocks; when I tried to upload images they never appeared as image. They just appeared as files that could be downloaded. Maybe I was doing something wrong.

So, I went a third route and did something which is pretty bad design. I used a website that I had access to in order to host the images and reference them as urls. This comes with the drawback that the images hosted on the website would need to be monitored for retention periods and cleanup. But, it’s a quick and easy way to get the image up there.

Below is a wrapper command which will use the image url in an PSSlack message. This will display the newly created graph in chat just as you would hope.

Script using the functions together:

Send-SlackHubotImageUrl:

Creating Charts/Graphs with Powershell for Slack–Pt2

Continued from Creating Charts/Graphs with Powershell for Slack – Pt1.

Generate a graph/chart

So, this was an adventure that went down a wild number of paths. What I needed was a program that could run from powershell and generate a graph in .png/.svg format. The image would be used later to send a message in slack.

Skip down below to chartjs/canvas for a solution.

Initially I wanted to be able to build a graph using powershell, so I searched the https://www.powershellgallery.com/ and github for what they had under 'chart’ or ‘graph’. Here’s some of the ones I tried and I why I didn’t settle on them:

• Using System.Windows.Forms.DataVisualization
  (https://learn-powershell.net/2016/09/18/building-a-chart-using-powershell-and-chart-controls/)
  
  I really wanted to find a solution that had this all figured out and wrapped up into an easy to use module.
  
• Log2chart (https://github.com/buscseik/log2chart/)
  
  ”html based graphs” … nope. I needed an image output.
  
• SvgChart (https://github.com/nicholasdille/PowerShell-SvgChart)
  
  I never actually got this to work. I used the example (slimmed down a bit) but could never get it to generate a chart.
  
• SeeShell (https://github.com/beefarino/seeshell)
  
  So, this had one of the coolest demos I’ve ever seen! It can take continuous input from a source (like processer utilization) and generate a live graph. Really Cool!
  
  BUT … I couldn’t figure out how to output in an image format.
  
  
  
• OxyPlotCli (https://github.com/horker/oxyplotcli)
  
  This was easy to setup and easy to create a graph, but I could not figure out how to force the Y axis to go from 0 to 100. Since one of the graphs I was making was a CPU graph I needed to have the traditional min/max values displayed.
  
• PSGraph (https://github.com/KevinMarquette/PSGraph)
  
  The very beginning of their description used the words graph, edge, and node which made me think it was a utility for better understanding networks (and not bar charts/line charts). A few people I work with suggested taking another look at this and I might do that again.
  

At this point, I changed directions and started looking for regular .NET or nuget packages which could do graphing. This also resulted in a dead end, usually because of price.

• Live Charts (https://lvcharts.net/)
  
  I didn’t see a way to generate an image.
  
• Highcharts .NET (https://www.highcharts.com/blog/products/dotnet/)
  
  This one came up a lot on stackoverflow answers and blog posts. I think it’s really good and the price tag shows that the developers believe that too.
  
• DotNetCharting (https://www.dotnetcharting.com/)
  
  I think this could do the job, but it also costs money.
  
• OxyPlot (http://www.oxyplot.org/)
  
  I don’t think I spent as much time on this as I should have. I was still a little sore at OxyPlotCli making it seem like it was impossible to force the setting of the X axis from 0 to 100. When working with the powershell version I used the OxyPlot source code to check if I could set the Y axis min/max values using reflection, but the X axis is always recalculated dynamically on line graphs. So, I assumed that would also be the case with the nuget version.
  

So, by this point I reached back in time to about 15 years ago and a friend was showing me the plot he had made in a just a couple hours with Gnuplot. And, he was using a Windows PC.

• Gnuplot win64 mingw pre-compiled
  (http://tmacchant3.starfree.jp/gnuplot/Eng/winbin/)
  
  I really should have spent more time with this one. I only spent about 30 minutes on it and didn’t get a graph produced, so I scratched it. But, looking back now, it was probably the most straight forward product with the least amount of dependencies. I would really like to take a second look at this one.

At this point, I decided to try to stick with the two technology stacks I was already using (powershell & nodejs). Since, I felt like I had exhausted the powershell stack, I took a look at npm for the nodejs stack.

• Google Charts (https://developers.google.com/chart/)
  
  Google Charts came up over and over again on stackoverflow and blog post answers. But, most of the answers had to do with an older (and recently deprecated 3/18/2019) version which allowed for web api calls to be made to google which would return the generated charts.
  
  The newer Google Charts runs completely in the browser and generates the graphs on the client side. To make this work, I would just need to be able to use a shim in nodejs to make it use the google charts as if it were in the browser. The recommended choice was JSDOM.
  
  However, before I really took the time to make this work I remembered that a co-worker got charting working using chartjs & canvas. So, I did a google search on that and …

chartjs/canvas on node charting

• chartjs (examples) (https://www.npmjs.com/package/chartjs)
• canvas (https://www.npmjs.com/package/canvas)

This finally did the trick. The software was pretty easy to use, with plenty of examples on their website and it was in a technology stack that had previous experience with.

The challenge with using this combination is that I wanted to make the charting functionality available from a Powershell command. So, doing the ridiculous, I built a Powershell module around ChartJS. (Powershell code which, if you remember from Pt 1, is being called from nodejs!)

Some notes on this code …

• packages.json
  
  To use this module you will need to use npm install to pull down the node dependencies found in package.json.
  
• ChartJs.psm1
  
  This will setup some variables which will be referenced else where.
  
• New-ChartJsUtilizationGraph.ps1
  
  Obviously, this is the one that actually generates the graph. It expects data to be either CPU or Memory utilization data; anything else will probably cause an issue.
  
  In order to make a useful graph, we want as many data points from VMware as possible to display (see previous post). Each data points X value will have a datestamp associated with it. Unfortunately, that can make the X axis display become very dense / unreadable as the multiple timestamps display on top of each other (or they don’t display at all). To have better control over this, lines 41-52 ensure that there are only 5 datestamps displayed: first in time, 1/4 of the way through, 1/2 way, 3/4 of the way, and the most recent timestamp.
  
  The function works by generating the javascript which can actually run in nodejs. This script is placed in a temporary folder and the node_modules needed to use it are copied into that same folder (for local reference/usage).
  
  At the end of the script, it should clean up the temporary folder and files, but it won’t do that if an error occurs. I wanted to be able to debug errors, and if the temporary was always cleaned up … well, you get it.
  
• GraphUtilizationTemplate.js
  
  This is a copy and paste of an example file I found on stackoverflow. Using the documentation, it didn’t take too long to change things to what I needed; but it also wasn’t straight forward. You’ll probably need to do some experimentation to find what you need.
  
  An important thing to note, xAxes > ticket > autoSkip: false is really important to make the labels on the x axis appear correctly.

Next up: Creating Charts/Graphs with Powershell for Slack–Pt3 (Sending an Image to Slack).

Creating Charts/Graphs with Powershell for Slack–Pt1

This adventure came about from troubleshooting a problem with coworkers through slack. While we had hubot commands available to us to look at CPU and Memory usage for an given point in time, we would go to vSphere’s UI to screen grab Memory usage over time. It became quickly apparent that we needed a command to just generate these charts as we needed the in chat. So, our use case was:

Using historical VMWare/vSphere data generate a graph of CPU or Memory usage over time and upload it to slack using a hubot command.

This is actually way harder than it sounds because of one reason. Software that generates charts & graphs is pretty complicated. So, breaking down the use case into smaller sub-problems this is what it look like:

• Build a Hubot (nodejs) command that will kick off the process.call into Powershell to execute the operations necessary.
• Retrieve historical CPU/Memory data about a machine from our VMWare/vSphere infrastructure using PowerCLI.
• Generate a graph/chart from the data using any graphing technology that will actually run (a) outside of a browser and (b) in our environment. This is much more difficult than it sounds. (A Future Post)
• Send the graph back into slack so that it can be displayed in the channel where it’s needed. (Yet Another Future Post)

Build a Hubot Command

We currently run a hubot (nodejs) instance to handle ChatOps commands through Slack. Even though hubot is build on top of nodejs, we are a Microsoft shop and the majority of knowledge with scripting is in Powershell. So, long ago, when we implemented hubot we used PoshHubot to bridge the technology gap and allow the nodejs platform call our Powershell modules.

If we had to do it over again, with the current technology that available today, we would probably use Poshbot instead.

Anyways, we’ve been doing this for a long time, so this part of things wasn’t too new or difficult.

Retrieve Historical CPU/Memory Data

Within the debugging procedures that started all of this, we were pulling in graphs using screen grabs of vSphere. So, the best place to get data from would be vSphere and VMWare does a great job of making that information available through PowerCLI.

To do this, we used the Get-Stat command with either the cpu.usage.average or mem.usage.average stat being retrieved.

Quick note: I’m having a difficult time getting Connect-VIServer to work with a PSCredentials object. I’m not sure what the issue is, but for now the authentication process to the server is working because Connect-VIServer allows you to store credentials on a machine and reuse them. That was pretty nice of them.

The Get-Stat data is somewhat wonky at times. In the “1h” timeframe I’m using an IntervalSecs of 20 simply because that’s the only interval allowed. Each data point is always 20 seconds apart at 00, 20, and 40 seconds. If you use a Start and Finish range of over a week along with an IntervalSec amount, you could wait a real long time to get your data back; but you will get back all the data.

Because of all the data that will come back if you use an Interval amount, when you start to get a time range longer than a few hours it’s best to just let the Get-Stat command figure out what’s the appropriate interval amount to send back. That’s why on the “1d”, “1w”, “1m”, and “1y” timeframes I just use the Start and Finish parameters without an interval.

Both the cpu.usage.average and mem.usage.average data points return a percentage value back. This is fine for the CPU data, because we normally think about CPU usage in percentages. But, for memory, we normally think of its usage in GBs. So, there’s a quick section which converts the Memory usage percentage over to the actual amount of GBs used.

Next time, I’ll dig into New-ChartJsUtilizationGraph.