AWS API Key Exposed in github

AWS doesn’t want you to accidentally expose your API Key through github. And as a courtesy, they scan the creation of all new repositories in github for AWS Keys. Supposedly it’s done with this tool: truggleHog. And I couldn’t be happier that they do this! Not only for their own sake, but also the piece of mind of their customers.

Some months back I was asked to update some secrets because of exactly that scenario. A programmer had accidently checked in their AWS Keys into a public github repository and AWS had emailed our account managers to report that:

1. The key had been detected in repository X …
2. And, it had been deactivated on AWS

That was fantastic. And they detected and shutdown the key in under an hour of the key being exposed on github.

Now comes the truly amazing part …

The reason AWS does this is because malicious parties are also scanning creation of all new repositories in github and they are also looking for the same keys. And, a malicious group had found our keys before AWS deactivated them.

The automated attacker used the keys to spin up instances of x-large EC2 instances with plenty of GPU/CPU power and SSD drives. We assume the instances they created where then fitted with crypto-mining software and they went to town.

After receiving AWS’ email about the keys being disabled, our AWS contact point looked through our account and found the illegitimate EC2 instances and started killing them. So, cased closed right? Nope …

Our AWS manager shut down the EC2 instances that were in our primary region, in US-WEST-2. But, we all forgot to check the other regions. And the attackers had spun up identical stacks in all regions of AWS. The next morning we awoke to the realization that it was probably running in other regions and built a script to shut them down as quickly as possible.

All told, the 12 hour period that the EC2 instances were running ended up being over $5000 in charges.

But, AWS to the rescue again. Because it was the first time this had happened, AWS forgave the bill with two conditions:

• If it happens again, we pay for it.
• We needed to setup Billing Alarms in case any of our services (legitimate or not) starts to create charges that we are not comfortable with.

All-in-all, AWS is really trying to help out their customers; and I kind of want to give them a big hug.

Let’s Encrypt, IIS Central Cert Store and Powershell

Let’s Encrypt is a pretty popular tool with a mission to generate free SSL certificates in order to create a more secure internet. The goal is to ensure that the price of SSL certificates does not stand in the way of using them. Unfortunately, when you don’t charge for a product you really have to cut down on the amount of money you spend on customer service.

Their website is a model for limited user interaction. They provide documentation, help guides, and then they point you away from their site and towards the sites of many supporting tool providers which implement their SSL generation platform. But, you will be hard pressed to find a “Contact Us” or “User Support Forum” area on letsencrypt.org. To summarize their site: Here’s how it works, here’s the client providers, read the client providers documentation please.

I don’t fully understand the ACME protocol, but to me it reads like a strict Process and API for validating requests and provisioning signed certificates. Normally there might be a handy website that will guide you through this process with step-by-step instructions but, because there are so many different types of computer systems and programming languages that can implement the ACME protocol, they leave those guides up to the implementers of the ACME clients for each of those systems.

My preference is Powershell, and I found the Posh-ACME guide gave me a good start, but didn’t help me through the final steps of installing the certificate for use with IIS. In this case, an IIS Centralized Certificate Store. So, hopefully this can help others with a start to finish script showing the end users process; instead of hunting down individual steps from different sites.

Invoke-WebServiceProxy and Ignore Errors

One of the frustrating parts of using Invoke-WebRequest, Invoke-Rest, and Invoke-WebServiceProxy is that when they “throw” errors, they don’t actually throw. Many of the original powershell functions don’t throw errors, instead they use Write-Error and and return control from the function. This is really strange functionality for anyone coming for C#, javascript, or other 3GL languages.

You can suppress the error message by using the [CmdletBinding] parameter –ErrorAction “SilentlyContinue”. However, there are two problems with this. When using ‘SilentlyContinue’, the error message is still written to the $global:Error collection. And, these functions don’t always implement the functionality the same way. For example, Invoke-WebRequest doesn’t care what the error action is, it’s still going to write the error to the screen and it’s going to update the $global:Error collection.

The output from these two examples is kind of hard to see, because the Write-Host from within the finally block writes to the screen before the error message from Invoke-WebRequest. But, when $global:Error.Clear() is run within the finally block it somehow only affects a scoped instance of $global:Error. Which is completely counterintuitive to the idea of ‘$global’.

Test-ClearInsideOfFinallyBlock.ps1:

Test-ClearOutsideOfFinallyBlock.ps1:

But, the way Invoke-WebServiceProxy was written, it does respect the –ErrorAction parameter. And it was seemingly designed to work like this:

• -ErrorAction Continue
  
  Writes an error message to screen. Updates $global:Error. And returns nothing.
  
• -ErrorAction SilentlyIgnore
  
  Does not write an error message to screen. Updates $global:Error. And returns nothing.
  
• -ErrorAction Ignore
  
  Does not write an error message to screen. Does not update $global:Error. And returns nothing.

This implementation makes sense when you understand it. But, because Invoke-WebRequest and Invoke-WebService behave differently you would never know it.

MFA Tokens–AWS Hardware or Google Authenticator

As security goes, MFA One Time Passwords are surprisingly simple. I think that’s one of the great things about them. MFA OTP are all based around a shared a secret. In general it’s a generated string that’s not too terribly long; which makes it easy to store and not too bad to type out every once in a while. That generated string has some mathematical properties that allow it to be combined with a timestamp to create a six digit code that changes every X number of seconds (generally 30 seconds). That’s it. It’s just a shared secret.

So, to use an MFA TOTP all you really need to do is share the secret between the service provider and service consumer. And that’s the comparison I want to make: Is there a difference between how you share the secret when using a hardware token vs Google Authenticator. And, I want to use AWS as the service provider.

Here is AWS’ page on the variety of MFA scenarios that they support, complete with links to purchase suggested hardware devices. It’s a pretty great starting point for anyone.

And, here’s my quick comparison:

Hardware MFA Device

If you go with a Hardware MFA device (for example the gemalto Safenet Display Card), and you start to setup the card in AWS’ IAM user account configuration, you’ll eventually run into this screen:

With that card, the Serial Number printed on the back of the card is the Shared Secret. The security is that you lock the card away and keep it safe; because if you can look at the back of the card, you can get the shared secret.

And, the way the secret is shared is that you send the Shared Secret to AWS over an https connection.

Google Authenticator

If you go with the Virtual MFA device (ie. Google Authenticator), and you start to setup the virtual device in AWS’ IAM user account configuration, you’ll eventually run into this screen:

The QR Code is kind of the classic way of getting the info into your phone. (And, I would suggest screen shoting the QR Code and the “Show secret key” value and storing the image into a password safe. The QR code contains a little extra info that labels the TOTP code in Google Authenticator. Also, it’s kind of annoying to buy a new phone and hand enter all the codes again.)

So, with the virtual device, AWS is generating the Shared Secret. And they are providing the secret to you by sending it to your browser over https. You are then expected to setup a lock on your phone to keep your device safe and secure.

Comparing the two

So is there really a difference? The transmission of the Shared Secret is still over https in both cases. That’s the moment where there is most likely going to be something/someone that could intercept the information.

Once the secret is shared, the hardware device will probably be the more at risk device to reveal the secret. Both the card and the cell phone can be stolen from your pocket or purse. Except, once stolen, the burglar can just read the shared secret off the back of the hardware card; where unlocking an iPhone is kind of a nightmare.