AWS API Key Exposed in github

AWS doesn’t want you to accidentally expose your API Key through github. And as a courtesy, they scan the creation of all new repositories in github for AWS Keys. Supposedly it’s done with this tool: truggleHog. And I couldn’t be happier that they do this! Not only for their own sake, but also the piece of mind of their customers.

Some months back I was asked to update some secrets because of exactly that scenario. A programmer had accidently checked in their AWS Keys into a public github repository and AWS had emailed our account managers to report that:

1. The key had been detected in repository X …
2. And, it had been deactivated on AWS

That was fantastic. And they detected and shutdown the key in under an hour of the key being exposed on github.

Now comes the truly amazing part …

The reason AWS does this is because malicious parties are also scanning creation of all new repositories in github and they are also looking for the same keys. And, a malicious group had found our keys before AWS deactivated them.

The automated attacker used the keys to spin up instances of x-large EC2 instances with plenty of GPU/CPU power and SSD drives. We assume the instances they created where then fitted with crypto-mining software and they went to town.

After receiving AWS’ email about the keys being disabled, our AWS contact point looked through our account and found the illegitimate EC2 instances and started killing them. So, cased closed right? Nope …

Our AWS manager shut down the EC2 instances that were in our primary region, in US-WEST-2. But, we all forgot to check the other regions. And the attackers had spun up identical stacks in all regions of AWS. The next morning we awoke to the realization that it was probably running in other regions and built a script to shut them down as quickly as possible.

All told, the 12 hour period that the EC2 instances were running ended up being over $5000 in charges.

But, AWS to the rescue again. Because it was the first time this had happened, AWS forgave the bill with two conditions:

• If it happens again, we pay for it.
• We needed to setup Billing Alarms in case any of our services (legitimate or not) starts to create charges that we are not comfortable with.

All-in-all, AWS is really trying to help out their customers; and I kind of want to give them a big hug.