Cloud Security and a $20000 AWS Charge

Cloud security is a hot topic right now. In the current state of the Software Revolution, pretty much everything in everyday life is moving to the cloud: movies (Netflix & Hulu), business communications (Google Hangouts & Slack), driverless cars (Google & Tesla), storage (DropBox & OneDrive), cloud robotics (Google & Industrial Perception) to name a few of the thousands of cloud-based technologies available today. There are so many smart-* apps and devices coming out that all of our daily computational power is moving to the cloud.

This is awesome. This allows software developers and engineers like me to work from any location as long as I have an internet connection. In fact, my current job (shameless plug for Yappee) allows me to work remotely because all of our technologies are based on the cloud - GitHub, Amazon AWS, and Slack are a developer's best friend.

Under the carpet, all of these applications have critical security features and complex API's which are crucial to the success of these technologies. On the front-end side of things, the users are presented with elegant interfaces with a ton of power at their fingertips. This is the future and it's awesome.

This magical ride with cloud computing is only going to get bigger and faster according to Moore's Law. This 2^* exponential growth is going to allow us to do things we never thought possible with software, but STOP RIGHT THERE...

We are becoming so enveloped within this digital world that we are failing to recognize the consequences that may occur from cloud security.

Two days ago, I was playing around with my GitHub repo and to make room for another private repository, I switched one of my previous Ionic projects to a public, open-source project because what the heck someone might find it useful. That someone wasn't who I expected.

I received an email yesterday morning, with the title Your AWS account is compromised. Wow what's going on here? I read into the email and saw that my GitHub project was listed with my AWS security credentials listed through my elastic beanstalk configuration. I took immediate action to change all my passwords and keys associated with this account as well as deleted my github repo from the cloud. It didn't take long until I received another email regarding a support case I didn't even open myself.

Thank you for taking quick action to delete your exposed access key. The hold on your account has been lifted.

However, prior to you deleting your access key your AWS account has been compromised and currently there's a charge of $13510.95 USD. To prevent further charges as well as for me to submit a concession request, please go through and delete any unauthorized resources.

Holy shit!!!! $13510.95 USD. My account was compromised... as fuck. These GitHub bots have no mercy. Overnight, they were able to acquire my AWS credentials through their scraper and launch up 500 SpotInstances as well as 500 more c3.8xlarge EC2 instances in all regions of the world - all using my billing account. Absolutely nuts.

I contacted Amazon AWS support immediately with my pressing issue. One of their employee's, Ben, let me know about all of "suspicious" activity that my account had undergone. Ben is my hero. He got my attention immediately through a phone call and voicemail, and immediately submitted a concession request on my behalf, assuring me that all of my expenses would be covered by Amazon. Amazon and their customer support receive my highest regards. 10/10.

Screen Shot 2015-10-29 at 3.38.57 PM

This is my billing account by the morning. Within 1 day of my GitHub repo going public, the bot was able to launch off 500 of the largest EC2 instances available on Amazon with spot requests to continually relaunch them if they went down, accruing over $20000 of computing costs. This is insane (also impressive from a hacker standpoint).

Admittedly, I am 100% responsible for leaking my credentials to a public zone, but trust me, if it can happen to me, it can happen to anyone. There are bots these days for everything. Keep your shit safeguarded.

tldr: Cloud Security is important. Keep your passwords safe.