Looks like AWS is under attack

turbin3

turbin3

BuSo Pro
Joined
Oct 9, 2014
Messages
613
Likes
1,285
Degree
3
Just a guess. Either that or a serious hardware/software failure. I'm guessing it's an attack, considering StackOverflow has been dealing with one since last night. Those of you using S3, your stuff is down right NOW. This highlights the importance of diversifying your backend, and not aggregating too much in any one place. Definitely worthwhile to consider having fallbacks in place.
 
Wow, sure looks like a possible attack/hardware thing. I thought about this very thing with one of my current projects. I'm using "minio" https://github.com/minio/minio as a simple CDN and pretty much anything S3 would be used for. It's compatible with S3 so setting up S3 mirrors is a non-issue.
 
Ya'll must be on the Amazon AWS's east coast regions. I've been all good on the west regions throughout the day, I've been actively testing and pushing data in and out of S3 all day.

Edit, looks like it's really jut east:
east.png
 
So if I visit a website and get the message
Error 503 Backend fetch failed
Is this the reason?
 
Yeah, I had an issue earlier with Gravity Forms, took a second to realize the problem. Caught it on the dev tools timeline.

I'd guess this is a continuation of the DDOS's where "they" shut down the east coast DNS nodes trying to block Wikileaks. It's almost the same pattern here but only AWS this time, where O'Keefe is hosting the CNNLeaks videos.

I hit them up to see if the outage affected CPM ads and Affiliate Tracking/Reporting... because you can never tell on the last / first day of the month as they crunch totals... their answer was "We don't know."

What a day.

EDIT: I think I have an answer for them:

asDVnrw.png
 
How a typo took down S3, the backbone of the internet
Not an attack but worse than that in my opinion ...

Let's see what we know:
-Human error when typing a command line while performing basic maintenance
-Their goal was to "debug" their billing system
-Their monitoring dashboard was not showing any signs of problems
-On top of that it took over 4 hours for the servers to come back online

If this is really what happened, hopefully it will raise red flags for all those businesses that are relying
on one provider ... "Never put all your eggs in the same basket, no matter what the basket looks like" :confused:
 
ryandiscord said:
What is the best way to protect yourself from something like this?
Click to expand...

It's interesting how huge a problem it was for service providers. I was lucky that our internal systems that depend on AWS didn't seem to be materially impacted but even having the ability to switch our system online on another cloud platform instantly wouldn't have protected us from so many services we use seemingly being wedded to AWS and having no plan for something like this. They would have still all gone down.
 
ryandiscord said:
What is the best way to protect yourself from something like this?
Click to expand...

Have several backups in other locations, on multiple hosts. For example, Linode, DigitalOcean, Vultr, hell have S3 buckets in ALL regions (they're cheap!). Have a reverse proxy server (NGINX works well for this) that's setup to direct things where to go. If necessary, when things go down, make some nginx.conf tweaks and set NGINX to proxy to your alternate location. That's a really simple and manual explanation. There's a lot of ways people automate this, and in fact a lot of existing services many hosts offer services to handle these sorts of functions.

You can simply code fallbacks, in order of priority, into the nginx.conf and largely handle that. The simple explanation is, as cheap as VPS' are these days, have a few on the side. When you're pushing/deploying new stuff, copy to your alternates. Have one running as your primary, functioning as the reverse proxy to route things where they need to go.
 
You must log in or register to reply here.
Back