2017-02-28
Amazon Outage Causing Failures (Resolved)
Recent Posts
- What's New For July 2024
- Automate Management of Files.com with Terraform
- What's New for June 2024
- Still using SFTP with Boomi? There's Something Better!
- What's New for May 2024
- What's New For April 2024
- What's New for Q1 2024
- Automation Configuration Changes may be Required
- A New Look for Files.com
- Permission Fence - A New Way To Manage Permissions
Update at 5:05 PM ET: As promised, an "all clear" update regarding this Amazon S3 downtime event. Amazon has been hard at work at a fix for their S3 problems and although they have not yet announced a fix, our internal monitoring shows our customers having success with all operation types.
Feel free to connect to BrickFTP to perform any operation at this point.
We are still slightly throttling connections on a per-IP basis while the system recovers, but these limits should not affect most normal users and we intend to remove them all entirely as soon as the Amazon S3 system can handle the volume.
With the engineering challenges behind us, we will be "all hands on deck" responding to emails and voicemails so if you had sent us a message, you should expect a response soon.
We will send out a post-mortem to our full list later this week after we've had some time to discuss internally the lessons learned by this event. Hopefully Amazon will complete an incident report as well and we will share all the relevant details with you.
Back to work!
___________________________________________________________________________________________________
Update at 4:12 PM ET: Amazon has been working hard to resolve this issue. At this time, all operations except for uploads are now succeeding reliably. That includes download, list, and delete.
Uploads are still mostly failing at this time and Amazon estimates an additional 1 hour ETA on a fix for uploads.
You are free to continue to try to upload during this time, but we are limiting FTP, SFTP, and WebDAV connections to 10 per IP address at this time so as to not further overload Amazon's systems.
We will let you know once uploads are operating reliably again.
___________________________________________________________________________________________________
The BrickFTP team is aware of an Amazon outage that is causing failures for our BrickFTP customers. As you probably know, BrickFTP currently stores hundreds of terabytes of data, (i.e. hundreds and hundreds of physical disk drives worth) and in order to do that we use Amazon.com as our backend storage vendor.
Amazon is the largest vendor of cloud services in the world, and thousands of popular websites are offline during this outage. They claim to be aware of the issue and to be working on it. Here's a post from Amazon:
We've identified the issue as high error rates with S3 in US-EAST-1, which is also impacting applications and services dependent on S3. We are actively working on remediating the issue.
Unfortunately, we don't have an ETA I can share at this time, however, our full staff is in and working on this. We will mail the full list again once we are operational.
We are doing everything we can, however our system is ultimately built on top of Amazon S3 and there isn't much we can do until they're up.
Questions? Need help?
Please let us know how we can assist you. We’re here and glad to help. Please contact us by email, chat (in your web interface when logged in), or phone (1-800-286-8372 ext. 2).
Get Instant Access to Files.com
The button below will take you to our Free Trial signup page. Click on the white "Start My Free Trial" button, then fill out the short form on the next page. Your account will be activated instantly. You can dive in and start yourself or let us help. The choice is yours.
Start My Free Trial