Several people have asked for additional details. We just posted a quick follow-on:
[UPDATE] A central theme of the recent AWS issues has been the Amazon Elastic Block Storage (EBS) service. We use EBS at Twilio but only for non-critical and non-latency sensitive tasks. We've been a slow adopter of EBS for core parts of our persistence infrastructure because it doesn't satisfy the "unit-of-failure is a single host principle." If EBS were to experience a problem, all dependent service could also experience failures. Instead, we've focuses on utilizing the ephemeral disks present on each EC2 host for persistence. If an ephemeral disk fails, that failure is scoped to that host. We are planning a follow-on post describing how we doing RAID0 stripping across ephemeral disks to improve I/O performance.
Ironically, this highlights one of the main issues we discuss in the post!
The Twilio Engineering blog is hosted off an external Wordpress site with a single IP that's forwarded from ngnix load balancer pool. Since the load balancers assume that the external service can fail, they won't tied resources blocking access to other parts of the site.
We just enabled caching on the ngnix proxy to the external site hosting our Wordpress install for the engineering blog. Hopefully that should help performance.
Hi Anthony, Twilio service status available via our public status page http://status.twilio.com/ We communicated degraded international SMS service to customers on August 16.
You bring up a good point that the information might not be readily discoverable. We'll work to make the status page more findable and to extend the API (http://status.twilio.com/documentation/rest) with features such as RSS to let customer subscribe up-to-the-minute status information.
When we launched the Twilio SMS Beta we tried hard to support sending SMS messages to both US and International destinations. When there were problems, we worked with our customers to collect forensic data on hundreds of carriers worldwide and pass it to our carriers partners to debug.
At Twilio we are dedicated to working with top quality carriers and technology. After months of working to fix problems, were not able to deliver the reliable International SMS service our customers have come to expect.
We apologize for any problems this has caused for our customer and we'll work to bring back International SMS service after were able to deliver on the quality we do the rest of Twilio services.
In a 100% ideal scenario an organization would only need developers as every possible failure/problem would be covered and handled by automation.
Obviously the real world is different. New code is deployed that has bugs, there are unpredicted events, there are complex failure scenarios that are difficult to automate etc.
A DevOps engineer may write automation software in conjunction with developers to automate the operational aspects of business logic. Thus, it's a partnership between the DevOps engineer whose metrics are driven by availability/reliability/scalability/security and the developer who is trying to attain some business objective.
Correct. Stashboard is simply a lightweight frontend display for your API/service status. It has a GUI and REST API that allow you to update status information. Using the API, one could wire Stashboard into Nagios or any other alerting system.
Props to Dave. All the flowery language, pictures, and hyperbole generate great discussion and traffic.
Dave is one of the most savvy investors and marketers around. When we were first pitching Twilio no one seemed to understand "marketing to developers" but he got it while half asleep from an all-nighter.
If you curious about Dave's geek cred, scope his linkedin profile:
* Investment Program Director at fbFund Incubator
* Investor at Bit.ly
* Investor at DailyBurn
* Advisor at Life360
* Startup Investor (FF Angel Fund, fbFund Incubator) at Founders Fund
* Advisor, Investor at Twilio
* Organizer & Founder at Startup2Startup
* Advisor at 750 Industries
* Advisor, Angel Investor at SlideShare
* Advisor, Angel Investor at TradeVibes
* Advisor, Angel Investor at EventVue
* Advisor, Angel Investor at Oortle
* Angel Investor at RichRelevance
* Advisor, Angel Investor at KISSmetrics
* Advisor, Angel Investor, Boardmember at TeachStreet.com
* Advisor, Consultant, Angel Investor at Mint Software
* Advisor, Angel Investor at Mashery
* Advisor at Kiva.org
* Co-Founder, Boardmember at Silicon Valley Microfinance Network
* Advisor, Angel Investor at Simply Hired
* Angel Investor at Canopy Financial
* Startup Advisor / Angel Investor at 500 Hats LLC
maybe i am too stupid but bookmarks are just too complicated for me... they are always on the wrong computer, organizing them is a pain, and i just don't trust them to a 3rd party to keep them. google always seems easier :)