Good points. Keep in mind the OP is talking about a 14x multiple of annual run rate, rather than trailing 12. He's calculating annual run rate as 12x his most recent month. Considering it's a quickly growing company, if he were calculating his multiple as a multiple of trailing 12, it would be higher than 14.
Rob, best of luck to you and the Twilio team while fixing and recovering from this incident.
We've been with Twilio for years and have received excellent service. Billing issues are always serious, but its nothing compared to a call routing issue, and when it comes to call routing Twilio has been flawless. I'd much rather call my bank and discuss an overdraft fee then call a client and explain why their calls were not routed properly.
In the years we've been with Twilio, we have not experienced any significant downtime. Compare that to one of their main competitors that experienced eight consecutive hours of downtime during business hours earlier this year, and I'm confident we're in good hands.
Seconding Willie's comments. I'm not your customer today, but seeing how y'all are handling this would make me feel confident to use Twilio should the need arise.
When you get some time (I'm sure you have none right now), read some of the Dreamhost discussion below. You folks appear to be doing an excellent job responding to this.
I completely agree. I've seen several public instances like this one over the past year here where Rob and the Twilio team have done what appears to be the Right Thing (from a customer service perspective). It's inspiring to see them do this, and to see that it is indeed possible to execute very well with a focus on customer happiness and "making things right".
I see your point about growth, but I disagree over the term bootstrap. I interpret bootstrap to be about external investment. Its possible to have a high-growth company (a "startup"), that receives no investment, or receives investment by the founders.
I second the vote for Blue Box. We moved from Heroku 8 months ago and have been extremely happy.
I agree with grrrando, there were a few instances of slow responses during setup, but I also agree thats because you're talking to an individual in that case, rather than a support team.
Two of my favorite things about Blue Box:
1) On the first sales call they had a knowledgeable engineer that discussed the specifics of our app.
2) With their online chat I get answers from an engineer within minutes, even on nights or weekends (they have phone support too, I prefer chat)
I use one daily and really like it, although its made of cheap plastic. The $99 list price is pushing it, you can probably find one on ebay for ~$30. I got mine for free at the Humanscale showroom when I went to look at their chairs.
The conclusion is that it will buy you a bit more time, but does not fix the underlying issue.
In Rap Genius' case, they're large enough that they would still have significant issues even if they switched to cedar and unicorn with 2-4 worker processes.
> they would still have significant issues even if they switched to cedar and unicorn with 2-4 worker processes.
Yeah, but how about config.threadsafe! with puma or thin in multi-thread mode?
It is odd to me that very little of this conversation on the nets recognizes that Rails _does_ support multi-threaded concurrent request handler, with the right app server stack (figuring out the right app server stack can be non-trivial, although it's getting better).
Agreed. I am working with a client who is slightly lower on the Heroku customer food chain and using Unicorn with four workers right now. Our next step is Puma though that will likely not be the end point.
The right vs wrong issue here is not the proper way to architect a router. The issue is that Heroku glossed over an extremely important aspect of their engineering documentation, because it painted their platform in a bad light. This is particularly damning, because as an engineer working on their platform, I could design around their shortcomings as long as they don't hide them from me.
Furthermore, I believe we could make an argument Heroku intentionally misled (both in documentation and in their support responses) clients as to how their router worked.
The reason I want to stay out of that discussion is that it often just amounts to some mud-slinging from one side upon the other. Intentionally mislead is quite an accusation and I don't think Heroku would indulge in it.
Personally, I think it is incredibly naive to build an application around a framework where you have no built-in concurrency. The main reason is that the queue you will build up in front of it is outside your reach so you have to sustain it.
It is also naive to think that your cooked up statistical model resembles reality in any way. Routing is a hard problem so it is entirely plausible that your model does not hold up in reality. Besides, the time it takes to construct those R models is a missed opportunity for improving the backend you have. And the R models doesn't say a lot, sorry. At best they just stir up the storm --- and boy did they succeed.
I agree it is unfortunate that Heroku's documentation isn't better and that New Relic doesn't provide the accurate latency statistics. But to claim that this is entirely Heroku's fault is, frankly, naive as well.
It is quite an accusation, but I'm far from a bystander in this issue. I have extensive, documented communication with Heroku engineers over the course of 1.5 years (Feb 2011 - June 2012).
I'm not discussing any sort of cooked up statistical models. I'm discussing the real-world experience I had scaling an application on Heroku.
"mud-slinging from one side upon the other"
You imply that I was uninvolved before Rap Genius's expose. I assure you that is not the case. I've chosen a side in this argument well before Rap Genius went public.
The "you" were addressed to RG, not your experience. I completely agree that scaling is one of the harder problems in computer science. Most people just run a stateless system and hope for the best, but this is a delicate matter and it is a hard problem.
The hard part being a customer or Heroku is that the problem might be on the other side of the fence. And how do you communicate that in a diplomatic way?
Personally my opinion is something along the lines of "If you use Ruby +rails in that configuration, then you deserve the problem".
I had a string of similar requests with Heroku between Feb 2011 and June 2012, before we migrated off their platform.
I would complain about h12 errors, they would tell me to upgrade my resources and/or that it was my problem and there was nothing they can do. We ended up with a solution that was easily 10x as expensive (over-powered DB, too many dynos) as our initial configuration, and it still didn't fix the issue.
I'm happy to provide the full text support requests, but they don't tend to be quite as juicy as the one you posted.
We moved our Twilio app off Heroku for the same reasons. Extensive optimizations and we would still get timeouts on Twilio callbacks.
The routing dynamics should be explained better in Heroku's documentation. From an engineering perspective, they're a very important piece of information to understand.
