Twilio blocked our account and with it hundreds of our customers [resolved]
164 comments
I work for Twilio and forwarded it internally. If you are 6 years with the company, you probably have an AE assigned to your company or have been in touch with any other human being along the way?
UPDATE: Your account should be active now.
UPDATE: Your account should be active now.
Yes, it's active now after almost 7 hours. Thank you!
I plan to update everyone on what happened as soon as I get some sleep first. I've been up since 7 am and it's 2 am now. The incident started at 7 pm.
I plan to update everyone on what happened as soon as I get some sleep first. I've been up since 7 am and it's 2 am now. The incident started at 7 pm.
and setup another system to back this one up immediately because whatever algorithm brought down your account will do so again! If you spread the load across two platforms it would be a lot of work but you would be half-alive instead all-dead.
We're still interested in an update!
Try to find an alternative service supplier at very minimum for back up and consider legal recourse.
It's a bit shit that they had to jump through so many hoops, with HN being the only one that actually worked (after 7 hours) in order to get in touch. Serious kudos to you for helping them but I know a lot of people who wouldn't touch your company with a bargepole for these kind of reasons. Having to kick up a fuss on social media in order to get service, how very 2021..
> Having to kick up a fuss on social media in order to get service, how very 2021..
Exactly. I'm locked out of a google account I have which links to a yt channel and a twitter account. The only truly viable option for me to get these back is work my way up the social media infulencing ladder and then complain how I lost access to my accounts. This is insane.
Exactly. I'm locked out of a google account I have which links to a yt channel and a twitter account. The only truly viable option for me to get these back is work my way up the social media infulencing ladder and then complain how I lost access to my accounts. This is insane.
Google is a different beast entirely. Complaining on social media doesn't generally work.
Not really true. If you manage to get a "Google locked me out for no reason" post to hover on the frontpage of HN for a couple of hours, you most likely will get help from some Google insider. There are been plenty of such posts on HN where the issue has been resolved by what I assume to be HN-googlers.
It is true. I've seen multiple posts where Googlers or ex-Googlers with contacts can't get access to a locked Google account.
I had a similar issue as you, what ended up working for me was to contact the support for payment issues, and saying I had issues with paying for something.
Got in contact with a real human (i think..) in chat in under a minute, which helped me get it resolved via filling out like 4 different forms.
No idea if this would work for you though, but worth a try?
Got in contact with a real human (i think..) in chat in under a minute, which helped me get it resolved via filling out like 4 different forms.
No idea if this would work for you though, but worth a try?
I've had to use this kinda method for other things. No way to get a hold of a real person for support, try their new customer contact details. you can get they answer those real quick!
I had an issue with a propane company contractor failing to secure permits before installing a line, which caused an expensive backyard project to grind to a halt, while still incurring cost. Every call to support was met with very friendly folks who promised a call back same day.
That never, ever happened and I had to resort to a nicely worded note to a corporate VP on LinkedIn to resolve it. Even then, I'm pretty sure that "unpermitted gas line work" was the real driver in getting it resolved, since that's a big deal.
That never, ever happened and I had to resort to a nicely worded note to a corporate VP on LinkedIn to resolve it. Even then, I'm pretty sure that "unpermitted gas line work" was the real driver in getting it resolved, since that's a big deal.
It makes sense.
If the problem really gets kicked up in the media and left unaddressed, then eventually politicians will be forced to add some real regulation to force companies to do right by their customers.
That'd be expensive. So instead, ignore customers who can't kick up social attention and quickly resolve those that can before it gets legs. No one cares about the unknown, disconnected hordes of betrayed customers who lost their access for unknown reasons.
And I'm sure no company is covertly doing this to their tiny, would be competitors, killing the baby in the crib so to speak.
In fact, if you're an employee that actually cares about your customers, you'd do well to not internally raise these issues. Let the PR hit the executives harder and make them squirm.
If the problem really gets kicked up in the media and left unaddressed, then eventually politicians will be forced to add some real regulation to force companies to do right by their customers.
That'd be expensive. So instead, ignore customers who can't kick up social attention and quickly resolve those that can before it gets legs. No one cares about the unknown, disconnected hordes of betrayed customers who lost their access for unknown reasons.
And I'm sure no company is covertly doing this to their tiny, would be competitors, killing the baby in the crib so to speak.
In fact, if you're an employee that actually cares about your customers, you'd do well to not internally raise these issues. Let the PR hit the executives harder and make them squirm.
> Having to kick up a fuss on social media in order to get service, how very 2021..
I'm kinda sad that the thread is about Twilio and not Google - my GMail account is blocked and I was hoping I could jump on the bandwagon.
Someone here should make a startup out of it.
I'm kinda sad that the thread is about Twilio and not Google - my GMail account is blocked and I was hoping I could jump on the bandwagon.
Someone here should make a startup out of it.
same. I had a issue with google payment. It was around $5. Emailed them 100 times even phoned them for hours. They didn't solved the issue .
After I shamed them in twitter/github the issue got solved in 1 day.
The world is has become too wild tbh.
After I shamed them in twitter/github the issue got solved in 1 day.
The world is has become too wild tbh.
Please bring this up with someone that cares in your company: you ABSOLUTELY NEED a well working support organization with EFFICIENT escalation procedures.
Support cannot be an afterthought. If a long time customer has a critical incident like this, it needs to be caught and forwarded to the relevant people within a few hours at most. This type of PR fallout is unacceptable and completely avoidable. I, for one, will avoid Twilio just based on this story alone.
Support cannot be an afterthought. If a long time customer has a critical incident like this, it needs to be caught and forwarded to the relevant people within a few hours at most. This type of PR fallout is unacceptable and completely avoidable. I, for one, will avoid Twilio just based on this story alone.
This thread is making my company seriously reconsider using Twilio. Please make sure Twilio’s leadership is aware of how damaging these kinds of situations are to your customers, and how much other customers pay attention to this kind of customer mistreatment.
What is preventing this from happening to someone else?
Why was it deactivated?
"I work for Twilio and forwarded it internally."
For the love of christ will you please give us an 'email' verb in twiml ?
Or do you need sendgrid signups for some business metric and, therefore, this dead simple integration has to be avoided ?
For the love of christ will you please give us an 'email' verb in twiml ?
Or do you need sendgrid signups for some business metric and, therefore, this dead simple integration has to be avoided ?
The customer should sue Twillio into oblivion, or at least try to.
Can you imagine running a business and having a critical aspect of it just entirely stopped without warning or recourse?
People could lose their jobs during a pandemic due to ineptitude and systematic arrogance?
If you factor these kinds of risks into business equations, it does not look good.
Depending on Tech is starting to feel like US Health Insurance: you're 'covered' until you're 'not' in which case you're going bankrupt.
Can you imagine running a business and having a critical aspect of it just entirely stopped without warning or recourse?
People could lose their jobs during a pandemic due to ineptitude and systematic arrogance?
If you factor these kinds of risks into business equations, it does not look good.
Depending on Tech is starting to feel like US Health Insurance: you're 'covered' until you're 'not' in which case you're going bankrupt.
>Can you imagine running a business and having a critical aspect of it just entirely stopped without warning or recourse?
I understand what you're saying, but OP cannot alleviate their responsibility in this. You can't outsource your critical infrastructure to a third-party SAAS vendor and not have a plan for when things go tits up. OP didn't even know who to call. That's on them because they should have identified this as a risk long time ago.
I understand what you're saying, but OP cannot alleviate their responsibility in this. You can't outsource your critical infrastructure to a third-party SAAS vendor and not have a plan for when things go tits up. OP didn't even know who to call. That's on them because they should have identified this as a risk long time ago.
If there was a reasonable recourse for the customer to take, and they were derelict, then obviously this is the case.
But it doesn't so much seem like this is the case, having tried to reach out through several channels.
There are two issues that stand out:
1) The power asymmetry - some large SaaS vendors provide critical infrastructure to customers - like financial service providers to small business. There are tons of regulations in that industry because that.
2) The lack of recourse as standard practice in new high tech. If your bank magically stops you from collecting VISA at your corner store - there's generally a human you can speak to, pretty much right away. You may be put on hold. I get relatively immediate customer service from my bank as a tiny retail customer. The lack of provisions here by the industry is a systematic problem.
But it doesn't so much seem like this is the case, having tried to reach out through several channels.
There are two issues that stand out:
1) The power asymmetry - some large SaaS vendors provide critical infrastructure to customers - like financial service providers to small business. There are tons of regulations in that industry because that.
2) The lack of recourse as standard practice in new high tech. If your bank magically stops you from collecting VISA at your corner store - there's generally a human you can speak to, pretty much right away. You may be put on hold. I get relatively immediate customer service from my bank as a tiny retail customer. The lack of provisions here by the industry is a systematic problem.
>But it doesn't so much seem like this is the case, having tried to reach out through several channels.
It was too late now. When things are on fire, you're in panic-mode now. The reality is that if Twilio had shitty support today, they had shitty support last week and 6 months ago (etc.). That should have been identified by OP as a risk to their business and rectified (either by forging a relationship with someone at Twilio so you can backchannel support request, or identifying your account manager and making sure they are responsive, or moving to another vendor with better support). Twilio is a critical supplier for OP's company. You can't just assume they care about your business as much as you care about your business.
>he power asymmetry - some large SaaS vendors provide critical infrastructure to customers
OK. So? There are lots of things you can plan for and not control. None of us are fully in control of our circumstances. We can't control how trillion-dollar companies behave. We can't control the weather. What you can control is your actions, your planning and ultimately your response to things you can't control.
For example, I peeked at OPs comment history and he mentioned that their support line was provided by them (they provide telephony services and therefore they dog-food their products) ... was that the right move though? Because if they are down (or Twilio is down), their customers will need to talk to someone and won't be able to reach them. Maybe it makes sense to have another provider handle their support line (or at least have a backup).
The point is that, sure, Twilio screwed up - fully agree with that. Maybe OP can recover some damages, and maybe not. Maybe it isn't worth chasing Twilio through courts for years and spending thousands of dollars. Regardless, ultimately it is OP that suffered the consequences of Twilio's screw up, so OP should prepare themselves for this in the future. And don't tell me that there's nothing OP could have done. That's bullshit.
It was too late now. When things are on fire, you're in panic-mode now. The reality is that if Twilio had shitty support today, they had shitty support last week and 6 months ago (etc.). That should have been identified by OP as a risk to their business and rectified (either by forging a relationship with someone at Twilio so you can backchannel support request, or identifying your account manager and making sure they are responsive, or moving to another vendor with better support). Twilio is a critical supplier for OP's company. You can't just assume they care about your business as much as you care about your business.
>he power asymmetry - some large SaaS vendors provide critical infrastructure to customers
OK. So? There are lots of things you can plan for and not control. None of us are fully in control of our circumstances. We can't control how trillion-dollar companies behave. We can't control the weather. What you can control is your actions, your planning and ultimately your response to things you can't control.
For example, I peeked at OPs comment history and he mentioned that their support line was provided by them (they provide telephony services and therefore they dog-food their products) ... was that the right move though? Because if they are down (or Twilio is down), their customers will need to talk to someone and won't be able to reach them. Maybe it makes sense to have another provider handle their support line (or at least have a backup).
The point is that, sure, Twilio screwed up - fully agree with that. Maybe OP can recover some damages, and maybe not. Maybe it isn't worth chasing Twilio through courts for years and spending thousands of dollars. Regardless, ultimately it is OP that suffered the consequences of Twilio's screw up, so OP should prepare themselves for this in the future. And don't tell me that there's nothing OP could have done. That's bullshit.
Sorry to hear. Twilio has changed since its early days. Last month, our bill went from $20 per month to $16,000. The reason: fraudsters use Twilio to setup premium numbers, then use Twilio customers (like us) to send OTPs to those numbers using bots. The result is us out of pocket, fraudsters and Twilio making money.
Twilio is also the only Pay as you Go SaaS vendor that doesn’t have spend limits on accounts and advises customers to implement their own any fraud measures themselves.
While this was going on, we got an email from their sales rep, congratulating us for increasing our spend by more than 8000% and wanted to have a chat. So I guess they do have the systems to monitor unusual patterns, they just turn a blind eye.
You’d never know, someone in Twilio might even be selling customer lists on the black market to the fraudsters so they know who to target.
Twilio is also the only Pay as you Go SaaS vendor that doesn’t have spend limits on accounts and advises customers to implement their own any fraud measures themselves.
While this was going on, we got an email from their sales rep, congratulating us for increasing our spend by more than 8000% and wanted to have a chat. So I guess they do have the systems to monitor unusual patterns, they just turn a blind eye.
You’d never know, someone in Twilio might even be selling customer lists on the black market to the fraudsters so they know who to target.
> While this was going on, we got an email from their sales rep, congratulating us for increasing our spend by more than 8000% and wanted to have a chat. So I guess they do have the systems to monitor unusual patterns, they just turn a blind eye.
At this point we're entering criminal territory. Enriching yourself when you should reasonably know something criminal is occurring (like someone being defrauded), and helping the perpetrator by providing your service anyways, can make you a criminal yourself very fast.
As far as judges will be concerned, you're now a perp yourself.
At this point we're entering criminal territory. Enriching yourself when you should reasonably know something criminal is occurring (like someone being defrauded), and helping the perpetrator by providing your service anyways, can make you a criminal yourself very fast.
As far as judges will be concerned, you're now a perp yourself.
In the United States I don’t think there are many jurisdictions where an automated marketing email will get anyone put on the most-wanted list.
I have a very hard time seeing the criminality of this. Can any lawyers provide more insight?
I have a very hard time seeing the criminality of this. Can any lawyers provide more insight?
You could try AWS SNS SMS. This is also PAYG. A wrapper like https://sendwithses.com might make your work easier.
EDIT: With AWS SNS SMS you can set a spending limit and your bill will never go over that.
EDIT: With AWS SNS SMS you can set a spending limit and your bill will never go over that.
I thought AWS SNS SMS was a wrapper around twilio? I recall a big song and dance on the Twilio side about them providing AWS with a service... Could be wrong.
AWS SNS SMS seems best suited for internal communications (we use it for alerts); it does not seem suitable for commercial use where you require targeted sends, you'd have to create a topic for each customer.
Source: use it and twilio.
Source: use it and twilio.
Clearly not.
Looks like we'd need https://docs.aws.amazon.com/pinpoint/latest/userguide/channe...
Which isn't bad, but is much more limited than what Twilio provides.
Looks like we'd need https://docs.aws.amazon.com/pinpoint/latest/userguide/channe...
Which isn't bad, but is much more limited than what Twilio provides.
FWIW: you can turn off automatic recharge and ask for a different reminder limit. Not ideal, but at least it prevents crazy bills.
They still bill you no matter what, and if the bill is big enough they will sic collections on you.
This surprises me. I thought they would disable the service when the balance goes negative. I need to add some more monitoring and a failsafe rate limit...
They also continue to bill for recurring services even if your account balance goes to zero, watch out.
That's painful for you, but a good thing to know as a community. Is it possible to prevent sending to premium numbers?
Yes, but it adds insult to injury. One of the suggestions by Twilio is to use their service to check the number and send only to mobile numbers. The issue? Twilio charges per call for that service! So you pay them to "possibly" stop you paying them more and fraudsters. The whole thing stinks
Is there some kind of regex for premium rate vs standard numbers?
In some countries.
For example, all UK mobile phone numbers begin 07 (i.e. +44 7), fixed/special rate 08, and premium rate 09.
But SMS also has "short codes", which is probably what's being abused here. 5 or 6 digit numbers, usually used for things like "Text 12345 to donate €2 to this charity" or "Send 'CXM' to 12345 and receive a message telling you when the next bus comes to this stop, costs 10¢".
https://en.wikipedia.org/wiki/Telephone_numbers_in_the_Unite...
https://en.wikipedia.org/wiki/Short_code
For example, all UK mobile phone numbers begin 07 (i.e. +44 7), fixed/special rate 08, and premium rate 09.
But SMS also has "short codes", which is probably what's being abused here. 5 or 6 digit numbers, usually used for things like "Text 12345 to donate €2 to this charity" or "Send 'CXM' to 12345 and receive a message telling you when the next bus comes to this stop, costs 10¢".
https://en.wikipedia.org/wiki/Telephone_numbers_in_the_Unite...
https://en.wikipedia.org/wiki/Short_code
Might libphonenumber be of some help?
How the the fraudster make money, not familiar with the concept of premium numbers. EDIT: Oh premium numbers charge the sender
The fraudsters setup premium numbers with some other telephony provider, possibly in another country. Then they use those numbers to receive 2FA (two factor auth) passwords (either SMS or read aloud in calls). Because the defrauded party's servers called those premium numbers through Twilio, the defrauded party has to pay Twilio.
AFAIK there's no easy way to always, securely detect premium numbers. The fraudster can setup a forward from a normal number to a premium one anyway. So checking number prefix is useless. One could listen for the "After the beep, this call will be billed at X c/min" recording that premium numbers in honest countries have. The fraudsters manage to find less honest premium number providers that skip these.
This is a well enough known issue that Twilio has a page about it: https://www.twilio.com/learn/voice-and-video/toll-fraud
AFAIK there's no easy way to always, securely detect premium numbers. The fraudster can setup a forward from a normal number to a premium one anyway. So checking number prefix is useless. One could listen for the "After the beep, this call will be billed at X c/min" recording that premium numbers in honest countries have. The fraudsters manage to find less honest premium number providers that skip these.
This is a well enough known issue that Twilio has a page about it: https://www.twilio.com/learn/voice-and-video/toll-fraud
The real fraud here seems to be the telephony industry's concept of a premium number. That there isn't an enforced standardized way to tell that a phone number will incur extra fees to contact is a failure of the industry. This shouldn't be a hard problem.
There is such a system, countries are supposed to report all premium number ranges to the ITU. The problem is that some countries/carriers don't actually do that and then still charge the high amount.
If Twilio wanted to stand up for their customers they would refuse to pay for any premium number that's not in an ITU registered premium range. It will cause a few lawsuits with telecom carriers, but then probably ends this problem once and for all.
Twilio claims this isn't possible, but most mobile operators offer an option to disable calling premium numbers. So either the providers are very customer friendly and eat the cost (unlikely, it would open them up for huge amounts of fraud) or the mobile operators figured out how to block it or push the cost back to the upstream provider.
If Twilio wanted to stand up for their customers they would refuse to pay for any premium number that's not in an ITU registered premium range. It will cause a few lawsuits with telecom carriers, but then probably ends this problem once and for all.
Twilio claims this isn't possible, but most mobile operators offer an option to disable calling premium numbers. So either the providers are very customer friendly and eat the cost (unlikely, it would open them up for huge amounts of fraud) or the mobile operators figured out how to block it or push the cost back to the upstream provider.
Couldn't you not support SMS 2FA for noncompliant countries, and then check that it's not in an ITU registered premium range?
Wow. That's a lot of words for Twilio to say "we won't help you mitigate toll fraud".
> the fraudsters can set up a forward from a normal number to a premium number
Doesn't this mean that your call from Twillo to the normal number would be charged at your usual rate, and the fraudster would pick up the cost for the forwarding?
Doesn't this mean that your call from Twillo to the normal number would be charged at your usual rate, and the fraudster would pick up the cost for the forwarding?
> The fraudster can setup a forward from a normal number to a premium one anyway. So checking number prefix is useless.
Would the forwarding party not have to pay the premium rate part in this case?
Would the forwarding party not have to pay the premium rate part in this case?
Yes, in all telco systems I know the one that sets up call forwarding pays for the part from forwarder to destination.
I think most consumer telephony companies let you block calling premium numbers, so presumably Twilio could just ask their telephony provider to do so.
[deleted]
Sucks. But then again, anything that puts a nail in the coffin of SMS/Voice call 2fa, probably for the best. I feel bad for the genuine uses of phone tech that are targeted though.
I've only briefly played with Twilio, but would it be possible for your app/TwilML to intercept the request to send a text to those numbers, and detect a volume of X messages being sent to the same number (highly unlikely) and abort?
That's one of their recommendations. So the idea of Twilio is just drop a library and we take care of telephony for you, but then they suggest you implement cooldown periods, exponential backoffs, use their other services (paid) to check if the number is a mobile etc. And they tell you this, after you get the nasty bill.
We had this issue as well. Out tens of thousands of dollars. For a tiny startup, this almost killed us.
We’re tiny too and now worried. We only offer 2FA with SMS. Did they abuse your SMS OTp?
[deleted]
Twilios base prices are way higher than their competitors so I don't know why you would use them in the first place
Welcome to the new world where tech companies rule. Where moral standards are gone. Where local laws do not apply.
This shit is happening a lot. Artists on Youtube, App builders on Playstore/iOS. Marketing companies on Google search, pretty women on Instagram, political men on Facebook or Twitter.
I feel for you. Hope it gets alright.
>Welcome to the new world where tech companies rule.
I understand what you're saying but, speaking specifically to this case (which is a little different than YouTube censorship), I think this highlights the dangers of outsourcing your critical infrastructure to third-party SAAS companies without a mitigation plan.
As companies, we have to get into the habit of putting together mitigations plans for times when the outsourced SAAS disruption has significant business impact. Here, Twilio blocked the account of this vendor, but Twilio could have gone bankrupt, or they themselves could have had a significant disruption and the end-result would have been the same. Was there a plan put in place for that? If Twilio isn't responsive with support, are any of their competitors better? Maybe it was a better idea to go with a smaller or more expensive competitor but one who gives you a dedicated account manager you can call anytime if something goes wrong. Twilio fucked up here, but ultimately the responsibility for business continuity rests with OP. They can't offload that responsibility on Twilio because if Twilio fucks up, Twilio doesn't suffer the consequences.
I understand what you're saying but, speaking specifically to this case (which is a little different than YouTube censorship), I think this highlights the dangers of outsourcing your critical infrastructure to third-party SAAS companies without a mitigation plan.
As companies, we have to get into the habit of putting together mitigations plans for times when the outsourced SAAS disruption has significant business impact. Here, Twilio blocked the account of this vendor, but Twilio could have gone bankrupt, or they themselves could have had a significant disruption and the end-result would have been the same. Was there a plan put in place for that? If Twilio isn't responsive with support, are any of their competitors better? Maybe it was a better idea to go with a smaller or more expensive competitor but one who gives you a dedicated account manager you can call anytime if something goes wrong. Twilio fucked up here, but ultimately the responsibility for business continuity rests with OP. They can't offload that responsibility on Twilio because if Twilio fucks up, Twilio doesn't suffer the consequences.
The entire value of these companies is in the outsourcing of the entire problem.
It's incredibly hard to have multiple vendors if you're using specific or advanced functionality, and if you're hosting a failover yourself then you might as well just use that instead.
It's incredibly hard to have multiple vendors if you're using specific or advanced functionality, and if you're hosting a failover yourself then you might as well just use that instead.
>It's incredibly hard to have multiple vendors if you're using specific or advanced functionality, and if you're hosting a failover yourself then you might as well just use that instead.
I'm not suggesting any specific solution and I didn't even suggest they run their own infrastructure. There are good reasons to outsource these kinds of operations. But what happened was the company got the rug pulled underneath their feet and realized they didn't know how to contact their critical supplier. That's not good and that's their failure in this mess.
I don't understand the resistance to identifying and mitigating business risks. Honestly.
I'm not suggesting any specific solution and I didn't even suggest they run their own infrastructure. There are good reasons to outsource these kinds of operations. But what happened was the company got the rug pulled underneath their feet and realized they didn't know how to contact their critical supplier. That's not good and that's their failure in this mess.
I don't understand the resistance to identifying and mitigating business risks. Honestly.
You're not understanding because there is no resistance. This was the risk.
Again, let's say Twilio is the only supplier that could have provided this service for them, they still need to understand what risks that entails and how to mitigate them.
What happened in this situation is that Twilio for one reason or another decided to block access to their service, ostensibly halting their business. OK. That's not good. You know what made it worse? The CEO realizing that they don't have any line of communication to Twilio and that Twilio had shit tech support and therefore he had to frantically trawl Twitter and HN and Reddit and emailing Jeff Lawson (Twilio) CEO. Are you telling me they couldn't prevent this? You sure there isn't an Account Manager assigned to their account? Twilio holds conferences, there are plenty of opportunities to forge some relationship with someone at Twilio so you could at least backchannel issues like this (in fact, some Twilio dev here at HN noticed this and escalated it). By the way, Twilio is not the only telephony provider. If Twilio has shit support, OP can find another critical supplier, maybe a smaller or a more expensive vendor that is more responsive.
How about this: I looked through OPs comment history and he mentioned that their support numbers is provided by their service (i.e. they dogfood their product). Is that not a risk? If they are down, or Twilio is down, their customers can't reach a human either.
Very few of us control our circumstances, but we can certainly control our response to them. Outsourcing isn't the problem here. It has benefits and detriments. Lack of planning, and foresight is the issue here. This could have been a 10 minute outage, instead of a full day outage.
What happened in this situation is that Twilio for one reason or another decided to block access to their service, ostensibly halting their business. OK. That's not good. You know what made it worse? The CEO realizing that they don't have any line of communication to Twilio and that Twilio had shit tech support and therefore he had to frantically trawl Twitter and HN and Reddit and emailing Jeff Lawson (Twilio) CEO. Are you telling me they couldn't prevent this? You sure there isn't an Account Manager assigned to their account? Twilio holds conferences, there are plenty of opportunities to forge some relationship with someone at Twilio so you could at least backchannel issues like this (in fact, some Twilio dev here at HN noticed this and escalated it). By the way, Twilio is not the only telephony provider. If Twilio has shit support, OP can find another critical supplier, maybe a smaller or a more expensive vendor that is more responsive.
How about this: I looked through OPs comment history and he mentioned that their support numbers is provided by their service (i.e. they dogfood their product). Is that not a risk? If they are down, or Twilio is down, their customers can't reach a human either.
Very few of us control our circumstances, but we can certainly control our response to them. Outsourcing isn't the problem here. It has benefits and detriments. Lack of planning, and foresight is the issue here. This could have been a 10 minute outage, instead of a full day outage.
Remove the tech-specific words from your comment, and you would probably find people saying the same thing throughout all of history.
Not sure if large car companies or clothes manufacturers (just two examples of large companies) were ever able to ignore legal rights of their customers. This has only become possible with digital services where companies refuse to acknowledge local laws and regulation.
For most of history, customers didn't have the kind of legal rights they have now. Businesses could refuse to serve you, warranties were non-existent until late 1800s. And even when consumers did get rights, there have been constant stories in newspapers of "big companies trampling on the rights of consumers".
> Not sure if large car companies or clothes manufacturers (just two examples of large companies) were ever able to ignore legal rights of their customers.
Ha, check out “unsafe at any speed” the book that launched Ralph Nader and the government’s role in consumer safety.
Or “The Jungle”, the book that launched food safety regulations (and some worker safety) a century ago.
Certain politicians and certain companies decry “regulation” generically but there are good reasons behind almost all of it.
Ha, check out “unsafe at any speed” the book that launched Ralph Nader and the government’s role in consumer safety.
Or “The Jungle”, the book that launched food safety regulations (and some worker safety) a century ago.
Certain politicians and certain companies decry “regulation” generically but there are good reasons behind almost all of it.
If you want an example of ignoring local law, how about the East India Company? They formed their own army, took over part of India, and set up their own government.
Exactly. If "putting innocent people in chains and enslaving them in a foreign country" doesn't count as ignoring people's legal rights, I'm not sure what does.
> clothes manufacturers (just two examples of large companies) were ever able to ignore legal rights of their customers
Clothes manufacturers have been notorious for mistreating their employees, though; e.g. Triangle Shirtwaist.
Clothes manufacturers have been notorious for mistreating their employees, though; e.g. Triangle Shirtwaist.
It's really frustrating. I've been trying for months to get access to a suspended twitter account that was suspended without informing me or giving a reason. The tickets get ignored and their support doesn't respond on Twitter. They also ignore GDPR (which would give me the right to have my data deleted). I have no idea what to do. The account isn't important but it shows up prominently on Google as a suspended account which isn't great.
Any small company ignoring privacy laws would get into trouble. If you're as big as Twitter or Facebook you can just ignore laws and get away with it..
Any small company ignoring privacy laws would get into trouble. If you're as big as Twitter or Facebook you can just ignore laws and get away with it..
> They also ignore GDPR (which would give me the right to have my data deleted
Report them to your local ICO.
Report them to your local ICO.
"What these corporations are doing is literally destroying the basis for a developed economy....
"In countries with strong rule of law:
"1. Property rights over land, equipment, and personal items are clear and protected by law.
"2. Contracts between people, businesses, and the government are effectively enforced by the legal system.
"3. Political accountability is high and corruption is low.
"4. Business regulations are clear and enforced in a transparent manner.
"In such environments people make long-term investments and build large organizations. In contrast, if the property rights and contracts are not enforced and the business regulations are not clear, most of the economy consists of small family owned firms with little modern equipment. A high-tech, prosperous economy would not develop.
"Effectively, there are no contracts anymore in the digital economy. There is no predictability anymore. There is no accountability. There is no responsibility. There are no requirements for performance anymore. In sum, the US digital economy is rapidly becoming the equivalent of a third-world economy, complete with crony capitalism and digital robber barons."
"In countries with strong rule of law:
"1. Property rights over land, equipment, and personal items are clear and protected by law.
"2. Contracts between people, businesses, and the government are effectively enforced by the legal system.
"3. Political accountability is high and corruption is low.
"4. Business regulations are clear and enforced in a transparent manner.
"In such environments people make long-term investments and build large organizations. In contrast, if the property rights and contracts are not enforced and the business regulations are not clear, most of the economy consists of small family owned firms with little modern equipment. A high-tech, prosperous economy would not develop.
"Effectively, there are no contracts anymore in the digital economy. There is no predictability anymore. There is no accountability. There is no responsibility. There are no requirements for performance anymore. In sum, the US digital economy is rapidly becoming the equivalent of a third-world economy, complete with crony capitalism and digital robber barons."
I have the same on Sendgrid, owned by Twilo. One day they just cut us off with no explanation, kneecapping our product. I raised two tickets, neither of which were answered. The next day, it all started working again.
I’ve always been a big advocate for using 3rd party SaaS, but with the level of customer service I’m seeing lately, I’m starting to recommend avoiding it where possible.
I’ve always been a big advocate for using 3rd party SaaS, but with the level of customer service I’m seeing lately, I’m starting to recommend avoiding it where possible.
That's 99% of why I'm generally trying to avoid SaaS. I always run into some bug or nuisance or there's some account trouble and unless you're spending insane amounts of money, I've found support to be severely lacking, often answering support tickets with canned responses without reading the issue, or not answering at all.
I'd rather spend 100 hours upfront to have something that I know and can maintain than using a service, running into trouble and spend 50 hours chasing the support around to see whether they'd fix it, or worse, rely on their system, have it fail or be banned, and then having to spend 100 hours building a replacement, often while things are exploding left and right.
Relying on SaaS feels incredibly fragile to me.
I'd rather spend 100 hours upfront to have something that I know and can maintain than using a service, running into trouble and spend 50 hours chasing the support around to see whether they'd fix it, or worse, rely on their system, have it fail or be banned, and then having to spend 100 hours building a replacement, often while things are exploding left and right.
Relying on SaaS feels incredibly fragile to me.
I think this should be regulated - companies often take massive amount of customers and while they are most of the time able to deliver the product, they have no capacity to service support queries. I think there should be an SLA guaranteed by law and for a company it should be illegal to take more customers if they are unable to provide SLA on support.
This way we wouldn't have a situation where you contact a company and in reality you talk to bots or nobody at all.
This would also deal with the problem of companies growing too big to fail and not having reasonable competition.
Or you could just, you know, buy from Saas'es that provide SLA's enforceable in your jurisdiction. Vote with your <insert local currency here>
I recently moved to Postmark. It’s been an extremely positive experience. They’re a small, dedicated team with amazing support.
It's an insane amount of engineering, but maybe one solution is to have two back end telephony SaaS. If one turns off, rely on the other until you get it fixed? Plivo, Mexmo?
I got this email from Twilio a few days back, it could be related:
As a current Twilio customer, we are notifying you that we are updating our Messaging Policy, available at https://www.twilio.com/legal/messaging-policy. We are sending this message to all customers, regardless of whether action is required to ensure compliance.
[...]
There are a few key sections we’d like to call out:
Applicability. We clarified that the Twilio Messaging Policy applies to all organizations using our services, and that Independent Software Vendors (ISVs) are responsible for ensuring their users are in compliance.
Messaging Policy Evasion. We clarified that content or techniques designed to avoid mechanisms that detect policy violations are not permitted.
Usage Limitations - Content We Do Not Allow. We clarified that you may not send content that is illegal in the jurisdiction where the message recipient lives. For example, in the United States, this includes cannabis and CBD messages.Maybe time for a new HN headline convention, like:
Tell FAANG: You blocked me, again
FAANG Support: Important please call back ASAP
Tell FAANG: You blocked me, again
FAANG Support: Important please call back ASAP
Could this be productised somehow? Would an investor be willing to invest in my startup which effectively un-blocks small companies from larger ones? It would do the leg work to compile the necessary information before contacting a real human.
I'll call it Support As A Service.
I'll call it Support As A Service.
I posited the interestingness of such an idea a couple months ago in a thread about app store blocks: https://news.ycombinator.com/item?id=25988410
IMHO, most emphatically *yes* - this is a completely unserved/unserviced area of the B2B and B2C markets.
I noted in the linked thread that such a service would likely start by leveraging individual contacts, and building a reputation for unwaveringly high signal.
It definitely does seem to me that the vacuum between corporate and customers represents a giant sleeper industry that fundamentally won't shift very quickly.
Which means you're unlikely to reach the usual core-problem-is-solved entrenchment/stagnation point that tends to happen when you ramp up on a complex-but-solve-once problem, then find yourself stuck with a bunch of intricate sausage machine bits that aren't doing anything...
IMHO, most emphatically *yes* - this is a completely unserved/unserviced area of the B2B and B2C markets.
I noted in the linked thread that such a service would likely start by leveraging individual contacts, and building a reputation for unwaveringly high signal.
It definitely does seem to me that the vacuum between corporate and customers represents a giant sleeper industry that fundamentally won't shift very quickly.
Which means you're unlikely to reach the usual core-problem-is-solved entrenchment/stagnation point that tends to happen when you ramp up on a complex-but-solve-once problem, then find yourself stuck with a bunch of intricate sausage machine bits that aren't doing anything...
Makes sense.
1. If someone works at a company, they can sign up on our website as a support agent.
2. We will only list companies with which we have an authenticated contact, maybe require people to verify they got our authentication token on their employee email?
3. When someone contacts us about an issue with a company we support, we verify the request is genuine. Perhaps charge a small fee to block spam. (This is the problematic part. How do we know a problem is genuine without asking for potentially confidential information?)
4. Add all vetted requests to a private channel that all the people who volunteered on step one. (Can we pay our volunteer support agents? How? Does paying make things worse? Maybe gamify this step and send our volunteers gifts?)
5. Monitor performance of support agents to avoid giving false hope to our customers.
1. If someone works at a company, they can sign up on our website as a support agent.
2. We will only list companies with which we have an authenticated contact, maybe require people to verify they got our authentication token on their employee email?
3. When someone contacts us about an issue with a company we support, we verify the request is genuine. Perhaps charge a small fee to block spam. (This is the problematic part. How do we know a problem is genuine without asking for potentially confidential information?)
4. Add all vetted requests to a private channel that all the people who volunteered on step one. (Can we pay our volunteer support agents? How? Does paying make things worse? Maybe gamify this step and send our volunteers gifts?)
5. Monitor performance of support agents to avoid giving false hope to our customers.
[deleted]
Try this link: https://investors.twilio.com/governance/person-details/defau...
Phone number on the page and an email address that may work for you. Let me know if that helps.
Phone number on the page and an email address that may work for you. Let me know if that helps.
> Since then, we are trying to get in contact with them through every channel we could find: email, Twitter, LinkedIn, [..] They don't have a support number we can call.
Tons of platforms seem to operate like this, and this kinda thing happens all the time. HN is turning into a desperate last resort as a PleaForHelp-as-a-Service (hoping that bad PR or a company insider is the trigger).
But by now business decision-makers should really wake up and acknowledge the risk of choosing such platforms. Of course they should demand better service, but this is also something to research upfront ("is there a support desk and at least a phone number.") and then decide if they want to pay the extra money, or go somewhere else that addresses their needs.
Tons of platforms seem to operate like this, and this kinda thing happens all the time. HN is turning into a desperate last resort as a PleaForHelp-as-a-Service (hoping that bad PR or a company insider is the trigger).
But by now business decision-makers should really wake up and acknowledge the risk of choosing such platforms. Of course they should demand better service, but this is also something to research upfront ("is there a support desk and at least a phone number.") and then decide if they want to pay the extra money, or go somewhere else that addresses their needs.
Terrifying to rely on another team's support staff for your business to exist.
In this time and age I think it's inevitable. Think AWS instead of Twilio for example.
AWS provides a service that you can use as a commodity, and with appropriate backups, can easily recover from. DNS is the magic here.
Unfortunately (with the exception of SIP dialling, which is extremely niche) there is no DNS for telephony. If your provider cuts you off, you can’t transparently move away, and number porting is slow.
Unfortunately (with the exception of SIP dialling, which is extremely niche) there is no DNS for telephony. If your provider cuts you off, you can’t transparently move away, and number porting is slow.
> there is no DNS for telephony.
In fact there is: https://www.networkworld.com/article/2332977/lan-wan-what-is...
Only most telephony providers sabotage it.
In fact there is: https://www.networkworld.com/article/2332977/lan-wan-what-is...
Only most telephony providers sabotage it.
afaik the numbers are associated to a provider, they get blocks of XXXX numbers. it would not help somebody to have enum on the provider level. because they are stuck there.
Numbers are more like IP addresses and less like domains.
Do you really think the typical aws customer can easily recover from being banned?
From what I've heard surrounding the Parler incident, AWS worked _with_ Parler staff to get their data off the platform. Sure, it's not a hot-swap to Azure or metal or whatever, but it was nice to see that they didn't hold Parler's data hostage or just plain delete it.
I'd assume the same courtesy would be extended to any customer, provided their ToS breaches aren't significantly worse (read: illegal) and it's not a billing-related suspension.
I'd assume the same courtesy would be extended to any customer, provided their ToS breaches aren't significantly worse (read: illegal) and it's not a billing-related suspension.
>I'd assume the same courtesy would be extended to any customer
Unpopular opinion.
You see, I dont want assumption. I want guarantees. I also dont want a platform owner to remove you without some days if not weeks notice. I mean your landlord and court give you time for eviction. At least I hope that is true, I am not sure if time to vacate is universal across the world. They also give little to no time to solve any dispute.
I also dont like it when media spin that AWS is at least helping them move their Data off the platform as if it is something good. I dont know why it is perceived that way in the US, but that was absolutely the least they need to do and expected in EU or UK.
It is increasingly a worry sign. I have been yearning for an iOS Time Capsule for more than a decade. And the recent event makes it even more important. I want to own my things.
Unpopular opinion.
You see, I dont want assumption. I want guarantees. I also dont want a platform owner to remove you without some days if not weeks notice. I mean your landlord and court give you time for eviction. At least I hope that is true, I am not sure if time to vacate is universal across the world. They also give little to no time to solve any dispute.
I also dont like it when media spin that AWS is at least helping them move their Data off the platform as if it is something good. I dont know why it is perceived that way in the US, but that was absolutely the least they need to do and expected in EU or UK.
It is increasingly a worry sign. I have been yearning for an iOS Time Capsule for more than a decade. And the recent event makes it even more important. I want to own my things.
Ok, then own your things. Your iOS time capsule is your computer with either macOS or iTunes installed on it; you've been able to backup and sync (even wirelessly, if I'm remembering correctly) more content than an iCloud backup would let you backup anyways.
Spin up your own servers, etc. I don't really get why so many people here think AWS is the only way to run a website nowadays; you can put metal in a closet anywhere that has a good enough internet connection. If you're planning on doing shit that'll get your cloud accounts suspended, why on earth are you using a cloud provider?
The only way to _guarantee_ that you'll have access to your data no matter what is to own it, so own it!
Back to your eviction policy- it's common in my area for leases to have a clause stating that landlords may recoup delinquent rent by entering your apartment and taking your property. Legal? Don't know, but my lease as well as the leases of most of my friends have this clause in them. I also know that university housing will kick you out with < 1 week notice at my uni if you're in violation of your housing contract.
Maybe we just have very different points of view/experiences, but in my eyes you can't "own" anything in a cloud provider. This is fine for most circumstances (i.e. my personal site is on Netlify but if I get kicked off I still have all the source), but if you need to own something just put it on your own server.
Spin up your own servers, etc. I don't really get why so many people here think AWS is the only way to run a website nowadays; you can put metal in a closet anywhere that has a good enough internet connection. If you're planning on doing shit that'll get your cloud accounts suspended, why on earth are you using a cloud provider?
The only way to _guarantee_ that you'll have access to your data no matter what is to own it, so own it!
Back to your eviction policy- it's common in my area for leases to have a clause stating that landlords may recoup delinquent rent by entering your apartment and taking your property. Legal? Don't know, but my lease as well as the leases of most of my friends have this clause in them. I also know that university housing will kick you out with < 1 week notice at my uni if you're in violation of your housing contract.
Maybe we just have very different points of view/experiences, but in my eyes you can't "own" anything in a cloud provider. This is fine for most circumstances (i.e. my personal site is on Netlify but if I get kicked off I still have all the source), but if you need to own something just put it on your own server.
An AWS customer who wants to prepare for such a scenario can do so. Regardless of whether most of their customers do, the point is that it’s possible—and quite reasonable.
That isn’t the case for PSTN. It’s outright impossible.
That isn’t the case for PSTN. It’s outright impossible.
AWS will (at a reasonable spend) sign a legally binding contract preventing them from pulling service from you unexpectedly.
You sure about that? They pulled the rug under parler.
But let's say there is a contract or TOS that says they won't, and they do - what are you going to do about it? Go to court for 2 years?
But let's say there is a contract or TOS that says they won't, and they do - what are you going to do about it? Go to court for 2 years?
Do you have a link or keywords to search for?
It’s a supply chain.
We had the same problem some months ago with Mailgun. They blocked our account after many years using them without apparent reason or prior notice and took them 2 days to unblock it, leaving our thousands of clients stranded without receiving any of our transactional emails.
Lesson learnt, always keep a backup for crucial services and move away from Mailgun
Lesson learnt, always keep a backup for crucial services and move away from Mailgun
We just moved to Mailgun because SendGrid did something similar to us! Fortunately we only used SMTP so it was a straightforward change, but having a backup for something like Twilio Video would be quite an effort.
Could this be the result of one of your customers breaching Twilio's TOS?
If so, does using Twilio's subaccounts feature do anything to prevent your whole account being suspended for the actions of one of your customers?
If so, does using Twilio's subaccounts feature do anything to prevent your whole account being suspended for the actions of one of your customers?
Yeah, they changed their TOS 2 days ago... This seems likely.
Do they send mail about breaching?
Port your phone numbers out ASAP, you can jump to Teli or BulkVS and pay quite a bit less with much broader numbering inventory available.
Even if we wanted to do this, this process can take days and sometimes even weeks.
A simple port request (1 phone number) has to be resolved within 24 hours, and complex port requests have similarly short SLAs.
The FCC has defined these SLAs and if you ever have a losing provider that is close to breaking the SLA on a port request, search for the providers local number portability escalation list and call each contact until resolved.
The FCC has defined these SLAs and if you ever have a losing provider that is close to breaking the SLA on a port request, search for the providers local number portability escalation list and call each contact until resolved.
thanks for those 2 providers. will also check them out. do you know (beside voxbone) some further international DID providers?
Prior to Voxbone joining Bandwidth.com they were a provider we had chosen to avoid.
Voxbeam and Anveo Direct are solid choices for international DIDs.
Bandwidth.com has a poor reputation FYI, they pissed off a few Public Utility Commissions and were blacklisted from getting new phone numbers in some states.
Voxbeam and Anveo Direct are solid choices for international DIDs.
Bandwidth.com has a poor reputation FYI, they pissed off a few Public Utility Commissions and were blacklisted from getting new phone numbers in some states.
Our company phone system has several VOIP providers (including Twilio) as well as multiple inbound analog lines.
Two weeks ago, all of the analog lines stopped working, which are local numbers, but our main 800 number forwards to those numbers.
Calling the local provider was futile. The provider’s office was also experiencing the issue and could not receive calls.
Although we do have the 800 number on business cards, we quickly updated our website and Intercom docs to use a recently provisioned 800 number from one of the VOIP providers.
It took 2 days for the other lines to begin working again. This is the first POTS outage we have experienced in several years.
Two weeks ago, all of the analog lines stopped working, which are local numbers, but our main 800 number forwards to those numbers.
Calling the local provider was futile. The provider’s office was also experiencing the issue and could not receive calls.
Although we do have the 800 number on business cards, we quickly updated our website and Intercom docs to use a recently provisioned 800 number from one of the VOIP providers.
It took 2 days for the other lines to begin working again. This is the first POTS outage we have experienced in several years.
Sorry to hear that. I hope everything works out.
Take this a learning experience. Once you get your business back, sit down with your team and come up with a mitigation plan because this may happen again. It's possible you may need to switch to another vendor, even one more expensive or smaller, but one with a dedicated support line and/or an account manager and stronger SLAs. Your business heavily relies on Twilio and you should have someone you can call at Twilio outside of the general support line. Maybe you need to attend their conferences and create a personal relationship with their developers or other employees so in cases like this you can backchannel your issue.
If you're going to outsource critical business infrastructure to third party SAAS vendors, make sure you have a plan for when things go tits up, because things will go tits up. It looks like Twilio screwed up here, but Twilio isn't going to suffer the consequences of this like your business will.
Take this a learning experience. Once you get your business back, sit down with your team and come up with a mitigation plan because this may happen again. It's possible you may need to switch to another vendor, even one more expensive or smaller, but one with a dedicated support line and/or an account manager and stronger SLAs. Your business heavily relies on Twilio and you should have someone you can call at Twilio outside of the general support line. Maybe you need to attend their conferences and create a personal relationship with their developers or other employees so in cases like this you can backchannel your issue.
If you're going to outsource critical business infrastructure to third party SAAS vendors, make sure you have a plan for when things go tits up, because things will go tits up. It looks like Twilio screwed up here, but Twilio isn't going to suffer the consequences of this like your business will.
Disclaimer: IANAL
As far as telephony is concerned (things like SMS are a different story), a phone provider cannot legally do this, even if you signed some ToS agreement. According to the FCC, phone providers, including voip providers, have legal requirements around when they are allowed to terminate service: https://www.fcc.gov/consumers/guides/when-your-telephone-com...
As far as telephony is concerned (things like SMS are a different story), a phone provider cannot legally do this, even if you signed some ToS agreement. According to the FCC, phone providers, including voip providers, have legal requirements around when they are allowed to terminate service: https://www.fcc.gov/consumers/guides/when-your-telephone-com...
The page you link to says that this doesn’t apply to mobile providers in general.
I was under the impression that voip is not considered "wireless" but I could be mistaken there.
I get the impression those protections are for residential phone service, not the kind of stuff twilio does. Leaving that aside, though, SMS (at issue here) is definitely in the wireless realm, even if the messages come from an API call rather than a mobile device's radio.
Somewhat related anecdote. Whilst we didn't get cut off, we found a huge drop-off in time to receive, and the quality of, support from our live chat/support messaging provider.
Having been a customer since our launch [1], and very early after theirs, it has been really sad and frustrating to see that friendly distrupter energy zap out of the service.
I get why businesses go through that changes like this during growth, but it feels like all the things to let slide, your persona is not one of them!
[1] https://mailosaur.com (somewhat related to the thread we let you test SMS messages)
Having been a customer since our launch [1], and very early after theirs, it has been really sad and frustrating to see that friendly distrupter energy zap out of the service.
I get why businesses go through that changes like this during growth, but it feels like all the things to let slide, your persona is not one of them!
[1] https://mailosaur.com (somewhat related to the thread we let you test SMS messages)
Yikes, talk about bad PR.
I am sorry that this happened to you, but thank you for sharing this, you are not the first one, nor the last one by the looks of it.
I intend to avoid Twilio in a wide range, don't need that kind of a "business partner" in my life.
I intend to avoid Twilio in a wide range, don't need that kind of a "business partner" in my life.
Many of these system now a days use automation with little human oversight. Algorithms classify and block accounts. The little human oversight is inadequate to even verify what happened. I believe support is spread thin with cost saving and automation.
P.S. I am having a similar problem, but with Viber. I posted the details here: https://www.reddit.com/r/viber/comments/mrcu6f/viber_block_f...
P.S. I am having a similar problem, but with Viber. I posted the details here: https://www.reddit.com/r/viber/comments/mrcu6f/viber_block_f...
Lesson learned: always have a number to call, always have the ability to get a hold of a flesh-and-blood human being. My company mandates this as a minimal requirement for all our SaaS providers.
With regards to having a mitigation strategy we take another tactic: what's the SaaS provider's mitigation strategy? How do they cope with data center loss, communications loss - what processes and procedures do they have in place? Any business-worthy SaaS provider has this information readily available. Any hemming or hawing is a sign of trouble.
With regards to having a mitigation strategy we take another tactic: what's the SaaS provider's mitigation strategy? How do they cope with data center loss, communications loss - what processes and procedures do they have in place? Any business-worthy SaaS provider has this information readily available. Any hemming or hawing is a sign of trouble.
[deleted]
a lot of us have this as nightmare scenario.
- long term we really should move to an industry wide due process system with legislation for compensation and other parameters. once there is some more structure there can also be insurances or similar ways to prevent something like this to ruin a company.
- there maybe is a market for more intermediaries that bundle twilio and messagebird or similar providers and have an official strong communication channel to the providers to resolve issues and provide failover
- long term we really should move to an industry wide due process system with legislation for compensation and other parameters. once there is some more structure there can also be insurances or similar ways to prevent something like this to ruin a company.
- there maybe is a market for more intermediaries that bundle twilio and messagebird or similar providers and have an official strong communication channel to the providers to resolve issues and provide failover
100%. Disappointment from seeing a once upstart (which I was rooting on from the sidelines) become a faceless big co aside, I think insurance is the right way to handle this problem and prevent it from being unfairly externalized.
We had an incident where our we got charged thousands (later refunded) by abusing dodgy routing and number verification process.
The problem was that Twilio will allow calls through when the call balance is below zero, if the calls were queued before it was zero. You then end up with a negative balance.
On the plus side, it does force you to start making defensive plans to protect your application.
On the plus side, it does force you to start making defensive plans to protect your application.
We asked for a phone number transfer to another Twilio account, and the support rep took the wrong action by mistake.
He wiped our entire account, causing us to lose tens of thousands of recordings, logs etc. We were told sorry, no way to recover them - only the underlying phone numbers.
It's cheap, but I'd never trust them again.
He wiped our entire account, causing us to lose tens of thousands of recordings, logs etc. We were told sorry, no way to recover them - only the underlying phone numbers.
It's cheap, but I'd never trust them again.
Twilio is a wet dream for spammers (robocalling, emailing, SMS).
They’ve turned spamming into a positive ROI business.
They’ve turned spamming into a positive ROI business.
Hmmmm they definitely have a phone number, I remember when I worked at a startup a few years ago I had many phone conversations with them regarding a bug in their mobile chat SDK. Perhaps we had a support contract tho...
They don't have a support number we can call.
I take it you've tried (844) 814-4627 ?
I take it you've tried (844) 814-4627 ?
Yes. When I select the option for support they just play a message saying that I should open a ticket via email instead.
Yeah looks like you're SOL. They also have some scheduled maintenance going on right now in the US, but doubtful that's to do with your situation. Also, it's 8PM here in SF, so I'm guessing e-mail will be the only option tonight.
Yeah, I guess so. I'll keep trying everything I can since our customers can't wait and are losing business because of the downtime.
Thank you!
Thank you!
While this does not resolve your current issue. Have a look at https://signalwire.com its from the same people who created FreeSWITCH.
It seems like there would be a market for reputable E2E vendors that provide you with a decent SLA.
On the other hand, that would be a bit more expensive, and might therefore be out-competed in a race to the bottom.
On the other hand, that would be a bit more expensive, and might therefore be out-competed in a race to the bottom.
This is why you have multiple vendors for every external function. Just like hardware manufacturers try to maintain at least two suppliers for every component.
Hey there, sorry to hear that. Are you able to login to your account at all? If so, can you open a ticket and share the number here so I can take a look?
Long time twilio user and evangelist here ... as I have mentioned here and in interviews[1], etc., I built my own little mini-telco out of twiml bins and have attempted, over the past 4-5 years, to move all of my telephony into the (twilio) cloud.
I was encouraged when I attended Signal 2018[2] by twilio product and technical managers who intimated that such personal "utility" uses of twilio were favored by the CEO (Jeff Lawson) himself.
Everything you need to know about how this has turned out - and which direction the company is going - can be imparted with two points:
1) Twilio straight up refuses to put an email verb in Twiml. This sounds like a nerdy, nit-picky point from my own personal use-case (and it is) but it's the most dead-obvious thing in the world, is technically trivial, and would make so many use-cases a world easier than they are.
Instead of adding an email verb to twiml (so you could, for instance, cc: an email address with emergency SMS alerts, or perhaps cc: yourself a log of all received SMS) they chose to buy an entire email company. Now you can create these use-cases provided that you sign up a sendgrid account, and tie the two together with code hosted at a third party and blah blah blah.
The reason they did this is providing telco utility tools is not interesting. Customer engagement at scale (or whatever bullshit) is what is interesting. Managing high value interactions. Whatever.
2) Q1 of 2021 they crammed phone only 2FA down the throats of all account holders. You may say "you can also use authy" but, of course, the initial setup of authy requires a real mobile number[3] and you can't log into twilio to enable authy without providing a mobile number first.
So, if you're planning on moving your telephony into the cloud, you can't - you'll need to tie real phones and SIM cards to these accounts. Have multiple twilio accounts ? Congratulations - you can either be super sloppy and share your mobile SIM across all of them or you can buy three phones.
This is for your security. Total bullshit.
In reality, twilio has a massive spam/scam problem that is almost unsolvable. I don't envy them in this position. However, instead of solving this problem, they have decided to just throw sand in the gears and slow down the scammers by demanding that everyone prove a SIM identity. This doesn't bother most users because they aren't, ironically, trying to shed SIM cards by moving their telco infrastructure to twilio ... but it fucks my shit up immensely.
Which is what happens when providing telco utility services has no path-to-unicorn and you need to turn yourself into an "engagement platform" that "enables customer interaction" for "high value use-cases".
[1] https://console.dev/qa/rsync-john-kozubik/
[2] "peak twilio"
[3] Which is so ironic since, in almost all cases, twilio numbers are NOT mobile numbers and cannot be used for "proper" 2FA since they cannot receive SMS from short-codes. So twilio demands a telephony use-case that they, themselves, cannot satisfy.
I was encouraged when I attended Signal 2018[2] by twilio product and technical managers who intimated that such personal "utility" uses of twilio were favored by the CEO (Jeff Lawson) himself.
Everything you need to know about how this has turned out - and which direction the company is going - can be imparted with two points:
1) Twilio straight up refuses to put an email verb in Twiml. This sounds like a nerdy, nit-picky point from my own personal use-case (and it is) but it's the most dead-obvious thing in the world, is technically trivial, and would make so many use-cases a world easier than they are.
Instead of adding an email verb to twiml (so you could, for instance, cc: an email address with emergency SMS alerts, or perhaps cc: yourself a log of all received SMS) they chose to buy an entire email company. Now you can create these use-cases provided that you sign up a sendgrid account, and tie the two together with code hosted at a third party and blah blah blah.
The reason they did this is providing telco utility tools is not interesting. Customer engagement at scale (or whatever bullshit) is what is interesting. Managing high value interactions. Whatever.
2) Q1 of 2021 they crammed phone only 2FA down the throats of all account holders. You may say "you can also use authy" but, of course, the initial setup of authy requires a real mobile number[3] and you can't log into twilio to enable authy without providing a mobile number first.
So, if you're planning on moving your telephony into the cloud, you can't - you'll need to tie real phones and SIM cards to these accounts. Have multiple twilio accounts ? Congratulations - you can either be super sloppy and share your mobile SIM across all of them or you can buy three phones.
This is for your security. Total bullshit.
In reality, twilio has a massive spam/scam problem that is almost unsolvable. I don't envy them in this position. However, instead of solving this problem, they have decided to just throw sand in the gears and slow down the scammers by demanding that everyone prove a SIM identity. This doesn't bother most users because they aren't, ironically, trying to shed SIM cards by moving their telco infrastructure to twilio ... but it fucks my shit up immensely.
Which is what happens when providing telco utility services has no path-to-unicorn and you need to turn yourself into an "engagement platform" that "enables customer interaction" for "high value use-cases".
[1] https://console.dev/qa/rsync-john-kozubik/
[2] "peak twilio"
[3] Which is so ironic since, in almost all cases, twilio numbers are NOT mobile numbers and cannot be used for "proper" 2FA since they cannot receive SMS from short-codes. So twilio demands a telephony use-case that they, themselves, cannot satisfy.
Vendor lock-in despite 6 years in the biz?
A little over 3 hours ago our entire account was suspended without reason. We received an email stating that fact without further explanation.
Since then, we are trying to get in contact with them through every channel we could find: email, Twitter, LinkedIn, and I even tried emailing Jeff Lawson, their CEO. They don't have a support number we can call.
We are not a small customer and we have been with them for more than 6 years. It's deeply frustrating that despite that fact, we don't seem to matter to them at all.
If you work at Twilio or if you can help us get in touch with them, I'd be indebted to you forever!