OVH CEO Octave Klaba apologizes and provides updates about SBG2(ovh.com)
ovh.com
OVH CEO Octave Klaba apologizes and provides updates about SBG2
https://www.ovh.com/fr/images/sbg/index-en.html
37 comments
The worst thing, in my opinion, is that there has been no communication from OVH directly to their customers. I am impacted and I have not received a single e-mail about the situation even now.
We received a mail. Fortunately we were using the managed database service from ovh and could quickly restore our websites from a backup.
Same in my case. No communication. Generally they are good with any networking issues, attacks, etc.
In this case nothing apart from the banner on the dashboard.
In this case nothing apart from the banner on the dashboard.
Octave has been putting out daily messages on twitter to customers explaining the situation.
Man I hate how that works.
I'll want to know what is going on with a thing and find out "oh they mostly post that kind of news on twitter" or "oh this person uses instagram", or "that news goes out via email", or youtube ... or ... who knows where ...
Like guises come on, where am I supposed to get information when you're using like 8 different channels and not consistently...
I'll want to know what is going on with a thing and find out "oh they mostly post that kind of news on twitter" or "oh this person uses instagram", or "that news goes out via email", or youtube ... or ... who knows where ...
Like guises come on, where am I supposed to get information when you're using like 8 different channels and not consistently...
This raises the question, "How could Octave be so tone-deaf?"
Only for people who want this incident to be part of a grand story arc about the character of OVH. The communication has been fine.
It was a bad pun.
Ah! I’m very sorry. (Doubly sorry? I know being misunderstood hertz?)
Should they be sending emails? Yes, of course. But, you’re being hyperbolic. It’s always, otherwise, Twitter or a status page on their website.
Did the fire take out their database of customer emails too?
Their status page was also showing everything being fine in the affected datacenter at least until the next day.
I did get an email yesterday around 16h00 GMT.
It was pretty generic, the only new thing it contained (for me at least) was the FAQ regarding this incident.
https://help.ovhcloud.com/en-gb/faq/strasbourg-incident/?xto...
https://help.ovhcloud.com/en-gb/faq/strasbourg-incident/?xto...
Following some links, I saw
SBG1 and SBG4: we are planning to bring these datacentres and the network room onlinenext week.
SBG3: we are planning to bring this datacentre online in 1-2 weeks.
SBG is OVH Strasbourg, a complex of data centres that are adjacent to each other, but which were not built at the same time and are architecturally not the same. Yes, SBG1-4 are the different buildings. Some of them are made from stacked shipping containers, but SBG2 was an actual building.
SBG2 basically burned to the ground. The fire also spread into SBG1 if I remember correctly: parts of SBG1 were destroyed but some parts are intact. SBG3 and 4 are also intact. Under the circumstances the entire complex is offline at the moment though, which is not surprising given the scale of the fire and the firefighter response. The photos reminded me of the WTC attack, which shut down lower Manhattan for months.
SBG2 was built like a vertical tunnel, a quadrangle of server rooms with a huge air shaft in the middle, intended to convection-cool the servers through the natural motion of the rising air through it as the air was warmed by the servers. In this case it may have resulted in something like a firestorm, where the fire maintained its own airflow through that column.
I'd be interested to know if there is any hope and/or intention in this situation to recover possibly-operable hard drives from the charred servers for data recovery purposes.
I couldn't believe the complaints about the audio in that mp4. I could understand him perfectly well except a few times his accent got in the way. Better audio wouldn't have mattered much.
SBG2 basically burned to the ground. The fire also spread into SBG1 if I remember correctly: parts of SBG1 were destroyed but some parts are intact. SBG3 and 4 are also intact. Under the circumstances the entire complex is offline at the moment though, which is not surprising given the scale of the fire and the firefighter response. The photos reminded me of the WTC attack, which shut down lower Manhattan for months.
SBG2 was built like a vertical tunnel, a quadrangle of server rooms with a huge air shaft in the middle, intended to convection-cool the servers through the natural motion of the rising air through it as the air was warmed by the servers. In this case it may have resulted in something like a firestorm, where the fire maintained its own airflow through that column.
I'd be interested to know if there is any hope and/or intention in this situation to recover possibly-operable hard drives from the charred servers for data recovery purposes.
I couldn't believe the complaints about the audio in that mp4. I could understand him perfectly well except a few times his accent got in the way. Better audio wouldn't have mattered much.
I am not directly impacted (not running anything that is advertised as physically there) but emails have been taking multiple hours to arrive for days. It would be nice to have a way of knowing whether I should change my MX servers or not.
I think it's fine for them to take some time to fix their infrastructure, but if they could let me know, I can easily work around it. Incoming email is kind of difficult to monitor and needs to work...
[edit] 3 separate tickets in 3 days... it seems dishonest to keep opening/closing them rather than let us know the infra is unreliable for the foreseeable future http://travaux.ovh.net/?do=details&id=49451& http://travaux.ovh.net/?do=details&id=49504& http://travaux.ovh.net/?do=details&id=49530&
I think it's fine for them to take some time to fix their infrastructure, but if they could let me know, I can easily work around it. Incoming email is kind of difficult to monitor and needs to work...
[edit] 3 separate tickets in 3 days... it seems dishonest to keep opening/closing them rather than let us know the infra is unreliable for the foreseeable future http://travaux.ovh.net/?do=details&id=49451& http://travaux.ovh.net/?do=details&id=49504& http://travaux.ovh.net/?do=details&id=49530&
maybe all their email systems were in that building, not even joking
duplicate of https://news.ycombinator.com/item?id=26427960
The video was badly prepared. He was talking about datacentre names as if the audience are his own employees and are fully aware of the different buildings.
He could at least have some graphics.
As a customer I understood the video perfectly. Those names appear on every invoice and during every order
And probably you as a customer are a main target of such video, but the fire was such a high-profile case (I think) that this should be more understandable for general audience.
edit: not sure why I got downvoted (and I don't really care that much), I just think that communication from CEO of big company after such event should be clear for everyone, not only for those directly affected.
edit: not sure why I got downvoted (and I don't really care that much), I just think that communication from CEO of big company after such event should be clear for everyone, not only for those directly affected.
But ... the video isn't for a general audience. He doesn't have any reason to apologize to a general audience. His customers are the only people that matter in this situation and this video is for them.
Honestly, nobody lost their lives, the impact is pretty much limited to OVH (who need to find a new building), their customers (who need to find new servers and restore backups), and the city of strasbourg (who run the emergency services and presumably will have some involvement in the cleanup). While we're all interested in a "Oh, are we prepared if that happened to our provider?" sense, I don't see why OVH is obliged to provide communications to us, rather than to their customers.
I would expect the communication to be directly targeted to customers not the general public.
OVH deserve points for being one of the only hosting providers I know that make their entire network weathermap available to the public, showing all their global interconnects and then letting you channel it down to a per-DC level: http://weathermap.ovh.net/#europe
Interesting. Are there popular tools to build such diagrams or is this a bespoke creation? ie are they using scripts+graphviz directly or is there some middlewear doing things?
It certainly wasn't well prepared. But it did everything there is to be done in such a situation. An quick apology, details about the situation, and what is being done. I didn't have any problem understanding what was being said. I don't know the details about the buildings but I don't think it is necessary to understand the general idea. Datacenter caught fire and was completely destroyed, some damage in the surrounding buildings, old buildings are worse than new buildings.
A nice presentation would have been great, but if there is a choice, I'd rather have the CEO work on the problem than polish his presentation.
Edit: He also named some people. I don't know who they are, I can only guess they are employees, and while I don't really care personally, I think it is tasteful to name them since they did the right thing by intervening quickly and not being reckless.
A nice presentation would have been great, but if there is a choice, I'd rather have the CEO work on the problem than polish his presentation.
Edit: He also named some people. I don't know who they are, I can only guess they are employees, and while I don't really care personally, I think it is tasteful to name them since they did the right thing by intervening quickly and not being reckless.
He looks as exactly as stressed out and sleep deprived as you might expect given the circumstances.
OVH wiped my server a couple weeks ago for non-payment while I was in the middle of a support ticket to sort out a billing issue with my invoice.
Was asked for more information from a support rep. Server was auto-wiped less than an hour later.
Ended an eleven year relationship and after this fire I can't recommend them in good faith to anyone.
Was asked for more information from a support rep. Server was auto-wiped less than an hour later.
Ended an eleven year relationship and after this fire I can't recommend them in good faith to anyone.
Had the same shit with them few months ago and also have troubles with load balancing.
I am on the step of choosing a new provider.
You still with OVH?
Maybe you could recommend something you really confident in.
I am choosing between Constellix (https://constellix.com/) and Azure (https://azure.microsoft.com/) now...
Pros, cons?
[deleted]
[deleted]
I'm glad they compartment their data centers. Having one data center for everything would be a big fail. I say this because you would be surprised at how many cloud services centralize everything in one spot.
They shouldn't advertise as 'cloud' then. I know it's a marketing buzzword and no guarantee for redundancy or anything, but still.
a 'cloud' provider should have, at the very least, multiple redundant datacenters in one availability zone.
a 'cloud' provider should have, at the very least, multiple redundant datacenters in one availability zone.
Literally no cloud does that. An availability zone is pretty much by definition a single point of failure, and you need to architect accordingly.
That's why you have multiple availability zones. Those are the physically separate locations that you have visibility into and can spread your infrastructure across. Having multiple data centers in a single availability without visibility into where your systems are is fairly useless. And if you do have visibility then you now simply have more availability zones so why bother renaming things.
Most cloud providers have availability regions and zones, with some services specifically having multi-region/multi-zone options (usually with extra cost). Is there any provider that hides this?