Google Cloud (Europe-west2) is experiencing downtime(status.cloud.google.com)
status.cloud.google.com
Google Cloud (Europe-west2) is experiencing downtime
https://status.cloud.google.com/incidents/fmEL9i2fArADKawkZAa2
68 comments
Summary: Cooling related failure in one of our buildings that hosts zone europe-west2-a for region europe-west2. The other zones are not impacted
I guess it's worth pointing out that europe-west2 is in London, which is currently at +35°C and was as high as +40°C at one point earlier today.
Equinix LD4/5/6 (Slough, Greater London) is the datacentre affected
It's a bit of a stretch to call europe-west2 "London" when it's Slough
It's also a bit of a stretch to call it "Google" when it's an Equinix datacenter. I know this is standard across all major cloud providers, and call me old-fashioned, but personally I'd prefer a bit more accuracy in naming of each region. Instead there is obfuscation. It's weirdly difficult to find a list of which datacenters correspond to which cloud region.
how about Slough-on-Thames?
So N+1 is not enough? https://www.equinix.com/data-centers/europe-colocation/unite...
Valid point, but I'm assuming that they never imagined 40-degree (outside) temperatures lasting multiple hours as a threat. Slough just got hit around 40.2 degrees, first time since records began.
Probably better to point out to move to Finland (europe-north1) as it's only 18 degrees.
If everyone here in the south follows this advice., Finland will sink
Belgium and Finland have better capacity than London/Zurich.
If it’s a single availability zone (which I presume west2-a is), then why are so many services down and recommending moving to other regions?
I hope there is a public postmortem that explains this. I have Cloud SQL instances with high availability enabled that have been down for over 5 hours with no workaround, when (according to docs) they should have failed over to the unaffected zones within minutes.
GKE user in that region, we are down.
I think it's deceptive to say it's a single zone. We have VMs across three zones, but anything stateful on zone a is stranded, we can't get the volumes back. It's not just that the zone is down, it's that services are not easy to fail out of the impacted zone or move that's the problem.
Kind on implies regional isn't good enough to secure high availability. You need multi regional deployments. Good luck with that when the bill comes due.
I think it's deceptive to say it's a single zone. We have VMs across three zones, but anything stateful on zone a is stranded, we can't get the volumes back. It's not just that the zone is down, it's that services are not easy to fail out of the impacted zone or move that's the problem.
Kind on implies regional isn't good enough to secure high availability. You need multi regional deployments. Good luck with that when the bill comes due.
No you just need regional PD
Nice, another entry for this month's https://outage.bingo/
Can't believe reddit hasn't had outage. They sure go down randomly.. Maybe that's micro outage that this doesn't consider?
Looks like I got a bingo on my first try: https://outage.bingo/?month=9&seed=10970
Note that you can use alphanumerical seeds too, for example your HN username https://outage.bingo/?month=9&seed=nemothekid
Atlassian should absolutely be on this, feels like every morning I sit down at my desk and get a 504 from hosted JIRA.
Lost AWS last Sunday as well in same region due to AC failure. Hot here :)
Has knock on effects for WPEngine which appears to use GCP (https://wpenginestatus.com/incidents/381158)
They aren't the only hosting company using GCP. GCP has been very aggressive targeting hosting partners.
Kinsta: https://status.kinsta.com/
SiteGround: https://twitter.com/siteground/status/1270492907078914052?la...
Both having same issue
Kinsta: https://status.kinsta.com/
SiteGround: https://twitter.com/siteground/status/1270492907078914052?la...
Both having same issue
I have HA Cloud SQL instances in europe-west2-a that are stuck in failover state but haven't failed over.
I assume this is due to the heatwave - the UK has broken temperature records today, hitting over 40°C (104°F) in London for the first time.
Oracle Cloud is suffering similar outages due to cooling infrastructure issues: https://ocistatus.oraclecloud.com/
I assume this is due to the heatwave - the UK has broken temperature records today, hitting over 40°C (104°F) in London for the first time.
Oracle Cloud is suffering similar outages due to cooling infrastructure issues: https://ocistatus.oraclecloud.com/
AWS London had an outage last Sunday too.
I’ve just lost my evening due to the Oracle Cloud, not their fault to be fair, and availability was good, but our apps weren’t happy with the performance.
What's crazy to me is that a bunch of servers in AC cooled room consumes more power than entire villages in Europe suffering from this heat.
Imagine being on your phone tweeting, instagram stories, whatsapp messages in the blistering weather while the databases and servers receiving and delivering data sits comfortably in AC cooled buildings. Does anybody find this contrast just eerie?
Imagine being on your phone tweeting, instagram stories, whatsapp messages in the blistering weather while the databases and servers receiving and delivering data sits comfortably in AC cooled buildings. Does anybody find this contrast just eerie?
And they have AC, whereas many private homes in villages have never had air conditioning.
Imagine if our workloads only did the minimum calculations necessary to achieve the associated user stories, instead of constantly rebuilding from massive dependency trees and all the other inefficiencies we put up with because it takes less dev effort that way and our performance budgets are being met either way.
Most places have emissions requirements for vehicles and so forth. Maybe some day we'll demand energy-efficient programming.
Most places have emissions requirements for vehicles and so forth. Maybe some day we'll demand energy-efficient programming.
> constantly rebuilding from massive dependency trees and all the other inefficiencies we put up with because it takes less dev effort that way and our performance budgets are being met either way
I've been trying to minimise redundant calculations in our CI/CD workflows, but none of our tools seem to like that.
Docker rebuilds all downstream if one bit of a step changes, GitHub Actions doesn't let you cache on local runners and requires uploading everything to their size-limited cloud cache.
I'm looking for better ways but of course no service advertises their energy efficiency. That tells something about the industry's priorities.
I've been trying to minimise redundant calculations in our CI/CD workflows, but none of our tools seem to like that.
Docker rebuilds all downstream if one bit of a step changes, GitHub Actions doesn't let you cache on local runners and requires uploading everything to their size-limited cloud cache.
I'm looking for better ways but of course no service advertises their energy efficiency. That tells something about the industry's priorities.
A tax on energy effectively incentiviszes efficient use of compute.
Today cloud providers deliberately site their facilities to make use of the cheapest tax-free electricity, which is effectively tax dodging.
Today cloud providers deliberately site their facilities to make use of the cheapest tax-free electricity, which is effectively tax dodging.
I would take this further and find out what is the most energy intensive computing task across these DCs and attack those first. This is a huge unknown for me as I have no statistics on what activities result in the most energy usage in these DCs but if I have to guess, it is probably GPU/Database.
I would also tax personal data. Current regulatory framework strangely assumes that personal data has no value yet entire companies are built around monetizing and claiming it as theirs. Those companies are also the biggest consumers of these DCs.
Even by taxing personal data that is being used by companies, you could realize significant energy savings as they have to discern between what they need and what they dont. Asking for someones driver license to verify their account? Tax them yearly. Suddenly companies that dont really need it find themselves searching for more energy efficient solutions since their tax dollars depend on this activity.
Unfortunately, we may not see this happen for quite some time.
I would also tax personal data. Current regulatory framework strangely assumes that personal data has no value yet entire companies are built around monetizing and claiming it as theirs. Those companies are also the biggest consumers of these DCs.
Even by taxing personal data that is being used by companies, you could realize significant energy savings as they have to discern between what they need and what they dont. Asking for someones driver license to verify their account? Tax them yearly. Suddenly companies that dont really need it find themselves searching for more energy efficient solutions since their tax dollars depend on this activity.
Unfortunately, we may not see this happen for quite some time.
> I would take this further and find out what is the most energy intensive computing task across these DCs
Nearly all compute power in the world is used doing things that theoretically take no compute. The reason they use so much compute is it was easier to pay for that compute than pay some engineer to figure out how to use less.
Nearly all compute power in the world is used doing things that theoretically take no compute. The reason they use so much compute is it was easier to pay for that compute than pay some engineer to figure out how to use less.
This makes a pretty big assumption that the only energy consumed by software is in what's required to execute the code.
That doesn't compute, because energy consumed by software is literally in what's required to execute the code by definition.
I believe the parent is talking about all the power overhead that's used even when there are no instructions running.
Simple statistics can be deceiving. Just because there are no 40C+ temperature records at some location it doesn't mean that won't change next year. Southern Europe has been learning it the hard way for a while.
luckily not one of the largest regions:
https://github.com/patmyron/cloud/#ip-addresses-per-region
https://github.com/patmyron/cloud/#ip-addresses-per-region
It's pretty bad -- I can't really find anything that works. VMs, storage, everything is hard down.
It's kinda bad, because cooling infrastructure when stretched can still cool some stuff, just less...
So they ought to be able to still keep some services up for some users.
They should have already practiced that as part of their emergency procedures, because they'd need to do the same is they had a severe limit on power use (eg. Due to a nationwide electricity or fuel shortage).
So they ought to be able to still keep some services up for some users.
They should have already practiced that as part of their emergency procedures, because they'd need to do the same is they had a severe limit on power use (eg. Due to a nationwide electricity or fuel shortage).
> So they ought to be able to still keep some services up for some users.
Wouldn't that depend on what exactly broke here? If the cooling of the power system failed, they might end up without power for example, impacting everything.
Wouldn't that depend on what exactly broke here? If the cooling of the power system failed, they might end up without power for example, impacting everything.
I suspect it wasn't that the cooling broke, but that the cooling couldn't keep up. Ie. The cooling could only remove XX kilowatts of heat, but that the equipment in a particular room was producing more than that.
I have to say, one thing about the U.S. is that there is AC everywhere - even up north. We have AC in elevators, on the busses - everywhere. Europe needs to start adjusting to the new normal of hot, hot summers.
It's a product of America's climate being more extreme on the highs and lows (a product of the jetstream). The one place that doesn't have extreme summers, the PNW, actually has very little AC too.
You say this as if it was not one of cause for hot hot summers
Air conditioning emits less CO2 than heating. Emissions would be lower if europe was hotter and everyone had AC
Air conditioning consumes a ton of energy. About 10% of all electricity consumed worldwide. https://www.iea.org/reports/the-future-of-cooling It emits about 4% of greenhouse gases https://www.sciencedirect.com/science/article/abs/pii/S25424... But the volume itself is not meaningful - the gases used by AC units are extremely powerful - up to 750 times the greenhouse effect of CO2. And of course, the AC unit itself heats the city - which is why they get much hotter than the countryside during heat waves, further exacerbating the demand for AC and making the problem even worse. It's a vicious circle.
> if europe was hotter
How do you suggest achieving that? Other than global warming?
> if europe was hotter
How do you suggest achieving that? Other than global warming?
> the gases used by AC units are extremely powerful - up to 750 times the greenhouse effect of CO2.
They are sealed systems. They only emit gasses when breached. Or when not properly disposed of.
> And of course, the AC unit itself heats the city
A few watts - whatever power it requires to operate. Say it uses 1.5kw. That's the max it will heat up the city. It will of course move 4x this amount of heat(maybe more!), being a heat pump. But the 'new' heat is only what's imported by the power grid. The rest of the heat was already in the city.
It's also worth noting that it's often quite sunny when it's hot. If you power AC units with sunlight, you are actually not introducing any more heat to the system at all. That sunlight would already hit your city and be converted into heat, if you use some of it to move heat around, you are not increasing the overall heat level.
They are sealed systems. They only emit gasses when breached. Or when not properly disposed of.
> And of course, the AC unit itself heats the city
A few watts - whatever power it requires to operate. Say it uses 1.5kw. That's the max it will heat up the city. It will of course move 4x this amount of heat(maybe more!), being a heat pump. But the 'new' heat is only what's imported by the power grid. The rest of the heat was already in the city.
It's also worth noting that it's often quite sunny when it's hot. If you power AC units with sunlight, you are actually not introducing any more heat to the system at all. That sunlight would already hit your city and be converted into heat, if you use some of it to move heat around, you are not increasing the overall heat level.
> They are sealed systems. They only emit gasses when breached. Or when not properly disposed of.
And mechanical devices having leaks is a folklore tale told to scare children?
> A few watts - whatever power it requires to operate. Say it uses 1.5kw. That's the max it will heat up the city.
Yeah. 1500W. Times the number of AC units in the city. That's what adds up to the temperature difference between the city and the countryside.
> If you power AC units with sunlight...
...with a solar panel above your own home, with no inefficiency in the electrical grid, the solar panel, or the AC unit itself. Please, this isn't a high school physics exam, you can't assume that there is no friction or that the AC unit is spherical. This is real life.
And mechanical devices having leaks is a folklore tale told to scare children?
> A few watts - whatever power it requires to operate. Say it uses 1.5kw. That's the max it will heat up the city.
Yeah. 1500W. Times the number of AC units in the city. That's what adds up to the temperature difference between the city and the countryside.
> If you power AC units with sunlight...
...with a solar panel above your own home, with no inefficiency in the electrical grid, the solar panel, or the AC unit itself. Please, this isn't a high school physics exam, you can't assume that there is no friction or that the AC unit is spherical. This is real life.
Take a look at heating if you want to see staggering energy consumption numbers. In a majority of places, people don't use heat pumps with a COP of say 4-5, but convert chemical energy with a cop of 0.5-0.9.
[deleted]
> That's what adds up to the temperature difference between the city and the countryside.
No, most of the difference is albedo difference meaning urban areas capture more radiant solar energy.
No, most of the difference is albedo difference meaning urban areas capture more radiant solar energy.
> They are sealed systems. They only emit gasses when breached.
Aircons in cars tend to need regassed every one or two years. (I'm not sure why car aircons need regassed more often than domestic aircons.)
Aircons in cars tend to need regassed every one or two years. (I'm not sure why car aircons need regassed more often than domestic aircons.)
> It emits about 4% of greenhouse gases [...] But the volume itself is not meaningful - the gases used by AC units are extremely powerful - up to 750 times the greenhouse effect of CO2.
Do AC units emit these gases? Aren't you mixing up CO2 from electricity production and gases that stay internal to the unit?
Do AC units emit these gases? Aren't you mixing up CO2 from electricity production and gases that stay internal to the unit?
They have two types of emissions: CO2 emissions from electricity production, and refrigerant gases that are supposed to stay inside the unit. But of course, units have leaks.
Many refrigerants have a greenhouse effect of literally zero. It’s safe and legal for me to vent my refrigerator because it’s filed with isobutane.
Good for you, but at least in the US, R600a (isobutane) is not used in AC units.
Besides, in the early XXth century, it was okay to vent CFC into the atmosphere. We didn't know that it destroyed the ozone layer. What makes you think that we won't discover something similar about isobutane in a few decades, besides wishful thinking?
Besides, in the early XXth century, it was okay to vent CFC into the atmosphere. We didn't know that it destroyed the ozone layer. What makes you think that we won't discover something similar about isobutane in a few decades, besides wishful thinking?
We study the greenhouse potential of gasses and have found r600a to have a potential of zero, in that it can not be ozone depleting.
We are under a smog alert atm. "caused by high ozone concentration"
https://www.rivm.nl/en/news/smog-alert-in-force-throughout-n...
https://www.rivm.nl/en/news/smog-alert-in-force-throughout-n...
... if that's true, that's a pretty mind-boggling fact considering what the common bias is.
Reversible heat pumps, not just AC.
I read 80% of Texas having air conditioning used as a political argument to ignore the climate.
Yeah indeed, Europe (and the rest of the planet) needs Americans to put the ACs away. Passive cooling is a medieval invention.
Aren't they adjusting customer thermostats, deferring plant maintenance, appealing people to conserve electricity and warning about blackouts in Texas precisely because AC is everywhere?
Are you trying to say American multinationals build their European data centers with less AC than their American data centers?