Google Sheets was down(sheets.google.com)
sheets.google.com
Google Sheets was down
http://sheets.google.com
115 comments
“ Google Drive service has already been restored for some users, and we expect a resolution for all users within the next 1 hours. Please note this time frame is an estimate and may change.” per their status page about 7 minutes ago
Same for me. Getting 'backendError' from the sheets api.
data: {
error: {
code: 503,
message: 'The service is currently unavailable.',
errors: [
{
message: 'The service is currently unavailable.',
domain: 'global',
reason: 'backendError'
}
],
status: 'UNAVAILABLE'
}
}I just want to say as someone in Finance who works in Spreadsheets all day that Google Sheets is just NOT a replacement for Excel. All the functions, size, and some of the external tools built for Excel are just not in Sheets. I know some of you say “well of course not for what you do” but there are Microsoft haters who just wont believe me when I say that.
I work at the Google office in Chelsea, New York. One day, my old manager and I were talking about the future of GSuite and how entrenced MS Office is. We were next to a south-facing window and he pointed out toward all the buildings in downtown Manhattan: "all those buildings run off of Excel."
I think the collaboration benefits of Google Sheets is high enough that it can be the default for most employees. Perhaps MS Excel should be used only by specialists, like how only a fraction of any company uses Photoshop.
I think the collaboration benefits of Google Sheets is high enough that it can be the default for most employees. Perhaps MS Excel should be used only by specialists, like how only a fraction of any company uses Photoshop.
Collaboration was certainly my driver for using Google Sheets, but most, if not all, of that functionality is in Excel Online now.
Is Excel Online still as slow as it used to be a couple of years ago? I remember reading (perhaps here on HN) about how it basically spun up a headless Excel instance for each user since they couldn't extract and decouple all the functionality necessary to run some of the more advanced features.
EDIT: https://news.ycombinator.com/item?id=17304660 (June 2018)
> Excel is a different world and a big-big elephant that won't likely be rewritten any years soon. Think of a code that is maintained over 30 years. Literally, developers are still maintaining code written in the 90s. Trying to bring Excel to the online world made MS create an architecture of html frontend which communicates with a dll session behind the scenes. As weird as it sounds, this dll lives as long as the browser has the spreadsheet opened. Think of it as MS raising VM for every excel file that is opened over the internet. While this is not very cost effective (saying the least) and not very performance friendly (hey ma! I made an understatement) this allowed them to move forward with Excel online very quickly by having a UI communicating with the bloatware dll that runs on the background in Azure. Summarizing Excel, it probably won't also be rewritten in js.
EDIT: https://news.ycombinator.com/item?id=17304660 (June 2018)
> Excel is a different world and a big-big elephant that won't likely be rewritten any years soon. Think of a code that is maintained over 30 years. Literally, developers are still maintaining code written in the 90s. Trying to bring Excel to the online world made MS create an architecture of html frontend which communicates with a dll session behind the scenes. As weird as it sounds, this dll lives as long as the browser has the spreadsheet opened. Think of it as MS raising VM for every excel file that is opened over the internet. While this is not very cost effective (saying the least) and not very performance friendly (hey ma! I made an understatement) this allowed them to move forward with Excel online very quickly by having a UI communicating with the bloatware dll that runs on the background in Azure. Summarizing Excel, it probably won't also be rewritten in js.
I have the utmost faith in MS being able to do JS/Web optimizations after what they've been able to do with Visual Studio Code.
The difference is that VS Code was built ground up entirely separate in every way from Visual Studio.
There was zero compatibility to maintain, zero code to port, etc.
There was zero compatibility to maintain, zero code to port, etc.
[deleted]
The free app in my existing email ecosystem is all it took for me
But there's probably at least one critical person in each of those buildings who uses PhotoShop.
So, you can't just get rid of PhotoShop, any more than you can just get rid of Excel.
So, you can't just get rid of PhotoShop, any more than you can just get rid of Excel.
The funny thing is, I'm a power-Photoshop user, and I find that photopea.com works reasonably well enough for me
I mean, ok lack of keyboard shortcuts and all when I'm wroking on masks, but when I'm not on a deadline it's fine
I mean, ok lack of keyboard shortcuts and all when I'm wroking on masks, but when I'm not on a deadline it's fine
A bit of a red herring but I’m not overly familiar with google sheets, I use Excel. A problem I have is others can edit formula on NFS shared documents without it being clear what’s changed.
In my mind, collaborative software is like git, with a clear commit history.
Is google sheets that collaborative? Or do you need to trust the people you share it with?
In my mind, collaborative software is like git, with a clear commit history.
Is google sheets that collaborative? Or do you need to trust the people you share it with?
You can set read and/or write permissions, as well as there is also a history which allows rolling back the sheet/viewing changes.
It’s not functions for me, you can create them and I adore google sheet for its versatility. But SPEED is the true problem. Whatever non-trivial you do, it becomes devastatingly slow.
That's where webassembly could really shine.
The part of Google Sheets that's syrupy slow is running in the cloud. Each the API calls from JavaScript Apps Scripts take on the order of tenths of a second to execute. It feels like they're all blocking remote procedure calls.
And scripts will time out and abort after just a few seconds (meaning they can only execute relatively few operations). So there is a very hard impenetrable wall blocking what you can do with scripting Google Sheets.
And scripts will time out and abort after just a few seconds (meaning they can only execute relatively few operations). So there is a very hard impenetrable wall blocking what you can do with scripting Google Sheets.
I could be described as a "Microsoft Hater" and I still prefer Excel over Google Sheets.
The trade-off of functionality and general control isn't worth for the "Share! Collaborate! Contribute!" hype.
"For what I use it for" I guess, but after going full Linux I've been on LibreOffice [or similar] alternatives. Just not Google Sheets if I can ever, ever help it.
The trade-off of functionality and general control isn't worth for the "Share! Collaborate! Contribute!" hype.
"For what I use it for" I guess, but after going full Linux I've been on LibreOffice [or similar] alternatives. Just not Google Sheets if I can ever, ever help it.
I was a heavy user of Excel for my personal life, finances, tracking, todo, everything for about 6 years because it was installed on my work computer. My timekeeping work was also 99% Excel sheets. My personal sheets were always on a USB Disk.
Now I moved to US & do not have dedicated desk, & strictly not allowed to plug anything in work pc, fireable offense. So I am back to use Google Sheets because I can access those at my phone, & Sheets is not the same. Maybe 50%, but many simple things in Excel are difficult in Sheets. The most being Calculate Formula button in Excel.
Now I moved to US & do not have dedicated desk, & strictly not allowed to plug anything in work pc, fireable offense. So I am back to use Google Sheets because I can access those at my phone, & Sheets is not the same. Maybe 50%, but many simple things in Excel are difficult in Sheets. The most being Calculate Formula button in Excel.
[deleted]
I think they are different tools:
Sheets -> Great collaborative experience for simple to moderate complexity use cases.
Excel -> The heavy artillery.
Sheets -> Great collaborative experience for simple to moderate complexity use cases.
Excel -> The heavy artillery.
I believe Sheets contains virtually all Excel functions, and while Excel supports 1M rows, Sheets supports 5M cells, so they can both outperform each other in size depending on how many columns you have.
But yes, add-on tools are going to be platform-dependent in both directions.
But yes, add-on tools are going to be platform-dependent in both directions.
Actually, in Excel, you can have unlimited rows by using power query.
Can you list some specific examples?
Joel Spolsky has some examples in his talk about excel: (https://www.youtube.com/watch?v=0nbkaYsR94c).
I particularly wish Google sheets had the tables feature.
I particularly wish Google sheets had the tables feature.
"All the functions" - I guarantee you that almost any function you want in Excel is in Sheets. Just because you don't know how to use it or find it doesn't mean it doesn't exist. The only reason you know how to use Excel better is that you have past experience with it. That's, effectively, the sunken cost fallacy.
"size" - what?
"some of the external tools" - duh? But what on earth are using using "external tools" integrated to Excel for?
"size" - what?
"some of the external tools" - duh? But what on earth are using using "external tools" integrated to Excel for?
> But what on earth are using using "external tools" integrated to Excel for?
Off the top of my head: Operation Research... i.e. serious calculations to decide where to spend a lot of money, like: Where should I build my power station? I know, for this I will get my mixed integer programming suite out. Where is that in Google sheets???
// note, I adore Sheets, but it definitely definitely cannot compete with excel for serious computation suites or even modest visualisations.
Off the top of my head: Operation Research... i.e. serious calculations to decide where to spend a lot of money, like: Where should I build my power station? I know, for this I will get my mixed integer programming suite out. Where is that in Google sheets???
// note, I adore Sheets, but it definitely definitely cannot compete with excel for serious computation suites or even modest visualisations.
The integrations in Excel are actually very important for power users. There's a reason Thinkcell can charge a couple hundred dollars per user.
Did anyone else initially misread the title as "Google Sheets is shutting down"?
Or was that just me?
Or was that just me?
You can be forgiven for that, considering how often "Google $GSERVICE is shutting down" appears on HN's front page.
Currently on HN's front page: Google's App Maker is shutting down
https://news.ycombinator.com/item?id=22160891
https://news.ycombinator.com/item?id=22160891
[deleted]
Thought for the day: spreadsheets have done more computing related to how business (and government and non-profits and schools) run, than any other programming language, including heavy lifters like COBOL or Java. Can I back that up with data? Of course not. But, having worked as a contractor in manufacturing (three different industries), university, advertising, magazine, real estate, gene sequencing, finance, and edutech, I have little doubt.
My preferred tool is python, I use javascript and/or SQL when I need to, I have dabbled in R, I have programmed in legacy applications on mainframes. Spreadsheets are more widely used than any of these, for doing computing that impacts how organizations actually run.
My preferred tool is python, I use javascript and/or SQL when I need to, I have dabbled in R, I have programmed in legacy applications on mainframes. Spreadsheets are more widely used than any of these, for doing computing that impacts how organizations actually run.
Something I think about all the time: “what is the spreadsheet equivalent for this domain?”
Would it be wrong to say that the language which the spreadsheet software is implemented in is also doing computing related to business?
Hey, it's all Assembler if you go down far enough.
It wasn't just me!
Though it was interesting that googling "google sheets down" and clicking News returned 3 results regarding winter weather bedding and 0 results about Google Sheets.
Though it was interesting that googling "google sheets down" and clicking News returned 3 results regarding winter weather bedding and 0 results about Google Sheets.
up for me now. Google SREs must be busy today!
Drive is down as well.
Seems like they got it up quickly. But would still love to read a post-mortem.
So were portions of Google Slides. I could not open some slide decks, while others were unaffected. This made me sweat as I was 15 minutes away from a very important presentation.
I always export my slides before an important presentation just in case, not gonna get burned by the cloud!
Google Docs, Drive, and Photos all seem to be affected.
Currently have to download my files from Google and work on them in Excel and Word to get work done.
Currently have to download my files from Google and work on them in Excel and Word to get work done.
class action coming up
I'm not aware of there being any SLAs for these services?
I'm hoping that was a joke. Most people don't even pay for it.
Did this affect gsuite as well? Either was I'm sure the TOS would never allow it. While you're at it suing google, might as well get a precedent set making TOS unenforceable.
Did this affect gsuite as well? Either was I'm sure the TOS would never allow it. While you're at it suing google, might as well get a precedent set making TOS unenforceable.
This did affect us on gsuite. We were receiving an error telling us we were blocked due to unusual suspicious activity coming from our ip. It was pretty misleading and took a little bit to realize that wasn't the case.
Did you have retry logic that triggered when Sheets started failing requests? Maybe that looked like a DDoS attack while the problem persisted.
Google internally use the DDos blocking system to block requests to reduce load on critical systems during outages. (It's common for systems to require reduced load during startup because caches are still filling and user requests after an outage are often more expensive to process)
user agreement should be enough, with proper attorney
Google Docs is having some issues too.
I’m having issues with web version of Google Calendar as well.
hmm I had assumed it's our firewall freaking out. Got warnings about unusual traffic that looked more well FW like.
Drive stayed up for us though
Drive stayed up for us though
Same. Gave me a good sized shock for a moment when I read it was saying the IPv4 address I was sourcing from wasn't one assigned to our company. Then I realized it was just the IP of our cloud security proxy.
[deleted]
An aside, but what do you do when your Google Sheet becomes too big?
I find things break down very quickly (based on number of collaborators or file size), and I'm writing (yet again) some type of Flask-Admin server. Retool (https://retool.com) is another option I've considered.
Any others?
I find things break down very quickly (based on number of collaborators or file size), and I'm writing (yet again) some type of Flask-Admin server. Retool (https://retool.com) is another option I've considered.
Any others?
I'm currently working on https://LightSheets.app - it's going to be a high performance spreadsheet application. Today I managed to successfully import a 5 GB CSV file with ~50 million rows of data (around 10x more than Excel or Sheets can manage). Next steps will be a serialization format for spreadsheets this size, not sure exactly how this will work yet.
What kind of limits do you find you hit regarding the number of collaborators?
What kind of limits do you find you hit regarding the number of collaborators?
Any way to subscribe to your development? I'd be willing/interested to beta test things.
Any chance of actually doing something about the response to microsoft's query about using python as a macro language?
Any chance of actually doing something about the response to microsoft's query about using python as a macro language?
Sure, mail me at [email protected] and I’ll keep you updated.
Which response was that? :)
Which response was that? :)
Love how fast your site loaded on my mobile phone. Props man.
2.39 KB landing page? A blurb above the fold not only addressing the purpose of this product but also the context that it's addressing? A digestible list of features and improvements over alternatives?
An exemplar to aspire to, for sure. Bravo!
An exemplar to aspire to, for sure. Bravo!
Really kind of you, thanks. Now my challenge is to make an app that lives up to the landing page - some of the things are definitely challenging!
[deleted]
Thank you!
Airtable is great for rapid iteration with a relational-database data model. I've seen it used as everything from a CRM/ticket-tracker to a Kanban tool. A Pro plan supports up to 50k rows per "base" (schema), fully real-time collaborative.
Salesforce is also a platform that lets you do practically anything without coding. Beware though: here lie dragons, a hellish programming experience... and incredible levels of career stability as a Salesforce consultant.
Salesforce is also a platform that lets you do practically anything without coding. Beware though: here lie dragons, a hellish programming experience... and incredible levels of career stability as a Salesforce consultant.
>An aside, but what do you do when your Google Sheet becomes too big?
A database.
A database.
I don’t understand all the posters talking about massive csv files in excel, sheets, etc.
How is that manageable or efficient and not error prone compared to a database or just csv processed with python or other code?
How is that manageable or efficient and not error prone compared to a database or just csv processed with python or other code?
Same reasons webdevs crush pages with 112 js and css files when you could, you know, just render the page like you want it in the backend and save 90% of that overhead. It's human nature to "stick with the working solution" even after it outgrows itself many times over.
You'd be very surprised at how many smaller businesses are basically run using Excel spreadsheets. Accountants that are only slightly technical absolutely adore them.
I know of a fair few banks that settle trades using Excel spreadsheets.
I know of a fair few banks that settle trades using Excel spreadsheets.
It's a familiar interface with well-understood operational primitives (copy, apply, combine, etc.). SQL/Python/writing declarative code introduces an interface which has a substantially steeper learning curve for anyone who isn't a power user, and is a step removed from the ease of use of drag-and-drop.
You move back to Excel. Every time I tried something heavyweight- it just can't handle.
This sounds snarky, but I don't intend it that way at all. If your sheet is too big for Google, then you should be using a real spreadsheet application instead.
Try Visidata https://visidata.org/
I've been fooling with some giant CSV files for model training and yeah Google Sheets and Excel just freeze up or stop importing. So what I've found really handy to play and filter those giant files is Wizard Pro if you're on a mac: https://apps.apple.com/us/app/wizard-pro/id497487206?mt=12
I had a massive monte-carlo type sheet and ended up moving a bunch of logic/calculation into attached scripts. Far more reliable and less likely to wedge.
Looks like you guys are working on some advanced use case of the sheets, is there anything I can look for such advanced usage. Like how people take more out of sheets. (I am using sheets with ifttt but nothing related to scripting in the sheets itself)
The docs for Apps Script are pretty good:
https://developers.google.com/apps-script
Start here for custom functions:
https://developers.google.com/apps-script/quickstart/custom-...
Array functions are weird in scripts, but just playing around with them should help understand how they work.
https://developers.google.com/apps-script
Start here for custom functions:
https://developers.google.com/apps-script/quickstart/custom-...
Array functions are weird in scripts, but just playing around with them should help understand how they work.
We had to move to Excel. :(
vmchale(3)
I dont remember Google's Search Engine had outages. ( May be once in the past 10 years ) Why is it all the other products, from GCP to Google Sheet had more outages?
And Google was suppose to compete with Microsoft Azure or Office 365?
And Google was suppose to compete with Microsoft Azure or Office 365?
There was a Google Search outage last September.
Eg.: https://twitter.com/iittmee/status/1173772524573908997 https://twitter.com/cryptoITGuy/status/1173773323970523136
Not even specific to Google, but a product like Search has less outages because it doesn't require authentication / authorization checks, one search requires very little network bandwidth (unlike say YouTube) and can even work if some / most / all systems needs to go into a read only mode.
Eg.: https://twitter.com/iittmee/status/1173772524573908997 https://twitter.com/cryptoITGuy/status/1173773323970523136
Not even specific to Google, but a product like Search has less outages because it doesn't require authentication / authorization checks, one search requires very little network bandwidth (unlike say YouTube) and can even work if some / most / all systems needs to go into a read only mode.
From Google's Site Reliability Engineering book[1]:
> SRE’s goal is no longer "zero outages"; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. This change makes all the difference. An outage is no longer a "bad" thing—it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.
I suspect Search has a lower error budget than Sheets.
[1] https://landing.google.com/sre/sre-book/chapters/introductio...
> SRE’s goal is no longer "zero outages"; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. This change makes all the difference. An outage is no longer a "bad" thing—it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.
I suspect Search has a lower error budget than Sheets.
[1] https://landing.google.com/sre/sre-book/chapters/introductio...
This interested me enough to read some of that chapter. Here are a few quotes that give more context:
> Traditional operations teams and their counterparts in product development thus often end up in conflict, most visibly over how quickly software can be released to production. At their core, the development teams want to launch new features and see them adopted by users. At their core, the ops teams want to make sure the service doesn’t break while they are holding the pager. Because most outages are caused by some kind of change—a new configuration, a new feature launch, or a new type of user traffic—the two teams’ goals are fundamentally in tension.
...
> The use of an error budget resolves the structural conflict of incentives between development and SRE. SRE’s goal is no longer "zero outages"; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. This change makes all the difference.
...
> ...the decision to stop releases for the remainder of the quarter once an error budget is depleted might not be embraced by a product development team unless mandated by their management.
> Traditional operations teams and their counterparts in product development thus often end up in conflict, most visibly over how quickly software can be released to production. At their core, the development teams want to launch new features and see them adopted by users. At their core, the ops teams want to make sure the service doesn’t break while they are holding the pager. Because most outages are caused by some kind of change—a new configuration, a new feature launch, or a new type of user traffic—the two teams’ goals are fundamentally in tension.
...
> The use of an error budget resolves the structural conflict of incentives between development and SRE. SRE’s goal is no longer "zero outages"; rather, SREs and product developers aim to spend the error budget getting maximum feature velocity. This change makes all the difference.
...
> ...the decision to stop releases for the remainder of the quarter once an error budget is depleted might not be embraced by a product development team unless mandated by their management.
> An outage is no longer a "bad" thing—it is an expected part of the process of innovation, and an occurrence that both development and SRE teams manage rather than fear.
As a user, outages are always bad things. That Google's SRE team thinks otherwise is chilling.
As a user, outages are always bad things. That Google's SRE team thinks otherwise is chilling.
Of course outages are bad. But if the choice is "our service occasionally goes down" and "we never release new features", you may accept the risk of occasionally going down.
So yeah, outages are always bad, but the alternatives can be worse.
So yeah, outages are always bad, but the alternatives can be worse.
> if the choice is "our service occasionally goes down" and "we never release new features", you may accept the risk of occasionally going down.
I don't think that I would, because I don't accept the premise of that being the necessary choice. It's just the choice that the providers deign to offer for economic reasons.
But my objection isn't that there should be zero downtime. My objection is the idea that a service provider considers any downtime to be acceptable.
I don't think that I would, because I don't accept the premise of that being the necessary choice. It's just the choice that the providers deign to offer for economic reasons.
But my objection isn't that there should be zero downtime. My objection is the idea that a service provider considers any downtime to be acceptable.
> My objection is the idea that a service provider considers any downtime to be acceptable.
If you don't view any downtime to be acceptable, the logical thing to do is invest all of your resources into reducing downtime. This means solely investing in reliability infrastructure, redundancy, and making few or no changes to the system, since change introduces failure.
Since no service does that, the logical conclusion is that very few people actually consider any downtime unacceptable. Broadly speaking, I can think of literally no service that advertises "zero downtime". Cold storage gets close, but even they offer a measly 12 or 16 9s of reliability.
In other words, reliability is a business goal, much like any other business goal. Trying to achieve "perfect" reliability with limited resources isn't a good time. So looking at error budgets empowers SREs. You can go to leaders and say "hey we're exceeding our error budget, so we not making any more changes and only working on reliability until we're back within our agreed reliability."
If you don't view any downtime to be acceptable, the logical thing to do is invest all of your resources into reducing downtime. This means solely investing in reliability infrastructure, redundancy, and making few or no changes to the system, since change introduces failure.
Since no service does that, the logical conclusion is that very few people actually consider any downtime unacceptable. Broadly speaking, I can think of literally no service that advertises "zero downtime". Cold storage gets close, but even they offer a measly 12 or 16 9s of reliability.
In other words, reliability is a business goal, much like any other business goal. Trying to achieve "perfect" reliability with limited resources isn't a good time. So looking at error budgets empowers SREs. You can go to leaders and say "hey we're exceeding our error budget, so we not making any more changes and only working on reliability until we're back within our agreed reliability."
I think I have utterly failed to successfully convey the point I was trying to make.
My point was not that I expect zero downtime or perfect reliability. My point is that I expect that companies don't consider downtime to be an acceptable and normal thing.
My point was not that I expect zero downtime or perfect reliability. My point is that I expect that companies don't consider downtime to be an acceptable and normal thing.
> My point is that I expect that companies don't consider downtime to be an acceptable and normal thing.
And my point is that if a company isn't doing this, they're idiots. SRE is entirely about planning for downtime. You have incident response procedures to minimize downtime when problems happen. You have tools like error budgets to make explicit your organizational goals. But all of these are predicated on the assumption that incidents (and downtime) are a "normal" thing that will happen.
Again, if SRE's goal is solely to minimize downtime at the cost of other organizational priorities, there's a very simple way to do that: disallow all new features and maintain the same app today. That would easily cut outages for most apps by a factor of ten.
> My point is that I expect that companies don't consider downtime to be an acceptable
So you think its unacceptable to have an SLA? That's a very common way of making explicit the amount of downtime the organization feels is acceptable. This kind of error budgets is just a non-public SLA that's used to guide development, as opposed to pay people. I'm curious what companies you use that publish 100% uptime guarantees, or similar SLAs.
And my point is that if a company isn't doing this, they're idiots. SRE is entirely about planning for downtime. You have incident response procedures to minimize downtime when problems happen. You have tools like error budgets to make explicit your organizational goals. But all of these are predicated on the assumption that incidents (and downtime) are a "normal" thing that will happen.
Again, if SRE's goal is solely to minimize downtime at the cost of other organizational priorities, there's a very simple way to do that: disallow all new features and maintain the same app today. That would easily cut outages for most apps by a factor of ten.
> My point is that I expect that companies don't consider downtime to be an acceptable
So you think its unacceptable to have an SLA? That's a very common way of making explicit the amount of downtime the organization feels is acceptable. This kind of error budgets is just a non-public SLA that's used to guide development, as opposed to pay people. I'm curious what companies you use that publish 100% uptime guarantees, or similar SLAs.
> So you think its unacceptable to have an SLA? That's a very common way of making explicit the amount of downtime the organization feels is acceptable.
Perhaps we mean different things by "acceptable". SLAs are a promise that downtime won't exceed certain levels. They are not a declaration that downtime is "acceptable", only that it's inevitable and is an attempt to characterize that inevitability.
What I mean is that when downtime happens, nobody at the company should be think "this is fine". They should be very concerned and engaging in urgent and speedy resolution to the problem.
The idea that a service is expecting and accepting downtime as part of normal operation and, even worse, as part of some sort of tradeoff with regards to developing new features is just bizarre and unacceptable to me.
It indicates a level of unconcern about customer needs and experience that renders the service untrustworthy.
Perhaps we mean different things by "acceptable". SLAs are a promise that downtime won't exceed certain levels. They are not a declaration that downtime is "acceptable", only that it's inevitable and is an attempt to characterize that inevitability.
What I mean is that when downtime happens, nobody at the company should be think "this is fine". They should be very concerned and engaging in urgent and speedy resolution to the problem.
The idea that a service is expecting and accepting downtime as part of normal operation and, even worse, as part of some sort of tradeoff with regards to developing new features is just bizarre and unacceptable to me.
It indicates a level of unconcern about customer needs and experience that renders the service untrustworthy.
But again, this just acknowledges reality. You only have a finite number of employees. If you aren't devoting all of them to reliability and stability, you're making a trade off with feature velocity.
Being aware of that trade off is more organizationally mature than not
> What I mean is that when downtime happens, nobody at the company should be think "this is fine". They should be very concerned and engaging in urgent and speedy resolution to the problem.
If you think this, you've entirely misunderstood. Error budgets aren't about outages when they happen. Individual outages should be dealt with quickly and without delay. But when making planning decisions for the next year or quarter, that's when error budgets matter.
Being aware of that trade off is more organizationally mature than not
> What I mean is that when downtime happens, nobody at the company should be think "this is fine". They should be very concerned and engaging in urgent and speedy resolution to the problem.
If you think this, you've entirely misunderstood. Error budgets aren't about outages when they happen. Individual outages should be dealt with quickly and without delay. But when making planning decisions for the next year or quarter, that's when error budgets matter.
"Chilling" is a bit of an overstatement, no? Maybe if we were talking about ATC systems or medical device firmware, etc., it would be more fitting.
You could set the failure budget really low if you never want any new features, but many users do want new releases now and then.
Office 365 has multiple outages daily, so are you sure that was a good example?
Well, actually Google search engine "outages" are fairly common, they are just less visible.
Examples:
Single data center outage:
Search -> load balancers handle that just fine
Cloud -> zone or sub-zone outage
Backend failure:
Search -> cache should handle that
Cloud -> outage
Data update failure:
Search -> stale search results
Cloud -> outage
Office 365 also has issues, they can be minimized but cannot be eliminated: https://twitter.com/msft365status?lang=en
Examples:
Single data center outage:
Search -> load balancers handle that just fine
Cloud -> zone or sub-zone outage
Backend failure:
Search -> cache should handle that
Cloud -> outage
Data update failure:
Search -> stale search results
Cloud -> outage
Office 365 also has issues, they can be minimized but cannot be eliminated: https://twitter.com/msft365status?lang=en
Without knowing anything about the internals, I would assume that Sheets is significantly more stateful than search.
I wouldn't notice Google rerouting my search queries to a slightly stale cluster in a different data center, but they can hardly serve me someone else's spreadsheets until consistency has been restored.
I wouldn't notice Google rerouting my search queries to a slightly stale cluster in a different data center, but they can hardly serve me someone else's spreadsheets until consistency has been restored.