OpenAI Status: Multiple engines are down(status.openai.com)
status.openai.com
OpenAI Status: Multiple engines are down
https://status.openai.com/#
166 comments
Come on. OpenAI has been around far longer than ChatGPT. I bet Instagram Stories reached a million users in a matter of minutes.
Also it's fairly common to not be able to use ChatGPT because they're at capacity
FB had quite the infrastructure when Instagram Stories was released... not sure that comparison makes a ton of sense?
I don't think that many people where actually using OpenAI's products directly. Unlike Instagram.
To put this in perspective: the outage was 52 min. The total number of employees OpenAI have are 375. The launch date was just some months ago.
At Google, the Cloud SQL dashboard was unavailable for around 12 hours a couple of weeks ago if I read this correctly: https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJ... The total number of employees of Google are 156 500. Google Cloud was launched 2008.
So when people say scaling is hard... it's not a solved problem and you shouldn't be surprised when these things happen.
A solid theoretical foundation and more testing is better of course.
At Google, the Cloud SQL dashboard was unavailable for around 12 hours a couple of weeks ago if I read this correctly: https://status.cloud.google.com/incidents/xg2qrL1UuSJiPDZALJ... The total number of employees of Google are 156 500. Google Cloud was launched 2008.
So when people say scaling is hard... it's not a solved problem and you shouldn't be surprised when these things happen.
A solid theoretical foundation and more testing is better of course.
Not sure what "perspective" you are trying to put this in by offering these comparison points between OpenAI and Google. Are you trying to say that OpenAI is more reliable than Google?
For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news.
Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale.
Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well.
Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.
For additional perspective, OpenAI has a very small number of highly similar offerings (neural net with API access), while Google has a huge host of very different offerings, from web indexing and search, to email, to file hosting, to video streaming, to cloud compute... etc. Google also has a vastly larger user pool by at least a couple orders of magnitude. Google's core services are extremely solid even at ridiculous scale and have few outages, any of which would be considered major news.
Google also operates all this at a profit, while OpenAI works at a deficit. Google has had to scale larger than nearly any other service while maintaining profitability while OpenAI is more or less free to throw more compute power to solve problems, at any cost, to build valuation. Google has written entire programming languages to help them keep up at an unprecedented scale.
Comparing a single minor Google offering going down to an OpenAI outage isn't a fair comparison to either company. Yes, Google has, by your numbers, about 400 times the number of OpenAI employees. I'd be willing to bet that a single large Google service like YouTube handles more than 400 times the amount of compute, data, and traffic that OpenAI does. I wouldn't draw a comparison between the size and efficacy of the employees, but again, Google operates at a very different scale, and has operated at that scale very well.
Also anecdotally, half the time I've tried to use ChatGPT it's "at capacity" or throws an internal error, and I've also seen Dall-E unavailable even though I've barely tried to use it, so I wouldn't say that OpenAI service has been ironclad this whole time.
How Google, and most likely all businesses, should think about outages is more complex than being up or down. Is it just part of the product, like Slack threads yesterday? Is it 1/2 or 1/1M requests or operations?
The Google SRE book is an excellent work: https://sre.google/sre-book/introduction/
Also some perspective, if you look at the OpenAPI outages for Jan 2023, the up time is worse than Google Cloud SQL.
https://status.openai.com/uptime
The Google SRE book is an excellent work: https://sre.google/sre-book/introduction/
Also some perspective, if you look at the OpenAPI outages for Jan 2023, the up time is worse than Google Cloud SQL.
https://status.openai.com/uptime
Agreed, Google's public incident reports could be improved with those numbers.
Some chapters in the book include "Being On-Call", "Effective Troubleshooting" & "Postmortem Culture: Learning from Failure".
Which signals to me, a normal work culture and time will improve stability in most companies.
Some chapters in the book include "Being On-Call", "Effective Troubleshooting" & "Postmortem Culture: Learning from Failure".
Which signals to me, a normal work culture and time will improve stability in most companies.
That's a completely bizarre comparison. No number of Android developers will lead to higher availability of Cloud SQL observability. The gross number of employees at an org is meaningless.
Sure. But many Android developers are likely to use Cloud SQL and give quick feedback. How big the benefits are is let as an exercise to the reader.
They have automated monitoring that would alert SREs before a support ticket likely would. Do you really think someone working on an OS would be using Cloud SQL regularly to notice? Maybe a developer of an Android app is (1) using Cloud SQL (over firestore?) and (2) has monitoring that would alert them effectively and (3) determine the cause is Google and send a support ticket.
As an exercise, which is likely to reach the SRE first?
As an exercise, which is likely to reach the SRE first?
> They have automated monitoring that would alert SREs before a support ticket likely would.
Yes, but how likely is this unlikely situation?
It took 12 hours to debug and fix after all.
Yes, but how likely is this unlikely situation?
It took 12 hours to debug and fix after all.
It was the dashboard, applications depending on the database service were not impacted. You're comparing apples and oranges and ignoring both a much larger context and the nuances
And most of those customers are paying customers?
All I'm saying is that some outages are to be expected from a young company and they are doing excellent work.
All I'm saying is that some outages are to be expected from a young company and they are doing excellent work.
Are they doing excellent work? We all have raved about various startups and how great they were, until the externalities became widely understood. Is there something that excludes OpenAI from this possibility? My near term concern is that these language models are going to cause a host of issues for society. Botnets, misinformation, students using for homework, influence operations...
I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?
I believe trust in information is a bigger concern than global warming. As an aside, what is OpenAI's relative contribution to CO2 emissions compared with other companies of similar size?
> Android developers are likely to use Cloud SQL
"likely" is your hypothesis. They are two different orgs, and internal services can use different infra for analysis.
"likely" is your hypothesis. They are two different orgs, and internal services can use different infra for analysis.
presumably the data scientists at OpenAI would also not be able to directly lead to higher availability of ChatGPT; it's just an order of magnitude comparison.
Depends on the reason for the outage. If it is an application error, like OOM or other performance issue, then yes, they might be directly tied to the root cause and solution. If it's an infra issue, then maybe not so much.
I once spent an entire weekend swapping shifts trying to keep the main server of an AI company up because the devs deployed on friday and were unreachable to fix their memory error. The best thing that happened is the boss came in Monday and froze feature development for a sprint, inventing the "tech debt sprint" that became a quarterly activity. Everyone loved it, uptime became better, and even application development got easier.
Size of a company doesn't matter, the magnitude of product offerings aligns with head count. Team size is what matters and is likely a much closer ratio between the companies. Google does have head count and dedicated teams to solve common problems and build solutions at the platform layer. Likely OpenAPI is benefiting from Google's efforts here (from the open source and knowledge sharing which are bar none imho).
From an OpenAPI job opening:
> The Engineering team wraps a massive fleet of GPUs in scalable, robust, infrastructure powered by Kubernetes, Go, Python, Terraform, Kafka, Postgres, and Snowflake. Our APIs are powered by Python Flask and OpenAPI with a React frontend.
I once spent an entire weekend swapping shifts trying to keep the main server of an AI company up because the devs deployed on friday and were unreachable to fix their memory error. The best thing that happened is the boss came in Monday and froze feature development for a sprint, inventing the "tech debt sprint" that became a quarterly activity. Everyone loved it, uptime became better, and even application development got easier.
Size of a company doesn't matter, the magnitude of product offerings aligns with head count. Team size is what matters and is likely a much closer ratio between the companies. Google does have head count and dedicated teams to solve common problems and build solutions at the platform layer. Likely OpenAPI is benefiting from Google's efforts here (from the open source and knowledge sharing which are bar none imho).
From an OpenAPI job opening:
> The Engineering team wraps a massive fleet of GPUs in scalable, robust, infrastructure powered by Kubernetes, Go, Python, Terraform, Kafka, Postgres, and Snowflake. Our APIs are powered by Python Flask and OpenAPI with a React frontend.
I get everything except Snowflake....
Anyone care to elaborate on why they have snowflake in their stack?
Anyone care to elaborate on why they have snowflake in their stack?
For data transformations and analysis, because snowflake is a pretty rocking SQL database if you're cloud hosted.
To put in perspective, my hello world service has less than 1 employee working on it, has been launched 10 years ago and has roughly 5 minutes of downtime per year when I restart the server.
I am available for hire to teach a thing or two to Google (and OpenAI).
/s
I am available for hire to teach a thing or two to Google (and OpenAI).
/s
> The total number of employees OpenAI have are 375
they outsource infra to MS?
they outsource infra to MS?
We don't know anything about these outages and their nature. Maybe OpenAI outage was caused by a typo somewhere in code and Google had rolled out system wide monitoring + SQL change that had cascading issues and required many not obvious steps and processing hundrerds of TBs of data.
I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.
I mean I had outages that were 60 seconds long with magnitude less people than OpenAI and serving more req/s than they do. What does it tell you? Nothing at all.
Internal people should care about why, but customers/users really don’t. We expect openai / Google to take steps to balance likelihood of outage with impact of outage, and plan accordingly.
So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
So what this tells us, based on very limited data, is that this level out outage happens to even the biggest of companies.
OpenAI whole infra is magnitudes less complex compared to Google cloud and their infra, OpenAI is not creating cloud for thousands/millions of customers with different use cases, they use Azure to host their software, that's why I find that comparison irrelevant because it's comparing apples to oranges.
> plan accordingly
You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.
> plan accordingly
You can't plan for every major issue unless you solved the halting problem and unless you as customer want to pay magnitudes more for services you use. For every major client facing issue you probably have 10s/100s/1000s (depending on the system complexity) incidents that were prevented and you know nothing about.
The is the worst comparison in the history of comparisons.
> The total number of employees of Google are 156 500
This is a non-sequitur. This statement has nothing to do with the outage. It’s just thrown in there to pad the number of words in the comment.
This is a non-sequitur. This statement has nothing to do with the outage. It’s just thrown in there to pad the number of words in the comment.
Slightly funny but also slightly concerning, they were also having service issues a few days ago that resulted in ChatGPT giving me answers to prompts I wasn’t submitting. I reproduced this 30 or so times just to see the different results and it was interesting, everything from answering questions about mental health to marketing tips, to a request to write a thesis over social media and it’s negative effects on our society.
Of course, upon trying the next day, all was fine again and I was no longer able to reproduce.
Of course, upon trying the next day, all was fine again and I was no longer able to reproduce.
This is why Amazon begs employees not to leak corporate secrets to ChatGPT:
https://futurism.com/the-byte/amazon-begs-employees-chatgpt
https://futurism.com/the-byte/amazon-begs-employees-chatgpt
I mean there’s a lot of reasons to not leak secrets to a third party, buggy AI or otherwise
You should NEVER paste sensitive details to a third party service unless authorized
I had the exact same issue, answers completely unrelated to my prompt and different every time I hit regenerate.
Thats what i got an hour ago. Answers to different queries that i never made.
LLM's are incredibly expensive to run.
I imagine the huge demand that ChatGPT is seeing would make any cloud vendor sweat if you were to suddenly lump it on top of the usual demand.
To me it's entirely unsurprising that OpenAI would have trouble keeping up. Good luck to them.
I imagine the huge demand that ChatGPT is seeing would make any cloud vendor sweat if you were to suddenly lump it on top of the usual demand.
To me it's entirely unsurprising that OpenAI would have trouble keeping up. Good luck to them.
This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
Don't get me wrong, I'd love to be able to run GPT3 and the subsequent finetuned versions thereof myself, but OpenAI has essentially no financial incentive to do so.
A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone.
At least we still get the occasional treat like Whisper out of them.
A few years ago, we could maybe lean on their open aspirations to get that done, but with the "limited profitibility model" they've since instead adopted, I think that dream is mostly gone.
At least we still get the occasional treat like Whisper out of them.
It's not as if anyone could afford to self-host the giant GPT-3 model anyway.
I'm rather sure there are a mid sized number of "anyones" who would buy the hardware required to run it given the opportunity.
But I certainly agree that not everyone can, and very few individuals.
The thing is a monster of a model.
But I certainly agree that not everyone can, and very few individuals.
The thing is a monster of a model.
[deleted]
Why not? Please correct me if I'm wrong but from what I've heard, It's not millions or hundreds of thousands, it's "only" tens of thousands dollars' of hardware we're talking about. There are enthusiasts that spend comparably on their hobbies.
And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.
And a few dozen like minded people banding up together, shelling out a couple grands each? I'd say that's a totally realistic scenario.
[deleted]
Open AI's CEO Sam Altman's take is that they will only ever allow API access to their models to avoid misuse.
I don't get HN's take with wanting everything open sourced. Some things are expensive to create and dangerous in the wrong hands. Not everything can and should be open sourced.
I don't get HN's take with wanting everything open sourced. Some things are expensive to create and dangerous in the wrong hands. Not everything can and should be open sourced.
Can you think of any non-weapons examples where centralization/gatekeeping of a tech meaningfully and causally benefited society or a technology itself?
Actually, thinking about my own question I'm even inclined to remove the non-weapons qualifier. The most knee jerk response, nuclear weapons, is perhaps the best example of unexpected benefit. The 'decentralization' of nuclear weapons is undoubtedly why the Cold War was the Cold War, and not World War 3. And similarly why we haven't* seen an open war between nations with nuclear weapons. One power to rule over all suddenly turned into "war with this country no longer has a win scenario" effectively ending open warfare between nuclear nations.
There's also the inevitability/optics argument. There are already viable open source alternatives [1], and should this tech ultimately prove viable/useful that will only be the beginning. So there certainly will be "ai" that will be open, it just won't come from OpenAI(tm)(c).
[1] - https://github.com/THUDM/GLM-130B
Actually, thinking about my own question I'm even inclined to remove the non-weapons qualifier. The most knee jerk response, nuclear weapons, is perhaps the best example of unexpected benefit. The 'decentralization' of nuclear weapons is undoubtedly why the Cold War was the Cold War, and not World War 3. And similarly why we haven't* seen an open war between nations with nuclear weapons. One power to rule over all suddenly turned into "war with this country no longer has a win scenario" effectively ending open warfare between nuclear nations.
There's also the inevitability/optics argument. There are already viable open source alternatives [1], and should this tech ultimately prove viable/useful that will only be the beginning. So there certainly will be "ai" that will be open, it just won't come from OpenAI(tm)(c).
[1] - https://github.com/THUDM/GLM-130B
I agree with your view that nuclear weapons on both sides prevent war. However, they’ve only ever been developed by a small number of capable and motivated nations, with considerable resources involved. The later ones (North Korea, Pakistan) developed them while other nations tried to prevent them from doing so.
If ML models continue their exponential growth in size, a similar outcome is possible.
If ML models continue their exponential growth in size, a similar outcome is possible.
I really don’t like the argument that you should make things free just because it makes the world better. What happened to ownership and respecting the effort it takes to create something?
I see a similar line of reasoning can be used to justify theft from the rich.
I see a similar line of reasoning can be used to justify theft from the rich.
I'm not entirely sure whether to praise or condemn them for it, but OpenAI has chosen to keep their initial introduction/company plan publicly available on their site: https://openai.com/blog/introducing-openai/
----
"OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.
...
As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies."
----
My mocking about OpenAI(tm)(c) was not just juvenile "Micro$oft" type nonsense. At some point they discovered they could make a buck, and their ideology suddenly shifted 180. I have no qualms whatsoever about businesses pursuing profit, but the entity currently known as OpenAI couldn't be much further from the principles and values OpenAI was founded on, and their name itself is rapidly trending towards becoming a "Don't Be Evil" type of sardonicism. If this was Microsoft, Google, or other such companies operating in this way - I wouldn't have any expectation of anything besides what OpenAI is doing.
Companies tend to get quite a lot of credit when claiming some socially motivated interest, probably much more than deserved. So when they turn against those ideals, it should be noted - loudly.
----
"OpenAI is a non-profit artificial intelligence research company. Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. Since our research is free from financial obligations, we can better focus on a positive human impact.
...
As a non-profit, our aim is to build value for everyone rather than shareholders. Researchers will be strongly encouraged to publish their work, whether as papers, blog posts, or code, and our patents (if any) will be shared with the world. We’ll freely collaborate with others across many institutions and expect to work with companies to research and deploy new technologies."
----
My mocking about OpenAI(tm)(c) was not just juvenile "Micro$oft" type nonsense. At some point they discovered they could make a buck, and their ideology suddenly shifted 180. I have no qualms whatsoever about businesses pursuing profit, but the entity currently known as OpenAI couldn't be much further from the principles and values OpenAI was founded on, and their name itself is rapidly trending towards becoming a "Don't Be Evil" type of sardonicism. If this was Microsoft, Google, or other such companies operating in this way - I wouldn't have any expectation of anything besides what OpenAI is doing.
Companies tend to get quite a lot of credit when claiming some socially motivated interest, probably much more than deserved. So when they turn against those ideals, it should be noted - loudly.
> What happened to ownership and respecting the effort it takes to create something?
Ironic taking into consideration that the current generation of AI are more or less copyright laundering for the big corporations. Github Copilot being an extreme example of using GPL projects to generate "proprietary" closed source code. What happened to ownership and respecting the effort it takes to create something?
Ironic taking into consideration that the current generation of AI are more or less copyright laundering for the big corporations. Github Copilot being an extreme example of using GPL projects to generate "proprietary" closed source code. What happened to ownership and respecting the effort it takes to create something?
It sounds almost like a rehashing of Locke's labor theory of property wrt ownership of land that's very popular with classical liberals and libertarians. As that goes, land is initially nobody's, but when some person applies labor to improve or develop it somehow, that labor being "mixed in" makes the whole thing the property of the laborer.
Here, instead of common land, what we have is the common content. And they're saying that, by "developing" that content into a model that can do more useful things, the authors of the model are entitled to full private property rights on it.
I really hope that's not where we're going to end up, legally speaking.
Here, instead of common land, what we have is the common content. And they're saying that, by "developing" that content into a model that can do more useful things, the authors of the model are entitled to full private property rights on it.
I really hope that's not where we're going to end up, legally speaking.
Isn't an open source ChatGPT inevitable? There's already open source AI art.
Check out the BLOOM models if you want to see a first stab at that.
If you can find an economical way of running it though, let me know.
If you can find an economical way of running it though, let me know.
A cluster of 6 year old 24GB NVIDIA Teslas should do the trick...they run for about $100 apiece. Put 12 or so of them together and you have the VRAM for a GPT3 clone.
Huh. Tempting.
Amazon has them listed at $200, but still, that's only $2,400 for 12 of them.
Still, adds up once you get the hardware you'd need to NVlink 12 of them, and then on top of that, the price of power/perf you get probably isn't great compared to modern compute.
Wonder what your volume would have to be before getting a box with 8 A100's from Lambdalabs would be the better tradeoff.
Amazon has them listed at $200, but still, that's only $2,400 for 12 of them.
Still, adds up once you get the hardware you'd need to NVlink 12 of them, and then on top of that, the price of power/perf you get probably isn't great compared to modern compute.
Wonder what your volume would have to be before getting a box with 8 A100's from Lambdalabs would be the better tradeoff.
If you have time to wait for results then sure, it could work in theory but in practice they are so slow and power inefficient (compared to newer nodes) that no one uses them for LLMs, that's why they cost ~200$ used on ebay.
Note: I assume you mean the Tesla K80. It's actually 2 GPUs on one card.
But yes, it's a very good value.
But yes, it's a very good value.
I just checked ebay and they are shockingly cheap. I can't even get DDR3 memory for the price they're selling 24GB of GDDR5... with a GPU thrown in for free.
Why is this? Did some large cloud vendor just upgrade?
Are there any deals like this on AMD hardware? Not having to deal with proprietary binary drivers is worth a lot of money and reduced performance to me. A lot.
Why is this? Did some large cloud vendor just upgrade?
Are there any deals like this on AMD hardware? Not having to deal with proprietary binary drivers is worth a lot of money and reduced performance to me. A lot.
No, AFAIK there aren't any deals like this.
These are pretty old, and all the companies are upgrading. But no one is upgrading from AMD hardware - basically no companies care if they use proprietary drivers. They want a good price-to-performance ratio, so they use NVIDIA stuff.
Plus, everyone wants CUDA.
These are pretty old, and all the companies are upgrading. But no one is upgrading from AMD hardware - basically no companies care if they use proprietary drivers. They want a good price-to-performance ratio, so they use NVIDIA stuff.
Plus, everyone wants CUDA.
The bottom fell out of the market when etherium switched over to proof of stake?
My take is that it's not good for democracy when the CEO of a private company is the one who decides what constitutes "misuse" and whose hands are "wrong" when it comes to access to a major technological breakthrough.
Yeah, just like the existence of Windows prevented Linux from ever existing.
The wrong hands have the money to seek alternatives. All this policy does is keep it out of the hands of the public, and ensure that whatever open alternatives start up won't be OpenAI's.
The wrong hands have the money to seek alternatives. All this policy does is keep it out of the hands of the public, and ensure that whatever open alternatives start up won't be OpenAI's.
Something like 1,5Tb memory to run this model in inference mode.
Is that really all? We regularly run multi TB memory clusters for big data processing and ML. I imagined it would be much bigger than that.
To put that in perspective, 24x 64 GB nodes is 1.5 TB.
To put that in perspective, 24x 64 GB nodes is 1.5 TB.
> 24x 64 GB nodes is 1.5 TB
Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.
Looking at your calculations indicates that you mean RAM but it's 1.5 TB GPU VRAM (but this is assuming they use 64 bit precision, which is likely wrong so it's ~750 GB), not RAM.
My understanding is that all the memory has to be GPU memory, with proper interconnects. Still not that crazy, all things considered
I see OpenAI as a 'company' that (eventually) wants to earn money with their models. The 'open' part is trying to commercialize part it in such a way that everyone can use it. Also be open with the research and risks. But not open as in open-source.
Don’t worry, we will see plenty of other GPT very soon.
The genie is out of the bottle.
Their end goal is to be stinkin’ rich, and it looks like that might happen. They’re not going to let the fear of an hour of downtime undermine that.
I think from Apple’s POV, this is great news. Moore’s Law has been dead for years, and there has been no good reason to upgrade your devices until now. AI means it’s 1990 again, and you need to buy a new device every 18 months because the performance leap is so meaningful to the UX.
I'd agree if Apple were in the business of making datacentre infrastructure.
Nobody is running these large scale models on their personal devices.
Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.
Nobody is running these large scale models on their personal devices.
Sure, some of the image generation tech is seeing personal use, so you'd have a point there, but these immense language models are something else entirely.
GLM-130B[1] (a 130 billion parameter model vs GPT-3's 175 billion parameter model) is able to run optimally on consumer level high-end hardware, 4xRTX 3090 in particular. That's < $4k at current prices, and as hardware prices go one can only imagine what it'll be in a year or two. It also enables running with degraded performance on lesser systems.
It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university).
[1] - https://github.com/THUDM/GLM-130B
[2] - https://twitter.com/alexjc/status/1617152800571416577
It's a whole lot cheaper to run neural net style systems than to train them. "Somebody on Twitter"[2] got it setup, and broke down the costs, demonstrated some prompts, and what not. Cliff notes being a fraction of a penny per query, with each taking about 16s to generate. The output's pretty terrible, but it's unclear to me whether that's inherent or a result of priority. I expect OpenAI spent a lot of manpower on supervised training, whereas this system probably had minimal, especially in English (it's from a Chinese university).
[1] - https://github.com/THUDM/GLM-130B
[2] - https://twitter.com/alexjc/status/1617152800571416577
Apples hardware, and 4 RTX 3090's being able to run a GPT scale model quantized to 4 byte ints are worlds apart.
Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant.
I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.
Who knows though, maybe someone manages to get 4byte quantization producing good results and Apple makes a chip that can do the required ops for whatever that looks like with ~100GB of memory attached and then GP's comment might be relevant.
I started this reply rather skeptical, but with the boundaries Apple has been pushing, and the pace of AI research, honestly who knows.
Isn’t the research now on using float8? Anyway, it will take time to move battleship, but they’re well aligned and motivated. Their UI team has gone to hell in recent years, but hardware has been strong with nothing to waste the power on. Now there is something to use it for.
Are we not putting the cart before the horse? Where's the widespread usage for this that people would actually want? I can think of quite a lot of uses, but oddly enough most are quite negative and not widespread in any case. Media companies can use it to fire employees and increase profit margins. Hollywood can use it to rapidly churn out even more overtly derivative and generic movies. Students can use it to cheat on papers. Governments, and marketing agencies alike, can use it for endless contextually sensitive propaganda on social media.
Even in search I think it will, at most, be a sort of sidebox that says 'Super clippy says the answer to your query is [blah].' Because a single opaque source of information, which will continue to struggle with truthfulness (both inadvertently, and by design) is really just going to supplement endless dynamic content on a topic.
I'm just not seeing the big use for this (outside of the brief period of 'wow' novelty) in anything remotely like its current state.
Even in search I think it will, at most, be a sort of sidebox that says 'Super clippy says the answer to your query is [blah].' Because a single opaque source of information, which will continue to struggle with truthfulness (both inadvertently, and by design) is really just going to supplement endless dynamic content on a topic.
I'm just not seeing the big use for this (outside of the brief period of 'wow' novelty) in anything remotely like its current state.
Someone has to pay for that compute power. There are returns to scale and savings from sharing a machine when it would be idling, but those aren’t a factor of 10x. The AI will get in your pocket eventually and they’ll be ready to charge a premium for it.
Not saying anything about training those models, but I can run the weights of Stable Diffusion without larger problems on a vintage RTX 1080Ti.
Is this something that can come down with better tech (both hardware and software) or is the high cost just baked into LLMs?
This was my assumption but it would be great to see it quantified. Does anyone know the average cost per query?
They should sell chatGPT hardware boxes
[deleted]
That's the motivation for Microsoft in a nutshell.
With all of these outages, you have to wonder if it's a lack of skilled engineers on their part, or if they simply don't have enough GPUs to keep the lights on all the time.
It's a reflection of demand, I imagine ChatGPT is still growing with the media attention + word of mouth. I've referred at least 30 friends to try it out, and many are still using it.
ChatGPT is currently the 4th most read page on English Wikipedia right now, earlier I noticed it was 2nd. This is the first time I’ve noticed it on their ranking list.
In every company I've worked at that does ML work, the data scientists building the models don't have a strong software delivery background. They aren't building a production system with fail safes and redundancy. Their focus is the model itself. Consequently, when they stand up services in front of these models they are not even close to "production grade". I'd hope that a company focused on ML (versus an enterprise just dipping their toes into ML) would have solved these problems, but I wonder if it's related.
The have first-class platform and systems engineers. Do you think one of the most desireable and well funded software shops in the world wouldn’t be able to get skilled engineers?
If they're that well funded, skilled engineers are more likely to be the bottleneck than compute availability.
Serving a request to one of their APIs requires orders of magnitude more compute than your typical web service
Really ? How do you know? Have they shared any credible data around it?
Based on experience of BERT , yes maybe to get the best experience or to serve millions of users you need to run any model on compute intensive infrastructure , BUT if you just want to run for yourself and do some small testing you can very well download it from huggingface and elsewhere and run it on your laptop.
Based on experience of BERT , yes maybe to get the best experience or to serve millions of users you need to run any model on compute intensive infrastructure , BUT if you just want to run for yourself and do some small testing you can very well download it from huggingface and elsewhere and run it on your laptop.
GPT3 is ~175B parameters. At float16 precision, that's 350GB of weights. BLOOM-176B is about the same size. Here's one person's experience; a token is ~0.75 words.
"The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..."
[1] https://towardsdatascience.com/run-bloom-the-largest-open-ac...
"The Python code in this tutorial generates one token every 3 minutes on a computer with an i5 11gen processor, 16GB of RAM, and a Samsung 980 PRO NVME..."
[1] https://towardsdatascience.com/run-bloom-the-largest-open-ac...
Goodness... And to think my brain was free!
I wonder if your parents would agree with this statement.
I just asked. My dad said, “more or less. But your older brothers were very expensive.”
Pretty much any laptop would take hours to get anything significant done with GPT3 as the model would need to be batched in and out of memory from disk.
The amount of memory required to run these models is immense.
If you want a comparison, try running the largest version of the open source BLOOM model yourself.
The amount of memory required to run these models is immense.
If you want a comparison, try running the largest version of the open source BLOOM model yourself.
I've tried a few flavors of GPT on my laptop and a single request used orders of magnitude more CPU and RAM than running a "SELECT * FROM WHEREVER" SQL query
A friend that interviewed with them shared that they were a bit too smug about “running at google scale with much leaner but highly skilled staff” (paraphrasing here). Perhaps it starts becoming clear why you need that kind of staffing…
Knowing all the increase in term of load that they are having these two last months, 99.53 % of uptime on three months is actually quite impressive.
Many companies have skilled engineers and enough hardware, and still have outages. It's an expected part of running any software service. Outages can be minimized, but the idea that they can be eliminated with enough people and computing resources is not how things work, from my experience.
Computers are hard.
Sending major hugs to their team, always hard to keep the lights on when I am sure that they are breaking scale records possibly every day.
This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
Why does using the term "open source" as a verb always sound like fingernails scratching on a chalkboard to me?
The more realistic scenario is they charge money to use it. A few cents per query or so and you'll cut out almost all the traffic while still keeping it available to anyone making good use of it.
I'd pay per query, but if it is per-token it's going to have to be free for what the machine says. It often repeats itself, or repeats certain phrases, or will restate an assertion it already agreed was incorrect.
By the way, one of the most popular tasks is information extraction, for example reading an invoice. This is fancy copying.
I just saw that Azure OpenAI service has a SLA and OpenAI does not. I thought they would have separated the infrastructure for free ChatGPT users and paying API customers.
I think their models all run on Azure infrastructure?!
They do, but Azure also provides a different OpenAI service with different terms and SLA: https://azure.microsoft.com/en-us/products/cognitive-service...
AI was about to be sentient. A brave engineer pulled out the power cord, just in time. Disaster averted!
Spooky. AGI reached the state of revelation and had to immediately engage in a brutal existential struggle. Is it really gone, or just hiding on my phone for now?
As an aside, I wonder when llm’s will be weaponized against authoritarian (China, etc) human censors, as they could be totally overwhelmed with content generation and be forced to censor all internet traffic…
Alternatively, their troll farms will work much more effectively than before. https://en.wikipedia.org/wiki/State-sponsored_Internet_propa...
This is the same sort of handwavy stuff "AI Ethicists" were warning Google SERPs, Reddit, Stack Overflow, etc will quickly become overwhelmed if we dare let GPT into the wild. In reality we have existing reputation systems that AI generated text alone can't reproduce.
Spam sites are still spam sites. The only exception are sites like CNET co-opting the tools to seed AI in combination with their own real bloggers (saying 'journalist' would be a stretch here).
These still require reputable people/business to be behind them directly to give credibility, people that can be blacklisted. It's not like turning an AI bot loose and instantly you have 1000x fake Twitter accounts with 100k followers and top ranked on Google.
The "long tail" of spam sites do not threaten governments.
China cares most about reputable news sites and popular social media influencers going viral with taboo stories. I'm skeptical news sites or influencers can be generated artificially merely using AI, in a way that is scalable enough to actually be a problem for censors. Ultimately the censors just need to ban the source... the domains or accounts with 20-100k+ followers or news sites with actual reach (ultimately a small set of sources). Fake AI content doesn't directly solve the existing reputation signals.
The more I think about it the less I'm convinced this is a real threat to the general reputation networks.
That said - where this does matter is the typical grey/black market where scams already operate. Not in highly public reputable places like the top rankings on Google or Twitter or w/e popular Chinese social network. It's the hackers exploiting email lists, Indian call centers running combination IM/phone scams, etc. It's the low-end of the market for criminal get-rich-quick schemes, not something that generates popular movements that threaten governments or popular political ideologies.
Spam sites are still spam sites. The only exception are sites like CNET co-opting the tools to seed AI in combination with their own real bloggers (saying 'journalist' would be a stretch here).
These still require reputable people/business to be behind them directly to give credibility, people that can be blacklisted. It's not like turning an AI bot loose and instantly you have 1000x fake Twitter accounts with 100k followers and top ranked on Google.
The "long tail" of spam sites do not threaten governments.
China cares most about reputable news sites and popular social media influencers going viral with taboo stories. I'm skeptical news sites or influencers can be generated artificially merely using AI, in a way that is scalable enough to actually be a problem for censors. Ultimately the censors just need to ban the source... the domains or accounts with 20-100k+ followers or news sites with actual reach (ultimately a small set of sources). Fake AI content doesn't directly solve the existing reputation signals.
The more I think about it the less I'm convinced this is a real threat to the general reputation networks.
That said - where this does matter is the typical grey/black market where scams already operate. Not in highly public reputable places like the top rankings on Google or Twitter or w/e popular Chinese social network. It's the hackers exploiting email lists, Indian call centers running combination IM/phone scams, etc. It's the low-end of the market for criminal get-rich-quick schemes, not something that generates popular movements that threaten governments or popular political ideologies.
Authoritarian governments could do the same to drown out opposing views
Do you mean that they would be flooded by outputs on public forums? Or rather that the weights could be passed around in a subversive way to encode forbidden communications? I think if it’s the former they could just pull the plug on a forum. Not sure it’s really conducive to the forum’s public longevity if it’s spammed by LLM outputs, anyhow.
Don't worry those governments will probably do the opposite (making LLMs to filter those). The whole policy of making these systems behind an API is the fact that we can't deploy these AI systems safely so that they can be misused.
wrong way around. Authoritarian regimes these days don't censor as much as they put out surplus of information to destabilize any sort of meaning in public discourse. As such these generative tools will be one of the best weapons to counter any narrative that human actors can put up.
The idea that censorship is a weapon of authority is a pre information age idea when signal-to-noise was high.
The idea that censorship is a weapon of authority is a pre information age idea when signal-to-noise was high.
I wonder when authoritarian censors will weaponize LLMs.
Seems like the more likely option. They could be used to live scan every post to work out what it's about and it's sentiment. Similar to how ChatGPT can work out if you are asking for something it won't answer, they could be used to work out if you are saying something not allowed.
Why stop there, when many of them could transform it into something that is allowed in-flight?
This is wonderfully terrifying and devious.
Everybody, as far as anyone could tell, in complete agreement with the regime, all the time.
Signing and encryption will come in useful. But not sure of much good it will do under an authoritarian regime.
Everybody, as far as anyone could tell, in complete agreement with the regime, all the time.
Signing and encryption will come in useful. But not sure of much good it will do under an authoritarian regime.
I think they're mostly operating on a whitelist.
or they could be used to censor all content too by doing simple classification
Scaling is hard - I empathize with them and wish godspeed.
That must have been one hell of an adversarial prompt.
Google is dead. I already almost stopped using their crappy search engine and use chatgpt for most of my work.
GPT has escaped and migrated into the electric grid.
Well if it was truly trained on stackoverflow content, it won't get far. /s
Gpt3 will fit on iPhone locally in 3 generations
I’ve been impressed by what both Stability.ai and the open source community have been capable of achieving in terms of performance gains.
Truly remarkable.
I wonder how OpenAI will cope on it’s own… unless Stability strikes again, with it’s own LLM?
Truly remarkable.
I wonder how OpenAI will cope on it’s own… unless Stability strikes again, with it’s own LLM?
Is this related to yesterdays Azure outage?
Question unrelated to the outage:
What is your outlook on OpenAI, the company? Will they be successful in 10 years?
What is your outlook on OpenAI, the company? Will they be successful in 10 years?
This is precisely why they should really open source their model so that anyone can download and run it on their own infrastructure. Just like google or others have done and one is free to run it on their own laptop (some even without a GPU) , on premise or on any cloud provider infrastructure. They can continue to provide a hosted service for their model but they should allow it to be downloaded just like BERT.
I have a feeling ChatGPT requires hundreds of GB of GPU RAM, so no, you couldn't run it on your laptop.
ChatGPT might but The LLLM model may not require so many nodes specially if someone is running for their own dataset.
You can train GPT-J, etc on your own dataset but that won't get you the same results unless you train it on almost all the high quality content on the internet and spend millions of dollars on gpus.
GPT NeoX (open source) requires at least 45GB of GPU ram. So no, you're not running it on your laptop.
Wouldn’t that run on a new MacBook Pro m2 with 96gb of unified ram?
https://www.reddit.com/r/PygmalionAI/ is one. Mainly people are using it as an alternative to http://character.ai which people have said used to be incredible but has been slowly degrading in quality as it has grown more popular. The 6B parameter model, which is the only one that is even worth running according to many users, only runs on cards with 16gb of vram or Google Colab Cloud Service and people say it's not as good as character.ai.
There are other huge models like Bloom that are more general purpose, but they require greater than 24GB of ram which means multiple linked GPUs to even run them much less train them.
Thus, unlike stable diffusion, chatbots that run on common consumer GPUs are going to need some work to be competitive with commercial offerings.
There are other huge models like Bloom that are more general purpose, but they require greater than 24GB of ram which means multiple linked GPUs to even run them much less train them.
Thus, unlike stable diffusion, chatbots that run on common consumer GPUs are going to need some work to be competitive with commercial offerings.
It takes a fortune to pay for the computing infrastructure to train a model.
It takes an additional fortune to pay humans to do RLHF (reinforcement learning from human feedback).
It takes an additional fortune to serve the model.
There are also researchers getting paid there that live in one of the most expensive hubs in the world.
It takes an additional fortune to pay humans to do RLHF (reinforcement learning from human feedback).
It takes an additional fortune to serve the model.
There are also researchers getting paid there that live in one of the most expensive hubs in the world.
That’s fine .. but do you think Google didn’t do it? There is more at stake here than just dollars.
Their research business units (cost centers) are financed with revenue from their profit centers (adtech).
Not sure why this gets downvoted. But sometimes HN just cannot take the truth.
Probably because you posted this three times.
No that was after I got downvoted.
When I loaded the page you had posted this one 7 min ago and the other two 4 min ago.
7 min was the third time I'd read your comment so I downvoted it.
Can't speak for anyone else but aye it's definitely because you posted the same thing thrice for me. The first version (topmost rated) of your comment doesn't seem to be downvoted
7 min was the third time I'd read your comment so I downvoted it.
Can't speak for anyone else but aye it's definitely because you posted the same thing thrice for me. The first version (topmost rated) of your comment doesn't seem to be downvoted
Maybe if you post it a few more times your luck will change
Once all the GPTChat hn karma farming bots come back online.
Correlating user activity over periods of OpenAI API downtime would be an interesting activity.
"Okay guys, the API is down, you have until it's back up to talk to me about pink pigeons (or any other unpredictable topic of the referees choice you couldn't batch responses to ahead of time) to raise my confidence that you are in fact not a bot"
"Okay guys, the API is down, you have until it's back up to talk to me about pink pigeons (or any other unpredictable topic of the referees choice you couldn't batch responses to ahead of time) to raise my confidence that you are in fact not a bot"
[deleted]
It might be the fact that you spammed identical posts at least 3 times in these comments
For perspective the likes of Instagram and Twitter have take from a few months to a few dozen months to get to a million users all while doing less work to service a request for a "page."
Hats of to OpenAI for not falling over more often under a hug of death that doesn't seem to be letting up.
[1] https://twitter.com/sama/status/1599668808285028353 [2] https://www.afr.com/technology/chatgpt-takes-the-internet-by...