EleutherAI announces it has become a non-profit(blog.eleuther.ai)
blog.eleuther.ai
EleutherAI announces it has become a non-profit
https://blog.eleuther.ai/year-two-preface/
108 comments
You're in luck! EleutherAI has trained and released open source weights of several LLMs, including GPT-Neo (2.7B parameters), GPT-J (6B parameters), and GPT-NeoX (20B parameters). This last model is currently tied for second on the list of the largest open source LLMs in the world.
We also developed VQGAN-CLIP and CLIP-Guided Diffusion, techniques for doing text-to-image synthesis that don't require training and can easily be run locally for inference.
We also developed VQGAN-CLIP and CLIP-Guided Diffusion, techniques for doing text-to-image synthesis that don't require training and can easily be run locally for inference.
I don't want to be too cynical, but OpenAI used to be more open too until they decided releasing weights was too dangerous (/not profitable enough?), what guarantee is there that Eleuther doesn't also close their doors at some point?
Yes but in their wake they still left great contributions (CLIP, whisper, etc). So far they still seem net positive, even if they’re now for profit
I fail to see creating and releasing models as black boxes and strictly monetizing them while harvesting public data under the name of academic research as net positive.
They suck in public data, create models, let them loose, let hyped people improve them and profit off of data and people's free work.
IOW, a mechanical Turk which doesn't pay.
They suck in public data, create models, let them loose, let hyped people improve them and profit off of data and people's free work.
IOW, a mechanical Turk which doesn't pay.
I mean, ultimately there isn’t one. I’m just providing examples of how we fulfill the things that the OP says they want, as they seem unaware of our work.
But I’m confused by the anti non-profit vibes in this comment section. We aren’t saying that becoming a non-profit makes us ethical people, that would be a silly argument. But people do realize that the alternative would be to become a for-profit entity right?
We’re still the same community-driven open collaborative research lab we’ve always been. But incorporating allows us to do things like hire full time staff, enter organizationally binding legal agreements, and protect our members. Between the options of becoming a for-profit and a non-profit, the later seems clearly better suited for our goals.
But I’m confused by the anti non-profit vibes in this comment section. We aren’t saying that becoming a non-profit makes us ethical people, that would be a silly argument. But people do realize that the alternative would be to become a for-profit entity right?
We’re still the same community-driven open collaborative research lab we’ve always been. But incorporating allows us to do things like hire full time staff, enter organizationally binding legal agreements, and protect our members. Between the options of becoming a for-profit and a non-profit, the later seems clearly better suited for our goals.
I think people are extrapolating from their experience with OpenAI, which is understandable albeit unfair to you. "Open"AI burned a lot of bridges with their bait-and-switch, so developers are weary of trusting another company making similar claims. Personally, I welcome any actually open company, and I will continue to chastise OpenAI for their duplicity every chance I get.
There is no guarantee, but it sounds like if they didn't incorporate they wouldn't be doing much of anything, public or not.
[deleted]
Sam doing a 180 on non-profit to for-profit was egregious, IMHO. Having said that, plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.
Also: https://twitter.com/elonmusk/status/1626516035863212034
Also: https://twitter.com/elonmusk/status/1626516035863212034
It's disruptive technology and an enormous amount of value is likely to be created. I would much rather the profits go to a nonprofit which can distribute them charitably rather than them giving them away freely to the megacorps which actually have the resources to run these massive models (edit^2: at scale). I do think the 100x cap on return on Microsoft's $1B investment was way too high (we don't know what the cap is on later investments but it's meant to be reducing over time).
Edit: Plus it's still the case that OpenAI aren't putting profit above all else. E.g. they just released the ChatGPT API and it's 10x cheaper per token than GPT3! And their Charter does count for something, it's binding on them.
Edit: Plus it's still the case that OpenAI aren't putting profit above all else. E.g. they just released the ChatGPT API and it's 10x cheaper per token than GPT3! And their Charter does count for something, it's binding on them.
A nonprofit wouldn't have to distribute profits to charity. It just can't distribute profits to private or public shareholders (because it can't have them). A nonprofit is obligated to reinvest any earnings towards its mission.
Not remotely an expert, this is just my basic understanding. "Nonprofit" is a pretty abused term.
Not remotely an expert, this is just my basic understanding. "Nonprofit" is a pretty abused term.
Not-profit just means it doesn't have shareholders in a traditional sense. But it can have real-estate, it can pay very high salaries, travel, distribute significant bonuses, buy expensive hardware, etc.
TIL Amazon was basically a non profit
No, Amazon has shareholders, nonprofits do not.
The use of the word “basically” means that the previous commenter is saying that Amazon operated like a non-profit because it re-invested the vast majority of its profits.
Understand and disagree, since there’s a fundamental difference between Amazon and a nonprofit — both in their intended missions and the literal outcomes.
I just looked up the the top CEO salaries of non-profit organizations in the USA - as of 2020, they ranged from about $16 million to $8 million. A non-profit appears to simply be a 'non-stock corporation' and can have the same ridiculouly pyramidal compensation structures that shareholder corporations do.
Overhauling the legal definition of non-profit to require the highest-paid employees to make no more than ~10X as much as the lowest-paid employees would make a lot of sense.
Overhauling the legal definition of non-profit to require the highest-paid employees to make no more than ~10X as much as the lowest-paid employees would make a lot of sense.
> Overhauling the legal definition of non-profit to require the highest-paid employees to make no more than ~10X as much as the lowest-paid employees would make a lot of sense.
Very easy to work around by outsourcing most operations. If you really want to prevent high salaries there you'd need to link it to some more fixed measure, such as a multiple of median salary.
Very easy to work around by outsourcing most operations. If you really want to prevent high salaries there you'd need to link it to some more fixed measure, such as a multiple of median salary.
A non-profit just means excess earnings are reinvested back into the organization instead of having the option of giving them to shareholders.
I think you are confusing non-profit with charity perhaps?
I think you are confusing non-profit with charity perhaps?
CEO salaries alone are the wrong metric. Not that I'm defending them at all (they're ridiculous) but another way to look at it:
As one example, the CEO of Kaiser was the highest paid nonprofit CEO at ~$18m[0] in 2021. That seems egregious until you realize Kaiser had $93b in revenue in 2021[1].
As one example from healthcare, the CEO of Moderna was paid $18m in 2021[2]. Moderna reported $18b in revenue in 2021[3].
You see this over and over in discussions about nonprofits, charities, etc. In most cases the executive leadership team is running a huge organization - in the case of Kaiser nearly $100b in revenue and 300k employees. Competent leadership at that scale is expensive.
In an ideal world this wouldn't be the case and individuals with these qualifications would be willing to take even more significant pay cuts for the "greater good" of whatever the mission of the nonprofit it. Per usual that's not the world we live in.
Speaking personally, I'd give pause to working with, donating to, etc an organization doing $100b in revenue with an executive team making 1/20 (or whatever) they would make elsewhere. In some limited cases for those who have "made their money" it would be a great thing. For anyone else I would assume they're either taking a HUGE pay cut for the cause (great thing, but unlikely) or completely incompetent, embezzling, etc.
[0] - https://www.erieri.com/blog/post/top-10-highest-paid-ceos-at...
[1] - https://about.kaiserpermanente.org/news/kaiser-foundation-he...
[2] - https://thehill.com/business-a-lobbying/business-lobbying/34...
[3] - https://investors.modernatx.com/news/news-details/2022/Moder....
As one example, the CEO of Kaiser was the highest paid nonprofit CEO at ~$18m[0] in 2021. That seems egregious until you realize Kaiser had $93b in revenue in 2021[1].
As one example from healthcare, the CEO of Moderna was paid $18m in 2021[2]. Moderna reported $18b in revenue in 2021[3].
You see this over and over in discussions about nonprofits, charities, etc. In most cases the executive leadership team is running a huge organization - in the case of Kaiser nearly $100b in revenue and 300k employees. Competent leadership at that scale is expensive.
In an ideal world this wouldn't be the case and individuals with these qualifications would be willing to take even more significant pay cuts for the "greater good" of whatever the mission of the nonprofit it. Per usual that's not the world we live in.
Speaking personally, I'd give pause to working with, donating to, etc an organization doing $100b in revenue with an executive team making 1/20 (or whatever) they would make elsewhere. In some limited cases for those who have "made their money" it would be a great thing. For anyone else I would assume they're either taking a HUGE pay cut for the cause (great thing, but unlikely) or completely incompetent, embezzling, etc.
[0] - https://www.erieri.com/blog/post/top-10-highest-paid-ceos-at...
[1] - https://about.kaiserpermanente.org/news/kaiser-foundation-he...
[2] - https://thehill.com/business-a-lobbying/business-lobbying/34...
[3] - https://investors.modernatx.com/news/news-details/2022/Moder....
Well, "they are doing something amazing, so ethics can be suspended for T amount time in the name of value" is equally egregious from my point of view.
[deleted]
> Having said that, plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.
Non-profit doesn’t mean the organization isn’t supposed to have surplus revenue, it means it doesn’t exist to return profits (e.g., via a claim on the assets of the company) to stockholders or some other beneficiary with a claim on them.
Non-profit doesn’t mean the organization isn’t supposed to have surplus revenue, it means it doesn’t exist to return profits (e.g., via a claim on the assets of the company) to stockholders or some other beneficiary with a claim on them.
>Having said that, plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.
That's not a contradiction. A non-profit can be profitable. The profit just needs to be reinvested back into the mission of the organization.
That's not a contradiction. A non-profit can be profitable. The profit just needs to be reinvested back into the mission of the organization.
OpenAI is a nonprofit that ownes a for-profit. Same as Mozilla. OpenAI was never a charity.
A privately owned for-profit company can do whatever it wants, good or evil. OpenAI is not publicly-owned.
Did you donate money to OpenAI?
A nonprofit is an organization that uses its income and profits for the organization's main goal that supports the mission. On the other hand, a charity is a type of nonprofit that engages in activities aimed at improving lives in the communities.
NFL was a non-profit trade association until ignorant whiners made it bad PR and they changed it to shut up the whiners.
Did https://openai.com/about
A privately owned for-profit company can do whatever it wants, good or evil. OpenAI is not publicly-owned.
Did you donate money to OpenAI?
A nonprofit is an organization that uses its income and profits for the organization's main goal that supports the mission. On the other hand, a charity is a type of nonprofit that engages in activities aimed at improving lives in the communities.
NFL was a non-profit trade association until ignorant whiners made it bad PR and they changed it to shut up the whiners.
Did https://openai.com/about
>plenty of extremely profitable organizations masquerade (literally and figuratively) as non-profits.
Isn't Ikea technically a non-profit? through some weird structuring?
Isn't Ikea technically a non-profit? through some weird structuring?
Sort of meta that Sam did something that Elon Musk would do…
If you want more open research and weights, you should be happy about this announcement. Incorporating as a nonprofit doesn't guarantee that an organization will act ethically, but it does make it more likely. Nonprofits have more restrictions around how they can spend their money. They also pay less, so the teams is more likely to be mission-aligned.
The profit motive pushes organizations to keep their work secret (which is what has happened with OpenAI).
The profit motive pushes organizations to keep their work secret (which is what has happened with OpenAI).
> Not sure why the AI community has a weird obsession…
Can you elaborate on this “obsession”? Aren’t a lot of AI things made/implemented by for-profit companies? Google, Meta, Midjourney, etc?
Can you elaborate on this “obsession”? Aren’t a lot of AI things made/implemented by for-profit companies? Google, Meta, Midjourney, etc?
Yeah lol. The obsession is understanding this is going to be pretty impactful technology and thinking it might be nice to not have that be closed source for once
I'm with you there on hoping for more concentration on local training and inference, especially on something like the new Orin (open to donations, NVidia!).
I think there's still something of a financial disincentive to promote selfhost over cloud capabilities for just about every party involved except the selfhoster. NVidia loses out if they're selling a few Nano or Orin platforms relative to the much pricier datacenter cards sold in huge lots. The cloud hosters are making up for the cost of hosting in terms of end-user pricing, snarfing all that lovely customer data and likely selling analytics, and probably other measures I can't understand yet. And the large companies that fund research and initial model development want to know what's being done with their models so as to gain any possible competitive advantage. They can't necesarily guarentee that intel from a self-hoster. Almost nobody is willing to spend the time explicitly to make it easier for the individual dev at a small lab or at home to do this because that's essentially a donation and not a business expense that might yeild obvious returns.
I think there's still something of a financial disincentive to promote selfhost over cloud capabilities for just about every party involved except the selfhoster. NVidia loses out if they're selling a few Nano or Orin platforms relative to the much pricier datacenter cards sold in huge lots. The cloud hosters are making up for the cost of hosting in terms of end-user pricing, snarfing all that lovely customer data and likely selling analytics, and probably other measures I can't understand yet. And the large companies that fund research and initial model development want to know what's being done with their models so as to gain any possible competitive advantage. They can't necesarily guarentee that intel from a self-hoster. Almost nobody is willing to spend the time explicitly to make it easier for the individual dev at a small lab or at home to do this because that's essentially a donation and not a business expense that might yeild obvious returns.
> easier ways to train models locally, better pruned models for embedded device inference (e.g. running on a Jetson Nano), easier ways to fine-tune for specific contexts
I think there are steps being taken in this direction (check out [1] and [2] for interesting lightweight transpile / ad-hoc training projects) but there is a lack of centralized community for these constrained problems.
[1] https://github.com/emlearn/emlearn [2] https://github.com/BayesWitnesses/m2cgen
I think there are steps being taken in this direction (check out [1] and [2] for interesting lightweight transpile / ad-hoc training projects) but there is a lack of centralized community for these constrained problems.
[1] https://github.com/emlearn/emlearn [2] https://github.com/BayesWitnesses/m2cgen
I was looking for open alternatives to self hosted, or crowd hosted finetuned LLMs like ChatGPT and found LAION Open Assistant. Then found resources to further optimize inference as well as training:
- Open source fine tuned assistants like LAION Open-Assistant [1]
- inference optimizations like VoltaML, FlexGenm Distributed Inference [2]
- training optimizations like Hivemind [2]
1 https://github.com/LAION-AI/Open-Assistant
2 https://github.com/underlines/awesome-marketing-datascience/...
- Open source fine tuned assistants like LAION Open-Assistant [1]
- inference optimizations like VoltaML, FlexGenm Distributed Inference [2]
- training optimizations like Hivemind [2]
1 https://github.com/LAION-AI/Open-Assistant
2 https://github.com/underlines/awesome-marketing-datascience/...
It seems like a rift between "Enlightenment" and "Authoritarian High Modernism." People want this progress to benefit reason and humanities, not contribute to our backsliding. Being nonprofit is at least symbolic of enlightenment.
> Not sure why the AI community has a weird obsession with being non-profit (or not)
A lot of times, I find the obsession isn’t a desire for collaboration or anything noble, it’s as simple as this: people want to use the end products for free or very cheaply.
A lot of times, I find the obsession isn’t a desire for collaboration or anything noble, it’s as simple as this: people want to use the end products for free or very cheaply.
> It's not like non-profits can't be evil
There's no shortage of non-profits that exist to pay saleries. As an example, growing up in the bible-belt, and I can confirm that religion can be good business.
There's no shortage of non-profits that exist to pay saleries. As an example, growing up in the bible-belt, and I can confirm that religion can be good business.
I can't help but wonder if non-profit makes it easier to get access to or train on certain datasets/models.
taxes!
I hope Eleuther will continue to focus on the development of truly open-source models, which seems to be where the big gap is. The quality gap between Neo-20X and even the original GPT3 is still rather large, and making headway there would be amazing.
there will always be gap, since corps can grab open model, and add resources to improvements on top of it.
This is not the case with Stable Diffusion. The open source community is moving faster and innovating faster, and the open models are by far the best. Midjourney is a contender, but it doesn't hold a candle to the best open models
Let’s be honest though; the number of people willing to devote their time to tireless effort generating porn (or suggestive images) is >> the number who will ever be willing to do it for text models.
Add the size barrier (an order of magnitude > SD for text models) and it’s pretty questionable the open source text models ecosystem is going to go anywhere.
Hardly anyone has done anything interesting with any of the existing ones and they’ve been out for longer than SD has.
Text models just aren’t that interesting to people.
Add the size barrier (an order of magnitude > SD for text models) and it’s pretty questionable the open source text models ecosystem is going to go anywhere.
Hardly anyone has done anything interesting with any of the existing ones and they’ve been out for longer than SD has.
Text models just aren’t that interesting to people.
I’m more interested in text models. So much potential there. I was fascinated by the person who hooked up a Siri shortcut to one of the openai models, asking it to return a json response. I think with a little imagination the potential applications are just as compelling as image models like SD.
Just one example, using these in games to generate unique conversations, actions, and events.
Just one example, using these in games to generate unique conversations, actions, and events.
9 days later and its now possible to run a 65B parameter LLM on a 64GB macbook pro, significantly outperforming GPT-3, OpenAssistant has announced that they got more than 100,000 samples of RLHF to replicate the 'Chat' component, Kobold Horde now supports Llama-33B for anyone to try in a playground with no censorship, and at least 4 other independent projects are spun up to develop at-home LLM tooling.
People want good text models at home.
People want good text models at home.
> Let’s be honest though; the number of people willing to devote their time to tireless effort generating porn (or suggestive images) is >> the number who will ever be willing to do it for text models.
Spot on. Everyone is talking about hos "fast moving" and "innovative" SD community is. Is it? I don't see many models that generate something that is NOT pretty anime girls. There are some... but if you exclude all "female focused" models, SD community doesn't seem particularly active.
Spot on. Everyone is talking about hos "fast moving" and "innovative" SD community is. Is it? I don't see many models that generate something that is NOT pretty anime girls. There are some... but if you exclude all "female focused" models, SD community doesn't seem particularly active.
The problem with LLMs is the big ones don't run on consumer hardware. If 12GB or 24GB ram would run them you would see just as much interest.
The open source models will keep up as hardware allows IMO.
The open source models will keep up as hardware allows IMO.
I think this space is in initial choke chaos.
Eventually commercial entities will build pipelines which will absorb open models and build on top of them using proprietary tech, data and expensive compute.
Eventually commercial entities will build pipelines which will absorb open models and build on top of them using proprietary tech, data and expensive compute.
They're the real OpenAI! Great team.
If only they had chosen a better name... "Eleuther" is hard to pronounce correctly for someone who doesn't know the word.
how do you pronounce it correctly
If it's from the greek word for freedom (ελευθερία) then i guess elefther: ele (like electric) f the (like thesis) r.
The English convention is to pronounce -eu- in Greek words as /ju/, not /ef/.
Yes, basically Greek words in English are pronounced as Latin loanwords, according to how the Greek words would have been pronounced in Latin around AD 1 (except that ph and th, but not ch, are pronounced as fricatives in English). Greek αυ and ευ were still pronounced as diphthongs at the time.
My name is Ελευθέριος and I transliterate it as Eleftherios in English.
There's a "standard" way of how greek words should be transliterated in English (ELOT 743). You can test it here https://www.passport.gov.gr/passports/GrElotConverter/GrElot...
And yes, Eleftherios is the correct transliteration :)
And yes, Eleftherios is the correct transliteration :)
I know. I've even seen my name written as Eleytherios.
Where are you getting an /f/ sound from?
Edit: Ah, it seems the digraph "ευ" is pronounced /ef/ in Modern Greek, despite (by convention) being transliterated "eu".
Edit: Ah, it seems the digraph "ευ" is pronounced /ef/ in Modern Greek, despite (by convention) being transliterated "eu".
Correct. Thee same pronunciation exists in ancient greek for example ευ ζην is ef zin (the good life).
Also the ευ situation is a little more complex ie sometimes it is pronunciated as ev (depending if the next letter is a vowel).
Also the ευ situation is a little more complex ie sometimes it is pronunciated as ev (depending if the next letter is a vowel).
Wikipedia suggests that "ευ" was pronounced /eu̯/ in ancient greek[0]. My suspicion is that "ευ ζην" is an Ancient Greek "fixed phrase" which is pronounced with Modern Greek phonology.
[0]https://en.wikipedia.org/wiki/Ancient_Greek_phonology#Diphth...
[0]https://en.wikipedia.org/wiki/Ancient_Greek_phonology#Diphth...
Huh, this seems to be the same linguistic phenomenon observed in British English with the word "lieutenant" but perhaps independently evolved.
I'm just going to keep saying YOO-luh-tuhr, thank you very much.
Reminds me of Yog-Sothoth, which would be a great name for an anthropomorphized statistical model...
> Yog-Sothoth knows the gate. Yog-Sothoth is the gate. Yog-Sothoth is the key and guardian of the gate. Past, present, future, all are one in Yog-Sothoth. He knows where the Old Ones broke through of old, and where They shall break through again. He knows where They have trod earth's fields, and where They still tread them, and why no one can behold Them as They tread.
> Yog-Sothoth knows the gate. Yog-Sothoth is the gate. Yog-Sothoth is the key and guardian of the gate. Past, present, future, all are one in Yog-Sothoth. He knows where the Old Ones broke through of old, and where They shall break through again. He knows where They have trod earth's fields, and where They still tread them, and why no one can behold Them as They tread.
You laugh. But, that has been the meme about GPT for a bit now
https://twitter.com/natfriedman/status/1625850766039842824?c...
RLHF = Reinforcement Learning from Human Feedback
https://twitter.com/natfriedman/status/1625850766039842824?c...
RLHF = Reinforcement Learning from Human Feedback
A meme that quickly became a long-form article: https://astralcodexten.substack.com/p/janus-simulators.
Reminds me of how there was a post on LessWrong some good decade ago, with a chilling premise: we somehow manage to get a safe, trustworthy, correct AI working, and as we fire it up and ask to tell us about our world, it starts talking about things we somehow can't perceive...
I couldn't find the IPA even in Wiktionary, but I found out where it comes from: https://en.wikipedia.org/wiki/Eleutheromania
Actually the origin is this word (with the last two letters transposed), and the Wikipedia article gives IPA
https://en.wikipedia.org/wiki/Eleutheria
https://en.wikipedia.org/wiki/Eleutheria
Like the Eleuthera island in the Bahamas I would think: https://youtube.com/watch?v=3-F-7WKXG14
So Eh-Luther AI.
So Eh-Luther AI.
I’d been saying E Luther. Is that all wrong?
I assume similar to luthier?
Which is everyone
Everytime I see it, I think of John Fowles' The Magus.
See, for example: https://pzo31b.p3cdn1.secureserver.net/wp-content/uploads/20... (via https://www.fowlesbooks.com/1966-fowles-letter-to-high-schoo... )
See, for example: https://pzo31b.p3cdn1.secureserver.net/wp-content/uploads/20... (via https://www.fowlesbooks.com/1966-fowles-letter-to-high-schoo... )
While I appreciate what Eleuther have done - why are they highlighting "research" consisting of removing a bit of whitespace from some source code... is it April 1st already?
Heres my addition! Or subtraction, i guess... (-4 chars by changing call to AdaptiveAvgPool2d):
Heres my addition! Or subtraction, i guess... (-4 chars by changing call to AdaptiveAvgPool2d):
from torch.nn import*
def c(h,d,k,p,n):S,C,A=Sequential,Conv2d,lambda x:S(x,GELU(),BatchNorm2d(h));R=type('',(S,),{'forward':lambda s,x:s0+x});return S(A(C(3,h,p,p)),*[S(R(A(C(h,h,k,1,k//2,1,h))),A(C(h,h,1)))for _ in[0]*d],AdaptiveAvgPool2d(1),Flatten(),Linear(h,n))[deleted]
Just having a bit of fun.
I followed along on their discord for a while but I wasn’t sure how to gradually increase my engagement to get involved.
Awesome. Remember your great work in GTP-Neo
What was special about GPT-nano ?
not nano, Neo, and its successor NeoX, one of the largest public GPT models available https://blog.eleuther.ai/announcing-20b/
Very big LLM. Was almost as good as GTP-3. And it was open source.
I really appreciate how the work of EAI has truly focused on making models accessible to the general public.
How do they pay for their compute?
We have a number of donors including Hugging Face, Stability AI, Nat Friedman, Lambda Labs, and Canva that make our work possible. We also have some orgs that provide sponsorship for computing resources specifically: Stability AI, CoreWeave, and Google Research.
are you bottlenecked by engineering
Yes, we are currently bottlenecked primarily by engineering manpower.
Have you reached the point of dogfooding your own models to augment work that would normally be considered engineering?
If so, I wonder about the implications of black-box models helping to study and develop solutions for interpretability. It gives me "reflections on trusting trust" vibes.
If so, I wonder about the implications of black-box models helping to study and develop solutions for interpretability. It gives me "reflections on trusting trust" vibes.
I can only imagine the more successful that activity was, the worse the bottleneck would become
Not access to GPUs for training? What would it take to get a 65B model out of eleuther?
Our policy is to not comment on timelines for future models, as our ability to meet those timelines is heavily influenced by factors outside of our control and we don’t want to lead people on.
A cr q
Donations and sponsors such as Stability AI, based on the announcement in their Discord server
With money. A non-profit can still have plenty of revenue.
Now I have Lenny Kravitz songs stuck in my head.
Eleuther has been at the forefront of AI, without them none of this would be possible ,thanks and kudos
sorry for killing the mood. great article and great progress.
but. didn't openai start as a nonprofit, too?
but. didn't openai start as a nonprofit, too?
OpenAI has some complicated setup about this. Part of it is non-profit and part of it isn't and there's some rules they laid out on how to keep it non profit first from what I've gathered
to my knowledge - started 100% non-profit, the split into partial profit came later when they started having a chance to make money
As a three letter acronym (TLA) aficionado, the page uses TPU wwhich isn’t explained.
tensor processing unit. a processor designed to do large, fast matrix operations.
TPU - Tensor Processing Unit. They're essentially Google GPU's.
What we do need is more weights being released in the public domain (hard to find even on Huggingface), easier ways to train models locally, better pruned models for embedded device inference (e.g. running on a Jetson Nano), easier ways to fine-tune for specific contexts, and so on. My big gripe, and for obvious reasons, is that we need to step away from cloud-based inference, and it doesn't seem like anyone's working on that.