The "tech bros" were right: The board absolutely were conspirators wrecking the company based on dogmatic ideology. "Tech bros" had clear evidence of Sutskever, Toner and McCauley's deep and well-known ties to the Effective Altruist (EA) movement and doomerist views. Nobody can doubt Sutskever's technical credentials, whatever his strange beliefs, but Toner and McCauley had no such technical background or even business achievements. These members instead were organizers in EA and researchers in the "field" of AI governance and safety. This is no expertise at all. These "disciplines" are built more on Yudkowsky's rants and hypotheticals and the precautionary principle gone mad than any empirical research. These ideas also come from a cult-like social movement (EA) accumulating power across tech and government with few scruples, as shown by SBF's implosion last year and many smaller incidents. How could Toner and McCauley assess briefings if they couldn't assess the technical fundamentals? How could they foresee the consequences of sacking the CEO if they didn't understand business? If they already believed AGI would kill all humans, how could they judge new advances on the merits without jumping to wild conclusions? Instead, us "tech bros" felt people with this background would fall back on uninformed, reactionary, and opaque decision making, leading to wrecking an $80 billion business with no decent explanation.
This now seems to be exactly what happened. The board saw Q* and decided to coup the company, to put all power in their hands and stop development. This by itself is bad enough if you care about open science or progress, but it gets worse. They didn't even want to hint at capabilities increases to avoid "advancing timelines" i.e. Open-ing knowledge about AI, so they made up some canard about Altman's "lying to the board" to hide their real reasons. This is vile libel for unscrupulous ends. When they realized this excuse wouldn't fly, they obfuscated and refused to explain their true concerns, even to their own handpicked CEO Emmett Shear. However, it turns out that destroying $80 billion and lying about why won't fly in the real world. The board had no second-order or even first-order thinking about the consequences of their actions, and were rolled up by bigger actors. These people were unprepared and unable to follow up their coup.
This failure is exactly what you'd expect from a brilliant scientist with little organizational experience (Ilya) and social-science academics and NGO organizers (Toner and McCauley). I don't care what gender they are, these people neither deserved their authority nor could use it effectively. Dismissing valid criticism of the board as "cyber bullying" or "tech bro sexism" merely underscores why most engineers hate DEI rhetoric in the first place.
Yes, the EAs and longtermists on the board probably disliked the commercial focus of “Open”AI or its rapid scaling; I’m not blaming them for Laundry Buddy. But make no mistake: the nonprofit board had even less interest in opening up their research or sharing their code. They and the “superalignment” team believe AGI can end the world and needs to be in safe hands (i.e. their own hands). The EA movement which they are embedded in is one of the leading forces advocating shutting down open AI development through regulation.
The board has strong EA and doomerist ties. Within OpenAI, Ilya Sutskever is on record as saying that the world will realize open-sourcing weights is foolish by 2025, and the Atlantic reported he literally burned an effigy of “unaligned” AI at the company retreat. Helen Toner is similarly involved in “AI governance” and not coincidentally took funding early in her career from OpenPhil, one of the key EA slush funds. The new CEO, their appointee, is quite literally a character written into Eliezer Yudkowsky’s rationalist fanfic in a cameo, and believes the probability of AGI killing humanity is 50%.
These people have absolutely no interest in decentralization and accountability, and were more than happy to let OpenAI accumulate power to “protect humanity”—until they pulled the plug for reasons they still refuse to disclose. Let me be clear: this is unacceptable. Not coincidentally, their so-called utilitarianism and altruism merely justifies accumulating all power in their hands, and taking any action (like the backstabbing we saw last week) to make it happen. For all MSFT’s faults, they play by traditional and predictable corporate rules of greed and can be reasoned with. The safety faction are true believers and implacably opposed to openness anywhere, and moreover happily gave the veneer of altruism to the regulatory capture of the commercial faction anyway before they realized they couldn’t control it. I know which one I’d pick.
“Open”AI already was an unaccountable big corp. They’ve already refused not only to open their weights but to publish most of their research, to create an insurmountable gap with the rest of the world, and to legislate it in with lobbying. “Openness” merely meant “the API is open to your money”, with opaque “content policies” we had no say in. They had the same unaccountable and opaque power and the same commercial drive, but “for our own good” as they define it unilaterally.
Moving to MSFT merely means they’ll do the same thing, but with a bit more reliability, a bit more fear of liability, and a bit less of the doomerist sanctimony of the original leadership. Better to at least have a bigco with coherent and stable values like money. The bigco “nonprofit” led by the current board has made erratic decisions, has insane longtermist and EA values, and refused to give any meaningful statement about why they did what they did. Inasmuch as they resist commercialization, it’s to have more opacity and more control not less. How can we trust these people with control over AGI? Better to junk the board and deal with straightforward greed rather than hubris.
Phishing emails don’t exactly take AGI.
GPT-NeoX has been out for years, Llama has been out since April, and you can set up an operation on a gaming desktop in a weekend. So if personalized phishing via LLMs were such a big problem, wouldn’t we have already seen it by now?
Just a brief reminder of the last hundred years of copyright: large lobbies like UMG, Disney, or Elsevier have hugely circumscribed the public domain for all but historical purposes (copyright lengths increased from 28 years to *128*). Open access in academic publishing (for public funded research, even!) has been a decades-long fight that is not fully won. The DMCA has created myriad abuses and denied the freedom to control your own hardware, like allowing companies like John Deere to prevent right to repair. Six studios own most movies, a few labels control most music. The cultural heritage of the last hundred years, which should be the collective property of humanity, is locked up in the coffers of a few megacorps: who, by the way, don't much care how well their artists are eating. Yes, there are edge cases, but spare me the sympathy for the poor artist on Fiverr with five images in LAION: this is a fight between tech titans and copyright monopolists, where the little people are always going to be an afterthought. Come out and say the original copyright term of 28 years (well long enough to eat!) should be brought back, condemn the DMCA and the naked abuse of democracy in the Sonny Bono act, and THEN maybe we can get to chiding the lapses of a few researchers just wanting to understand the world better.
I’m not sure who the “we” is in your statement, “we do know with certainty that IVF is immoral”. As evidenced by the entire rest of the thread, this opinion is hotly contested. If you believe IVF is morally wrong, please at least explain why. As it stands, you’re simply condemning families and researchers as evil and holding future research to an arbitrarily high, nebulous ethical bar with no explanation.
How does open source compete with the Claude API? Easy: actually let you use the model. From the signup page:
> Anthropic is rolling out Claude slowly and incrementally, as we work to ensure the safety and scalability of it, in alignment with our company values.
> We're working with select partners to roll out Claude in their products. If you're interested in becoming one of those partners, we are accepting applications. Keep in mind that, due to the overwhelming interest we've received so far, we may take a while to reply.
No thanks, I'd much rather not wait months to see if my app deserves their oh-so-limited attention, or "aligns with the values" of a company taking $400m from Sam Bankman-Fried.
To be more charitable to your underlying point, Claude 2 is free to chat with via Anthropic's website, Poe, or Slack, and the GPT-4 API is open to use. If you're building a prototype or just need a chatbot, these do have better results and dev experience, at least for now. But I don't think picking on your Claude API example is unfair. These companies could randomly refuse your prompts via some opaque "moderation API" (that all GPT fine-tuning data goes through!), train on your company's proprietary data, spy on your most intimate questions, or just not find you worth the trouble and cut you off, at any time. THAT is why open source beats proprietary hands down: My device, my data, my weights, my own business.
Governments are far harder to remove than tech firms. You may be able to ditch Google, but there's only one government in your country. And "democracy" doesn't prevent state surveillance.
Most politicians back it, so voting differently makes little difference. Labour and the Conservatives support the Online Safety Bill, the Patriot Act was bipartisan, and voters have very little control over the EU and can’t stop Chat Control. And most of “government” isn’t directly elected: you can’t vote out the NSA, and Congress has little power over them either. The government blunts corporate abuses but doesn’t stop them: revolving doors ensure authorities target small fry while big companies like Visa keep going unimpeded. And finally, most voters don’t mind surveillance that much, since government and media manufacture consent for it. Don’t count on ordinary people to “vote it out” until it’s too late.
Lobbying against government surveillance helps marginally, but it's an eternal struggle. Governments take as much power as they can get, while abuses are exponentially harder to detect and stop than refusing to grant that power in the first place. The “slippery slope” isn’t a fallacy, it’s the record of the last twenty years. Don’t let them track speech and money with a central ID and digital currency, just because you don’t like a few tech bros or online trolls.
Basic economics: The optimal number of people who die tragically in a burning building is not zero. Every person trades off some amount of safety for cost, that’s just part of living. It’s possible that this particular tradeoff isn’t worth it, but human life isn’t infinitely sacred. I assume if it were in your power, you also would rather let some people burn to death than ban wooden houses, for example. This, well, inflammatory emotive rhetoric really doesn’t lead to good discussion, or good policy for that matter.
The article makes a good point: we should prevent “open-washing” and draw a distinction between well-intentioned restrictive licenses like “Open”RAIL and true open source. However, I worry the name “ethical source” is itself a bit question-begging. While outfits like Bloom may believe in good-faith ethical principles, their definition of ethics isn’t necessarily everyone’s. If restricted models are “ethical”, is releasing open weights “unethical”? Conversely, is releasing a model with PII or artist styles in it “ethical” if a few known use cases are forbidden? There’s no one right answer. Labeling any one set of restrictions as “ethical” off the bat makes discussion harder and puts open source on the back foot to justify “not being ethical”. Better to just call them “restricted models” or “guarded models”, and leave it to individuals to decide if these restrictions are beneficial or not.
The source seems to be a February article from Automotive News interviewing several Toyota execs off-the-record: https://archive.is/0LZYf. The relevant quotes are midway through
So “if you have nothing to hide, you have nothing to fear”? Would you tell that to abortion rights activists in Mississippi, journalists reviewing the Panama Papers, or dissidents in China? Radical new idea here, but maybe you shouldn’t believe that everyone in authority from now till you die will only fire and jail people for the ideas that you personally don’t have or like, or suggest data privacy rules for others accordingly.
Shocking that this kind of knee-jerk anti-innovation sentiment seems to be the dominant mood on Hacker News these days. Whatever legitimate criticisms of SV startups there are, this is mere contentless anti-tech sniping.
Yes, as a matter of fact, I do. If you look at even the US regulatory environment, for example: FDA approval for new drugs takes decades and billions, and enough said on America's "War on Drugs". The NRC has practically strangled nuclear power in its cradle, and environmental review adds years and millions to practically all new infrastructure. Health insurance regulation and financial regulation have enshrined a few oligopolistic players, with nearly impossible regulatory burdens for any new entrants. It's not merely that any one regulatory agency has run amok. All such agencies are fundamentally incentivized to increase their power and reduce risk by adding regulation in a one-way ratchet, while they have very little incentive to allow innovation except sustained external pressure, which new industries rarely can sustain. Large corporations rarely counterbalance this regulatory creep: they welcome regulation, since they can capture agencies to their will through lobbying and the "revolving-door" and afford a legal staff, while startups and individuals still face impossible regulatory burdens. Don't even get started on copyright. Regulation in America is fundamentally broken, and while it admittedly does manage to "protect" consumers from some of the worst potential abuses, it does so by choking innovation and essentially freezing us in the 1970s.
Why should we believe that AI regulation would be any different? Whatever agency is given power won't just stop at reasonable guidelines. They will likely be pressured by big players like Microsoft to choke off open source, by copyright giants like the RIAA and Disney to stop generation, and by every imaginable constituency to protect their jobs from change. Most importantly, individual open-source development will become prohibitive, and AI will be locked behind corporate APIs. In fact, if you look at sites like LessWrong, AI doomers are openly welcoming regulation precisely because they know an FDA-like agency will stop progress dead in its tracks. Make no mistake, AI regulation will hand enormous power to governments and corporations while denying it to the individual, whatever the good intentions were at the beginning. It is a devil's bargain, and if we as an industry are to take it in the name of "safety", we should at least do it with eyes open, instead of pretending that all that is being asked for is "guardrails" and "common sense".
For those skeptical of the above comment, this technique absolutely works and powers production-grade models like Anthropic’s Claude. There’s plenty of literature on this, but here are a couple papers that might be helpful for people doing their own training:
- Constitutional AI: by Anthropic, an “RLAIF” technique that creates the preference model for “finding errors” based on a set of around 70 “principles” the AI uses to check its own output, not human feedback like in ChatGPT. This technique taught the Claude bot to avoid harmful output with few to no manual harmfulness labels! https://arxiv.org/abs/2212.08073. Not sure if there’s a HuggingFace implementation with LoRA / PEFT yet like there is for regular RLHF, so somebody may need to implement this for Llama still
- Self-Instruct: Creates artificial training data on instruction tuning from an untuned base model, from a tiny seed of prompts, and filters out the bad ones before fine-tuning. Manages to approach Instruct-GPT performance with only ~100 human labels. https://arxiv.org/abs/2212.10560
Yes, in an ideal world a competitor could just start up their own ChatGPT. However, over the last decade we've seen over and over that "just build your own X" isn't a real solution. Nearly all web-based businesses rely on a few oligopolies which can and do deplatform people they don't like:
- Apple and Google (Parler)
- Payment processors like Visa and Mastercard (porn) or, heck, the banks (Canada truckers, Kanye West)
- Crowdfunding (Unstable Diffusion)
- Cloud hosting like AWS (Parler again)
- Infrastructure like Cloudflare, DNS (Kiwifarms)
And that's not even counting network effects, social media activism, and the difficulty of raising money for a controversial business. Starting a large profitable tech business opposed to SV values is next to impossible, unless you're in China or are content selling to all 2 Monero users on the dark web.
And you can still theoretically run a website out of your basement. LLMs, on the other hand, need ridiculous amounts of compute to train, which practically requires cloud computing. Fine-tuning isn't an answer either: almost all open-source foundation models are conveniently "ethically-sourced" so the groups releasing them can sue any users they don't like. How would a conservative make ParlerGPT if AWS or Google won't rent them A100s or TPUs - "just build your own exascale cluster"??? Even then I suspect Nvidia would deplatform them and it would be game over.
Some companies are infrastructure, and if they're not treated like utilities, your rights are a dead letter. Yes, I do insist that Mastercard and Visa, or Apple and Google, or Nvidia and AMD must always succumb to every group's values who want to buy their services for any legal use. And though I wouldn't necessarily go this far, there's a case to be made that foundation models (which after all are trained on everybody's data) should be politically neutral as well.
From my perspective, our magic algorithm does indeed make us immune to the “consequences” of scraping copyrighted work. Legally, diffusion models learn from work rather than copying any specific piece, and copyright does not protect a style or give artists the right to retroactively restrict learning from open content. Ethically, our society accepts legally automating people out of jobs, and there’s no good reason that art is different - simply social prestige and classism. Current models, even fine-tuned ones on specific artists, are not wrong, however sad or harmful for the artist they may be. There are winners and losers, that’s capitalism, for better or worse.
I am sorry for artists and think we should probably have a subsidy for human creativity. But for me, Twitter’s bad-faith arguments about “stealing” or “copyright violation”, and maximalist demands for bans on AI art, make compromise undesirable and probably impossible.
No, I don’t accept that it’s “the people” boycotting Kickstarter? It’s a small minority of media and activists threatening platforms with clickbait headlines, doxxing, and regulation if they don’t give up neutrality, and a larger group who believes whatever the media tells them or has no incentive to speak up. The choice Kickstarter made wasn’t “Do my customers have a principled stance against Unstable Diffusion?” It was, “Do I stand up for my principles - or do I have to put up with $MEDIA_OUTLET running wall to wall stories claiming I hate artists, 200 activists dogpiling my every twitter post, and ordinary people unconsciously associating me with whatever caricature they read on Twitter?” In this case and most others, neutrality is impossible because a minority has chosen to make it impossible, not because either Kickstarter or its customers made a principled decision. This is why this entire thread exists: to counterbalance this and to make clear that cowardice comes with its own costs.
Are you aware of the equally common objection to “I can tolerate anything except intolerance”: that if allowed to censor in the name of tolerance, our self-appointed guardians will simply call whatever they don’t like “intolerance” and censor it? (Or “stochastic terrorism”, or “micro-aggression”, or “online harms”, or “misinformation”, or “trolling”, or “sealioning”, or “toxicity”, etc). Certainly one must draw the line somewhere. But the speech debate doesn’t end at the Paradox of Tolerance, and at this point I assume anyone who brings it up in isolation is doing so in bad faith.
This now seems to be exactly what happened. The board saw Q* and decided to coup the company, to put all power in their hands and stop development. This by itself is bad enough if you care about open science or progress, but it gets worse. They didn't even want to hint at capabilities increases to avoid "advancing timelines" i.e. Open-ing knowledge about AI, so they made up some canard about Altman's "lying to the board" to hide their real reasons. This is vile libel for unscrupulous ends. When they realized this excuse wouldn't fly, they obfuscated and refused to explain their true concerns, even to their own handpicked CEO Emmett Shear. However, it turns out that destroying $80 billion and lying about why won't fly in the real world. The board had no second-order or even first-order thinking about the consequences of their actions, and were rolled up by bigger actors. These people were unprepared and unable to follow up their coup.
This failure is exactly what you'd expect from a brilliant scientist with little organizational experience (Ilya) and social-science academics and NGO organizers (Toner and McCauley). I don't care what gender they are, these people neither deserved their authority nor could use it effectively. Dismissing valid criticism of the board as "cyber bullying" or "tech bro sexism" merely underscores why most engineers hate DEI rhetoric in the first place.