Tell HN: I'm Putting Together Some Censored Language Models and Training Sets
6 comments
Nice! I'm creating a next-gen search engine. Email sent.
Awesome! I'm on a bit of a high-G burn for $DAY_JOB, but I'll try to get back to you shortly!
Why would I want a model obsessed with race-war and anti-semitism?
Because you had done even a single Google search and turned up, for example this paper: https://aclanthology.org/2022.ltedi-1.6.pdf titled "Detoxifying Language Models with a Toxic Corpus" (Yoon A Park and Frank Rudzicz at Toronto), which while not exactly groundbreaking, is a well-footnoted exploration of how to use adversarial or quasi-adversarial training ensembles in a language setting?
To be clear, I'm aiming for a tiny amount of snark/sarcasm above, the minimum amount. I want to provide a useful answer, but I'm a mechanism designer by trade and incentives matter. Don't do that.
The rest of my snark is at the offices of The Facebook, where people smarter than me are doing work that no one writing this comment, including and especially myself, are intellectually or creatively capable of ;)
To be clear, I'm aiming for a tiny amount of snark/sarcasm above, the minimum amount. I want to provide a useful answer, but I'm a mechanism designer by trade and incentives matter. Don't do that.
The rest of my snark is at the offices of The Facebook, where people smarter than me are doing work that no one writing this comment, including and especially myself, are intellectually or creatively capable of ;)
Yeah, appealing to hero-worship and faith in Facebook that creating a fascist AI is good actually isn't a convincing argument.
It's just a film reference: https://www.youtube.com/watch?v=mehUC5l-lGM&t=43s.
I've been pretty clear on my "hyper-FAANG ML model/courpus posture", e.g.
- https://news.ycombinator.com/item?id=32626783 - https://news.ycombinator.com/item?id=32597111 - https://news.ycombinator.com/item?id=32537134
I'm going on the permanent record under my real name with shit like: "Fogging up the windshield with a bunch of feigned alarm about AI apocalypse but ramming the R&D through at full thrusters is a dick move in either case."
I'd love to persuade you to help out! But if you're not interested, just declining to participate is really the move here.
I've been pretty clear on my "hyper-FAANG ML model/courpus posture", e.g.
- https://news.ycombinator.com/item?id=32626783 - https://news.ycombinator.com/item?id=32597111 - https://news.ycombinator.com/item?id=32537134
I'm going on the permanent record under my real name with shit like: "Fogging up the windshield with a bunch of feigned alarm about AI apocalypse but ramming the R&D through at full thrusters is a dick move in either case."
I'd love to persuade you to help out! But if you're not interested, just declining to participate is really the move here.
I understand that this is very controversial, but I'm personally opposed to censorship of previously public ML checkpoints, and more generally models trained on publicly-crawled corpuses.
I've decided to do my little bit on this and I've started getting together a few things things that are technically available but censored on the major hubs and therefore comparatively high-friction for lightweight experimentation and research. For obvious reasons it's not as simple as creating a GitHub repository, hence this post.
I'm not indifferent to arguments that these things could cause harm, but I believe that good-faith participation in the HN community is a sufficient bar for access to anything I could personally assemble, so what little (and hopefully growing) amount of this I get together I'm willing to share with anyone on HN who has even an eyeball-plausible comment history of caring about the community.
I don't have much yet, but I've tracked down the torrents for e.g. GPT-4Chan and am starting to archive it's tuning corpus, the HN comments corpus, some brand-name newspapers: low-hanging fruit like that. I've got some spare bandwidth on a modest number of GPUs so I'm planning to do a variety of fine-tuned checkpoints and throw together Docker and Nix environments for loading them up.
If you want what I've put together so far (which admittedly isn't much) or to help with this: [email protected]