> 1. The AI answers many questions, some of which are potentially harmful, and generates first draft answers.
...
> 4. The system repeats this process until it collects a large dataset of first draft answers, and rewritten more-ethical second-draft answers.
> 5. The system trains the AI to write answers that are less like the first drafts, and more like the second drafts.
Actually, there are 2 separate models involved in this finetuning step: AI in steps 1-4 is a pre-existing LLM trained to converse and follow instructions (I call it "advisor"). The second AI starts out as a pre-trained LLM (so only trained to provide likely completions on a general dataset), and is trained in step 5 to generate completions that look more like the dataset constructed in 1-4.
So this process is using the helpfulness in the adviser to bootstrap additional values onto a new model.
> What if you had overseer AIs performing Constitutional AI Feedback on trainee AIs, or otherwise tried to separate out the labor?
That is literally what the paper is about, though its a bit hidden. First paragraph of 4.1 Methods:
"We continue to utilize human feedback labels for helpfulness as in prior work, but replace human feedback
labels with model feedback labels for harmlessness. That is, instead of asking crowdworkers to provide
comparison labels for harmlessness, we simply present the same task to an independent model, called the
feedback model (typically a pretrained LM). Once the desired comparison labels are obtained, the remainder
of the training pipeline (i.e., preference model training and RL) is exactly the same as RLHF."
> Factually, its first draft made a best guess (prediction) of what’s accurate.
A well pre-trained LLM makes every word a good guess of what the authors of its training set would say, following the words that preceded... in a different context, possibly a few years ago, possibly as part of a web-fiction. "accurate" is the wrong word here.
To make an LLM prefer, when answering factual questions, to rely on factual sources and approximate the real world, you have to use an additional process that does so. RLHF and CAI might be part of the solution to that, or not, depending on the feedback they provide.
Dividends lower stock price by moving cash from company to owner, buybacks increase stock prices[1] by de-diluting, so only one of those is evidence of the CEO (whose compensation is often tied to stock price) acting on his own interests. Also taxation is different.
That sounds like an argument about "if one's time is worth less than minimum wage, one cannot find a job". And sure that is true for some people, but saying it benefits _only_ ... sounds like it is never good for any worker, which is way too strong a conclusion, because other situations exist.
For example: hiring one of two candidates, Nick and Joe, would be profitable under 30$ per hour. Nick asks for 15$, its Joe's first job so he's willing to take 10$, I go to Nick and say, "look Joe will do it for 10, so I really can't justify paying more, but I'd prefer to pay you those 10, since you're experienced". If Nick has a better offer elsewhere, he has no problem. If not, Nick gets just 10.
If the state says 15 is the minimum wage, not only Nick but also Joe must get 15, so my best move is to take Nick at 15. Clearly, a higher minimum wage _can_ benefit the worker, and not only the very weakest one.
Capitalism allows each participant to seek only the best available deal that is agreeable to them, sure, but availability is subject to negotiating power, and that is distributed very unevenly.
Actually, there are 2 separate models involved in this finetuning step: AI in steps 1-4 is a pre-existing LLM trained to converse and follow instructions (I call it "advisor"). The second AI starts out as a pre-trained LLM (so only trained to provide likely completions on a general dataset), and is trained in step 5 to generate completions that look more like the dataset constructed in 1-4.
So this process is using the helpfulness in the adviser to bootstrap additional values onto a new model.
> What if you had overseer AIs performing Constitutional AI Feedback on trainee AIs, or otherwise tried to separate out the labor?
That is literally what the paper is about, though its a bit hidden. First paragraph of 4.1 Methods:
"We continue to utilize human feedback labels for helpfulness as in prior work, but replace human feedback labels with model feedback labels for harmlessness. That is, instead of asking crowdworkers to provide comparison labels for harmlessness, we simply present the same task to an independent model, called the feedback model (typically a pretrained LM). Once the desired comparison labels are obtained, the remainder of the training pipeline (i.e., preference model training and RL) is exactly the same as RLHF."