Ask HN: How do companies like OpenAI, Perplexity fine tune rich output?
3 comments
Anything with a linter means, at minimum, free verifiable rewards for RL (though whether something parses versus looks good is another story). That, plus, they have more data than anyone, and also it seems somewhat reasonable that stronger models could learn 'more' from a given instance or set of examples.
Can you elaborate on "linter means and verifiable rewards for RL"? Is this something others would find extremely difficult to do ?
They’re saying that they can use linters to check the output from a reinforcement learning model and reward it for correct output.
One curious question is how do they fine tune rich data (markdown, html outputs, tables, graphs etc) at scale. Currently, performing fine tuning involves the laborious process of carefully editing inputs (prompts) and outputs one by one. Becomes more difficult as the data context increases and one has to carefully examine the input data and provide the right output including things like formatting, grammar, UI etc.
Considering such a wide variety of questions they are processing, it amazes me how are they doing it at scale. Any thoughts?