kcorbitt

kcorbitt

5 tháng trướcdiscuss

And lately, the sweet spot has been moving upwards every 6-8 weeks with the model release cycle.

kcorbitt

6 tháng trướcdiscuss

Is it?

kcorbitt

năm ngoáidiscuss

Dang, hadn't seen that. Namespace collision strikes again.

kcorbitt

năm ngoáidiscuss

I really like RLPR for when you have a known-good answer to compare to as well!

kcorbitt

năm ngoáidiscuss

No, we don't do anything. Theoretically we could judge several times with different ordering.

We could measure order bias really easily though; we just need to look at the average score by rollout position across many runs. I'll add that to my list of experiments!

kcorbitt

năm ngoáidiscuss

Thank! If there are any topics that you'd find particularly interesting, let me know and I can try to find time. :)

kcorbitt

năm ngoáidiscuss

Looks cool! With vLLM v1, prefix caching is enabled by default and seems quite performant. Is the advantage of LMCache the fact that you can offload to CPU and disk as well? How much is throughput/latency affected if you need to pull a large KV cache from disk/cpu instead of GPU RAM?

Also, how realistic would it be to share the KV cache across vllm nodes within a data center? It would be really nice to be able to freely distribute requests to a pool of vLLM workers without worrying about prefix-aware routing, but maybe that isn't the right approach because moving the KV cache around would be too slow?

kcorbitt

năm ngoáidiscuss

I was curious about this so I had o3 do a bit of research. Turns out 300 L40s have more compute than any supercomputer before 2013 (and arguably before 2016, depending on how you count reduced-precision FLOPs).

https://chatgpt.com/share/685dea79-26ec-8002-bd62-7ed83aedf4...

kcorbitt

năm ngoáidiscuss

The real answer is that nobody trusts their automated evals enough to be confident that any given automatically-trained release actually improves performance, even if eval scores go up. So for now everyone batches up updates and vibe-checks them before rolling them out.

kcorbitt

năm ngoáidiscuss

It seems like the speedups here are most useful for small models, since on larger models a smaller fraction of the total time would be spent swapping between kernels? Would be interesting to see at least theoretical results for LLMs in the 14-70B parameter range, which is what most folks deploy in practice.

And of course the effect on throughput at larger batch sizes, which they allude to at the end.

Overall a very interesting result!

kcorbitt

năm ngoáidiscuss

There are many industries where you need lots of experience before you're a net contributor to productivity. This is true for everything from hairdressers to doctors. We have ways of dealing with this (eg. taking out loans to undergo years of training).

The problem comes if the number of years of experience you need to outperform the frontier AI models advances at more than 1 per year, which is not out of the question.

kcorbitt

năm ngoáidiscuss

I wonder if they've trained the model to operate with a shallower stack; eg. the full model may be composed of 24 transformer blocks, but they've also trained it to accept embeddings at layer 8, so it can be operated with just 16 transformer blocks on lower-resourced devices.

Experimenters in the open source tinkering community have done the opposite (copy/pasting layers in existing models to make them deeper) and it seems to work... fine, with minimal post-training on the new, deeper model required to exceed the performance of the original model. So it's not a crazy idea.

kcorbitt

năm ngoáidiscuss

It's very unlikely that they're doing their own pre-training, which is the longest and most expensive part of creating a frontier model (if they were, they'd likely brag about it).

Most likely they built this as a post-train of an open model that is already strong on coding like Qwen 2.5.

kcorbitt

năm ngoáidiscuss

For "that last 10% of reliability" RL is actually working pretty well right now too! https://openpipe.ai/blog/art-e-mail-agent

kcorbitt

năm ngoáidiscuss

Ok good questions here.

By fine-tuning in this context I assume you mean "supervised fine-tuning", or SFT. SFT trains a model to produce a specific string of output tokens, given an input. With SFT, if you were trying to train an assistant to solve math problems using a code interpreter, you might train it on a dataset that looks like:

    input: 'What is 934+1208'  
    output: `print(934+1208)`

    input: 'how many "r"s in strawberry'
    output: `print(len([l for l in "strawberry" if l == 'r'])`

etc, etc.

RL, on the other hand, just means training a model not to produce a concrete string of output tokens, but rather to create an output that maximizes some reward function (you get to decide on the reward).

For the example above, you might create the following dataset for RL training:

    input: 'What is 934+1208'
    ground_truth: 2142

    input: 'how many "r"s in strawberry'
    ground_truth: 3

You would then train the model to write python code that produces the ground_truth output. Your training code would take the model's output, run the python it produced, and then check whether the output matches the expected ground_truth. Importantly, this doesn't require you actually writing the code to solve the problem (you don't even have to know if it's solvable, technically!). Over time, the training loop would make the model more likely to produce outputs that get high rewards, which hopefully means it gets better at producing valid and applicable python.

This is useful in lots of domains where it's easier to check the answer than actually produce it. In the blog post[1] linked above, we train the agent to effectively use keyword search to try to find the correct emails in an inbox. As the model trainer, I didn't actually know what the right strategy was to choose keywords that would most quickly find the relevant email, but through training with RL, the model was able to figure it out on its own!

[1]: https://openpipe.ai/blog/art-e-mail-agent?refresh=1746030513...

kcorbitt

năm ngoáidiscuss

Figured now was a good time to post this since we recently got surprisingly good results on training an email research agent. Link is above, but will put it here as well since I think it's a good example of RL's promise: https://openpipe.ai/blog/art-e-mail-agent

kcorbitt

năm ngoáidiscuss

[dead]

kcorbitt

năm ngoáidiscuss

We may be in a simulation, but your odds of being alive to see this (conditioned on being born as a human at some point) aren't that low. Around 7% of all humans ever born are alive today!

kcorbitt

năm ngoáidiscuss

Yep. And tbh you probably don't even have to do this; the R1 paper found that just running SFT the base model with a relatively small number of monolingual reasoning traces was enough for it to get the idea and iirc they didn't even bother selecting for language specifically in the RL training looop itself.

kcorbitt

năm ngoáidiscuss

To be honest, I don't expect the performance to generalize to other task types with this specific training regime. If we had a panel of like 30 logic puzzles and cross-trained against all of them simultaneously it might though.

I think there's a lot of benefit to discovering a training regime that allows small specialized models to do extremely well in one narrow task; if we can figure out how to make small models that beat SOTA on a specific task and are cheap to train and run, that's in some ways a more useful outcome than a very large model that is good at many tasks (but is more expensive to run for each of them).

Submissions

Codex, File My Taxes. Make No Mistakes

A Pocket Guide to Surviving the Robot Apocalypse

Show HN: RULER – Easily apply RL to any agent

Everything I know about reward hacking

Show HN: ART – a new open-source RL framework for training agents

ART·E: how we built an email research agent that beats o3

Using GRPO to Beat o1, o3-mini and R1 at “Temporal Clue”

Analyzing OpenAI's Reinforcement Fine-Tuning: Less Data, Better Results

Using reinforcement learning and $4.80 of GPU time to find the best HN post

Show HN: Agent.exe, a cross-platform app to let 3.5 Sonnet control your machine

DPO fine-tuning outperforms SFT

OpenPipe Mixture of Agents: Outperform GPT-4 at 1/25th the Cost

What we've learned in 3 days of Llama 3

Mixtral Curious? Comparing Mistral 7B and Mixtral for fine-tuning

S-LoRA: Serving Thousands of Models from One GPU for Fun and Profit

Is AI the next crypto? Insights from HN comments

Fine-tune your own Llama 2 to replace GPT-3.5/4

Show HN: Automatically convert your GPT-3.5 prompt to Llama 2

YC's Startup School Relaunching as Continuous Program

comments