I think what you're really getting at is that it's only useful if the benchmarks are predictive of your workloads. If it predicts well (for example, your tasks are equally easy), then the fact that a larger model can complete it more quickly means that you may be able to complete the task more cheaply, depending on the token cost.
If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.
I remember reading this blog post when it was first published, but the subsequent updates are better than I would've ever expected this to turn out. Worth checking it out again if you've seen it before :)
I would hope this is true both in the context of LLMs and more broadly, but I think this is especially not the case for LLMs. It's hard to take the idea that companies are trying to hire people with reservations about LLMs seriously when many companies have LLM use mandates. It is counterproductive in the eyes of the employer to hire employees that will be combative on LLM from day one.
Are you going to ask your employees for their 2% cashback if you reimburse them for a purchase they made with their credit card too?
At the end of the day, it seems more practical to focus on ensuring that they picked the appropriate value product/service. If they picked appropriately, does it matter if the company gives them employee a kickback? Sure, the employee will then be incentivized to get the most expensive thing they can, but this was already the case because it's not their money and they want the best thing they can get away with.
Is what you suggest about training even possible? Most exploitation techniques are really just about having in-depth knowledge of how components work. For example, I imagine a sufficiently powerful model could fairly easily re-invent the ROP chain from first principles if it just knew how the stack works. This same principle applies to much more complex attack too; exploitation is often just an exercise in knowing vastly too much trivia, which LLMs tend to have in spades.
Is interactive use for coding something that actually works today? With unsafe mode, even frontier hosted models are slow enough I end up just tabbing out to work on other tasks. It would need to be much faster if I am to sit and stare at it while it churns. Local models might be a lot slower but workflow-wise it doesn't change much for me.
Reading this thread, I'm starting to think that I did not fall out of a coconut tree and that I exist in the context of all in which I live and what came before me.
Another trick is that people are usually nicer to you if you talk to them after having gone to the same place at the same time for a while. If you smile and waive at them a few times before you go and talk to them, you've built a bit of familiarity by nature of being a "regular" and aren't just cold approaching people you've never seen.
>The ergonomics of using a laptop on an economy-class tray table are not worth it. You're sitting there like a T-rex
The trick I've found is to pack a bluetooth keyboard. If you put your laptop on the tray table, you can put the bluetooth keyboard on your legs _under_ the tray table and have your arms fully and comfortably extended. This works especially well if you're a vim/emacs/other keyboard driven editor user as you very rarely need to reach up to poke the trackpad .
>If they are implemented as a "rendez-vous" absolute date
Do people actually do that? What's the plan for when the user sleeps their machine? All the events just inexplicably happen all at once when they wake it?
I wonder how much of the cost is coming from the cache misses vs the more frequent indirections/ILP drop?
For example, I wonder what this test looks like if you don't randomize the chunks but instead just have the chunks in work order? If you still see the perf hit, that suggests the cost is not from the cache misses but rather the overhead of needing to switch chunks more often.
What's the concern? Harness tends to be fairly common in the context of "shim program which manages some other program" (see: "test harness", "fuzzing harness", etc.)
This has also been a common theme in recent decades with respect to privacy.
In the US, the police do not generally need a warrant to tail you as you go around town, but it is phenomenally expensive and difficult to do so. Cellphone location records, despite largely providing the same information, do require warrants because it provides extremely cheap, scalable tracking of anyone. In other words, we allow the government to acquire certain information through difficult means in hopes that it forces them to be very selective about how they use it. When the costs changed, what was allowed also had to change.
Does it have the same issue with key repeat? It's a long standing iOS bug, but some apps seem to fix it (somehow). Without the fix, vim ends up being essentially unusable since I can't press and hold on the cursor movement keys. iSH, for example, hits the bug and it severely limits the usefulness of the app.
Wouldn't that apply only to a truly unlimited subscription? Last I looked all of their subs have a usage limit.
If you're on anything but their highest tier, it's not altogether unreasonable for them to optimize for the greatest number of plan upgrades (people who decide they need more tokens) while minimizing cancellations (people frustrated by the number of tokens they need). On the highest tier, this sort of falls apart but it's a problem easily solved by just adding more tiers :)
Of course, I don't think this is actually what's going on, but it's not irrational.
You don't have to fly in acro mode lol. The common hobbyist drone firmwares have full support for even things like autonomous GPS missions. You also don't need expensive gimbal stabilized cameras; you're not making a cinematic film, so you can just hot glue a 360 camera to the bottom and deal with the slight oscillations.
The original example is very common in ISAs at least. Both ARMv8 and RISC-V (likely others too but I don't have as much experience with them) have the idea of requiring software to treat reserved bits as if they were zero for both reading and writing. ARMv8 calls this RES0 and an hardware implementation is constrained to either being write ignore for the field (eg read is hardwired to zero) or returning the last successful write.
This is useful as it allows the ISA to remain compatible with code which is unaware of future extensions which define new functionality for these bits so long as the zero value means "keep the old behavior". For example, a system register may have an EnableNewFeature bit, and older software will end up just writing zero to that field (which preserves the old functionality). This avoids needing to define a new system register for every new feature.
If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.