jimberlage

386 karmajoined vor 7 Jahren

[email protected]

comments

vor 5 Tagendiscuss

I can understand why this might make for a bad benchmark, but

Overly strict tests enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.

Underspecified prompts omit requirements that hidden tests enforce and that are not reasonably inferable.

Low-coverage tests under check the requested feature, so incomplete fixes can pass.

A misleading prompt points models toward the wrong behavior or contradicts what tests require.

If the goal is, "how does my model compare to real SWEs", these are pretty reasonable situations that your model will have to encounter. It's a little like making a nursing exam and then flagging that some of the tests required you to ask the attending doctor for additional information that's not in the chart, or that the patient's family didn't fully explain their aging grandma's medical history.

I can understand why they might want a tighter benchmark, but if you're OpenAI and you promised your model as a replacement for real workers, this isn't the best look. It seems like you would want to test these things.

vor 13 Tagendiscuss

You can be correct, but on different axis.

You can be correct that your method makes code more DRY, and miss the point that the other person believes that things are going to diverge significantly over time and doesn’t value DRY.

You can be correct that your method is more resilient to failure, and miss that the other person believes that some level of failure is OK and wants an option that is less technically complex.

I’ve seen people get upset that they were correct and yet the room shifted against them. Most times, it seems like they are correct. But they are correct on a narrow axis, that misses the motivations of the other people in the room.

This is part of the reason high level account reps focus on the mix and viewpoints of people in the room over technical specs. Get the lay of the land first, and then you can tailor your pitch to be correct in the way that the audience will be receptive to.

letzten Monatdiscuss

I would love to have a unicode character for representing the start/end of a data block so that LLMs could at least send data meant to be uncorrupted down a different path at tokenization.

letzten Monatdiscuss

Managers will document for employees. That’s maybe the better comparison.

vor 2 Monatendiscuss

Plague, Inc (an iOS game where you control the parameters of a pandemic and try to get a 100% infection rate) will give you a really good feel for the math behind this.

The most successful strategy is to make a virus that spreads fast, with few visible symptoms until the late stages of the disease. A deadly virus, early will just cause borders to be locked and the international research community to swarm on a cure.

vor 3 Monatendiscuss

Yeah this was me. I just got a message that I hit my limit and now I am looking into what it takes to run Qwen on local hardware.

vor 3 Monatendiscuss

I don't think this was the British. (Not to apologize for them - they certainly made things worse, not better.) Sudan sits on a historical chattel slavery route that stretches back to Roman times. It's hallmarked by the Northern population raiding the south, along racial lines.

Scholarly article for reference if you want to learn more: https://www.jstor.org/stable/827888

vor 3 Monatendiscuss

Assuming that all homes are at equal risk of being burglarized. In practice the neighborhoods I’ve seen are either at much higher risk or much lower risk.

vor 5 Monatendiscuss

William Shatner has the most experimental, wild Spotify I've ever seen. If you haven't ever seen it, look at his discography. He does a lot of almost spoken-word poetry over soft rock, punk, etc. You get the sense that he views acting as his side hustle and is waiting for his musical career to take off.

vor 6 Monatendiscuss

And from the comments below, sounds like they might be aggressively crawling still, but unidentified or with a different crawler identity. So perhaps they are hoovering up everything in the AI era.

vor 6 Monatendiscuss

I remember back in the day, when SEO was a more viable channel, being surprised at how much of the game was convincing Google to crawl you at all.

I naively assumed that they would be happy to take in any and all data, but they had a fairly sophisticated algorithm for deciding "we've seen enough, we know what the next page in the sequence is going to look like." They value their bandwidth.

It led to a lot of gaming of how you optimally split content across high-value pages for search terms (the 5 most relevant reviews should go on pages targeting the New York metro, the next 5 most relevant for LA, etc.)

I'm surprised again, honestly. I kind of assumed the AI race meant that Google would go back to hoovering all data at the cost of extra bandwidth, but my assumption clearly doesn't hold. I can't believe I knew all that about Google and still made the same assumption twice.

vor 7 Monatendiscuss

So half the population would benefit? Half the population is more than enough reason to do all that and more.

vor 2 Jahrendiscuss

The security vulnerabilities that affect my PII kinda contradict the idea that the kernel is an implementation detail. It’s not an implementation detail if I do, in fact, have to care about it.

vor 2 Jahrendiscuss

Is anyone using Rama at scale? This isn't a complaint about it being closed-source, people have to make their money somehow. But I've seen a lot about Rama's scale when the only things to check the claim are a private beta and a single-process dev build.

I'm super curious to hear from people who have tried it for larger projects!

vor 2 Jahrendiscuss

Sourced from https://www.ojp.gov/ncjrs/virtual-library/abstracts/united-s....

vor 2 Jahrendiscuss

If he’s eligible for parole at 1/3 of a 25 year sentence, he will spend anywhere from between 8 years, 4 months in prison or 25 years. That would make him somewhere between 40 and 57 when he gets out.

vor 2 Jahrendiscuss

A reasonableness test (like what a court system uses) makes this quiz really easy to answer. Basically everything is either allowed or a margin case except for the person driving their Civic in the park.

TBH, I think if you have this quiz to a non-engineer, they would come away wondering what the point is.

Perhaps the answer is that content moderation should just let a judge make a decision and maybe appeal?