HackerTrans
トップ新着トレンドコメント過去質問紹介求人

BalinKing

867 カルマ登録 10 年前
Third-year Ph.D. student at CMU, working on programming languages and formal verification; CS undergrad at Caltech (BS '23, Venerable); ex-professional software developer.

https://github.com/jgrosso

投稿

ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases

arxiv.org
2 ポイント·投稿者 BalinKing·4 か月前·0 コメント

コメント

BalinKing
·12 時間前·議論
I ran into that issue too w/ sentence-based flashcards, where I almost immediately memorize the sentence itself and the whole thing becomes self-defeating. Similarly, I thought to use LLMs to generate fresh sentences on-the-fly, but the output was never reliable enough for my use-case... e.g. I came across some grammatical construction that the LLMs refused to use correctly. But, this probably depends on the language in question, since it sounds like you had success with it! Maybe I should try again at some point, now that the consumer models are a lot beefier than they were even six months ago.
BalinKing
·13 時間前·議論
But Wikipedia[0] says Ῥωμαῖοι transliterates to Romaioi, i.e. "Romans".

[0] https://en.wikipedia.org/wiki/Byzantine_Empire#Nomenclature
BalinKing
·5 日前·議論
On the subject of nostalgic paint apps, I distinctly remember one of the Kid Pix drawing tools/games that I had growing up... what a weird piece of software that was!
BalinKing
·8 日前·議論
Welcome to HN. Just a heads up that, from the site guidelines[0]:

> Don't post generated text or AI-edited text. HN is for conversation between humans.

(I'm not a mod or anything; I just noticed that most of your comments have been flagged/killed, and I suspect this might be why.)

[0] https://news.ycombinator.com/newsguidelines.html
BalinKing
·19 日前·議論
IIRC linguists (like, actual academic researchers) prefer to “break up” kana and analyze the consonant and vowel separately when dealing with conjugations. Treating kana as indivisible units is AFAIK only really a thing in Japanese linguistics historically done in Japan. All this to say, I’m pretty sure this “just change the vowel” approach is perfectly fine from a theoretical perspective (and as a fellow learner, also very aesthetically satisfying :-) )
BalinKing
·24 日前·議論
The lead says "how I approach IIS targets during bug bounty" (emphasis mine), so (assuming the author is being truthful) I'm guessing the tone of the title is just for fun.
BalinKing
·先月·議論
Excluding supergeniuses, pure mathematics—even at a very basic, undergraduate level—simply can't be understood passively. Even with an infinitely patient AI teacher who could answer any question on-demand, it'd still require a massive amount of work to actually understand anything in research-level mathematics. Basically every single word in a mathematical definition is a term of art, and (IME) if one doesn't grok each of those words at a fairly deep level, the new definition never really makes too much sense. And this applies recursively: each of the words has some thoroughly inscrutable definition of their own.

Of course it'd be super helpful to have, say, a teacher who could tailor explanations to anyone's precise background (e.g. where possible, using examples that come from the student's field of study when explaining some abstract concept). Or, if some definition comes with some precondition that has no obvious purpose, perhaps an omniscient teacher could explain why it's there with concrete counterexamples.[0] But even granting all this, I think that mathematical intuition is necessarily based on a lot of hard work actually exploring definitions on one's own, with pencil-and-paper and a lot of thought. That is to say, even though the process could probably be sped up a lot with a nigh-omniscient teacher[1], I doubt that a student wouldn't still need years of training to even have a clue what's going on.

(I'm saying all this, by the way, as someone who is terrible at all this and has very little mathematical maturity[2]—I'm speaking from my own frustrating experience....)

[0] c.f. Lakatos' excellent book Proofs and Refutations

[1] without the "curse of knowledge," or else we're back to square one of "answers that are correct but useless"

[2] e.g. the "post-rigorous stage" described in https://terrytao.wordpress.com/career-advice/theres-more-to-...
BalinKing
·2 か月前·議論
> horrific things are going to happen to them without the their/or their families consent

Indeed, that is (allegedly) the case with organ donation: https://www.nytimes.com/2025/07/20/us/organ-transplants-dono...
BalinKing
·2 か月前·議論
Towards the end of the article: "the company plans to eventually remove the anesthesia from some brain slices"

Here's hoping the idea is that the slices will be really small, or something, because frankly the whole thing is utterly horrifying enough as-is.
BalinKing
·2 か月前·議論
The advantage(?) of take-home exams à la Caltech is that they can be open everything and 3–5 hours long :-P (For what it's worth, being able to listen to music during an exam, ctrl+F a digital textbook, etc. was super awesome; it would deeply sadden me if that becomes infeasible in the future once enough students stop caring about the Honor Code....)
BalinKing
·2 か月前·議論
Things may have changed, but I don't recall any group exams during my time at Caltech, and conversely I do recall a strong sense of pride in the Honor Code. Also, if your professor allows collaboration, then it's definitionally not cheating: There is a vast moral difference between "the professor made the assignments difficult with the specific expectation that people will collaborate" and "the professor doesn't want collaboration but people did it anyway".

Frankly, this comment feels almost entirely foreign to my experience—I suppose things could've changed over the years (although my impression is that things have gotten much worse recently, not better), or it could be major-specific, or I just got lucky with the specific people I happened to hang out with?
BalinKing
·2 か月前·議論
From the site guidelines (https://news.ycombinator.com/newsguidelines.html):

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
BalinKing
·2 か月前·議論
I'm not the other commenter (and I believe you that it's not AI), but I'd guess it's mostly the first line: a short affirmation followed by "The problem is ...." feels like the sort of formula the LLMs love to use. (Not trying to imply that there's anything inherently wrong with it, of course.)

While we're at it, I'm under the impression that the recent LLMs have also co-opted "genuinely", which I'll never forgive them for—first they stole my em-dashes, and now they're stealing my adverbs too?!
BalinKing
·2 か月前·議論
In general, I think this phenomenon is called "phono-semantic matching": https://en.wikipedia.org/wiki/Phono-semantic_matching.
BalinKing
·3 か月前·議論
It's just that, in my (uninformed) opinion, Anthropic is incentivized a priori to claim things like this about their models. Like, it's probably really good marketing to say "our product is so smart, and we're so concerned about ethics, that made sure a psychiatrist talked to it". I guess it's ultimately a judgment call, but to me the conflict of interest seems big enough that I'm really wary of this sort of argument. (I'm reminded of when OpenAI claimed GPT-5(?) was "PhD-level"—I can personally attest that, at least in my field, this is totally inaccurate.)
BalinKing
·4 か月前·議論
In the paper itself, the abstract actually does have a paragraph break, so it's probably just an autoformatting issue or something.
BalinKing
·4 か月前·議論
AI-generated comments are disallowed by HN guidelines: https://news.ycombinator.com/newsguidelines.html#generated
BalinKing
·4 か月前·議論
I think a steelman interpretation of the parent is that entirely LLM-generated projects should be disallowed. There's a lot of submissions on Show HN that seem completely vibe-coded to me (like, including the README), which is a very different situation IMO from someone who simply used Claude to write some—or even most—of the code. When even the human-facing portion of a submission is LLM-generated, it bothers a lot of people (myself included).
BalinKing
·4 か月前·議論
Sorry, updated my original comment—I meant to qualify it to only those cases where it's blatantly obvious. Obviously a lot of ambiguous comments will slip through as a result, but I agree with you that false negatives are better than false positives.
BalinKing
·4 か月前·議論
I furthermore wish that "posting an LLM-generated comment (i.e. and passing it off as your own)" was worthy of an instant ban, because I see this sort of behavior from non-green accounts as well.

EDIT: I meant (but totally forgot) to qualify that my "proposal" would only apply when the LLM-ness is self-obvious—idk, make up a "reasonable person" standard or something. Presumably, the moderators would err on the side of letting things slide. Even so, many comments I've seen are simply impossible for any reasonable person to claim as "human-written"—the default ChatGPT style is simply too distinct.