Looked at the repo but it says NOTHING about what value this project offers.
I mean, I get that it's "fun" to store information within the digits of pi. But is this just amusement, or is there a value prop for production use here?
(Speaking as a math major, by the way. I'm sympathetic to the cause.)
This site will probably defeat its purpose. You discover an empty showing and are excited to have your own "private theater", but thanks to this site, somebody else will have the same idea and you'll both have to share your private theater.
A few other commenters have talked about the paper review process.
I wasn't thinking of this at all. Important to understand: the peer review process takes up only a minor part of a professor's mindshare. It's considered a chore. Much more important is to read lots of new papers (including pre-prints) for continual education, to know what's going on in your field and adjacent fields.
Here's an important aspect to understand: successful professors don't read papers in full. They're too busy for that. They only take a look at the title, abstract and introduction — and perhaps they will glance at the figures. This is why telling a compelling story is so important.
The fact that the strategy makes zero returns suggests that Polymarket is unbiased — this is moderately interesting.
Has anybody looked into the repo in more detail? I imagine it's useful for infrastructure inspiration to build your own bot pursuing more differentiated trading strategies.
An interesting side effect might be that only people locked out from using LLMs will learn how to program in the future, as vide coding doesn't teach you the fundamentals.
I know what you're thinking — when the calculator came about, being forced to compute in your head wasn't an advantage. But LLMs are different: a calculator is a strictly improved substitute for mental arithmetic, whereas an LLM is only an approximate solution — and it is far from clear whether LLMs will ever become a perfect solution, given the nuanced challenges around context management, interpreting intent, etc.
That's a great way to look at it. The paper is a reality check for anyone who thinks of benchmarks as these monolithic, oracular judges of performance. It highlights the soft underbelly of benchmarking.
Yes, I did see that section. We've known for a while that reward hacking, train/test data contamination, etc. must be taken seriously. Researchers are actively guarding against these problems. This paper explores what happens when researchers flip their stance and actively try to reward hack — how far can they push it? The answer is "very far."
I'm not sure if the paper's findings are all that actionable. The paper doesn't say "here's how benchmarks are currently being gamed." It says "here's how benchmarks could in theory be gamed."
Whether benchmark results are misleading depends more on the reporting organization than on the benchmark. Integrity and competence play large roles in this. When OpenAI reports a benchmark number, I trust it more than when that same number is reported by a couple Stanford undergrads posting "we achieved SOTA on XYZ benchmark" all over Twitter.
To anybody who wants to try this at home: consider wearing a face mask while you do the filing. The author mentions taping off speakers etc. to protect the machine's internals from aluminum dust, but don't forget to protect your own body, too. Aluminum exposure has been linked to Alzheimer's, and inhalation likely poses particularly high risk, compared to ingestion.
This is an interesting catalog of vulnerabilities, but I'm not sure how groundbreaking the main insight is.
Evaluating AI models has always relied largely on trust. If you want to game the benchmarks, you can. Simply train on your test data.
When an AI agent has autonomous control over the same computing environment where its scores are recorded, it's not surprising that it can, in principle, falsify its scores. A more interesting question would be whether agents behave in this way automatically, without manual tuning by the researcher.
That said, the main takeaway of "don't trust the number, trust the methodology" is valid. It's already a truism for researchers, and spreading the word to non-researchers is valuable.
It's interesting that Monet—the painter who was later criticized for his inhuman, all-too-realist depictions of his fellow creatures—started his career drawing caricatures.
I agree for the most part. DND isn't perfect, though. When you're bored, your mind naturally searches for things to do, and you'll be tempted to proactively check your lock screen, which unhelpfully informs you about "3 messages received while in Do Not Disturb." Now you really want to know what those messages are.
This is why I tend to keep my phone physically far away from me, and out of sight.
Intermittent idleness is appealing and even productive, as it often surfaces valuable ideas from your subconscious. That said, today's society is badly equipped for idleness. With phone notifications going off every few minutes, it's difficult not to be constantly interrupted with the "task" of looking at a text. Let's throw out our phones first, then we can experience true mental repose.
If I understand this right, the difference between the author's suggested approach and simply chatting with an AI agent over your files is hyperlinks: if your files contain links to other relevant files, the agent has an easier time identifying relevant material.
This take confuses the value of a project at inception with its value at maturity. Vibe-coded projects are at the beginning of their life. When Slack was at a comparably stage, it similarly didn't have hundreds of engineers running it. So the question facing vibe coding is not whether it can substitute for a mature tech product. The question is if vibe coding can substitute for genuine engineering expertise at the very beginning of a budding, immature project.
The privacy angle is interesting. I'm curious how people view the pricing strategy of taking a one-time payment for lifetime access. My first thought was that it encourages the developer to focus more on recruiting new users rather than keeping existing ones happy - makes me wonder what will become of the product if new user growth stalls.