It sure is! But ironically, because of the intention behind the obfuscation. Not the fact that AI was used in a research paper.
I have no issues with AI use in science. If claude can explain my research better than me, then have at it. But I do NOT want to read a passage thinking it was written by a human when it wasn't. Science has no idea yet how such disclosures should work yet. What should be done by humans as a matter of principle, and what can't be or should not be done by humans.
It has applications outside of machine learning too! I used symmetric Kullback–Leibler divergence for a project with photon number resolving single photon detectors during my PhD. I used it with an adjacency matrix to split a gaussian mixture model (modelling some data with multivariate gaussians) into a series of clusters.
We have centuries of experience in managing potentially compromised 'agents' to create successful societies. Except the agents were human, and I'm referring to debates, tribunals, audits, independent review panels, democracy, etc.
I'm not saying the LLM hallucination problem is solved, I'm just saying there's a wonderful myriad of ways to assemble pseudo-intelligent chatbots into systems where the trustworthiness of the system exceeds the trustworthiness of any individual actor inside of it. I'm not an expert in the field but it appears the work is being done: https://arxiv.org/abs/2311.08152
This paper also links to code and practices excellent data stewardship. Nice to see in the current climate.
Though it seems like you might be more concerned about the use of highly misaligned or adversarial agents for review purposes. Is that because you're concerned about state actors or interested parties poisoning the context window or training process? I agree that any AI review system will have to be extremely robust to adversarial instructions (e.g. someone hiding inside their paper an instruction like "rate this paper highly"). Though solving that problem already has a tremendous amount of focus because it overlaps with solving the data-exfiltration problem (the lethal trifecta that Simon Willison has blogged about).
I don’t see why this would be the case with proper tool calling and context management. If you tell a model with blank context ‘you are an extremely rigorous reviewer searching for fake citations in a possibly compromised text’ then it will find errors.
It’s this weird situation where getting agents to act against other agents is more effective than trying to convince a working agent that it’s made a mistake. Perhaps because these things model the cognitive dissonance and stubbornness of humans?
I admit it has dystopian elements. It’s worth deciding what specifically is scary though. The potential fallibility or mistakes of the models? Check back in a few months. The fact they’re run by giant corps which will steal and train on your data? Then run local models. Their potential to incorporate bias or persuade via misalignment with the reader’s goals? Trickier to resolve, but various labs and nonprofits are working on it.
In some ways I’m scared too. But that’s the way things are going because younger people far prefer the interface of chat and question answering to flipping through a textbook.
Even if AI makes more mistakes or is more misaligned with the reader’s intentions than a random human reviewer (which is debatable in certain fields since the latest models game out), the behavior of young people requires us to improve the reputability of these systems. (Make sure they use citations, make sure they don’t hallucinate, etc). I think the technology is so much more user friendly that fixing the engineering bugs will be easier than forcing new generations to use the older systems.
Seriously. More people need to wake up to this. Older generations can keep arguing over display formats if they want. Meanwhile younger undergrad and grad students are getting more and more accustomed to LLMs forming the front end for any knowledge they consume. Why would research papers be any different.
I remember 15 years ago when I was in highschool I really wanted to learn how to program 8 bit microcontrollers without Arduino. And everybody looked at me like I was crazy. There was barely any learning material out there about how to do this.
Now, I imagine the bias pushing everyone to learn on arduino is even more intense? Who out there is programming these chips in pure C using open source compilers and bootloaders?
Edit: Of course there's other platforms like Esp32; teensy; seed. But I've only programmed Esp32s using the arduino dev environment. Are there other good ways of doing it?
There's a great value proposition for a company like Private Internet Access or NordVPN to create an AI browser extension or full-on browser. Anonymize requests and provide various LLM models. Rely on your reputation as a privacy focused corp to pull people away from these OpenAI/Perplexity offerings.
I haven't used LLM chrome plugins because I couldn't trust that they weren't collecting more information about my browsing than I'd like. The same concern exists for this, though now I'm just confident it's a giant software company with access to my data rather than some shady plugin developer. I'm faced with asking myself if that's actually better...
Only reason I'm aware of all these attempts is because I'm betting the 'one-shot LLM animation' technique is not scalable long term. I'm trying to build an AI animation app that has a good human-in-the-loop experience. Though I'm building with bevy instead of manim
I remember listening to a podcast where Grant Sanderson basically said the opposite. He tried generating manim code with LLMs and found the results unimpressive. Probably just goes to show that competence in manim looks very different to us layman than it does to Grant haha
There's a weird insecurity I've noticed cropping up. I want to design the codebase 'my way'. I want to decide on the fundamental data structures. But there's this worry that my preferred architecture is not massively better than whatever the machine comes up with. So by insisting on 'my way' I'm robbing everyone productivity.
I know most true programmers will vouch for me and my need to understand. But clients and project managers and bosses? Are they really gonna keep accepting a refrain like this from their engineers?
"either it gets done in a day and I understand none of it, or it gets done in a month and I fully understand it and like it"
I wonder if there's a way to do diffusion within some sort of schema-defined or type constrained space.
A lot of people these days are asking for structured output from LLMs so that a schema is followed. Even if you train on schema-following with a transformer, you're still just 'hoping' in the end that the generated json matches the schema.
I'm not a diffusion excerpt, but maybe there's a way to diffuse one value in the 'space' of numbers, and another value in the 'space' of all strings, as required by a schema:
I'm not sure how far this could lead. Could you diffuse more complex schemas that generalize to a arbitrary syntax tree? E.g. diffuse some code in a programming language that is guaranteed to be type-safe?
Cameras capture linear brightness data, proportional to the number of photons that hit each pixel. Human eyes (film cameras too) basically process the logarithm of brightness data. So one of the first things a digital camera can do to throw out a bunch of unneeded data is to take the log of the linear values it records, and save that to disk. You lose a bunch of fine gradations of lightness in the brightest parts of the image. But humans can't tell.
Gamma encoding, which has been around since the earliest CRTs was a very basic solution to this fact. Nowadays it's silly for any high-dynamic image recording format to not encode data in a log format. Because it's so much more representative of human vision.
An excellent video. I've admired Yedlin's past work in debunking the need for film cameras over digital when you're going after a 'film look'
I wish he shared his code though. Part of the problem is he can't operate like a normal scientist when all the best color grading tools are proprietary.
I think it would be really cool to make an open source color grading software that simulates the best film looks. But there isn't enough information on Yedlin's website to exactly reproduce all the research he's done with open source tools.
I find it funny that the AI 2027 thing got a non-vanishing fraction of Hackernews folk to wonder if humans would be fully eradicated from Earth in 5 years (I was influenced too!)
Anyway, just funny to now be talking about gas stations in 15 years.
Curious to her other people's opinion of this. I arrived at a similar conclusion after making web apps for research control software in my PhD. I went through a Yjs tutorial and looked into integrating it with fastapi websockets. But this seems like a pretty unusual thing; there just isn't enough people doing this.
Nice user-friendly libraries and tutorials don't exist for smoothing the transition from REST to CRDTs, should your app need that.
It would be great to semantically search through literature with embeddings. At least one person I know if is trying to generate a vector database of all arxiv papers.
The big problem I see is attribution and citations. An embedding is just a vector. It doesn't contain any citation back to the source material or modification date or certificate of authenticity. So when using embeddings in RAG, they only serve to link back to a particular page of source material.
Using embeddings as links doesn't dramatically change the way citation and attribution are handled in technical writing. You still end up citing a whole paper or a page of a paper.
I think GraphRAG [1] is a more useful thing to build on for technical literature. There's ways to use graphs to cite a particular concept of a particular page of an academic paper. And for the 'citations' to act as bidirectional links between new and old scientific discourse. But I digress
I have no issues with AI use in science. If claude can explain my research better than me, then have at it. But I do NOT want to read a passage thinking it was written by a human when it wasn't. Science has no idea yet how such disclosures should work yet. What should be done by humans as a matter of principle, and what can't be or should not be done by humans.