Ask HN: Best practices for research code?
5 comments
I've seen a very broad spectrum of research code. In general research code translates O(1e1-1e2) lines of mathematics into O(1e3-1e4) lines of code. I find mathematics easier to understand than code, so that's going to color my opinion.
My favorite research code tends to look like the mathematics it implements. And that's really hard to do well. You need to pick abstractions that are both efficient to compute and easy to modify as the underlying model changes. My favorite research code also does the reader a lot of favors (eg documents the shape of the data as it flows through the code, uses notation consistent with the writeup or standard conventions in the field).
Industry research code... I'm happy to see basic things. Version control (not a bunch of Jupyter notebooks). Code re-use (not copy+paste the same thing 20x). Separation of config and code (don't litter dozens of constants throughout thousands of lines of code). Functions < 1000 lines apiece. Meaningful variable names. Comments that link the theory to the code when the code has to be complicated.
Overall it's probably most helpful to find a researcher in your field whose code you like to read, and copy the best aspects of that style. And ask readers of your code for feedback. I really enjoy reading Karpathy's code (not my field), but that may be an exception because a lot of what I've read is intended to teach a more or less codified approach, rather than act as a testbed for iteration in a more fluid design space.
My favorite research code tends to look like the mathematics it implements. And that's really hard to do well. You need to pick abstractions that are both efficient to compute and easy to modify as the underlying model changes. My favorite research code also does the reader a lot of favors (eg documents the shape of the data as it flows through the code, uses notation consistent with the writeup or standard conventions in the field).
Industry research code... I'm happy to see basic things. Version control (not a bunch of Jupyter notebooks). Code re-use (not copy+paste the same thing 20x). Separation of config and code (don't litter dozens of constants throughout thousands of lines of code). Functions < 1000 lines apiece. Meaningful variable names. Comments that link the theory to the code when the code has to be complicated.
Overall it's probably most helpful to find a researcher in your field whose code you like to read, and copy the best aspects of that style. And ask readers of your code for feedback. I really enjoy reading Karpathy's code (not my field), but that may be an exception because a lot of what I've read is intended to teach a more or less codified approach, rather than act as a testbed for iteration in a more fluid design space.
I feel like SWE skills are underappreciated in research code. I've seen a lot of bugs creep in due to poor design or bad testing practices. Leading to the wrong conclusions. Not to mention that its harder for readers to consume if its unreadable code.
Researchers that think their code is "throwaway" dramatically limit their reach.
Researchers that think their code is "throwaway" dramatically limit their reach.
It makes sense. I can’t speak for the AI/ML field, but a lot of the software jobs I’ve seen in scientific research areas were pretty obvious they wanted scientists who could do a little code, as opposed to developers who can do a little science.
If people can comprehend your code they can point out flaws in it that invalidate your experiments. But be a good researcher and don't think like that. :)
for llm's it's a github repo - spec driven development prompt or skill with a "WIP" (work in progress) status and a broad context summary with <AGENT> instructions to chunk the document.
What's your favorite resource on how to write code in research? What are the research-code-specific equivalents of Rich Hickey's talks or SPJ's posts or the many many SWE blogposts posted to HN?