My New Year’s Resolution – read a research paper every weekday(blog.acolyer.org)
blog.acolyer.org
My New Year’s Resolution – read a research paper every weekday
https://blog.acolyer.org/2016/12/29/my-new-years-resolution-read-a-research-paper-every-weekday/
47 コメント
I'm not sure that one can derive significant value from reading a research paper per day. Great research is not easily grokked or appreciated upon first read. I think one would derive far more value from reading a single paper for an entire week.
Yes, I'm planning to cut back too. - An academic
You should try writing these on authorea.com (get DOI and permanent archival). Basically, turning them into official pieces of open post-publication peer review. Could cross-post to your blog too still!
I don't know about CompSci research. However I do http://outcomereference.com/ which is just a side-project and read about 2-3 medical research papers everyday and the biggest challenge is open access to papers. It is possible to find the PDFs on various research sites but the vast majority is costly (avg $40 per paper). Not sure if there is free & open access for CompSci research?
I like the idea of scientifically vetting foods, but as you said it might be hard coming up with a balanced viewpoint if you are not considering all the research. Also might be helpful to consider the impact factor of each paper you are citing.
Let me get my smart-ass instinct out of the way first: journals have impact factors, not papers :)
But it doesn't really matter because it's a somewhat-empty vanity-metric anyway. For assessing clinical trials (and aggregating their results) the Cochrane collaboration is probably the gold standard. Here's something I found on their method to evaluate trial quality: http://www.bmj.com/content/343/bmj.d5928
But it doesn't really matter because it's a somewhat-empty vanity-metric anyway. For assessing clinical trials (and aggregating their results) the Cochrane collaboration is probably the gold standard. Here's something I found on their method to evaluate trial quality: http://www.bmj.com/content/343/bmj.d5928
In CompSci it's pretty rare for there not to be open access versions of a major paper. Off the top of my head there have been a few in Nature but outside that almost everything ends up as open access.
A lot of CS papers are available on http://citeseerx.ist.psu.edu/
[deleted]
You can read one paper a day, but I am not sure you can understand one paper a day.
At least for me during my PhD I spent more than a full-time day to completely understand papers that were interesting for my research.
At least for me during my PhD I spent more than a full-time day to completely understand papers that were interesting for my research.
I've found that there's generally two levels of paper understanding. The first is achievable in a day, and consists of recognizing what the authors did and roughly how their technique works and why it's different. The second, much more nuanced level of understanding comes when you really dig into the paper and analyze every sentence, and commonly occurs when you need to extend or implement it. The writing is rarely wrong, but many things that are not relevant are either easier said than done or painful but boring implementation details. At that point, you could rewrite their paper from scratch if needed.
I have 2 comments saying just the opposite that good/bad papers require more time to read than bad/good papers.
So a bit of clarification to my comment. If I spend months reading papers about the same topic, the first papers will take me a lot of time but the last papers will take me less time because I have a lot of background and I don't have to check references, review concepts, understand maths proof or whatever. Probably those papers are very related and share a lot of information.
Good papers (for whatever you think it is good) require time to read because you really want to understand it completely. Bad papers (for whatever you think a bad paper is) require time to read because you cannot be sure if that paper is relevant for you unless you understand it completely. Although at the end you have a ton of papers to read and you check the abstract and decided whether spend time on the paper or not.
So a bit of clarification to my comment. If I spend months reading papers about the same topic, the first papers will take me a lot of time but the last papers will take me less time because I have a lot of background and I don't have to check references, review concepts, understand maths proof or whatever. Probably those papers are very related and share a lot of information.
Good papers (for whatever you think it is good) require time to read because you really want to understand it completely. Bad papers (for whatever you think a bad paper is) require time to read because you cannot be sure if that paper is relevant for you unless you understand it completely. Although at the end you have a ton of papers to read and you check the abstract and decided whether spend time on the paper or not.
I'm slowly finishing up my PhD and I tend to agree with you. However, he will probably be reading quite a different selection of papers than what you encounter during a PhD.
In my experience, time required varies quite a lot between papers and authors. If you take a paper from, say, top 1% (or 0.1%) of papers published on some broad topic, you can probably read through it once and get some relatively good understanding of what the author was trying to say (and very likely enough to write a short re-cap).
This is definitely not true for the remaining 99% of the published scientific literature. At one point I had the idea of keeping up with my field of research by setting up Google Scholar alerts. This resulted in about 1 paper per 2-3 days that seemed to be worth studying based on the title and abstract. Still at that rate I found it quite impossible to keep up with all the content that dredged up.
In my experience, time required varies quite a lot between papers and authors. If you take a paper from, say, top 1% (or 0.1%) of papers published on some broad topic, you can probably read through it once and get some relatively good understanding of what the author was trying to say (and very likely enough to write a short re-cap).
This is definitely not true for the remaining 99% of the published scientific literature. At one point I had the idea of keeping up with my field of research by setting up Google Scholar alerts. This resulted in about 1 paper per 2-3 days that seemed to be worth studying based on the title and abstract. Still at that rate I found it quite impossible to keep up with all the content that dredged up.
I took classes in my PhD where we would read 4 papers a week, which is close to one a weekday. Some of the papers were very theory-heavy, when those came up you would spend half a day just reading.
Of course, for a seminar class, you do the strategic thing and read enough to not be caught flat-footed during seminar, and to come up with a few salient points of discussion.
I looked some at the papers read for this project and they seemed to be very systems-heavy. Having taken both theory-heavy and systems-heavy seminar classes, the systems-heavy seminar reading loads were much easier. You could probably read 5-8 systems papers a week, to comprehension, if you really wanted to.
Of course, for a seminar class, you do the strategic thing and read enough to not be caught flat-footed during seminar, and to come up with a few salient points of discussion.
I looked some at the papers read for this project and they seemed to be very systems-heavy. Having taken both theory-heavy and systems-heavy seminar classes, the systems-heavy seminar reading loads were much easier. You could probably read 5-8 systems papers a week, to comprehension, if you really wanted to.
I'd state it with a bit more nuance.
From my experience in physics:
You can read a paper in 15-30 minutes and get the main points and contributions.
For good papers, you can spend multiple days to really understand the paper.
And for really good papers you can spends months to completely understand them, then extend them, and then publish your dissertation.
From my experience in physics:
You can read a paper in 15-30 minutes and get the main points and contributions.
For good papers, you can spend multiple days to really understand the paper.
And for really good papers you can spends months to completely understand them, then extend them, and then publish your dissertation.
Reminds me of the 3 pass approach in the paper, "How to read a paper"
http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/p...
http://blizzard.cs.uwaterloo.ca/keshav/home/Papers/data/07/p...
[deleted]
Just read or read AND understand ?
I've been reading Adrian's blog for about a year now, I honestly I don't know how he finds the time.
I would say that he's insane, but his blog already summarizes a research paper every weekday. I had thrown around the idea of doing this with my own field, but I found that it took me the better part of a day to really understand the paper, much less commit a summary to prose.
This should be a movement.
I will try to read at 1 paper per month. Good enough for a beginner.
I will try to read at 1 paper per month. Good enough for a beginner.
You might be interested in Fermart's Library:http://fermatslibrary.com
Question: What are the selection criteria people use to determine which papers to read.
I went through a period where I tried to get in the habit of reading research papers regularly. Two issues I found that lead me to eventually dropping this:
(1) At least in my industry (generative architecture & building performance modeling) it's frustrating how often papers don't include technical details, and resort simply to a high-level overviews of implementation and go straight to detailing their outcome. I don't learn much from such papers.
(2) Low return of value of simply just reading the paper. I went through a period where I tried to read a paper everyday. By the end of this period I had a massive stack of highlighted, and annotated research papers I kept in a folder on my desk. And that folder has remained there untouched to this day. So I find there is low value in simply consuming research, and the key for me at least is to actually struggle through and implement the research I read. Which is why point (1) is key for me.
I'm inspired by your post though, I want to start getting back into a slightly modified version of this. Identify research papers with algorithms I can play with/experiment - and then implement it, aiming for something like a paper per week or so? This process means I won't be able to do a paper per day, but it'll be a lot more valuable/interesting to me. This calls for a new git repo...
I went through a period where I tried to get in the habit of reading research papers regularly. Two issues I found that lead me to eventually dropping this:
(1) At least in my industry (generative architecture & building performance modeling) it's frustrating how often papers don't include technical details, and resort simply to a high-level overviews of implementation and go straight to detailing their outcome. I don't learn much from such papers.
(2) Low return of value of simply just reading the paper. I went through a period where I tried to read a paper everyday. By the end of this period I had a massive stack of highlighted, and annotated research papers I kept in a folder on my desk. And that folder has remained there untouched to this day. So I find there is low value in simply consuming research, and the key for me at least is to actually struggle through and implement the research I read. Which is why point (1) is key for me.
I'm inspired by your post though, I want to start getting back into a slightly modified version of this. Identify research papers with algorithms I can play with/experiment - and then implement it, aiming for something like a paper per week or so? This process means I won't be able to do a paper per day, but it'll be a lot more valuable/interesting to me. This calls for a new git repo...
A fun game is following the reference trail right back to the source works, often decades ago. Pick something popular, say convolutional neural nets. Do a brief literature review and try and read the most highly cited papers as far back as you can go. You'll usually hit some really interesting (and very readable) old papers that cover really groundbreaking things. Using the convnet example, at some point you'll come across the Nobel winning paper on how cat brains interpret visual stimuli (Hubel, 1963).
I used to have a subscription to nature and just leaf through it, going as deep as I got into each paper.
The subscription was quite cheap (may have been using student discount), and it has a sort-of "popular science" section in the front summarising the major papers for audiences from other fields. Nature publishes papers from all fields of science and it can be quite entertaining to take a deep dive into vulcanology. You'll also get to notice when different branches of science end up using the same methods.
Of course that's more for entertainment value (and possibly annoying your friends). for work I stsck everything that appears interesting into an app (app-tly named "Papers") and search through it when I think there could be something in there.
The subscription was quite cheap (may have been using student discount), and it has a sort-of "popular science" section in the front summarising the major papers for audiences from other fields. Nature publishes papers from all fields of science and it can be quite entertaining to take a deep dive into vulcanology. You'll also get to notice when different branches of science end up using the same methods.
Of course that's more for entertainment value (and possibly annoying your friends). for work I stsck everything that appears interesting into an app (app-tly named "Papers") and search through it when I think there could be something in there.
>resort simply to a high-level overviews of implementation and go straight to detailing their outcome
Those don't sound like research papers but articles... What Journal(s) are you finding these papers in?
Those don't sound like research papers but articles... What Journal(s) are you finding these papers in?
Is it only a research paper if it is published in a journal? What about conference papers?
Here's an example I was reading recently which doesn't go into much detail: http://www.ibpsa.org/proceedings/asim2012/0097.pdf from the International Building Performance Simulation Association (IBPSA) conference.
Here's an example I was reading recently which doesn't go into much detail: http://www.ibpsa.org/proceedings/asim2012/0097.pdf from the International Building Performance Simulation Association (IBPSA) conference.
Conferences certainly count if they are refereed (peer reviewed).
I don't know anything about that field, but that paper certainly doesn't pass the first glance test that others have mentioned in this thread. Good figures, etc.
I don't know anything about that field, but that paper certainly doesn't pass the first glance test that others have mentioned in this thread. Good figures, etc.
I think it's almost impossible to regularly select only papers that will be helpful to your work or thought provoking in your field. The nature of the beast is that you're going to read a lot of irrelevant or simply bad papers so you have to develop strategies to discard those quickly. The other thing is unless you're following a specific thread of research from citations or a group of authors you're probably not going to know which papers are important/worth reading ahead of time.
Some strategies I've seen conference reviewers use and that I use to quickly evaluate whether a paper is worth a deeper dive:
1. Figures. Can you understand the figure from the caption or associated text blurb? Is it neat and labeled correctly? I've found that excellent figures are a strong signal that the authors put a lot of thought and work into the paper. If you look at Yann LeCun's early works they all have really excellent diagrams with clear descriptions. Some of his figures are so classic that they regularly appear in ML presentations to this day.
2. Read the related work section. A lot of people seem to skip this but it can be dynamite if you have knowledge of the field. You'll probably be aware of other papers referenced in this section and when the authors of the current paper point out differences and similarities to other methods this will give you a great idea of whether this paper's approach is worth understanding.
3. Save the Method and Result sections for last. For me, and I would assume most people, these heavily technical sections are very time consuming to understand and ought to be read very last. Read abstract, intro, related works, and conclusion first. I would bet I discard half of the papers I pick up without ever readin the middle chunk.
4. Conference or Journal it was published in. IEEE, NIPS, CVPR are all places where the best stuff in my area gets published. Papers not from top shops should have to do more to earn your attention. Strong figures being the one I usually go back to.
One thing I would not do, that I used to do, is worry about the number of citations a paper has. Awesome new research and old but potentially newly important work will not have many citations at first.
This is just my list. I'd appreciate other filters people have for reading papers!
Some strategies I've seen conference reviewers use and that I use to quickly evaluate whether a paper is worth a deeper dive:
1. Figures. Can you understand the figure from the caption or associated text blurb? Is it neat and labeled correctly? I've found that excellent figures are a strong signal that the authors put a lot of thought and work into the paper. If you look at Yann LeCun's early works they all have really excellent diagrams with clear descriptions. Some of his figures are so classic that they regularly appear in ML presentations to this day.
2. Read the related work section. A lot of people seem to skip this but it can be dynamite if you have knowledge of the field. You'll probably be aware of other papers referenced in this section and when the authors of the current paper point out differences and similarities to other methods this will give you a great idea of whether this paper's approach is worth understanding.
3. Save the Method and Result sections for last. For me, and I would assume most people, these heavily technical sections are very time consuming to understand and ought to be read very last. Read abstract, intro, related works, and conclusion first. I would bet I discard half of the papers I pick up without ever readin the middle chunk.
4. Conference or Journal it was published in. IEEE, NIPS, CVPR are all places where the best stuff in my area gets published. Papers not from top shops should have to do more to earn your attention. Strong figures being the one I usually go back to.
One thing I would not do, that I used to do, is worry about the number of citations a paper has. Awesome new research and old but potentially newly important work will not have many citations at first.
This is just my list. I'd appreciate other filters people have for reading papers!
Are there any good feeds of newly written machine learning research papers that are worth reading and that is regularly updated?
I agree wih the rest : this is too hard. Some papers will take a long time to grasp. More so after we have been conditioned to read things fast on he web.
You're right, that does sound crazy.
Andrew Ng on this topic:
"...if you seriously study half a dozen papers a week and you do that for two years, after those two years you will have learned a lot."
http://www.huffingtonpost.com/2015/05/13/andrew-ng_n_7267682...
"...if you seriously study half a dozen papers a week and you do that for two years, after those two years you will have learned a lot."
http://www.huffingtonpost.com/2015/05/13/andrew-ng_n_7267682...
Anyone know of good online libraries to find research papers?
Don't bother trawling through libraries. Look at the list of top conferences in your field and just go through the proceedings of those (e.g. for systemsy conferences look at the list given in the article as a good starting point). You should be able to get free versions of most papers you find through Google.
sci-hub?
I wonder if a higher level equivalent of this could be to read a full IEEE article every day (from the right publications, of course), since these often seem to be more accessible writeups of academic papers anyway and a lot of the hard work of curating and picking out the key details has been done.
Spend some time on context. Reading papers in isolation maybe makes you appreciate the idea that it presents, but understanding how that idea came about and what alternative ideas it replaced or subsumed primes you for extending or applying it. For that it's perhaps more valuable to scan more background material loosely, rather than just adding today's references to your reading list. Also it's very helpful to pay attention to names and affiliations when doing this.
Another pitfall I've had with regular reading like "a paper a day/week" is not to discard papers fast enough. Often the abstract either oversells the results, or the details are simply too obvious to you to learn anything new from them. Learn to recognize "weak" (to you) papers early and skim through the rest of it quickly or look for better sources on the topic.
I'd suggest aiming to understand an idea a week, which maybe involves reading 2-4 papers, and skimming 15-20 more, plus background material in textbooks and Wikipedia. Doing that in 8-12 hours would already be pretty efficient.
Another pitfall I've had with regular reading like "a paper a day/week" is not to discard papers fast enough. Often the abstract either oversells the results, or the details are simply too obvious to you to learn anything new from them. Learn to recognize "weak" (to you) papers early and skim through the rest of it quickly or look for better sources on the topic.
I'd suggest aiming to understand an idea a week, which maybe involves reading 2-4 papers, and skimming 15-20 more, plus background material in textbooks and Wikipedia. Doing that in 8-12 hours would already be pretty efficient.
> If you like the idea of being exposed to more research and cutting-edge ideas over 2017...
Why not just read abstracts? It doesn't seem like his goal is more than "Let's see what people are up to."
In the context of AI / ML / DL, white papers were useful for me when I had a very specific task in mind and wanted to see the cutting edge approaches. Outside of that, I wasted untold hours I wish I had back reading white papers because they seemed important. I don't think I really got anything useful at the end of the day, in terms of advancing my projects. I also believe some deep learning papers completely exaggerate their results while being intentionally vague on how they implemented, and I wish had those hours back too trying to replicate.
I don't know, I disagree with the notion that it's inherently good to read a bunch of white papers. Better I think is work on your project, and when there are gaps go searching for papers.
Why not just read abstracts? It doesn't seem like his goal is more than "Let's see what people are up to."
In the context of AI / ML / DL, white papers were useful for me when I had a very specific task in mind and wanted to see the cutting edge approaches. Outside of that, I wasted untold hours I wish I had back reading white papers because they seemed important. I don't think I really got anything useful at the end of the day, in terms of advancing my projects. I also believe some deep learning papers completely exaggerate their results while being intentionally vague on how they implemented, and I wish had those hours back too trying to replicate.
I don't know, I disagree with the notion that it's inherently good to read a bunch of white papers. Better I think is work on your project, and when there are gaps go searching for papers.
This is a good stackexchange thread
http://cstheory.stackexchange.com/questions/1168/what-papers...
In my field (Software), I find more profitable to me to watch one 40 min video from a conference rather than trying to understand a paper where the author had to compress all the information in a silly 10 pages limit from a random conference.
As an author it is amazing how I feel I can communicate my work more efficiently, and with pedagogy in a talk with slides (or in a blog post maybe) than in the constraints of an academic paper.
As an author it is amazing how I feel I can communicate my work more efficiently, and with pedagogy in a talk with slides (or in a blog post maybe) than in the constraints of an academic paper.