No Paper Is That Good(econlib.org)
econlib.org
No Paper Is That Good
https://www.econlib.org/no-paper-is-that-good/
104 comments
It's an anti-intellectual stance. These become more and more popular today, even among scholars. The same attitude is often behind rejecting foreign lean words or technical jargon, it's often based on the idea that dumbing things down could somehow make you smarter. It won't. You are as smart as you are, getting gradually a bit dumber but also more experienced over the years, and the rest is education. If you reject education, the only result is that you'll be uneducated.
In my opinion, a real intellectual is interested in the topic and assorted problems and will critically read whatever fuels their interest. Even papers that are not good can be interesting and worthwhile reading. Wise men know the principle "garbage in, garbage out", though, and that their time during life is limited, so they choose their readings carefully. However, that doesn't mean that they shouldn't include fairly bad literature, too. For example, it's necessary to read seminal papers that are part of a canon, no matter how stupid they seem to be.
When I started studying philosophy a long time ago, some tutor whose name I've forgotten gave us two pieces of advice that would have been very valuable to me if I had taken them seriously earlier:
1. Never dismiss a book or paper quickly because you believe you've found a mistake. Read them until the end and take the authors seriously!
2. Aim for writing 1 DIN A4 page a day, no matter what you write (could be personal, could be professional, doesn't matter). You will write much less but as a goal about 1 page a day is good.
In my opinion, a real intellectual is interested in the topic and assorted problems and will critically read whatever fuels their interest. Even papers that are not good can be interesting and worthwhile reading. Wise men know the principle "garbage in, garbage out", though, and that their time during life is limited, so they choose their readings carefully. However, that doesn't mean that they shouldn't include fairly bad literature, too. For example, it's necessary to read seminal papers that are part of a canon, no matter how stupid they seem to be.
When I started studying philosophy a long time ago, some tutor whose name I've forgotten gave us two pieces of advice that would have been very valuable to me if I had taken them seriously earlier:
1. Never dismiss a book or paper quickly because you believe you've found a mistake. Read them until the end and take the authors seriously!
2. Aim for writing 1 DIN A4 page a day, no matter what you write (could be personal, could be professional, doesn't matter). You will write much less but as a goal about 1 page a day is good.
I'm curious if you read the reasoning as you seem to be presenting an argument based purely on the title. Certainly the original proposal doesn't seem anti-intellectual.
For example he says "full of mistakes", not one mistake.
For example he says "full of mistakes", not one mistake.
To me it is. You read the paper and then make up your mind on it. You do not dismiss a whole bunch of literature on the basis of two papers someone has provided to you.
There is nothing wrong with choosing your literature carefully, but that's far from the original suggestion. You can and should use your bullshit detector.
There is nothing wrong with choosing your literature carefully, but that's far from the original suggestion. You can and should use your bullshit detector.
The original proposal was not, "For any field of study, I will dismiss it if anyone provides me with two papers that do not meet my standards." Rather, the original proposal was "If you wish to assert a field of study is worthy of my attention, I will acquiesce and spend time studying it so long as you can provide me with two papers that meet my standards."
Because of this formulation, the standards are set rather high. Bryan Caplan's point is that if the best two papers out of a field of hundreds are not above these standards, then there is little value to be gained by slogging through the hundreds.
Because of this formulation, the standards are set rather high. Bryan Caplan's point is that if the best two papers out of a field of hundreds are not above these standards, then there is little value to be gained by slogging through the hundreds.
Not very laudable either. First you rely on someone else to give you the best papers and assume that this person knows them, instead of using your own brain. Second, anyone can wish to assert a field of study is worth someone's attention and that still doesn't oblige you to consider it worthy of your attention. Just tell them "fine, do your stuff, I'm not very interested in it" What's wrong with that?
>Bryan Caplan's point is that if the best two papers out of a field of hundreds are not above these standards, then there is little value to be gained by slogging through the hundreds.
And why would anyone need criteria for that? What kind of person would need to justify to themselves or others that they prefer to study topology instead of, say, financial mathematics?
Caplan apparently needs some criterion to be able to be dismissive about someone else's work or discipline rather than pursuing his own scientific goals and interests. Yes, I find that kind of anti-intellectual, or at least small-minded.
>Bryan Caplan's point is that if the best two papers out of a field of hundreds are not above these standards, then there is little value to be gained by slogging through the hundreds.
And why would anyone need criteria for that? What kind of person would need to justify to themselves or others that they prefer to study topology instead of, say, financial mathematics?
Caplan apparently needs some criterion to be able to be dismissive about someone else's work or discipline rather than pursuing his own scientific goals and interests. Yes, I find that kind of anti-intellectual, or at least small-minded.
The original proposal, which is not by Caplan, tries to optimize time spent reading, which is crucial as a researcher.
> And why would anyone need criteria for that? What kind of person would need to justify to themselves or others that they prefer to study topology instead of, say, financial mathematics?
We need criteria to assess the value to papers, because generally the whole assumed point of writing a paper is to add value.
> Caplan apparently needs some criterion to be able to be dismissive about someone else's work or discipline rather than pursuing his own scientific goals and interests. Yes, I find that kind of anti-intellectual, or at least small-minded.
Ad hominem much. Also, Caplan doesn't say that, and you are talking about the original proposal. You don't seem to have read the article further than the title and the premise, though interestingly you seem able to judge the author itself and convict him with bad thinking. This very much contradicts your own previous piece of advice
> 1. Never dismiss a book or paper quickly because you believe you've found a mistake. Read them until the end and take the authors seriously!
which, to me, seem rather like virtue signaling than a real advice; all the more when you don't apply it thoroughly in reality.
> And why would anyone need criteria for that? What kind of person would need to justify to themselves or others that they prefer to study topology instead of, say, financial mathematics?
We need criteria to assess the value to papers, because generally the whole assumed point of writing a paper is to add value.
> Caplan apparently needs some criterion to be able to be dismissive about someone else's work or discipline rather than pursuing his own scientific goals and interests. Yes, I find that kind of anti-intellectual, or at least small-minded.
Ad hominem much. Also, Caplan doesn't say that, and you are talking about the original proposal. You don't seem to have read the article further than the title and the premise, though interestingly you seem able to judge the author itself and convict him with bad thinking. This very much contradicts your own previous piece of advice
> 1. Never dismiss a book or paper quickly because you believe you've found a mistake. Read them until the end and take the authors seriously!
which, to me, seem rather like virtue signaling than a real advice; all the more when you don't apply it thoroughly in reality.
Look, I've never claimed I've read anything, that was your invention, and I've also said that you should use your bullshit detector and wise men choose their literature carefully.
I simply objected to the idea on the linked web page that you can ask someone to give you the two best articles in a field, then assess these, and use your resulting opinion to somehow evaluate the whole field as an outsider.
That idea is silly and small-minded. If that feels like an ad hominem attack to you, so be it.
I simply objected to the idea on the linked web page that you can ask someone to give you the two best articles in a field, then assess these, and use your resulting opinion to somehow evaluate the whole field as an outsider.
That idea is silly and small-minded. If that feels like an ad hominem attack to you, so be it.
Well, the counter point is the article is that the standards is ridiculous. And that you are just effectively setting those standard so that you'll never have to learn anything from that field, because no paper in that field will pass the ridiculous standard.
> It's an anti-intellectual stance.
It's not - it's a realization that life is short and much research is just writing.
It's not - it's a realization that life is short and much research is just writing.
It's not about dismissing the field, as much as the claim that you must have read the literature to constructively participate in a debate.
I think it’s reasonable to expect familiarity with “the literature” beyond “two exemplary papers” to be given any respect in an academic debate. If you aren’t reading the lit reviews and broadly in the topic how can you possibly contribute to the discussion productively? Almost certainly any points you want to make will have been examined and discussed already. This rule is about FOMO for wannabe rennaisssance-man know-it-alls who think their gut reactions are God’s own truth and demand a response from actual experts. If you don’t know the topic and don’t want to learn about it then please find something else to do.
Here's an example.
You stumble on a bunch of psychology papers in which psychologists noticed that whenever 6+ kids get together, there are either three mutual friends, or three mutual non-friends.
The psychologists think this has something to do with child psychology and have written 10,000 pages on it.
You immediately recognize it has nothing to do with psychology and is just basic Ramsey Theory [1]. Should you have to review thousands of pages before being allowed to chip in?
[1] https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(...
A less hypothetical example: You're Bertrand Russel and you discover Russel's paradox, a one-liner which negates thousands of pages of Frege's not-yet-published logic textbook. Do you have to wait for him to publish it, and then you read it, before raising your voice?
You stumble on a bunch of psychology papers in which psychologists noticed that whenever 6+ kids get together, there are either three mutual friends, or three mutual non-friends.
The psychologists think this has something to do with child psychology and have written 10,000 pages on it.
You immediately recognize it has nothing to do with psychology and is just basic Ramsey Theory [1]. Should you have to review thousands of pages before being allowed to chip in?
[1] https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(...
A less hypothetical example: You're Bertrand Russel and you discover Russel's paradox, a one-liner which negates thousands of pages of Frege's not-yet-published logic textbook. Do you have to wait for him to publish it, and then you read it, before raising your voice?
> The psychologists think this has something to do with child psychology and have written 10,000 pages on it.
If you mean that 10000 pages in total in 10-page papers, you have read the literature, and become the hero child psychology needs by writing a paper refuting this.
On the other hand, if you did not read the literature wider, how do you know that you are first to do so, or that the misconception is even shared by other than the authors of those papers. Of course, the implicit assumption in all this is that psychologists are stupid.
If you mean that 10000 pages in total in 10-page papers, you have read the literature, and become the hero child psychology needs by writing a paper refuting this.
On the other hand, if you did not read the literature wider, how do you know that you are first to do so, or that the misconception is even shared by other than the authors of those papers. Of course, the implicit assumption in all this is that psychologists are stupid.
In my experience, the vast majority of scientists would love to have that Ramsey’s Theory conversation with you. Some people might get prickly if you lead with “This is trivial and I can’t believe you don’t know that....” but I suspect the vast majority of my colleagues would hear you out.
On the other hand....Ramsey’s theorem applies to compelete (undirected) graphs. Friendship is not necessarily symmetric: Alice could consider Bob a friend, but not vice versa. It could also be context-dependent, especially with kids: Alice and Charlie get along, but not when Bob is around.
The thousands of pages that you don’t want to read probably address some of these “details” that your spherical cow model totally ignores. I think it’s totally fair for someone to say “That’s interesting, but have you read X,Y, and Z? They show that your model doesn’t apply because....”
On the other hand....Ramsey’s theorem applies to compelete (undirected) graphs. Friendship is not necessarily symmetric: Alice could consider Bob a friend, but not vice versa. It could also be context-dependent, especially with kids: Alice and Charlie get along, but not when Bob is around.
The thousands of pages that you don’t want to read probably address some of these “details” that your spherical cow model totally ignores. I think it’s totally fair for someone to say “That’s interesting, but have you read X,Y, and Z? They show that your model doesn’t apply because....”
This kind of proves the point. You're misunderstanding the mathematics. Asymmetry of friendship is irrelevant. Whether the friendships have non-local dependency is irrelevant. The kids' friendship and non-friendship IS a colored complete graph: between any two nodes (kids), there is either a red edge (if those two are friends--in the context of the group, if you want), or there is a blue edge (if those two are not friends--in the context of the group, if you want).
Asymmetry is relevant because your bound comes from undirected graphs. The number for a complete digraph is different.
Oh, now I see what you're saying. Ok, you're right, to be pedantic, the original phenomenon should be reworded: Whenever 6+ kids get together, either there are 3 mutual friends, or there are 3 kids K1,K2,K3 such that for any two K_i, K_j, either K_i is not K_j's friend, or K_j is not K_i's friend. I was implicitly using "A and B are friends" to mean "A is B's friend, and B is A's friend". Thanks for pointing that out :)
No problem!
My bigger point was that the literature on any given topic is chock full of discussions like this. Is friendship directional? Is it binary, or does it make more sense to consider weights on these edges (casual acquaintance/best friend)? Can an outside observer infer friendship, or do you need to rely on self-reports? If so, how consistent are they? And so on.
The more patient researchers are happy to walk you through these sorts of considerations, especially when you're introducing them to a new tool or something. However, if you catch someone who's busy—or grumpy—you might get blown off with "That won't work. Haven't you read the literature?" Personally, I think this happens too often, but you can sorta see how people might get sick of regurgitating the same arguments over and over.
This is not to say that "read it and come back" can't be used as a moat, or that ideas "from the literature" shouldn't be revisited and questioned, but I think insisting that people do a bit of reading is not usually meant in bad faith.
My bigger point was that the literature on any given topic is chock full of discussions like this. Is friendship directional? Is it binary, or does it make more sense to consider weights on these edges (casual acquaintance/best friend)? Can an outside observer infer friendship, or do you need to rely on self-reports? If so, how consistent are they? And so on.
The more patient researchers are happy to walk you through these sorts of considerations, especially when you're introducing them to a new tool or something. However, if you catch someone who's busy—or grumpy—you might get blown off with "That won't work. Haven't you read the literature?" Personally, I think this happens too often, but you can sorta see how people might get sick of regurgitating the same arguments over and over.
This is not to say that "read it and come back" can't be used as a moat, or that ideas "from the literature" shouldn't be revisited and questioned, but I think insisting that people do a bit of reading is not usually meant in bad faith.
Not a psychologist, but I don't see the immediate connection to Ramsey Theory other than matching numbers. Is there a proof somewhere that human relationships are isomorphic to colored complete graphs? Why not for example hypergraphs, where Ramsey combinatoric figures should be different?
The isomorphism is: let the kids be nodes. Between kids x and y, place a red edge if x and y are friends, or place a blue edge if x and y are not friends.
My point, above, is that this isn't necessarily a good abstraction. The edges could be directed (A considers B a friend, B does not consider A a friend). The graph may not actually be complete (maybe absence of friendship is different from non-friendship), and so on.
So yes, if kids' friendships form a complete undirected graph, then sure, your result is not at all surprising. But maybe they don't.
So yes, if kids' friendships form a complete undirected graph, then sure, your result is not at all surprising. But maybe they don't.
Psychologic findings are statistical. Cliques do not form perfectly reliably; there are kids who would have no friends, kids who have common friend but hate each other, or yes all six could be friends indeed.
R(3,3) doesn't seem universal enough here to be touted as "isomorphism"; it is not giving any particular insight except the most trivial (that child cliques sometimes can be represented as monochromatic sets of a complete graph). Am certain one can do a number of other graph-theoretic or algebraic relationships here with zero insight or predictive force.
R(3,3) doesn't seem universal enough here to be touted as "isomorphism"; it is not giving any particular insight except the most trivial (that child cliques sometimes can be represented as monochromatic sets of a complete graph). Am certain one can do a number of other graph-theoretic or algebraic relationships here with zero insight or predictive force.
R(3,3) doesn't say a clique necessarily forms. It says a clique forms or an anti-clique forms (or both). If all six are friends, then there are (6 choose 3) cliques. If all six hate each other, there are (6 choose 3) anti-cliques. If one kid is hated by everyone, that in some sense decreases the odds of a clique but simultaneously increases the chances of an anti-clique, and R(3,3)=6 guarantees that in some sense these increases/decreases balance each other out.
Some scientific subfields are ridiculous on their face.
But there's a crucial difference between that and the way original rule is phrased.
The fields one has to dismiss are characterized by things one knows is false by physics or combinatorics or other common sense things. Theories of telepathy, the flat earth, creation science or whatever.
The "you can't show me one good paper on the subject" is an appealing-seeming pronouncement but it's really dumb way to do it. Just say, "that doesn't make sense and you'd need huge evidence to prove it". Paper quality isn't really the question.
But there's a crucial difference between that and the way original rule is phrased.
The fields one has to dismiss are characterized by things one knows is false by physics or combinatorics or other common sense things. Theories of telepathy, the flat earth, creation science or whatever.
The "you can't show me one good paper on the subject" is an appealing-seeming pronouncement but it's really dumb way to do it. Just say, "that doesn't make sense and you'd need huge evidence to prove it". Paper quality isn't really the question.
Regarding telepathy...
https://www.sheldrake.org/research/animal-powers/a-dog-that-...
https://www.sheldrake.org/research/animal-powers/a-dog-that-...
Sheldrake is my go-to example of bad science.
In one of his experiments monitored by another scientist, when the dog didn't respond in a way that met Sheldrake's criteria, Sheldrake changed the criteria in mid-experiment.
That one incident alone should be enough to dismiss any results coming from Sheldrake. He's not only dishonest but blithely unaware that he's being blatantly so.
In one of his experiments monitored by another scientist, when the dog didn't respond in a way that met Sheldrake's criteria, Sheldrake changed the criteria in mid-experiment.
That one incident alone should be enough to dismiss any results coming from Sheldrake. He's not only dishonest but blithely unaware that he's being blatantly so.
What immediately comes to mind is the 1994 paper "A Mathematical Model for the Determination of Total Area Under Glucose Tolerance and Other Metabolic Curves", which drew a lot of attention immediately after it was published, with such follow-up articles as "Tai's Formula Is the Trapezoidal Rule".
I don't know of any actual person that believes this they are always happy to jump in a debate with little to know knowledge
That's ostensibly silly, though. What if someone mentions paper X and you haven't read it? How would you know a priori that X is bad?
You always need to read all relevant literature to be able to constructively participate in a debate.
You always need to read all relevant literature to be able to constructively participate in a debate.
The whole point is to clear what he calls "mud moats", a vast amount of literature that protects an idea from criticism.
Non-adherents can't attack an idea because someone simply says "well have you read X paper? No? Well, then you need to read more.".
And then when you can back and say "but this paper is wrong too", the process just repeats.
Non-adherents can't attack an idea because someone simply says "well have you read X paper? No? Well, then you need to read more.".
And then when you can back and say "but this paper is wrong too", the process just repeats.
Maybe that's a problem in pub discussion with certain kinds of people, but those have nothing to do with science anyway. You can always criticize whatever you like by publishing a critical paper - preferably one in which you also lay out a better approach or present more solid data and statistics.
However, if you show you haven't engaged with the relevant literature and don't address the arguments in sufficient detail, then your paper is likely not going to pass peer review, and rightly so.
However, if you show you haven't engaged with the relevant literature and don't address the arguments in sufficient detail, then your paper is likely not going to pass peer review, and rightly so.
> How is tying your own learning to someone else's ability to find the best papers in any way a smart thing to do?
If they are supposed to be an expert in the field, that seems like a fair test.
If they are supposed to be an expert in the field, that seems like a fair test.
What the author wants to read is probably a survey ...
Yes - most papers just present one idea, as part of a field.
Although, the original paper of an idea, I often find, has the most human-readable explanation. Pearl's 1986 paper of Bayesian Networks, Gauss' 1800-whatever paper on the normal distribution, the original bitcoin/blockchain paper by that Japanese-sounding name, etc. They're still trying to sell the idea so they really need to talk to the idiots.
Although, the original paper of an idea, I often find, has the most human-readable explanation. Pearl's 1986 paper of Bayesian Networks, Gauss' 1800-whatever paper on the normal distribution, the original bitcoin/blockchain paper by that Japanese-sounding name, etc. They're still trying to sell the idea so they really need to talk to the idiots.
Yeah it’s silly. If you don’t want to do any research/generate new knowledge, then sure apply this rule. But then, who cares?
Claude Shannon's paper on information theory [1] is possibly an example of such a paragon of work. No citations, 50 pages of pure awesome, spawned a whole research field of its own and 70 years on just as valid.
[1] A Mathematical Theory of Communication - http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
[1] A Mathematical Theory of Communication - http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
You said it. He proposed the model for digital AND analog communication still in use today. Although being a digital software person I admit I've never been motivated to read the part about continuous channels.
I was motivated however to install an archway for my house from the exact shape in Figure 7 (binary entropy function)
I was motivated however to install an archway for my house from the exact shape in Figure 7 (binary entropy function)
> No citations
The paper contains citations. But they are hidden in the footnotes - see page 1 or page 5 for example.
The paper contains citations. But they are hidden in the footnotes - see page 1 or page 5 for example.
And it's easily readable with basic math/engineering knowledge. Highly recommended.
Erdos has a few papers that are "that good" as long as you know all the theory leading to the new insight, but none (that I read) are as clear and freestanding as Shannon's AMToC
Erdos has a few papers that are "that good" as long as you know all the theory leading to the new insight, but none (that I read) are as clear and freestanding as Shannon's AMToC
Math is different from empirical work. Yes, Shannon's paper is great, but that's not really relevant here because it's in a completely different paradigm.
First paper I thought of too.
On the psychology replication crisis
>Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies.
https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...
Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever.
I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place.
I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.
>Overall, 36% of the replications yielded significant findings (p value below 0.05) compared to 97% of the original studies that had significant effects. The mean effect size in the replications was approximately half the magnitude of the effects reported in the original studies.
https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...
Of course if you point out that psychology research is strongly biased, the immediate response is that you must be an anti-intellectual. As if the system that produced such misleading results is a representation of all intellectual pursuits, and not just a mistake. The same way you get called an anti-intellectual for questioning the post-modern-analyses of whatever.
I'm really sick of people claiming that criticizing obviously bad science is "anti-intellectual". When you produce that many useless papers you're obviously, as a field, lacking an understanding of why science is good and useful in the first place.
I'm presuming that economics has similar problems with replication, and that you can only trust the most basic and obvious of their findings.
A lot of this is simply that p < 0.05 is a very low bar for research. If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about. Without this thing, it's not science.
A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened.
Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless.
In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this
A good mechanism that accurately explains some part of the world can be validated with a much better result than p < 0.05, because it can be properly isolated from other factors and its effect size can be strengthened.
Without a mechanism to explain how things work, the experimental results that get published are probably just noise. And because p < 0.05 is such a low bar to pass, lots of noise gets published. Trying to reproduce an experiment whose results were drawn from /dev/urandom is pointless.
In fact I'm surprised the replication was so high. I was expecting it to be around 5% (i.e. results are totally random). So maybe there is some hidden merit to all of this
> If we want to understand how the world works, we need to be able to construct mechanisms that explain why the results we measure in experiments come about.
This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.
This, 1000 times over. Unfortunately we seem to teach that statistical analysis is sufficient to infer causation in a lot our undergraduate science programs.
I wouldn't expect totally random because people have common sense. I'd expect it to be similar to the folk-wisdom of the time, since a lot of it is going to be repackaging of that same folk-wisdom, and negative results tend not to get published, or get massaged until they're positive.
I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.
I'd expect social-psychology to mirror the dominant opinions of people in other segments of the humanities, although it is of course hard to determine the direction of that relationship.
A serious problem is when people assume that because the replication crisis in psych is widely publicized that it is exclusive to psych and then they dismiss all of the findings in the field. In my experience the replication crisis is rarely used to criticize method but instead used to dismiss research out of hand, whether or not it is supported by other research that limits the likelihood of error.
I think that psych should be commended for attacking this problem. As a field they are funding replications, which basically never happens in any other field. We don't even know what the "replication crisis" would be in empirical CS. Given discussions with ML people I'd wager that the "replication crisis" in ML is at least as dramatic.
Science is messy. But this is why we should defer to experts who have vision over an entire field rather than focusing on individual papers as conclusive data points.
I think that psych should be commended for attacking this problem. As a field they are funding replications, which basically never happens in any other field. We don't even know what the "replication crisis" would be in empirical CS. Given discussions with ML people I'd wager that the "replication crisis" in ML is at least as dramatic.
Science is messy. But this is why we should defer to experts who have vision over an entire field rather than focusing on individual papers as conclusive data points.
To be consistent, you'd have to leverage your arguments against other fields with replication crises, which include neuroscience and oncology. It's really true of the biomedical sciences in general.
Psychology has a history of being sort of self-analytical (which makes sense if you're a field studying human behavior). Meta-analysis, for example, was largely incubated in psychology before it started expanding into other disciplines.
The criticism I'm aware of is really being leveled at the idea that this is somehow unique to psychology. It's like killing the messenger, or someone who is trying to fix the problem.
Psychology has a history of being sort of self-analytical (which makes sense if you're a field studying human behavior). Meta-analysis, for example, was largely incubated in psychology before it started expanding into other disciplines.
The criticism I'm aware of is really being leveled at the idea that this is somehow unique to psychology. It's like killing the messenger, or someone who is trying to fix the problem.
[deleted]
I would agree that "no paper is that good" -- on the first try or even the second try. Just as we need constantly refactor code, just as we need constantly re-edit a book, a paper that explains a scientific idea also needs constant re-write to become good. The sad reality is that scholars simply do not do that.
It is understandable though. Once the paper is published, then it is no-longer novel, and there is obviously lack of reward and motivation to write the same idea again -- just to write it better. ... unless you are actually writing a review paper, but then, it appears there is little value unless the review is "complete". A good illustration of the idea need be able to high-light the key idea while avoid having minor ideas obscure the key. Therefore, a "complete" review paper rarely provide a good read.
Compared to papers, a good text-book often is a much better read than all the original papers. It is not really because of the size, rather it is because a good text-book is written from the reader's point of view and focuses on conveying the idea itself (vs. selling the idea). And a good text-book takes many rounds to develop.
There is no reason papers cannot be developed in a similar way as text books: once a ground-breaking paper is published, it shall be constantly updated, each new edition reflects what the author has newly learned and incorporating new development including the entire community...
Alas, that is not the culture, and there is no motivation to do such.
It is understandable though. Once the paper is published, then it is no-longer novel, and there is obviously lack of reward and motivation to write the same idea again -- just to write it better. ... unless you are actually writing a review paper, but then, it appears there is little value unless the review is "complete". A good illustration of the idea need be able to high-light the key idea while avoid having minor ideas obscure the key. Therefore, a "complete" review paper rarely provide a good read.
Compared to papers, a good text-book often is a much better read than all the original papers. It is not really because of the size, rather it is because a good text-book is written from the reader's point of view and focuses on conveying the idea itself (vs. selling the idea). And a good text-book takes many rounds to develop.
There is no reason papers cannot be developed in a similar way as text books: once a ground-breaking paper is published, it shall be constantly updated, each new edition reflects what the author has newly learned and incorporating new development including the entire community...
Alas, that is not the culture, and there is no motivation to do such.
In addition to the incentive problem, you would also need to get people to peer review any changes to a paper. Most of the changes would be minor and annoying to peer review.
I think that courses remain the best way of distilling and communicating knowledge. The only problem is that they're not always available to a lay audience, since there's little incentive to make them available.
The other alternative for the lay audience is science/economics journalism (Economist, Scientific American, Discover, etc). This works to some extent but mostly just scratches the surface, since even regular journalism is struggling to be profitable these days. With deep technical topics there are too few readers and too few qualified, willing writers.
I think that courses remain the best way of distilling and communicating knowledge. The only problem is that they're not always available to a lay audience, since there's little incentive to make them available.
The other alternative for the lay audience is science/economics journalism (Economist, Scientific American, Discover, etc). This works to some extent but mostly just scratches the surface, since even regular journalism is struggling to be profitable these days. With deep technical topics there are too few readers and too few qualified, willing writers.
I think it's more useful for papers to be what they are: diffs. A textbook is a snapshot view. Both kinds of embodiment are necessary in their native form for different purposes.
I assume this relates mostly to economics or the humanities (social science, if you insist) in general. These apply to sciences too, but are less debilitating in the long run.
Fifth, most researchers’ priors are heavily influenced by some extremely suspicious factors.
Academics in the humanities identify themselves "I am an X." X can be post structuralist, rational materialist, classical liberal or some other broad, hairy, intellectual identity. This is bad news for objectivity. Everyone has a dog in the fight.
Fifth, most researchers’ priors are heavily influenced by some extremely suspicious factors.
Academics in the humanities identify themselves "I am an X." X can be post structuralist, rational materialist, classical liberal or some other broad, hairy, intellectual identity. This is bad news for objectivity. Everyone has a dog in the fight.
It's interesting. As an engineering type (EE, primarily, but I had several other undergrad degrees), everything was about what is true. How do electrons move with their respective related fields across geometries.
This other stuff seems to be about "what is useful to us as humans in a group matrix (culture/community". That seems like it should be studied - but I don't think we all have the same objectives. Also - some people keep trying to make it "true" vs. "useful".
What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"?
I don't know - it just seems (to me) that we spend too much time on the truth as opposed to (subjectively) communally useful. i.e. What should we decide to believe in as a community to hang together (because otherwise, we shall surely hang separately)?
This other stuff seems to be about "what is useful to us as humans in a group matrix (culture/community". That seems like it should be studied - but I don't think we all have the same objectives. Also - some people keep trying to make it "true" vs. "useful".
What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"?
I don't know - it just seems (to me) that we spend too much time on the truth as opposed to (subjectively) communally useful. i.e. What should we decide to believe in as a community to hang together (because otherwise, we shall surely hang separately)?
I'm in. Not everything needs to be true in the electrons-in-a-circuit sense to be useful. A lot of things mgs are subjective, both in the sense that it's arguable, but also in the sense that how people see it is important that in itself.
Man may not be the measure of *all" things, but he (she) is the measure of some things and that's ok. I think a lot of arguments (politics and political ideologies in particular) would get a lot nicer if we just accepted that we aren't arguing about absolute truth.
Man may not be the measure of *all" things, but he (she) is the measure of some things and that's ok. I think a lot of arguments (politics and political ideologies in particular) would get a lot nicer if we just accepted that we aren't arguing about absolute truth.
I've always wondered why most people are not passionately opinionated about methods for heart surgery, and yet are highly opinionated about economics (for example). The latter seems more complicated, in many ways, and yet people have no problem having an opinion about it. Why do you think that is?
It's probably because economics relates to every adult's life experiences. These experiences give people a reason to form theories about how economics works.
> What say ye all? Do you think that "True" has any place in the social sciences vs. the observational "I observe this to be useful in this context" or the almost psycho-analytical "I notice that when people believe X together, society does better"?
The problem with that is whoever takes it upon themselves to discard truth and consider 'what will have the best effect on people/society if they were to believe it' —is that you have to manipulate other people into believing it's true on the hope that your assessment of how it's going to affect people is accurate. It's almost certainly not.
It's interesting considering the human brain's criteria for accepting beliefs. It works pragmatically, not with strict adherence to truth—but, from what I can tell (and I've seen some research supporting this), the amount it's willing to deviate from (what it estimates to be) truth is proportional to the immediate demands of present circumstances. If you are in a crisis, it is willing to compromise and adopt a belief that conflicts with evidence, as long as it solves some immediate problem you're facing. But when that happens, you are accumulating something like technical debt. You become sort of 'out of harmony' with your circumstances, and problems will arise given enough time.
Why would it evolve to act in that way? It seems clear the answer is: because it's generally most beneficial to survival to adopt beliefs which best fit the observed data. It's a fair assumption that society holding true beliefs is also going to be most effective for survival.
So when someone manipulates someone else's beliefs because they think it will be beneficial, they are forcing that technical debt on them, and that's a decision only they should make for themselves. The same is true but amplified when considering spreading these ideas to wider audiences.
That said, much of contemporary social science and the humanities are rife with the opposite sort of thinking. This is heavily fueled by the fact that much of it doesn't consider 'correspondent' truth to be a real thing, so in their framework it is meaningless to say they are compromising the truth. Which truth? You can see this in Pragmatism's approach of literally redefining 'truth' to refer to something like 'the most effective belief to adopt'; and even more pervasive are all the schools of thought influenced by a kind of radical relativism which denies the possibility of truth. (It also clarifies a lot of philosophy once you realize the philosophers are actually thinking about the best mind-programs to put into people, rather than seeking truth as a typical person would recognize it. If only the sneaky bastards would say so to their readers' faces. But of course it would not be pragmatic to do so.)
It's a bad idea, and I sincerely hope the fields presently employing it fall into ruin, or are otherwise convinced to abandon it.
The problem with that is whoever takes it upon themselves to discard truth and consider 'what will have the best effect on people/society if they were to believe it' —is that you have to manipulate other people into believing it's true on the hope that your assessment of how it's going to affect people is accurate. It's almost certainly not.
It's interesting considering the human brain's criteria for accepting beliefs. It works pragmatically, not with strict adherence to truth—but, from what I can tell (and I've seen some research supporting this), the amount it's willing to deviate from (what it estimates to be) truth is proportional to the immediate demands of present circumstances. If you are in a crisis, it is willing to compromise and adopt a belief that conflicts with evidence, as long as it solves some immediate problem you're facing. But when that happens, you are accumulating something like technical debt. You become sort of 'out of harmony' with your circumstances, and problems will arise given enough time.
Why would it evolve to act in that way? It seems clear the answer is: because it's generally most beneficial to survival to adopt beliefs which best fit the observed data. It's a fair assumption that society holding true beliefs is also going to be most effective for survival.
So when someone manipulates someone else's beliefs because they think it will be beneficial, they are forcing that technical debt on them, and that's a decision only they should make for themselves. The same is true but amplified when considering spreading these ideas to wider audiences.
That said, much of contemporary social science and the humanities are rife with the opposite sort of thinking. This is heavily fueled by the fact that much of it doesn't consider 'correspondent' truth to be a real thing, so in their framework it is meaningless to say they are compromising the truth. Which truth? You can see this in Pragmatism's approach of literally redefining 'truth' to refer to something like 'the most effective belief to adopt'; and even more pervasive are all the schools of thought influenced by a kind of radical relativism which denies the possibility of truth. (It also clarifies a lot of philosophy once you realize the philosophers are actually thinking about the best mind-programs to put into people, rather than seeking truth as a typical person would recognize it. If only the sneaky bastards would say so to their readers' faces. But of course it would not be pragmatic to do so.)
It's a bad idea, and I sincerely hope the fields presently employing it fall into ruin, or are otherwise convinced to abandon it.
I tend to be biased towards wanting to agree with you 1000% on 99% of your post. As I age, though, I become increasingly skeptical of the idea that the human brain is any good at discovering or holding onto the truth. I do believe that the brain can be managed with good mental hygeine and process (https://www.amazon.com/Uncommon-Sense-Heretical-Nature-Scien...), but I find this particular discipline rare in the social science departments I have visited. (I probably just need to get out more).
> It's a fair assumption that society holding true beliefs is also going to be most effective for survival.
This is the part I disagree with. People are social. And their environment is other people. Imagine a nation or tribe populated by Aztecs, or Mormons (No hate - just an example). A person's ability to survive in a territory populated by Aztecs or Mormons is going to be significantly determined by their ability to conform to group beliefs re: normative behavior. The easiest way to do that (thanks mirror neurons!) is to believe it yourself...
> It's a fair assumption that society holding true beliefs is also going to be most effective for survival.
This is the part I disagree with. People are social. And their environment is other people. Imagine a nation or tribe populated by Aztecs, or Mormons (No hate - just an example). A person's ability to survive in a territory populated by Aztecs or Mormons is going to be significantly determined by their ability to conform to group beliefs re: normative behavior. The easiest way to do that (thanks mirror neurons!) is to believe it yourself...
> A person's ability to survive in a territory populated by Aztecs or Mormons is going to be significantly determined by their ability to conform to group beliefs re: normative behavior
I agree, but I see that as a separate issue from what I was attempting to address. The specific thing I had in mind was about authority figures in a position to promulgate some new idea to society: is it wise for them to spread ideas known to be false but estimated to have positive effects?
I agree, but I see that as a separate issue from what I was attempting to address. The specific thing I had in mind was about authority figures in a position to promulgate some new idea to society: is it wise for them to spread ideas known to be false but estimated to have positive effects?
I would say assuming sciences are somehow not affected by these sorts of factors is worse than what the humanities do - at least they admit a bias.
From my own experiences in the hard sciences, the human-factor dominates over objectivity, from the individual through to peer review.
One conclusion you could draw from this: economics might actually be a giant mud moat.
You’d be wrong. I’m not all that impressed with macroeconomics and it’s far from a mud moat. Microeconomics is making more progress and is on much sounder footing, hardly surprising given how many more empirical observations there are.
If you’re only going to read one microeconomics textbook read Varian’s Intermediate Microeconomics if you can do calculus. If you can’t Cowen and Tabarrok’s Principles is pretty good and has a load of wonderful accompanying videos.
Comparative Advantage
https://www.mruniversity.com/courses/development-economics/c...
If you’re only going to read one microeconomics textbook read Varian’s Intermediate Microeconomics if you can do calculus. If you can’t Cowen and Tabarrok’s Principles is pretty good and has a load of wonderful accompanying videos.
Comparative Advantage
https://www.mruniversity.com/courses/development-economics/c...
Here's Smith's definition of a mud moat:
And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows.
https://leaders.economicblogs.org/noah-smith/2017/smith-lite...
I was only half-serious in my original comment. What mud moat exists today is maintained less by economists and more by politicians and businesspeople, who find certain "classical" economic conclusions convenient. "Free markets are good - just read the economics literature!"
And in the world of intellectual debate, this vast literature can function as a mud moat. That is a term I just made up, sticking with the metaphor of political arguments as medieval castles requiring a defense. A mud moat is just a big pit of mud surrounding your castle, causing an attacking army to get trapped in the mud while you pepper them with arrows.
https://leaders.economicblogs.org/noah-smith/2017/smith-lite...
I was only half-serious in my original comment. What mud moat exists today is maintained less by economists and more by politicians and businesspeople, who find certain "classical" economic conclusions convenient. "Free markets are good - just read the economics literature!"
Economics is several separate sciences, awkwardly placed in the same department.
Most departments are heterogeneous, with mutually unintelligible subfields. CS encompasses both information theory and robotics. Cultural anthropology has little in common with medical anthropology.
Just like cognitive science has a much higher replication rate than social psychology, you will get different results with different subfields.
Well, Watson & Crick (Nature 1953) is that good. It does contain an error (the actual DNA structure is slightly wrong) And Avery (JEM 1944) http://jem.rupress.org/content/79/2/137) which "proved" that DNA is the molecule of heredity, is also there.
But to be qualified to read these papers and appreciate why they are points of quality within a sea of crap? That's hard.
But to be qualified to read these papers and appreciate why they are points of quality within a sea of crap? That's hard.
I was sort of on board with the two-paper rule until I got to the arguments about the reviews. Maybe in his mind integrating literature into a cohesive summary (ala meta-analysis) is a novel contribution, but if not, the attitude behind the two-paper rule is part of why science is in such crisis. The author is right, that the interpretation of the literature should be based on an accumulated read, and not one or two papers (unless they're reviews or meta-analyses).
Allow me to disagree. Chris Okasaki's thesis, "Purely Functional Data Structures" is That Good for the field of functional programming. There are other exemplary papers, like Godel's incompleteness theorem, Cantor's diagonal argument, or Einstein's statement of special relativity. Or Satoshi's paper on Bitcoin.
These papers are not exhaustive summaries of a field. But a reader comes away understanding the type of problems a field is devoted to solving and many of the existing ideas. And I believe that each is a paragon of their field.
These papers are not exhaustive summaries of a field. But a reader comes away understanding the type of problems a field is devoted to solving and many of the existing ideas. And I believe that each is a paragon of their field.
The article limits itself to discussing “empirical paper”s not theoretical papers.
In a theoretical paper, it’s possible to make statements that stand on their own merits. In empirical science, a single paper is never really enough to support an entire field. Empirical sciences generally rely on the development of scientific consensus.
In a theoretical paper, it’s possible to make statements that stand on their own merits. In empirical science, a single paper is never really enough to support an entire field. Empirical sciences generally rely on the development of scientific consensus.
That seems like an artificial distinction made up for economics. In physics there are plenty of theory papers that make very clear and concise predictions. If these papers hold then they are very much both. E.g. the original special relativity paper goes out of its way to make predictions and calculations at the state of the art of the time, which are quite jarring today.
Dirac's early papers are, from what I have heard, amazing pieces of work. One commentary about the quality is given at https://michaelberryphysics.files.wordpress.com/2013/07/berr... .
Hmm. I guess Mendal wrote a journal, not a paper per se. Millikan's oildrop experiment?
[deleted]
Papers aren't meant to be this crystallized nugget of truth, they are a progress report on an ongoing piece of living research. They aren't meant to be infallible.
On a slightly tangential note: how do you guys read research papers? Do you go through them word by word or you only skim them to get a general understanding?
I try to do the former but the work seems so boring that I am hardly motivated to do this more.
I try to do the former but the work seems so boring that I am hardly motivated to do this more.
In mathematics:
Read the title and abstract. Then read the statements of the main theorems and corollaries (skip the lemmas and interstitial commentary at first). At each step, evaluate whether you want to continue or abandon the paper for whatever reason (not interesting, not relevant, whatever).
Finally, if you made it this far, read the whole paper word for word with a pen and paper (or chalk + chalkboard) handy.
For longer papers or books/theses (say, 20+ pages), do this whole process on each individual section/part/chapter that you want to consume.
It also helps to have a reason to read the paper besides curiosity. I find things that are relevant to current work to be easier to get through than more peripheral sorts of things. Having a colleague to discuss things with also helps.
Read the title and abstract. Then read the statements of the main theorems and corollaries (skip the lemmas and interstitial commentary at first). At each step, evaluate whether you want to continue or abandon the paper for whatever reason (not interesting, not relevant, whatever).
Finally, if you made it this far, read the whole paper word for word with a pen and paper (or chalk + chalkboard) handy.
For longer papers or books/theses (say, 20+ pages), do this whole process on each individual section/part/chapter that you want to consume.
It also helps to have a reason to read the paper besides curiosity. I find things that are relevant to current work to be easier to get through than more peripheral sorts of things. Having a colleague to discuss things with also helps.
FWIW, I don't think this article really applies to areas like maths or computer science, where the main output is theorems or algorithms that can be verified line by line. It's more about empirical science where one is trying to prove/disprove hypotheses through rigorous experimentation.
No, I don’t think so, either. In math, there are excellent papers that cover .01% of what needs to be said about the subject, and ordinary or even bad papers that say much more. Provided there are no serious logical errors, the difference is simply how important is the question being answered.
Edit: Although it may go without saying, all else being equal, a better paper is easier to read than a lesser one. But, really, content is king.
Edit: Although it may go without saying, all else being equal, a better paper is easier to read than a lesser one. But, really, content is king.
Most of the time, skimming through, picking out the parts that are different from others in the field.
For physics: "Okay, this paper is from so and so, yup, same facility and equipment as the previous one. Theory section, same basic overview, skip to the end for what makes this experiment worthwhile? Anything interesting in their analysis, and how good are their statistics? Did they show the usual set of diagnostics plots for the type of analysis they did, and if not, what are they trying to downplay?"
For physics: "Okay, this paper is from so and so, yup, same facility and equipment as the previous one. Theory section, same basic overview, skip to the end for what makes this experiment worthwhile? Anything interesting in their analysis, and how good are their statistics? Did they show the usual set of diagnostics plots for the type of analysis they did, and if not, what are they trying to downplay?"
For the most part, the way to read papers in biology is to go through the figures. When I was quickly trying to figure out if a project I was about to start up just got scooped (it did, and good for them for saving me the effort), I did it by glancing at figure 1, then reading the title of figure 1, then closely examining figure 1 while cross-referencing with the detailed caption. Then move on to 2, 3, etc. By figure 5, I had a hard time telling what was going on, so I found the part of the text that referenced it, and read it carefully.
I've heard arguments that one should leave the abstract until the end, since it sometimes pushes a story harder than the data necessarily warrants, and so if you come to your own conclusion about what's going on in the paper, then check to make sure the authors claim what you feel is supported, you're less likely to be misled. Probably this makes sense for reviewing a paper, but for most things, I'd rather find out quickly if the paper is even relevant to what I'm working on.
I've heard arguments that one should leave the abstract until the end, since it sometimes pushes a story harder than the data necessarily warrants, and so if you come to your own conclusion about what's going on in the paper, then check to make sure the authors claim what you feel is supported, you're less likely to be misled. Probably this makes sense for reviewing a paper, but for most things, I'd rather find out quickly if the paper is even relevant to what I'm working on.
I've found looking at figures and captions to be really misleading. I find that I have to read the methods section first to convince myself that the data collection was correct before I can trust any figures.
You have to read the abstract. It's the intro and conclusion you can skip.
You have to read the abstract. It's the intro and conclusion you can skip.
It's worth to note that "reading papers" consists of two activities - actually reading a paper indenting to understand it, and skimming a large number of papers or abstracts to filter out which papers to actually read in detail - and it's the second part that seems to take the majority of time, at least for me.
> how do you guys read research papers
my phd advisor started by looking only at all the displayed formulas. If there was nothing interesting or new, he would not bother reading any sentence of the paper.
my phd advisor started by looking only at all the displayed formulas. If there was nothing interesting or new, he would not bother reading any sentence of the paper.
Is there any cross disciplinary repository of excellent research reviews?
It seems like a lot of commenters here are under the impression that the "two paper rule" (if someone says you should read the literature in field X, ask them for the two best papers in that field they can think of, and if those aren't impressive then don't bother looking further) is a proposal of Bryan Caplan, who wrote the OP here.
That is incorrect. The two-paper rule is Noah Smith's, and Bryan Caplan is disagreeing with it.
(It's scarcely possible to read any of Caplan's post without realising that; I conclude that many commenters here have not bothered to read the OP before commenting.)
That is incorrect. The two-paper rule is Noah Smith's, and Bryan Caplan is disagreeing with it.
(It's scarcely possible to read any of Caplan's post without realising that; I conclude that many commenters here have not bothered to read the OP before commenting.)
Following the link through to the original proposal, this seems like a misinterpretation of the suggestion. The problem this was supposed to "fix" was people using the existence of their being "vast literature" to shut down arguments.
From that perspective, the suggestion seems fine. You should be able to dig out two examples that show your field isn't nonsense. I don't think it was meant to be a high bar:
> There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.
From that perspective, the suggestion seems fine. You should be able to dig out two examples that show your field isn't nonsense. I don't think it was meant to be a high bar:
> There are actual examples of vast literatures that contain zero knowledge: Astrology, for instance. People have written so much about astrology that I bet you could spend decades reading what they've written and not even come close to the end. But at the end of the day, the only thing you'd know more about is the mindset of people who write about astrology. Because astrology is total and utter bunk.
[deleted]
>The best papers get up to around .20. Again, No Paper Is That Good. If you demur, consider this: In twenty years, will you still hold up the best papers of today as “paragons” or “exemplars” of compelling empirical work?
If the field is not totally vague, then yes. We can still consider certain papers in physics, or chemistry, or medicine, computer science etc. as exemplary decades, or even centuries, later, even when they deal with empirical work.
Soft sciences need not apply.
If the field is not totally vague, then yes. We can still consider certain papers in physics, or chemistry, or medicine, computer science etc. as exemplary decades, or even centuries, later, even when they deal with empirical work.
Soft sciences need not apply.
Nonsense. Plenty of old computer science papers don't hold up well to scrutiny or are largely irrelevant nowadays, such as those that were trying to optimize for bottlenecks in hardware that no longer exist or have shifted to different places. Even in pure mathematics, something that was considered a great insight a long time ago might be considered fairly trivial now, and not just because the other paper came out first. Just because a paper is correct doesn't make it interesting or worthwhile--a lot of the best papers have errors in them, but propose something that's a genuinely new contribution to the field.
>Nonsense. Plenty of old computer science papers don't hold up well to scrutiny or are largely irrelevant nowadays
Nonsense, knee-jerk answer.
First, we don't need all of them to "hold up well to scrutiny" but just 2 (as per TFA challenge). And we have way more than just 2 -- hundreds of great papers.
Second, the papers "that were trying to optimize for bottlenecks in hardware that no longer exist or have shifted to different places" can still be perfectly valid as per the challenge we have, which was:
1) that they were not "full of mistakes and bad reasoning", 2) that they did not "contain little or no original work" (and where thus just references and meta-papers)
The question wasn't if we have "plenty of" papers that are bad, or plenty of papers that were very good but have been super-ceded by changes in technology.
Just that we have at least 2 (and I argue we have way more than two) seminal papers that have original work, and are not full of mistakes and bad reasoning.
Nonsense, knee-jerk answer.
First, we don't need all of them to "hold up well to scrutiny" but just 2 (as per TFA challenge). And we have way more than just 2 -- hundreds of great papers.
Second, the papers "that were trying to optimize for bottlenecks in hardware that no longer exist or have shifted to different places" can still be perfectly valid as per the challenge we have, which was:
1) that they were not "full of mistakes and bad reasoning", 2) that they did not "contain little or no original work" (and where thus just references and meta-papers)
The question wasn't if we have "plenty of" papers that are bad, or plenty of papers that were very good but have been super-ceded by changes in technology.
Just that we have at least 2 (and I argue we have way more than two) seminal papers that have original work, and are not full of mistakes and bad reasoning.
> Soft sciences need not apply
Why not? Historians still revere Braudel and The Mediterranean was published nearly 70 years ago. Are you certain that there are no such exemplary works outside of the fields you like?
Why not? Historians still revere Braudel and The Mediterranean was published nearly 70 years ago. Are you certain that there are no such exemplary works outside of the fields you like?
No, but I'm certain that whether revered or not, Braudel's work is still "full of mistakes and bad reasoning" -- it's just that in history those are harder to tell.
Absolutely. Others have cited Shannon's paper on information theory. I would add Einstein's paper on Special Relativity. I read a translation of it on the 100th anniversary of its release, and I learned something, even though I already knew Special Relativity.
(What did I learn? I don't remember precisely - it's been 15 years - but I think it was that something on the electromagnetic side was caused by time dilation.)
(What did I learn? I don't remember precisely - it's been 15 years - but I think it was that something on the electromagnetic side was caused by time dilation.)
'We are all in a mud moat, but some of us are looking at the stars', as comrade Oscar Wilde almost said, then?
I suspect this is limited to economics and the social sciences. There is absolutely no way this holds up in the natural sciences like math, physics, etc.
People forget the original name for what we call economics was "political economy". That alone should tell you all you need to know about the dangers of treating that field like a science. If you ever want to know why expert economists can't seem to agree on things that happened 50-100 years ago or make accurate predictions for the future, the original name is very telling. Wouldn't it be ridiculous if we were still debating the validity of f=ma or e=mc^2? Wouldn't it be crazy for someone to claim general relativity is just flat out wrong, even though GPS systems would not work properly without humans accepting it?
Why is nothing remotely approaching reasonable standards used in the social sciences before acceptance?
People forget the original name for what we call economics was "political economy". That alone should tell you all you need to know about the dangers of treating that field like a science. If you ever want to know why expert economists can't seem to agree on things that happened 50-100 years ago or make accurate predictions for the future, the original name is very telling. Wouldn't it be ridiculous if we were still debating the validity of f=ma or e=mc^2? Wouldn't it be crazy for someone to claim general relativity is just flat out wrong, even though GPS systems would not work properly without humans accepting it?
Why is nothing remotely approaching reasonable standards used in the social sciences before acceptance?
Bitcoin's paper was that good.
Simple.
Easy to read.
Easy to replicate (a poc in python is easy)
It is a recent key innovation (the blockchain tech).
Simple.
Easy to read.
Easy to replicate (a poc in python is easy)
It is a recent key innovation (the blockchain tech).
What happens when you apply this author’s standard to this work?
Heh. From the article:
Does all of this hold for my papers, too? Of course. The most I can claim is that I am hyper-aware of my own epistemic frailty, and have a litany of self-imposed safeguards. But I totally understand why my critics would look at my best papers and say, “Meh, doesn’t really prove anything.”
Does all of this hold for my papers, too? Of course. The most I can claim is that I am hyper-aware of my own epistemic frailty, and have a litany of self-imposed safeguards. But I totally understand why my critics would look at my best papers and say, “Meh, doesn’t really prove anything.”
I stopped reading before I got to that point thanks to the author’s argument on account of I took his advice. Thanks for the datum.
He's asking for two papers to demonstrate that it is worth delving into the literature of an entire field. It's not clear that it is even meaningful to ask how it applies to the work of any particular individual author within the field.
This is one of worst ideas I've ever read. Wouldn't you just be cheating yourself?
How is tying your own learning to someone else's ability to find the best papers in any way a smart thing to do? It would be much better to doubt the person who gave you the papers (perhaps he miscalculated which two papers were the best), than to dismiss the entire field.