But it also fuzzes value. So what you end up with rather than "value" is network effects, people who are good at advertising, and random chaotic trends that blow like the wind. That is, it's not just enough to be the best, it's required to create the appearance of the best, or to capture whatever the whims of the network are.
People are also not always very good judges of long-term value. For the longest time the response I'd get whenever I expressed some sentiment of supporting small business, all things being equal, was something along the lines of "well you should support whoever can deliver the good at the lowest price." Although that's true to some extent, in the long term it leads to personal costs as local services are gone, and there's less and less competition. Short-term benefits, long term costs and so forth.
In general, I think there's far too little focus on maintaining healthy competition. The pendulum in public discussions often swings from public to private services and employment, which is a good conversation to have, but there's almost never a serious, critical discussion about increasing competition and what those barriers are. US society is full of things that stifle competition and they don't get 1/10 the attention they deserve. It seems as if when it's brought up, you're forced into these ultra-libertarian or anti-capitalist stereotypes.
The other day I was looking for a cycle restoration shop. There's a great one and they should be thriving right now. But their business was destroyed in riots. Not the business owner's fault. You could make an argument that it's just one more downstream effect of a racist cop killing someone over a counterfeit bill, a very significant one. This effect has played out many many times across the country now I'm sure. But who will foot the bill? The Mpls police dept?
It seems we used to have this intuitive understanding of "bad things happen to good people" and "the community should pitch in collectively to support these things" but it's fading away.
The darwinian model is flawed, unless you're focuses solely on the net benefits to those who benefit, which is circular in its reasoning.
I sincerely appreciate this blog post and discussion, as it raises a lot of important and compelling points, but I wish it had had some discussion of alternatives and why.
To me, messaging is a mess at the moment, somewhat like IoT because of lack of solid widely adopted standards (either de facto or de jure).
It's extremely difficult to get friends and family to use something. Most decisions are driven by secondary considerations, like it comes with an OS, or as part of an email or office system, or a gaming system. In some cases it's because "it's what everyone is using".
This shifts the threshold a bit in terms of concerns. What I mean by that is given the inertia involved in moving people to use a messaging system, the bar gets raised in terms of moving people off because of network effects. It's hard enough to get any friends or family to use Signal as an alternative to other things; convincing them to switch again introduces other problems.
I'd prefer something that can be used in more decentralized way, but that has its own issues in terms of syncing and always-on problems. And as security increases, more and more inconveniences are introduced -- it might be worth it, but the case still has to be made implicitly or explicitly to friends and family.
Again, not saying these kinds of discussions shouldn't happen, but they often seem kind of theoretical to me or like they're missing the point because of bigger issues with the messaging ecosystem in general. If you're not going to be able to use Signal anyway because everyone you know is using Whatsapp or iMessaging, or feel like messenger use is driven by "whatever is most popular" it feels like it's difficult to weigh things like "won't put on fdroid". I'd love to see it on fdroid but where does that rank?
Science has become so political and subject to hype and fads, at least in biomedicine, that everyone should be deeply skeptical of every claim that is made. That doesn't mean that you should distrust everything completely, just that critical skepticism should be the norm.
If by "science" you mean that "rigorous, logical empirical study" should be part of the decision-making, yes, I agree.
If by "science" you mean that the statements of scientists with prestigious academic appointments should be given priority over other considerations, I say no. Outsider criticism should be welcome, and I think nonscientific considerations are sometimes required.
One of the reasons society is in the mess it is in (at least in the US) is because of a failure to realize that certain segments, like the biomedical-scientific community, are just as driven by human failures as any other.
I feel odd being so gushing about Best Buy, because for the longest time I really didn't have the most positive impression of them at all, but I'm not exaggerating when I say that over the last few years, I've had the best customer service experience from them that I've had anywhere in any store of any sort. It's really phenomenal, like enough to make me make a point of purchasing things there when I have the option. I don't even know where to begin, but I feel like they've been super attentive and genuinely helpful without dumbing things down.
Whether that's worth it or not to any given person is a personal thing but I was really surprised by it.
10 million confounds is a lot to eliminate. Randomization only works because it decreases the probability of a spurious association existing. At some point the number of confounds eliminated would probably be more worthwhile. I'd definitely pay more attention to a study that eliminated 10 million confounds than most randomized controlled trials.
I agree a randomized controlled trial would be nice now. But even that is potentially fraught. For example, someone else mentioned the possibility that glucosamine reduces joint pain, which increases mobility, which increases longevity. Randomizing wouldn't really control for that sort of scenario. And that's not even getting into preregistration, meta-analysis etc.
I think the horse has left he barn in terms of the suggestions in the comic. I love the comic and the attention it's getting but the issues discussed are such a small part of the problem. Many of the comments here are on point.
At this point in time, many of the people in power (full professors, administration, etc) see papers as essentially useless, in part because of all the problems being discussed here. p-hacking, guest authorship, etc and so forth and so on. They're seen as a dime a dozen and somewhat ignored.
This might sound good until you understand that this means that they're dismissing the entirety of the scientific dialogue essentially, and that their alternative is grant money. My institution had training workshops for grad students where faculty would tell students that research is essentially worthless unless grant money is attached to it. The idea is that papers are a dime a dozen, and that if something is worthwhile, the feds will put their money where their mouth is so to speak.
The problem with this, of course, is that the grant system is horribly nepotistic and distorted. The biggest predictor of successful grant application last time I looked (based on empirical research in peer-reviewed journals) is co-authoring papers with someone on the review panel. Grant receipt is only weakly related to citation metrics, and from personal experience I can say that institutions are constantly encouraging researchers to inflate grant costs to bring in more indirect costs.
You could fix everything about journals and the publication process and it would do nothing about the shadow scientific world that exists in parallel that drives all the rest of the problems.
Eliminate indirect funds, require tenure of all researchers, set aside funding mechanisms for researchers that aren't tied to specific grant applications, randomize grant rewards, depriortize journals, ... there's a lot of things that need to happen.
IIRC, Hungary has implemented something like this and it's been floated as an idea, to distribute grant dollars as a fixed amount to everyone who has an H index or citation count above a certain amount.
When I've talked about this with colleagues, they've argued this would make problems worse, and it might, but I think the idea was more that once you reach a certain threshold you should be funded some amount.
Of course, I'd argue that this is basically the idea behind tenure at an R1 institution, that once you reach a certain level the state or private board of trustees is paying your salary to do research. But nowadays tenure at an R1 institution is proxy for external grant dollars, which defeats the purpose and is redundant.
The real problem is indirect funds, which create profit for institutions. The federal government needs to eliminate indirect funds, so that grant dollars aren't perversely incentivised and tenure is based on research quality rather than profit.
Maybe I read the original research article incorrectly, or maybe this BBC piece is discussing something additional, but my impression after reading about it more closely was that this test is actually discriminating Alzheimer's from other forms of dementia, not Alzheimer's vs no dementia.
I haven't thought this through yet but it seems interesting and important but also not quite how it's being portrayed in the media, and not quite as significant in some ways.
I agree that business isn't really the right model for academics but the problems imho is that what an "outstanding academic track record is" has become so ambiguous that it's practically meaningless -- and I say this as a former tenured prof at an R1 institution.
I could write a book about this stuff. The stories I could tell about what's behind those "outstanding academic track records"...
The problem, if anything, is trying to apply a business model to academics, equating research quality with federal grant dollars, taking away real intellectual freedom protections, and then ignoring all the ponzi scheming and exploitation that occurs. Everyone has their heads in the sand, knows academics (at least biomedical research) is full of BS, and just goes on pretending like it's not because no one knows of a good alternative, or doesn't have the courage or power to change things.
What's funny [sad?] to me is that your description of PMs sounds exactly like the most credentialed, accomplished researchers I know on paper.
It's interesting to me regarding some of the examples in the linked piece. Ghostwriting reviews, for example, is actually seen as a good practice in a lot of circles because it provides experience to grad students with the review process. Those guest authorships? Very grey area between that and collaborative authorships. It's not the grunt work, it's the idea, right? Or is it that ideas are a dime a dozen, and actually doing the work is important? I can't tell which it is anymore -- it seems to depend on what benefits those in power.
Someone else posted something about how 1% of research is fraud, and 80% is bad. I think the percent of fraud is probably higher, the percent bad research is lower, and the difference is much more fuzzy than you'd think initially. The really difficult thing is that tiny incremental contributions is how things actually work. No one wants to admit this though. Bad research is actively incentivized, and there's credit bubbles everywhere.
The worst problem is that this credentialing bubble is everywhere with everything, as another posted noted. The problem isn't the credentialing per se, it's how it's detached from reality, the real demands of the tasks. Having a credential doesn't mean that the person is competent for all the tasks it nominally encompasses; conversely, those tasks don't necessarily require the credential that's often demanded.
This poem often reminds me of geological timescale, which occurs over millions of years or more. For example, the Carnian Pluvial Event was a relatively humid rainy era that lasted for millions of years, beyond most of our comprehension in practical terms (https://laughingsquid.com/why-it-rained-for-two-million-year...), even though it's a relatively discrete period of time in the grand scheme of Earth. Even the Permian extinction, the largest extinction event in the history of known life, unfolded for over 60000 years. That is a flicker on geological timescales, but supercedes by a wide margin the period of recorded human history.
The harder story to tell that this piece is avoiding because of the difficulty is the institutional dynamics. It's hinted at in the article more or less directly, but the entire mindset at many places is so far from what's conveyed in this New Yorker piece that it sort of misses the ball completely.
At the time the story in this article begins, administrators at many places were completely unfamiliar with the general idea behind open content distribution. In such places, online systems for course content were seen as a way of locking down the intellectual property that courses represent, not to distribute it. Where I was at, faculty were explicitly admonished not to distribute any online course material because it was seen as leaking university IP. Course management systems were seen as a way to deliver materials online securely so as to ensure that they weren't widely distributed. Even if someone wanted to put the effort into something that might now be called a MOOC, it was doubly frowned upon because you were putting effort into teaching rather than research (especially if you were untenured), and triply so because you were putting effort into distributing courses online, which was a fringe activity.
By the time MOOCs came about and got a lot of press, and many of us were sort of shaking our heads at what seemed inevitable, it was a little too little too late. In online space, where social factors like reputation become amplified 100-fold, it's difficult to compete or add to the reputational weight of places like MIT or Harvard. Adding to this was the constant monetizing mindset, of trying to lock down materials, not recognizing the advertising/popularizing role that open content distribution enables. Then too was a sort of "so what" feeling, because in the absence of something like a pandemic, who really cares, from the perspective of the administration -- students come to campus and that's where the dollars come from; MOOCs and online course distribution of whatever form were seen as experimental, the realm of the extension service which is valued but only to a point.
There's lots of issues the piece doesn't get into, or only gets into in a half-hearted way, which is how these types of online teaching dynamics really mess with the traditional structures. So instead of having an interesting conversation about how unfamiliar it is for universities to be dealing with Course A at Institution 1 competing with Course B at Institution 2, it recapitulates the discussion in terms of Malan, which misses the bigger picture entirely. These issues are not about one instructor of one course at one university.
Maybe I'm just getting jaded, but I feel like this piece does a disservice to the behemoth of problems at universities represented by the online instructional debacle we've witnessed. Like lots of things, the pandemic is exposing widespread systemic problems, not creating them.
Peer review is a cornerstone of academics, and there continues to be a prestige associated with it as well as with certain journals. This is especially true in certain circles.
As far as I can tell though, functionally this is breaking down. People can find preprints and archived papers, and do, if they're searching by topic.
So journals at this point are providing a peer review portal, and formatting. I happen to think the formatting does provide value. My sense is that at a good journal, there's a kind of stochastic improvement in errors and formatting, so that the numbers of errors go down on average, and the formatting improved, on average, over iterations back and forth with the copyeditor.
As for peer review, I'm not so sure anymore. My sense is that it does provide some kind of stamp of approval from experts in the area, so if you don't know much about an area, it provides some sense that at least some small group of people in the area believe it meets some kind of basic standards. But that says very little, and the amount of noise in the review process is large.
I think the core of the academic communication system is slowly being hollowed out, and being replaced by blogs, things like twitter and mastodon, and archives. At this point the peer review journal process provides some value, but it's being propped up by tradition. Already, with COVID, we're increasingly seeing the focus on preprints. Journalists and others are careful to note something hasn't been peer-reviewed yet, but everyone knows it matters little because they can turn to experts to find out what they think of it.
If there aren't formal attempts to create an alternative, I think we'll just be left with people posting and passing around preprints and discussing them on twitter, mastodon, blogs, and message groups. If people want the nice formatting, and some stamp of approval, I think something else will have to be worked out. But the journals are starting to feel like they're getting in the way, in general, and represents some kind of power or status structure more than quality control system.
Paying reviewers I think creates bad incentives as the author of the post points out. So do author-pays systems. What is maybe missing from the piece is some recognition that in the past, reviewers reviewed and editors edited in part as part of their job. That is, you were paid as a faculty member at a university, and that was what people understood you did. Pre-internet, this was all valuable service. Now that universities and others are more focused on faculty bringing in profits rather than paying for their services -- and questions are being raised about the value of journals in general -- we are seeing these questions about what reviewers get paid.
I think in the future there will be value in article hosting and searching, and providing website frameworks for discussion and peer review, but I'm not sure they will look like journals per se. You'll see things like arxiv.org, but with commentary, rating, discussion, and approval infrastructure over them. That's what large libraries and research centers will be donating money to or paying for. I think journals per se will eventually start to seem kind of stodgy and old fashioned.
It's not just here. It seems like a common sentiment in a lot of places. It's confusing to me as well. There's a direct line in my mind from calls for social media to be self-regulate or be regulated (interestingly in different ways from conservatives and liberals), to these kinds of comments from Erdogan. It might be causal, it might just create a cultural context, but it's hard not to conclude that echoing calls for regulation of social media in the US aren't being coopted or leveraged in this case in a way that's very predictable.
First, there's a sort of framing issue with "subjective answers to questions" per the press release. These subjective experiences are the very nature of the thing of interest: feeling desperate, crying for help, loss of energy, and so forth. To dismiss them is, at some level, is akin to a chemist dismissing oxygen gas as an appropriate target of study because it's all subatomic particles at some level. These types of press releases often start with this notion, in my mind, because it's necessary to create some sense that the study in question is revolutionary.
I do think they're something of a bootstrapping problem with supervised approaches, which is why in psychopathology and psychiatry research there's a trend toward modeling symptom/behavioral presentations using "unsupervised" approaches. This is partially the impetus behind NIH RDoC, for example, and when you do this, you end up with similar but not identical constructs as DSM diagnoses. The recent DSM (DSM-5) was supposed to address this in part but got mired in politics.
If you look at the actual results, by far the most important features in predicting future symptom state are previous values of those symptoms. So follow-up symptom A is best predicted by baseline symptom A. This autocorrelation importance is a sort of law of behavioral individual differences. The EEG variables are adding in the prediction, but relatively weakly, which further speaks to the central importance of the subjective variables, which is informationally mediating whatever is going on under the skull.
This is really further underscored if you start thinking about things like this: are the EEG signals adding most to prediction of "physical" symptom improvement? There's also a literature about severity of symptoms; it's possible to make predictions of symptom response based on other criteria from previous studies and this paper does little to address that in a head-to-head comparison. K-fold cross validation is also good, but isn't the same as replicating something on entirely different sets of people with different sorts of heterogeneity due to unknown sources.
It's an interesting paper but people need to be cautions about the hype that goes on in these areas (biological psychiatry and AI).
My experience is that there is phenomenal cheese on both sides of the Atlantic, so much so it's ridiculous in my mind to argue about who has the best cheese, because they both have the best cheese. It's a win-win situation.
The problem in both regions, if you want to call it a problem, is that much of the variety is in local cheeses you'll never or rarely see going outside of a zip code, state, or country (in the case of Europe). So someone will move to, say, Virginia in the US and conclude that they've sampled US cheese, not being aware of all the local cheeses in another state that never travel outside of the area of a few counties. The same thing can happen in the UK and Europe. The cheeses I've loved most have been from these local dairies, cheeses that don't quite fit into traditional categories, and that I can never find unless I'm physically in the area.
So, I left a tenured position. The linked piece resonates with me.
I'm actually not so sure it's that common relative to the number of people who actually leave. After I left my position I started learning about other tenured faculty who left their positions also, some of whom I respected quite a bit and who I never imagined would have any reason to leave their position. One place interviewed me for a faculty position later, the primary problem being that they had multiple tenured faculty leave and had holes to fill. None of these individuals who left their positions wrote about it.
I do think you're right in the sense that it's this thing that no one knows what to do with. You've built up these relationships, your colleagues are all in academics, and many of them are happy. So they don't understand. You're kind of a glitch in the matrix, and no one knows how to approach it because it's not supposed to happen. I've been told everything from that I'm a total badass to crazy and insane. I myself haven't figured out which is more accurate yet. Maybe both.
The fact that these experiences compel some kind of cathartic explanation speaks volumes about the problems in academics though. It's like there's the commonly agreed-upon story in academics, the face being presented, and then there's reality, and they don't line up, so people just kind of pretend the glitches don't exist.
Not too long ago on HN there was a blog piece going through one person's (an economist?) exploration of whether or not biomedical research was in crisis. (I've tried to find it but can't.) It was frustrating to me because so much of what they detailed was true, but in the end they just decided something like "oh well, it's obvious this field is in crisis, but research is getting done in the end so it doesn't matter." It's like this ship in flames, where people are burning and there's emergency sirens going off, and as long as the individuals in power are happy, and there's a good narrative being presented to the public, with results in the media to talk about, no one cares.
I just can't tell where this is headed: if the ship is going to finally sink so far that it becomes something you can't hide, or if it's just going to keep on like this indefinitely, or if it's going to get fixed, or what. Universities have survived much worse than current circumstances, so it seems implausible to me anything will fundamentally change.
I don't disagree with these points, but I do think many individuals entering college are looking for some kind of extended [rite of] passage or something into adulthood, and I do think college provides that for many.
I don't necessarily think it has to be that way, and I don't necessarily think it's the best way, certainly not for everyone, but I think for a lot of people it provides one way. Right now I think it's a broken path, but I'm not sure what other paths there are to replace it at the moment, given that employers (for the most part, maybe not entirely) seem to be abandoning that role.
To me the bigger underlying subtext in a lot of these discussions is the vacuum that currently exists in terms of providing a safety net for people as they find their way. I think this could be provided by employers or something but I really don't think it is at the moment; colleges might also be flawed in being too expensive or not really delivering on their promises or something, and maybe we're at an inflection point. But if you think of spaces where society is ok with people figuring things out, or learning, or not really being fully developed yet, college is a major place. Outside of that it increasingly feels like people are treated as machines, interchangeable and replaceable, with little to no sense of potential or anything of that sort.
Believe me, I feel like universities are very broken at the moment in many many ways. I just don't really see them going away in the near future, even if they are remade, and I think there's a good opportunity for society to really remake them in a way that's more useful, whatever that is. It may involve restructuring what a college education entails, or paying for it with public funds, or whatever, but that opportunity is there.
As a former tenured professor I've been watching all of this with a bit of empathy, anxiety, concern, and schaudenfreude. Many of my best friends and colleagues are at the university and it still forms a huge part of my life.
In speaking with many students and colleagues, one thing has struck me, which is that many of the current undergrads are enrolled because they want to get away from home. They want some true transition to adulthood, where they live on their own and develop into an independent adult with a vocation and adult identity. This is maybe what I've heard articulated here and elsewhere about being exposed to new ideas, etc. but maybe isn't quite the same. It's something I never really quite appreciated the importance of until recently, but looms large in their minds. Many of them, even though they're taking courses "off-campus" and online, are still in their college town living in their apartment, and so forth. They didn't go home, they just stayed put.
The analogy with books and music is telling because for awhile there too there was a lot of hype about how the internet would lead to some radical reorganization and disruption of those industries. And in many ways it did. But many of the power inequities, the monopolies, and so forth are the same, and the fundamentals are the same. We've migrated a lot from radio and physical media to streaming services, but the players are still there. I suspect something similar will happen with universities; we'll just see a lot more diversity in what form an undergraduate education actually looks like. For some it will involve 4 years in a dorm / on-campus setting. For others it might be more eclectic. There are also some things that just can't really be duplicated at home: an undergrad isn't going to go out and purchase a mass spectrometer for the most part, or PCR equipment, or have storage room full of embalmed anatomy specimens to dissect.
One thing that's important to be mindful of is that this on-campus experience many of us remember is in many ways evaporating anyway. I don't mean to say that's ok, or acceptable, or whatever, but sometimes we talk about the experience of being on campus and discussing things with others in person, forgetting that a major societal issue right now is how that that used to be the norm for everything and is increasingly not. So we have these chats online, on HN comment sections, or over IM, or videoconferencing anyway, even if you're not in college. Whatever might be lost from the on-campus experience is being lost everywhere to some extent; I point this out only to suggest that if we're ok with it in some domains, why should college be special? Or is college the canary that we should be paying closer attention to?
I agree with the perspective of the author of the article that something fundamental needs to change, but my guess is 10 years from now everything will look very different and the same at the same time. I also think there's many problems with universities and higher education that have nothing to do with the cost-benefit calculus in some abstracted sense, but rather to do with how it should be paid for, whether universities are funded well enough, whether they're getting those funds from the right places, whether administration is too top-heavy, whether employers and society abuse interpretation of degrees, and so forth and so on.
It's important to keep that in mind re: improvements in inequality, but at the same time, for example, there is a refugee crisis that's been steadily increasing over time worldwide in recent times. GDP per capita is not the only metric of inequality, nor is China the only exception to the US.
I basically agree with everything you're saying, I just think something is going on that's not nearly captured by a lot of traditional metrics.
People are also not always very good judges of long-term value. For the longest time the response I'd get whenever I expressed some sentiment of supporting small business, all things being equal, was something along the lines of "well you should support whoever can deliver the good at the lowest price." Although that's true to some extent, in the long term it leads to personal costs as local services are gone, and there's less and less competition. Short-term benefits, long term costs and so forth.
In general, I think there's far too little focus on maintaining healthy competition. The pendulum in public discussions often swings from public to private services and employment, which is a good conversation to have, but there's almost never a serious, critical discussion about increasing competition and what those barriers are. US society is full of things that stifle competition and they don't get 1/10 the attention they deserve. It seems as if when it's brought up, you're forced into these ultra-libertarian or anti-capitalist stereotypes.
The other day I was looking for a cycle restoration shop. There's a great one and they should be thriving right now. But their business was destroyed in riots. Not the business owner's fault. You could make an argument that it's just one more downstream effect of a racist cop killing someone over a counterfeit bill, a very significant one. This effect has played out many many times across the country now I'm sure. But who will foot the bill? The Mpls police dept?
It seems we used to have this intuitive understanding of "bad things happen to good people" and "the community should pitch in collectively to support these things" but it's fading away.
The darwinian model is flawed, unless you're focuses solely on the net benefits to those who benefit, which is circular in its reasoning.