I love what you’re trying to accomplish here! I encourage you to experiment aggressively and try to focus on testing the biggest risks, rather than prioritizing the easiest things to make progress on but whose success or failure won’t impact your overall odds of achieving your goal.
One thing which I’ve noticed in your response to other comments is that your plan for controlling bad behavior by participants in your system often boils down to either having a strong culture means no one will behave poorly, or that voting and crowd consensus is an effective system for both motivating good behavior and punishing bad behavior. This probably works fine as long as (1) the community is very small such that everyone actually knows everyone else, or (2) there isn’t much actually at stake. You reference Stack Overflow and Wikipedia - these are examples where not much is at stake. If accumulating points and reputation in either of these systems would determine who gets tenure and the culmination of lifelong career ambitions, then I wonder if those systems would be as robust to bad actors. A white-hot risk I think you should focus on to experiment is that peer-reviewed academic science is not a friendly collegial system but an aggressive and high-stakes game where people have very very big incentive to game the system. The current system has tons of problems, but it does function in the face of participants who would like to cheat if they could. The big influential editors and reviewers act like a leviathan in the Hobbesian sense which causes many problems but also solves many.
I encourage you to think about how you could solve some of these potential issues that the current system doesn’t have so much, particularly around preventing various forms of bad behavior which are easily caught and punished in a more hierarchical system like the current one, but which a more crowd-sourced system might fall prey to. Things like collusion rings to upvote each others’ papers, using lay popularity or scientific fame to overwhelm legitimate complaints about issues with a paper, even bot nets or paying to win by farming upvotes - all things which current journals and peer review are basically immune to. People will try this with their careers and livelihoods on the line in a way Wikipedia editors or Stack Overflow users just wouldn’t care to.
These issues can be overcome, but for what it’s worth I think this is the hardest part of what you’re trying to accomplish, and more important for you to focus on than the tech stack or getting a first few papers published on your setup. It will be hard to simulate, but I think you should focus on how to test the behavior of a scientist who is desperate and has their back against the wall in the publish or perish game, and how you can design to reward good behavior and prevent bad behavior in the worst case, not just in the best case or even the average case.
Very cool effort. If you pull this off the positive externalities could be huge.
One thing which I’ve noticed in your response to other comments is that your plan for controlling bad behavior by participants in your system often boils down to either having a strong culture means no one will behave poorly, or that voting and crowd consensus is an effective system for both motivating good behavior and punishing bad behavior. This probably works fine as long as (1) the community is very small such that everyone actually knows everyone else, or (2) there isn’t much actually at stake. You reference Stack Overflow and Wikipedia - these are examples where not much is at stake. If accumulating points and reputation in either of these systems would determine who gets tenure and the culmination of lifelong career ambitions, then I wonder if those systems would be as robust to bad actors. A white-hot risk I think you should focus on to experiment is that peer-reviewed academic science is not a friendly collegial system but an aggressive and high-stakes game where people have very very big incentive to game the system. The current system has tons of problems, but it does function in the face of participants who would like to cheat if they could. The big influential editors and reviewers act like a leviathan in the Hobbesian sense which causes many problems but also solves many.
I encourage you to think about how you could solve some of these potential issues that the current system doesn’t have so much, particularly around preventing various forms of bad behavior which are easily caught and punished in a more hierarchical system like the current one, but which a more crowd-sourced system might fall prey to. Things like collusion rings to upvote each others’ papers, using lay popularity or scientific fame to overwhelm legitimate complaints about issues with a paper, even bot nets or paying to win by farming upvotes - all things which current journals and peer review are basically immune to. People will try this with their careers and livelihoods on the line in a way Wikipedia editors or Stack Overflow users just wouldn’t care to.
These issues can be overcome, but for what it’s worth I think this is the hardest part of what you’re trying to accomplish, and more important for you to focus on than the tech stack or getting a first few papers published on your setup. It will be hard to simulate, but I think you should focus on how to test the behavior of a scientist who is desperate and has their back against the wall in the publish or perish game, and how you can design to reward good behavior and prevent bad behavior in the worst case, not just in the best case or even the average case.
Very cool effort. If you pull this off the positive externalities could be huge.