Show HN: A Platform For Self-Experimentation(doubleblinded.com)
doubleblinded.com
Show HN: A Platform For Self-Experimentation
http://doubleblinded.com/
44 comments
That's a valid concern, and I agree that it's alarming. We will disclose where we are obtaining the L-Theanine from and provide a Certificate of Analysis.
Also, since this is a double blinded trial, if there are significant differences between placebo and real powders then that affirms that the powder was real.
Also, since this is a double blinded trial, if there are significant differences between placebo and real powders then that affirms that the powder was real.
I kinda wish the FDA had some certification to the effect that "this contains what it says it contains and its contents are no more likely to kill you than a peanut butter and jelly sandwich."
Yeah, that would be nice. There is such thing as a Certificate of Analysis which we are going to try to get from our supplier. http://www.sigmaaldrich.com/united-kingdom/technical-service...
For day to day things, this is what I consult.
https://labdoor.com/
https://labdoor.com/
Good call. consumerlab.com is good too.
In the first link, all of those stores are places I'd never buy supplements from. Many stores sell supplements from brands I never see when I research what to take to fix some problem.
When I realized that, that was the end of ever considering buying supplements from almost all stores.
Whole Foods and similar types of stores are reliable. However, I still prefer researching and buying online. Analyzing comments, both good and bad, are very revealing and can sometimes give you an idea if it will work for you.
But now I just try to eat my veggies and so on. Diversification of food choices gives you a more solid spectrum of nutrients and some foods/nutrients amplify the effects of other nutrients.
When I realized that, that was the end of ever considering buying supplements from almost all stores.
Whole Foods and similar types of stores are reliable. However, I still prefer researching and buying online. Analyzing comments, both good and bad, are very revealing and can sometimes give you an idea if it will work for you.
But now I just try to eat my veggies and so on. Diversification of food choices gives you a more solid spectrum of nutrients and some foods/nutrients amplify the effects of other nutrients.
The first link is bogus; it turned out to be a failure of testing and they had to abandon all the accusations and settled for nothing: https://www.reddit.com/r/Nootropics/comments/3ice6t/on_suppl...
Isn't there a huge potential for crowd sourced research platforms to take off and help advance the state of the art?
So far it seems like most studies are done by scientists / academics in a very closed off kind of way, but the true nature of science is open to everyone and with the right platform we might be able to use normal people to gather enough data to confirm or deny our hypotheses in a way that's much more open than the standard study.
Essentially, anyone would be able to test their hypotheses and either help advance scientific knowledge or at least provide extremely interesting markers that could be used as a basis to get funding for more rigorous / controlled studies. This approach obviously won't work well for every kind of question but just as the OP has highlighted: it's particularly well-suited for making self-experimentation more rigorous and I imagine it would work well for the social sciences.
Imagine if we gave independent researchers like Gwern the tools they needed to help answer more questions. If it was designed right: it would also be an incredible learning tool for the scientific method (since you could participate in the experiments + propose new ones.)
Are there any platforms like this in existence, I wonder? U-Uber for science?
So far it seems like most studies are done by scientists / academics in a very closed off kind of way, but the true nature of science is open to everyone and with the right platform we might be able to use normal people to gather enough data to confirm or deny our hypotheses in a way that's much more open than the standard study.
Essentially, anyone would be able to test their hypotheses and either help advance scientific knowledge or at least provide extremely interesting markers that could be used as a basis to get funding for more rigorous / controlled studies. This approach obviously won't work well for every kind of question but just as the OP has highlighted: it's particularly well-suited for making self-experimentation more rigorous and I imagine it would work well for the social sciences.
Imagine if we gave independent researchers like Gwern the tools they needed to help answer more questions. If it was designed right: it would also be an incredible learning tool for the scientific method (since you could participate in the experiments + propose new ones.)
Are there any platforms like this in existence, I wonder? U-Uber for science?
There are some in existence already:
- quantifiedmind.com for self-testing your mind, which has a few group experiments as well. I'm looking into integrating with them as part of doubleblinded.com.
- PACO (by Google) https://www.pacoapp.com/ Pretty cool and flexible platform, although not the most user friendly or easy to figure out. It is open source and has an app, which is cool.
- A few sites like 23andme and ubiome have a lengthy list of survey questions they ask you, which I only imagine they would pass through pattern recognition algorthms to detect correlations between whatever information you give them and your Genome (or gut genome in the case of 23andme).
So yeah, there is stuff out there. We want to do this for nootropics and supplements (and eventually much more).
- quantifiedmind.com for self-testing your mind, which has a few group experiments as well. I'm looking into integrating with them as part of doubleblinded.com.
- PACO (by Google) https://www.pacoapp.com/ Pretty cool and flexible platform, although not the most user friendly or easy to figure out. It is open source and has an app, which is cool.
- A few sites like 23andme and ubiome have a lengthy list of survey questions they ask you, which I only imagine they would pass through pattern recognition algorthms to detect correlations between whatever information you give them and your Genome (or gut genome in the case of 23andme).
So yeah, there is stuff out there. We want to do this for nootropics and supplements (and eventually much more).
The "eventually much more" part is what most interests me.
You guys might actually be some of the first researchers in the world to experiment with this kind of model. I can see this being very big if its marketed right.
You guys might actually be some of the first researchers in the world to experiment with this kind of model. I can see this being very big if its marketed right.
Small typo in the url for Quantified Mind, it's quantified-mind.com
IIRC, https://experiment.com/ is funded by by YC
This makes me wonder if there is a simple procedure for doing a pseudo-double-blind study on oneself (without a third party). That is, trying something that can be prepared in pill form in such a way that you don't know during any given day/week/month if it was a placebo, but that you can still determine that later to collate the data. I know a lot of people that swear by particular unregulated supplements or dietary choices and it would be really interesting to have an easy way to see if something they're taking is having the effect they think it does.
Yes, independent self experimentation is the way to go. You are not hoping that you can trust the people doing the study. You don't need to hope they did things the way they were reportedly done. You don't need to hope someone did not get paid off to tweak the numbers at the end. You just need to educate yourself on how to do proper studies and how to avoid false conclusions and techniques to bypass your own biases. But you also need to realize the results only apply to you at this part of your life. You can't promote your discoveries as being an absolute cure for other peoples problems. Even your own body can change over time and what used to work for you may no longer work.
I, too, have thought about conducting a pseudo-double-blind experiment on myself. I think I could pull it off, but I'd also be willing to pay a third party to help me. For example, DoubleBlinded could partner with a respect supplement maker, and offer personal experiments in any supplement made by their partner. The consumer wouldn't have to worry about setting up the experiment correctly, and they would get their own results immediately upon completion. Aggregate studies can wait until enough customers have self-experimented on the same supplement, and DoubleBlinded can afford to wait that long, because the partnership model keeps their costs low.
This is definitely one potential path we are considering. There's a question of whether we want to keep this specific to supplements/pills or try to broaden it to other things like exercise or diet (which are not exactly double-blindable)
If you can pill it, you could add unique markers (maybe apply pseudo-random patterns to pill blanks) and take pictures before randomization and immediately before consumption. Or build something that dispenses a random choice from multiple boxes and keeps a log.
If you're willing to involve a friend for 30 seconds, get them to label pillboxes and make a note of which pillboxes were labelled.
As far as I can tell, gwern.net does exactly that (and it's well documented in his website - under the nootropics category).
A very cool idea @ the high level, but one fatal flaw:
"At the end of each day you will fill out a short survey with a few questions evaluating the effects of the pills."
Self-reporting is an unsuitable mechanism to draw out scientific results.
There's an excellent detailed explanation available (1) but in TL;DR here are four of the most compelling factors at work:
1 - Honesty/Image management
2 - Introspective ability
3 - Understanding / Question Interpretation
4 - Response bias
Take Image Management & Response Bias - participants know that they will be able to see their results vs. the control group and it's not a leap to realize how easily our ego and even subconscious need for validation could dramatically skew the full study results.
(1) http://www.sciencebrainwaves.com/the-dangers-of-self-report/
"At the end of each day you will fill out a short survey with a few questions evaluating the effects of the pills."
Self-reporting is an unsuitable mechanism to draw out scientific results.
There's an excellent detailed explanation available (1) but in TL;DR here are four of the most compelling factors at work:
1 - Honesty/Image management
2 - Introspective ability
3 - Understanding / Question Interpretation
4 - Response bias
Take Image Management & Response Bias - participants know that they will be able to see their results vs. the control group and it's not a leap to realize how easily our ego and even subconscious need for validation could dramatically skew the full study results.
(1) http://www.sciencebrainwaves.com/the-dangers-of-self-report/
I'm not sure I see that as a fatal flaw... ultimately, the user is sitting in front of a digital box that can guide and prompt them in all the same ways a researcher can--the only limit is in performing tests that require a medical professional to assess biomarkers. If the pace of medical device development continues, it's even reasonable to think something like a Theranos-that-works could commoditize the process while being intrinsically tamper-resistant.
Regardless, users can be prompted to perform any software action (knowingly or unknowingly, to affect bias) and that action can be measured by the system. It may so happen that every critical measurement occurs unbeknownst to the user, before they self-report anything (if at all). As we are currently undergoing a period of sensor-proliferation (fitness/health devices, wearables, internet of things, etc...) it's not unrealistic to think we will soon be able to instantly correlate data from a smartphone camera, blood/tissue, and the cloud.
Now there's always the problem of intentional fraud/deception, but I think the aggregate nature solves that problem. A small percentage will try to "break" the system, and that small percentage will never surpass a critical threshold with enough volume. In terms of ML/SVM's, we're now very good about filtering outliers or "misrepresented data"... while the responsibility is on you to develop a reliable classifier (for data-consistency more than arbitrary measurement), I imagine at scale you could infer trends with the same relative accuracy of traditional academia and research.
It's a really fascinating new direction--even if only an adjunct to traditional research--and I'll definitely be keeping an eye on the project.
Regardless, users can be prompted to perform any software action (knowingly or unknowingly, to affect bias) and that action can be measured by the system. It may so happen that every critical measurement occurs unbeknownst to the user, before they self-report anything (if at all). As we are currently undergoing a period of sensor-proliferation (fitness/health devices, wearables, internet of things, etc...) it's not unrealistic to think we will soon be able to instantly correlate data from a smartphone camera, blood/tissue, and the cloud.
Now there's always the problem of intentional fraud/deception, but I think the aggregate nature solves that problem. A small percentage will try to "break" the system, and that small percentage will never surpass a critical threshold with enough volume. In terms of ML/SVM's, we're now very good about filtering outliers or "misrepresented data"... while the responsibility is on you to develop a reliable classifier (for data-consistency more than arbitrary measurement), I imagine at scale you could infer trends with the same relative accuracy of traditional academia and research.
It's a really fascinating new direction--even if only an adjunct to traditional research--and I'll definitely be keeping an eye on the project.
Just read through that. I agree with the points about self-reporting being unreliable.
Even with the bias of self reporting, differences between real and placebo will still be meaningful. You can't entirely deceive yourself if you don't know what you just took.
Also, there is some degree of honesty required for this experiment. If someone really wanted to, they could open up the pills and taste them to see if it's placebo or not. They could also intentionally lie. But the goal here is to learn about yourself through experimentation, so I expect most people that participate to have some level of desire for truth and honesty. I would also argue that while these biases do exist, even large randomized clinical trials have the same pitfalls, so we are not necessarily worse off than what's already out there.
Even with the bias of self reporting, differences between real and placebo will still be meaningful. You can't entirely deceive yourself if you don't know what you just took.
Also, there is some degree of honesty required for this experiment. If someone really wanted to, they could open up the pills and taste them to see if it's placebo or not. They could also intentionally lie. But the goal here is to learn about yourself through experimentation, so I expect most people that participate to have some level of desire for truth and honesty. I would also argue that while these biases do exist, even large randomized clinical trials have the same pitfalls, so we are not necessarily worse off than what's already out there.
AFAICT, there isn't a control group. You have 4 weeks of pills and each week it could be real or fake. So you are comparing against yourself.
I would not mess with Saint Johns wart. It will make you feel happy and cheery; but side effects are brutal. And if you participate in other drugs it might have harsh side effects.
[deleted]
Are you mainly referring to it being a CYP3A4 inhibitor?
And it's been banned in EU in 2011.
It's still available for sale from the biggest herbal supplement retailer in the UK [1]. That fact alone makes this claim seem unlikely.
[1] http://www.hollandandbarrett.com/shop/product/holland-barret...
[1] http://www.hollandandbarrett.com/shop/product/holland-barret...
A quick google search makes it sound more like all herbal medicines are now more strictly regulated but not banned.
I have done a lot of alternative med stuff. I know good protocols for how to test the waters, introduce new supplements, figure out what they are doing, etc. I would not use the service you are offering. It sounds to me like you are asking me to be a guinea pig.
To make this useful, I think you need to do the following things:
1) Educate people on good practices generally. Your current site looks to me to be an invitation to be someone's guinea pig and pay for the privilege instead of being paid.
2) Offer doubleblinded kits for a wide variety of supplements that people can choose from. The one supplement you are currently offering is one I never heard of. So I don't care. But I might care if you offered me the chance to pick and choose from a list of supplements.
I am no longer taking supplements, but I did take a lot of supplements at one time. I can see a service like this having a use for people, but not in its current format.
Also, I would remove the phrase "self experimentation." That sounds incredibly Dr. Jeckyl and Mr. Hyde to me. "Self testing" might be okay, but that's a terrible phrase and should be stricken from company vocabulary.
To make this useful, I think you need to do the following things:
1) Educate people on good practices generally. Your current site looks to me to be an invitation to be someone's guinea pig and pay for the privilege instead of being paid.
2) Offer doubleblinded kits for a wide variety of supplements that people can choose from. The one supplement you are currently offering is one I never heard of. So I don't care. But I might care if you offered me the chance to pick and choose from a list of supplements.
I am no longer taking supplements, but I did take a lot of supplements at one time. I can see a service like this having a use for people, but not in its current format.
Also, I would remove the phrase "self experimentation." That sounds incredibly Dr. Jeckyl and Mr. Hyde to me. "Self testing" might be okay, but that's a terrible phrase and should be stricken from company vocabulary.
Hey, thanks for the feedback. I totally agree about educating people on good practices. This is a big part of what I want to do over the next few years - I think educating people on the scientific method, especially in regards to self experimentation is important. We want to work to create best practices and protocols in the self-experimentation/quantified self world. We are planning on putting together some content (videos and such) on the topic and incorporating that educational aspect in our platform.
As far as the supplements go, I agree with that as well. We're planning on adding more once the L-Theanine trial is sorted out. This is just our practice round to get the protocol right. What other supplements would you be curious to see?
As far as the supplements go, I agree with that as well. We're planning on adding more once the L-Theanine trial is sorted out. This is just our practice round to get the protocol right. What other supplements would you be curious to see?
Did you come up with this idea, or why was doubleblinded registered just after this Reddit post?
https://www.reddit.com/r/Nootropics/comments/3p0y2x/crowdfun...
https://www.123-reg.co.uk/order/domain?X-CSRF-Token=45b50019...
That doesn't seem very trustworthy
https://www.reddit.com/r/Nootropics/comments/3p0y2x/crowdfun...
https://www.123-reg.co.uk/order/domain?X-CSRF-Token=45b50019...
That doesn't seem very trustworthy
This sounds like it could very easily be abused as a way to peddle supplements while skimping on the experimentation aspect. It's a good idea though. Do you have plans to include something other than supplements? For instance, common foods (to be consumed daily for long periods of time) or simple activities/exercises? You could still derive revenue by selling the equipment/ingredients through your platform but it would allow you to vary the product while maintaining the consumer base. Also, doing it with something other than supplements could be cheaper and more newsworthy if you do discover something. Very interesting!
Thank you for your informative and critical feedback - skepticism is at the heart of science! Very true, the system could potentially be used to promote a specific agenda. However, we very much want to avoid a system that allows something such as this as much as possible. Ideally through implementing open-source / 3rd party / community-input based protocols. If you have suggestions in this regard we'd love to hear them!
And yes, we plan on including food at a later date!
And yes, we plan on including food at a later date!
I just reported results from experimenting on myself at a bio-tech conference called "Basel Life Science Week 2015".
Cocoa seems to have made me two years younger.
The poster I presented is at
http://morse.kiwi.nz/kingsley/lib/exe/fetch.php?cache=&media...
Cocoa seems to have made me two years younger.
The poster I presented is at
http://morse.kiwi.nz/kingsley/lib/exe/fetch.php?cache=&media...
I wonder why they picked L-Theanine for their first experiment - the effects are commonly known (caffeinated tea vs caffeinated coffee). Maybe this knowledge was used deliberately so people "feel" a drug the first time around, and want to keep going.
We picked L-Theanine because of its safety profile and mild psychactive properties, meaning you can tell if you've taking it or not but it's not an overbearing feeling. We wanted the first experiment to be fairly predictable while we work through all the kinks of launching this.
Is there motivation to cheat? If I want to sell this stuff what's to stop me from signing up a bunch of people, having the pills tested, then reporting what I want you to find so I that you'll tell the public my product is effective?
This won't be an issue because we are supplying the supplements which are going to come from an established and reputable supplier. This is not a platform where anyone can sell their own supplements.
I think you missed my point. I sign 100-1000 people up for your study. You send the supplements to them. I quickly test the supplements to find out which is which. I then send you a report that makes the days my people took the supplements look positive for efficacy. You then report the supplements are effective. I can now point to your report to sell my supplements. It doesn't matter they are not your supplier. I still benefit from people thinking a supplement is effective.
If this can somehow preempt experimentation on animals, I'm all for it.
http://well.blogs.nytimes.com/2015/02/03/sidebar-whats-in-th...
http://www.nytimes.com/2013/11/05/science/herbal-supplements...
Want to do a double-blind study on whether St. John's Wort improves your mood? None of the bottles tested from Walmart, Target, GNC or Walgreens actually contained any St. John's Wort in 2015. That's despite all the reports years earlier about the same problem. There's little reason to believe things have improved or will improve so long as supplements remain largely unregulated.