Fabricated data in research about honesty(npr.org)
npr.org
Fabricated data in research about honesty
https://www.npr.org/2023/07/27/1190568472/dan-ariely-francesca-gino-harvard-dishonesty-fabricated-data
51 comments
This is the latest in my most frequently learned lesson:
Nobody thinks they are doing anything wrong. They think they are shoring up a weakness in a fundamentally good argument or cause.
Nobody thinks they are doing anything wrong. They think they are shoring up a weakness in a fundamentally good argument or cause.
While I generally go with this, it definitely isn't an absolutely true statement and we should even modify it. People know they are doing wrong, they just justify it with reasons why it is okay to do the wrong thing. I'm pretty sure that these people knew they were doing a big wrong, but I'm sure they justified it with "well everyone fudges the data a little." (this wasn't little)
I'm going to go with both of you.
They know what they're doing is wrong. But I don't think they did it because "everyone does it", they did it because they think they're ultimately right.
They believe in their results, they know in their heart of hearts that their conclusions are true. It's just that not all of the data lines up. It's incomplete, inconclusive, etc. Yes, it is wrong to fabricate data, however, over time, results will bear out the truth of their statements and it won't matter.
They are of the camp that process doesn't matter as long as you reach the right conclusion. Which is wrong. Process is kind of the only thing that matters.
They know what they're doing is wrong. But I don't think they did it because "everyone does it", they did it because they think they're ultimately right.
They believe in their results, they know in their heart of hearts that their conclusions are true. It's just that not all of the data lines up. It's incomplete, inconclusive, etc. Yes, it is wrong to fabricate data, however, over time, results will bear out the truth of their statements and it won't matter.
They are of the camp that process doesn't matter as long as you reach the right conclusion. Which is wrong. Process is kind of the only thing that matters.
Ouch. That's not just scientific dishonesty. It's also politics, and the legal system, and advertising, and internet conversations, and...
"the greater good"
"THE GREATER GOOD"
https://www.youtube.com/watch?v=5u8vd_YNbTw
"THE GREATER GOOD"
https://www.youtube.com/watch?v=5u8vd_YNbTw
Yep. There are very few true villains in reality. Just a lot of people with different values and priorities. Even Vlad Putin probably thinks he’s on the side of the angels.
Related. Others?
Fraudulent data raise questions about superstar honesty researcher (2021) - https://news.ycombinator.com/item?id=36726485 - July 2023 (33 comments)
UCLA professor refuses to cover for Dan Ariely in issue of data provenance - https://news.ycombinator.com/item?id=36684242 - July 2023 (131 comments)
Harvard ethics professor allegedly fabricated multiple studies - https://news.ycombinator.com/item?id=36665247 - July 2023 (215 comments)
Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments)
Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments)
A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 - Aug 2021 (90 comments)
Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments)
Fraudulent data raise questions about superstar honesty researcher (2021) - https://news.ycombinator.com/item?id=36726485 - July 2023 (33 comments)
UCLA professor refuses to cover for Dan Ariely in issue of data provenance - https://news.ycombinator.com/item?id=36684242 - July 2023 (131 comments)
Harvard ethics professor allegedly fabricated multiple studies - https://news.ycombinator.com/item?id=36665247 - July 2023 (215 comments)
Harvard dishonesty expert accused of dishonesty - https://news.ycombinator.com/item?id=36424090 - June 2023 (201 comments)
Noted study in psychology fails to replicate, crumbles with evidence of fraud - https://news.ycombinator.com/item?id=28264097 - Aug 2021 (102 comments)
A Big Study About Honesty Turns Out to Be Based on Fake Data - https://news.ycombinator.com/item?id=28257860 - Aug 2021 (90 comments)
Evidence of fraud in an influential field experiment about dishonesty - https://news.ycombinator.com/item?id=28210642 - Aug 2021 (51 comments)
The podcast “If Books Could Kill” has a great episode that breaks down and debunks the book “Nudge”. The podcast also factchecks various other pop-science and so called “airport books”.
I am glad that with internet I could cut out those word salad crap as much as possible, and stick with more reputable and updated sources.
Basically we have a better package management system to reduce the supply chain attack
Basically we have a better package management system to reduce the supply chain attack
Now we just have to apply that concept to the watering-down which occurs when they try to take white-papers and disseminate them to the public through news articles and etc.
It is important for the public to learn - and they will need assistance to understand findings that are outside of their usual realm of knowledge/experience. Unfortunately, the resources for this are shaky, profit-motivated, and arbitrary.
It is important for the public to learn - and they will need assistance to understand findings that are outside of their usual realm of knowledge/experience. Unfortunately, the resources for this are shaky, profit-motivated, and arbitrary.
'debunks Nudge?'
Someone should tell the politicians and PR consultants...
Someone should tell the politicians and PR consultants...
Recommended video on the matter: https://youtu.be/d2Tm3Yx4HWI
Kinda ironic that my most remembered detail about Dan Ariely's book is that "everyone cheats a little".
https://www.wired.com/2009/02/ted-1/
https://www.wired.com/2009/02/ted-1/
Wouldn't that be the opposite of irony?
I guess that Alanis Morissette tainted my perception of irony.
Obviously he was studying himself
> The two worked together on a paper about how to "nudge" people to be more honest on things like forms or tax returns. Their trick: move the location where people attest that they have filled in a form honestly from the bottom of the form to the top.
Another strike against the entire concept of "nudges". Its become clear that it was a bit of a liberal pipe dream saying you could accomplish huge changes in people or even society with imperceptible "nudges" that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change.
"Be more honest by changing a form." "Make people less fat by moving where the produce is in the store." Turns out things are more complicated.
Another strike against the entire concept of "nudges". Its become clear that it was a bit of a liberal pipe dream saying you could accomplish huge changes in people or even society with imperceptible "nudges" that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change.
"Be more honest by changing a form." "Make people less fat by moving where the produce is in the store." Turns out things are more complicated.
> Make people less fat by moving where the produce is in the store
It doesn't? Not even a little bit ? Genuine question. One of the example of data exploitation I was given in university is that retail companies look for patterns in the thing their customers buy, and when they see that people who buy X-kind of thing also tend to buy Y-kind of thing, they tend to put X and Y right next to each other to push the customers to buy X and Y. Wouldn't doing the opposite work ?
It doesn't? Not even a little bit ? Genuine question. One of the example of data exploitation I was given in university is that retail companies look for patterns in the thing their customers buy, and when they see that people who buy X-kind of thing also tend to buy Y-kind of thing, they tend to put X and Y right next to each other to push the customers to buy X and Y. Wouldn't doing the opposite work ?
They may be referring to the research of Brian Wansink, who was exposed a few years ago for at the very least shoddy research practices and at worst active data manipulation [1, 2, 3]. A large body of his work centered around how to rearrange school lunchrooms to promote better dietary decisions. His advice quite literally includes placing healthy food first [4].
[1] https://www.wsj.com/articles/a-cornell-scientists-downfall-1...
[2] https://web.archive.org/web/20180307074049/http://www.timvan...
[3] https://www.buzzfeednews.com/article/stephaniemlee/who-reall...
[4] http://www.brianwansink.com/for-school-lunches.html
[1] https://www.wsj.com/articles/a-cornell-scientists-downfall-1...
[2] https://web.archive.org/web/20180307074049/http://www.timvan...
[3] https://www.buzzfeednews.com/article/stephaniemlee/who-reall...
[4] http://www.brianwansink.com/for-school-lunches.html
Moving single digit volume of sales and "solving" obesity are two orders of magnitude different things.
> conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change
Who is claiming this?
Who is claiming this?
Good paper on this topic:
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4046264
Chater, N., & Loewenstein, G. (2022). The i-frame and the s-frame: How focusing on individual-level solutions has led behavioral public policy astray. Behavioral and Brain Sciences, 1–60. https://doi.org/10.1017/S0140525X22002023
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4046264
Chater, N., & Loewenstein, G. (2022). The i-frame and the s-frame: How focusing on individual-level solutions has led behavioral public policy astray. Behavioral and Brain Sciences, 1–60. https://doi.org/10.1017/S0140525X22002023
Are there countries where behavioral public policy has not been led astray?
(I have not read the article)
> bit of a liberal pipe dream
Fail to see any connection here.
Fail to see any connection here.
I believe it’s a reference to these initiatives:
https://www.politico.com/agenda/story/2015/10/obamas-effort-...
https://www.politico.com/agenda/story/2015/10/obamas-effort-...
This article is talking about things like defaulting options to organ donor or automatically enrolling people into a 401k plan. I'm not sure changing the default option is the same as changing someone's behavior. Sounds like it's about explicitly NOT changing the behavior but rather the outcome of that behavior.
Something you may notice about these fraud cases is that they are simple things that result in _surprising_ outcomes. I don't think people are surprised that texting teens works better given that the only people that call them are parents and telemarketers. There's two fundamental lines in the article too
> government has long been behind corporate America in following up to make sure its policies actually work.
> The SBST team’s report showed that 13 of its projects worked; one didn’t; and one was hard to tell.
And if we're getting down to it, this doesn't sound like an Obama or "liberal" thing, it sounds like a _government_ thing. Can we stop taking general concepts or universal behavior and pretending that when its bad only "my enemies" do it and when good only "my allies" do it? This is just dumb, and it happens even outside the context of politics.
Something you may notice about these fraud cases is that they are simple things that result in _surprising_ outcomes. I don't think people are surprised that texting teens works better given that the only people that call them are parents and telemarketers. There's two fundamental lines in the article too
> government has long been behind corporate America in following up to make sure its policies actually work.
> The SBST team’s report showed that 13 of its projects worked; one didn’t; and one was hard to tell.
And if we're getting down to it, this doesn't sound like an Obama or "liberal" thing, it sounds like a _government_ thing. Can we stop taking general concepts or universal behavior and pretending that when its bad only "my enemies" do it and when good only "my allies" do it? This is just dumb, and it happens even outside the context of politics.
> And if we're getting down to it, this doesn't sound like an Obama or "liberal" thing, it sounds like a _government_ thing. Can we stop taking general concepts or universal behavior and pretending that when its bad only "my enemies" do it and when good only "my allies" do it? This is just dumb, and it happens even outside the context of politics.
I'm all for this, although it does require an agreement on epistemic principles, which seems like a tall order in the current state of our civilization. Which is why we are left with will to power.
I'm all for this, although it does require an agreement on epistemic principles, which seems like a tall order in the current state of our civilization. Which is why we are left with will to power.
Looking for solutions that do not involve any structural change is inherent to liberal politics in our society, since our current political economy is deeply liberal in structure (for a definition of liberal that contains both mainstream "liberals" and conservatives, closer to classical liberalism).
You can see this plainly by looking at the pattern of recuperation and dilution of any proposed structural change - making up non-structural alternatives that don't work but might and then haranguing over them for years is a common pattern in liberal democracies. See for example trying to avoid public healthcare by creating "optimized marketplaces", the entire idea of nudging in economics, the idea of self-regulation, avoiding providing free national tertiary education by instead giving progressively stringent and oddly targeted scholarships, etc...
You can see this plainly by looking at the pattern of recuperation and dilution of any proposed structural change - making up non-structural alternatives that don't work but might and then haranguing over them for years is a common pattern in liberal democracies. See for example trying to avoid public healthcare by creating "optimized marketplaces", the entire idea of nudging in economics, the idea of self-regulation, avoiding providing free national tertiary education by instead giving progressively stringent and oddly targeted scholarships, etc...
In my experience doing research, this question should go *at the bottom*.
Why: A basic 101 thing one learns in Market Research is that personal questions _always_ go last.
So, if you want to ask an interviewed person if their answers were honest, researchers should be placing them last, towards the end of the questionnaire.
But, this only works if the questionnaire is oriented towards asking first impersonal questions, then more personal ones.
As another commented said: This is a lot of work. Doing research, asking people questions, gathering and then analyzing data is no walk in the park. It's sad that this happened. And, most important of all: Always being honest, no matter what the results are, even if you don't like them.
Why: A basic 101 thing one learns in Market Research is that personal questions _always_ go last.
So, if you want to ask an interviewed person if their answers were honest, researchers should be placing them last, towards the end of the questionnaire.
But, this only works if the questionnaire is oriented towards asking first impersonal questions, then more personal ones.
As another commented said: This is a lot of work. Doing research, asking people questions, gathering and then analyzing data is no walk in the park. It's sad that this happened. And, most important of all: Always being honest, no matter what the results are, even if you don't like them.
There are some types of nudges that have had measurable and durable societal benefits.
The primary one I'm aware of is making 401k a default, people are likely to stay enrolled and increase enrollment.
The primary one I'm aware of is making 401k a default, people are likely to stay enrolled and increase enrollment.
That's not even a nudge, you're not attempting to change anyone's behaviour (which is: do nothing) you're trying to take advantage of their default behaviour. Basically just putting the bin where people dump trash anyway isn't a "nudge".
Sure, I bring it up though because it's discussed in those behavioral psych books.
>Its become clear that it was a bit of a liberal pipe dream saying [...] that conveniently meant you didn't need to address systemic issues and stone wall any attempts at grander change.
Really? I thought liberals were all about addressing systemic issues (eg. systemic racism), and all about "grander change" (eg. defund the police/ICE, eat the rich, etc.).
Really? I thought liberals were all about addressing systemic issues (eg. systemic racism), and all about "grander change" (eg. defund the police/ICE, eat the rich, etc.).
The word "liberal" is terribly overloaded to mean all sorts of things. The OP's intended meaning seems to be liberal in the sense of a lack of coercive measures, preferring incentives instead. Your meaning seems to mean "leftist".
FWIW, I use it to describe that which is Lockean in substance (which entails a kind of radical, atomistic individualism that takes the individual to be the basic unit of society, and not the family), which basically means that most of the political spectrum is implicated.
FWIW, I use it to describe that which is Lockean in substance (which entails a kind of radical, atomistic individualism that takes the individual to be the basic unit of society, and not the family), which basically means that most of the political spectrum is implicated.
[deleted]
I’m a former academic who did a bit of work that involved statistical analysis of data in a similar way as these studies.
Fuck. These. People.
We put so much work making sure the analysis is right. Sweat the design, sweat the data collection, the consent forms, and dropped the studies where the result wasn’t there, as well as reporting null results. I specifically remember having to have a terrible conversation with a phd student that, after doing a power analysis, setting up a pilot study, and collecting enough data, we got p=0.054 on the relevant test. “Sorry, we can’t add additional participants; we can’t try a new analysis. We did the right thing and we should report a non significant result. It sucks.”
And these fuckers just go ahead and forge data to land TED talks?
Yes, let’s also have a conversation about incentives and the system. But individually, I reserve my sympathy for better people. Salt the earth they walk on, and rake them over coils. This is beyond reprehensible. Fuck. Them.
Fuck. These. People.
We put so much work making sure the analysis is right. Sweat the design, sweat the data collection, the consent forms, and dropped the studies where the result wasn’t there, as well as reporting null results. I specifically remember having to have a terrible conversation with a phd student that, after doing a power analysis, setting up a pilot study, and collecting enough data, we got p=0.054 on the relevant test. “Sorry, we can’t add additional participants; we can’t try a new analysis. We did the right thing and we should report a non significant result. It sucks.”
And these fuckers just go ahead and forge data to land TED talks?
Yes, let’s also have a conversation about incentives and the system. But individually, I reserve my sympathy for better people. Salt the earth they walk on, and rake them over coils. This is beyond reprehensible. Fuck. Them.
> This is beyond reprehensible.
It is easy enough to make honest mistakes and spread misinformation before it can be properly replicated that we do still require scientists to be honest. Active deception should come with a high cost. There should be a scientific equivalent of disbarment.
It is easy enough to make honest mistakes and spread misinformation before it can be properly replicated that we do still require scientists to be honest. Active deception should come with a high cost. There should be a scientific equivalent of disbarment.
> Active deception should come with a high cost
Yes, I wish so too. For what's worth, I'd buy "either of Dan Ariely and Francesca Gino write a NYT bestseller book in 5 years, go on a redemption tour, and nothing new happens" for $0.80 on a prediction market.
Yes, I wish so too. For what's worth, I'd buy "either of Dan Ariely and Francesca Gino write a NYT bestseller book in 5 years, go on a redemption tour, and nothing new happens" for $0.80 on a prediction market.
These people are called psychopaths, sociopaths, and narcissists. They sit at the top of almost every single organization because our/America culture breeds them and their personality disorders are effectively a human virus to do exactly that.
These people would be doing this anywhere else regardless of the incentives. Titles and power to harm others and bolster themselves are their only motivators.
People who have done real research like you and me, we'll never sit where they sit.
These people would be doing this anywhere else regardless of the incentives. Titles and power to harm others and bolster themselves are their only motivators.
People who have done real research like you and me, we'll never sit where they sit.
http://datacolada.org/109
http://datacolada.org/98
This latest one has great sleuthing examples:
http://datacolada.org/112
Their analysis suggested that the data (originally collected using Qualtrics) had been modified, i.e. that the data in the data files would not match the original data (still stored on Qualtrics' servers).
It turned out they were right: