Researcher at MIT analyzed 33K NYT articles on Palestine and Israel using ML(twitter.com)
twitter.com
Researcher at MIT analyzed 33K NYT articles on Palestine and Israel using ML
https://twitter.com/h_jackson_/status/1395406943192297481
24 comments
This sounds pretty biased itself, first of all: we would need to compare the reported news to "neutral" facts, if the Hamas actually did initiated let's say 20% more violence (illustrative figure) than Israel then it would make sense that roughly 20% reports were made to report such events. ML in this case is just automated bias, you can't look at real world in absolute, a better thing (maybe) would be to compare the NYT's reports to other new outlets.
Obviously this depends a lot on how you define the metric.
It seems to me (superficially) that "like-for-like" metrics such as "use of passive voice" should be fairly independent of reality; for a similar example, see the American tendency to refer to "officer-involved shootings" (but, in a clear case of bias, to never say "dog-involved bitings").
On the other hand, as you note, "use of violent terms" could be a reflection of real-world differences in violent actions, so it might be a more fraught metric.
I'm really intrigued by the idea that we can bring this kind of quantitative analysis to bear to reveal hidden bias, though; it sounds like it can be a powerful approach. But you're certainly right about potential pitfalls.
It seems to me (superficially) that "like-for-like" metrics such as "use of passive voice" should be fairly independent of reality; for a similar example, see the American tendency to refer to "officer-involved shootings" (but, in a clear case of bias, to never say "dog-involved bitings").
On the other hand, as you note, "use of violent terms" could be a reflection of real-world differences in violent actions, so it might be a more fraught metric.
I'm really intrigued by the idea that we can bring this kind of quantitative analysis to bear to reveal hidden bias, though; it sounds like it can be a powerful approach. But you're certainly right about potential pitfalls.
It's genuinely surprising to me that a researcher at MIT lacks the awareness to attempt at least the pretence of impartiality in a paper about bias.
While I'm a big fan of this NLP analysis, it's disingenuous to state ML can "prove" something. Journalistic bias is fundamentally a subjective judgement; 90% accurate ML models don't make the claim true.
That said, the paper itself is a good read.
http://web.mit.edu/hjackson/www/The_NYT_Distorts_the_Palesti...
That said, the paper itself is a good read.
http://web.mit.edu/hjackson/www/The_NYT_Distorts_the_Palesti...
> I prove a history of disproportionate bias against Palestinians in the New York Times.
This is a really bold statement for any analysis and seems odd to come put of academic work that’s supervised.
This preprint is submitted to the Journal of Palestinian Studies so it will be interesting to see how their peer review checks this out.
I don’t quite understand this conclusion based on the paper as it seems like the author looked for existence of some words and for use of active vs passive voice.
I don’t understand how active vs passive voice is a sign of bias unless you control for many other factors not mentioned. Are specific authors more likely, is time period affected, are other topics similar, etc etc. I don’t get the logic that this shows a bias in general and don’t see anything presented about why these 33k articles are different.
Similarly, I’m not sure how tracking the mention of Palestinian or Israeli people just by word count means anything. Wouldn’t this be impacted by events, this isn’t addressed in the paper. Are there more deaths or attacks in different groups. This seems important to address, but I couldn’t find anything. I’m also not sure how the 33k articles were selected so whether articles were included or excluded would impact these results.
Similarly not sure if the sentiment analysis of the NLP kits she used is similar or dissimilar from NYT in general.
And the determination of violent language seems like a manual exercise with no validation and would like to know more about the methods used to create the set of violent words. I don’t see the actual training_data.csv that the author created and would be interested to see what was actually made since that file seems to be the output of a manual process to sample 500 articles and pick words.
This is a really bold statement for any analysis and seems odd to come put of academic work that’s supervised.
This preprint is submitted to the Journal of Palestinian Studies so it will be interesting to see how their peer review checks this out.
I don’t quite understand this conclusion based on the paper as it seems like the author looked for existence of some words and for use of active vs passive voice.
I don’t understand how active vs passive voice is a sign of bias unless you control for many other factors not mentioned. Are specific authors more likely, is time period affected, are other topics similar, etc etc. I don’t get the logic that this shows a bias in general and don’t see anything presented about why these 33k articles are different.
Similarly, I’m not sure how tracking the mention of Palestinian or Israeli people just by word count means anything. Wouldn’t this be impacted by events, this isn’t addressed in the paper. Are there more deaths or attacks in different groups. This seems important to address, but I couldn’t find anything. I’m also not sure how the 33k articles were selected so whether articles were included or excluded would impact these results.
Similarly not sure if the sentiment analysis of the NLP kits she used is similar or dissimilar from NYT in general.
And the determination of violent language seems like a manual exercise with no validation and would like to know more about the methods used to create the set of violent words. I don’t see the actual training_data.csv that the author created and would be interested to see what was actually made since that file seems to be the output of a manual process to sample 500 articles and pick words.
These are good points. I think the claims are a bit bold.
I do think the "active vs passive voice" is an interesting metric, since it seems intuitively a bit harder to know why that would differ for the two groups.
As I noted below, in the US it's common to note that the phrase "officer-involved shooting" (i.e., passive voice) is often used to describe cases where police officers shoot someone, but "dog-involved biting" is rarely used to describe the case where a dog bites someone. This strikes me as a convincing (though perhaps not ironclad) argument for how such metrics—which are seemingly independent of the rate of mentions (which itself could be reflecting world events)—can shed light on authorial bias.
I think you could construct similar "event-independent" metrics; for example, are victims of violence more or less likely to be named when they are of one group or another? Are direct quotes from spokespeople more likely to be used from one group or another? Etc.
I generally argue against trying to make all journalism just "impartial he-said, she-said", so I'm a little uncomfortable with this exercise, but, again, the "officer-involved shooting" case is one where very clear language bias seems to affect the discourse.
I do think the "active vs passive voice" is an interesting metric, since it seems intuitively a bit harder to know why that would differ for the two groups.
As I noted below, in the US it's common to note that the phrase "officer-involved shooting" (i.e., passive voice) is often used to describe cases where police officers shoot someone, but "dog-involved biting" is rarely used to describe the case where a dog bites someone. This strikes me as a convincing (though perhaps not ironclad) argument for how such metrics—which are seemingly independent of the rate of mentions (which itself could be reflecting world events)—can shed light on authorial bias.
I think you could construct similar "event-independent" metrics; for example, are victims of violence more or less likely to be named when they are of one group or another? Are direct quotes from spokespeople more likely to be used from one group or another? Etc.
I generally argue against trying to make all journalism just "impartial he-said, she-said", so I'm a little uncomfortable with this exercise, but, again, the "officer-involved shooting" case is one where very clear language bias seems to affect the discourse.
I think the active/passive thing is worth expiration. But to 1) conclude that passive=bad seems to need evidence for why that is and 2) is there a significant difference for the Israeli/Palestine article set.
Both of those seem just stated without evidence so it seems odd that the MIT supervisors would have a student write this us.
Both of those seem just stated without evidence so it seems odd that the MIT supervisors would have a student write this us.
> This is a really bold statement
It's obvious. Any entity or agency that sides with the Zionist state, which was formed on stolen and occupied Palestinian lands, is biased. The tides finally seem to be changing, and truth is starting to dawn on people.
It's obvious. Any entity or agency that sides with the Zionist state, which was formed on stolen and occupied Palestinian lands, is biased. The tides finally seem to be changing, and truth is starting to dawn on people.
How is bias defined in an asymmetrical conflict?
I think the data the paper presents doesn't warrant the conclusion.
On the contrary, being "not biased" is predefined here as it would require you to use the same amount of active/passive voice and terms for violence without regard to what is actually happening.
That can be dangerously wrong. It could also mean that Palestinians commit more violence for example. Doesn't mean the NYT is biased. You would need to quantify reality first to be able to compare it to language used.
I think the data the paper presents doesn't warrant the conclusion.
On the contrary, being "not biased" is predefined here as it would require you to use the same amount of active/passive voice and terms for violence without regard to what is actually happening.
That can be dangerously wrong. It could also mean that Palestinians commit more violence for example. Doesn't mean the NYT is biased. You would need to quantify reality first to be able to compare it to language used.
Especially since this conflict is extremely complicated. As an example among many, there is more than one Palestinian side.
"disproportionate use of passive voice..." seems to be a healthy definition of bias to me. I did not have the time to properly analyse all 33000 articles myself, so I'm not in a position to validate the data.
Seems quite natural to me. When dealing with sensitive issues, you may want to use less assertive instruments of speech, like "20 people were killed on the Israeli side" rather than "Palestinians killed 20 Israelis." For proper research you'd need to compare at least against other pieces of military conflicts reporting.
English isn't my native tongue, but I don't really understand the bias conclusion.
After trying to look it up I found this article (2003):
http://itre.cis.upenn.edu/~myl/languagelog/archives/000236.h...
I thought that funny.
But I think it remains true, that you cannot infer bias by only observing language.
After trying to look it up I found this article (2003):
http://itre.cis.upenn.edu/~myl/languagelog/archives/000236.h...
I thought that funny.
But I think it remains true, that you cannot infer bias by only observing language.
It’s not not biased unless used differently. Are these differences due to other variables than just Israel/Palestine?
Also oddly, the paper didn’t include a statistical analysis of whether these differences are even significant.
Also oddly, the paper didn’t include a statistical analysis of whether these differences are even significant.
I think you are right in your critique, but you cannot then draw your own conclusions without providing some data that at least partially supports it.
Yes, we need to have a ground truth with regard to violence committed. But that data is already published elsewhere
https://www.un.org/unispal/document/auto-insert-208380/
Yes, we need to have a ground truth with regard to violence committed. But that data is already published elsewhere
https://www.un.org/unispal/document/auto-insert-208380/
I don’t think your linked ground truth is useful if the author wanted to try to establish reality or a “non-biased” baseline.
I don’t see any attempt in the paper to try to do this or to use your source, or others.
The number of fatalities would be hard, I think, to try to show what the violent wording would be used since it quantifies deaths (eg, 200 deaths in a single incident would be reported different than 200 events with one death each).
But perhaps trying to determine a correlation of reporting to deaths, or reporting to events. Or any attempt.
The biggest problem I have with the paper is that there’s no null hypothesis to disprove, so it “proves bias” without establishing or even estimating what a non-biased corpus looks like.
I don’t see any attempt in the paper to try to do this or to use your source, or others.
The number of fatalities would be hard, I think, to try to show what the violent wording would be used since it quantifies deaths (eg, 200 deaths in a single incident would be reported different than 200 events with one death each).
But perhaps trying to determine a correlation of reporting to deaths, or reporting to events. Or any attempt.
The biggest problem I have with the paper is that there’s no null hypothesis to disprove, so it “proves bias” without establishing or even estimating what a non-biased corpus looks like.
Putting numbers on the suffering is important work, but I don't like the conclusion that are drawn from it. They are mostly invalid.
If Israel attacks Palestine and kills a citizen, Palestine retaliates and kills 5 in return. Should they have stopped after killing just one person? It that a non-biased approach? What is the ethical approach? Kill just one? Kill nobody? Kill as much as you can?
Doesn't allow for conclusions about bias either, especially not in an armed conflict.
If Israel attacks Palestine and kills a citizen, Palestine retaliates and kills 5 in return. Should they have stopped after killing just one person? It that a non-biased approach? What is the ethical approach? Kill just one? Kill nobody? Kill as much as you can?
Doesn't allow for conclusions about bias either, especially not in an armed conflict.
I'm not making a political statement, I'm looking at this as a data science problem
I think these statistical methods would find bias on any topic.
This seems like an interesting claim. One way to validate it might be to run this analysis on the same topic on different news sources and see if the relative biases of the sources matches our intuitions.
For example, I would expect to find that Fox News uses more violent descriptions of BLM protestors, and that the Washington Post uses fewer; I would expect the converse when describing the Jan 6 US Capitol riots.
This kind of comparison doesn't say whether the BLM protests are more or less violent than the Capitol riots, but it would say whether such an analysis hews to our intuition about relative bias between news sources.
For example, I would expect to find that Fox News uses more violent descriptions of BLM protestors, and that the Washington Post uses fewer; I would expect the converse when describing the Jan 6 US Capitol riots.
This kind of comparison doesn't say whether the BLM protests are more or less violent than the Capitol riots, but it would say whether such an analysis hews to our intuition about relative bias between news sources.
The question is rather, was significant bias found.
That depends on the subjective weighted coefficients of your cost functions:)
Love that students are thinking about how they can apply what they learn to real problems.
However, choosing such a charged topic needs to be managed by an advisor - @h_jackson_ is about to get a lot more than she probably bargained for in terms of people arguing good faith, twisting her results to their narrative, and several arm chair scientists that are going to tear her down despite probably never actually reading what she's worked on.
Is her analysis perfect, no. Is it likely flawed, yea probably. This is where a professor can try to tone down the flames and push toward what might be next in the science of getting a right answer.
However, choosing such a charged topic needs to be managed by an advisor - @h_jackson_ is about to get a lot more than she probably bargained for in terms of people arguing good faith, twisting her results to their narrative, and several arm chair scientists that are going to tear her down despite probably never actually reading what she's worked on.
Is her analysis perfect, no. Is it likely flawed, yea probably. This is where a professor can try to tone down the flames and push toward what might be next in the science of getting a right answer.
or is her analysis flawed due to her own bias.
i.e. when you go in with this belief to quote her "The results of this study are now as relevant as ever. Today, the New York Times continues its legacy of Palestinian erasure."
how can one expect the results to be anything but something that reflects that. either the author would continue to tweak and tweak and tweak till they get the results desired or simply not publish something that contradicts their worldview.
i.e. when you go in with this belief to quote her "The results of this study are now as relevant as ever. Today, the New York Times continues its legacy of Palestinian erasure."
how can one expect the results to be anything but something that reflects that. either the author would continue to tweak and tweak and tweak till they get the results desired or simply not publish something that contradicts their worldview.