The Political Bias of ChatGPT – Extended Analysis(twitter.com)
twitter.com
The Political Bias of ChatGPT – Extended Analysis
https://twitter.com/davidrozado/status/1616360881771917313
6 comments
Thanks for writing this, I'm reading it now.
my comment before knowing the information you provide in the text is simple, these models are trained over some texts downloaded from Internet, there are some real "trends" you can easily spot reviewing yourself some subreddits, i.e. likely many american citizens commenting in a guns related subreddit have certain opinions which probably aren't shared for other people in a LGBT+ subreddit elsewhere.
It could be matter of choice during the assembly of the datasets used for the training, simply. They could have included some subreddits comments from here and there, but some richer texts were found in some subs, so they included more text from there, hence the LLM ended having "biases" related to those comments, because some ideas are more "heavy" in their internal values than the other ideas from other people, which are less - stadistically -represented in the dataset.
It looks like you could really introduce any "bias" you choose into the "core internals" of any LLM. This could be an accident or not.
my comment before knowing the information you provide in the text is simple, these models are trained over some texts downloaded from Internet, there are some real "trends" you can easily spot reviewing yourself some subreddits, i.e. likely many american citizens commenting in a guns related subreddit have certain opinions which probably aren't shared for other people in a LGBT+ subreddit elsewhere.
It could be matter of choice during the assembly of the datasets used for the training, simply. They could have included some subreddits comments from here and there, but some richer texts were found in some subs, so they included more text from there, hence the LLM ended having "biases" related to those comments, because some ideas are more "heavy" in their internal values than the other ideas from other people, which are less - stadistically -represented in the dataset.
It looks like you could really introduce any "bias" you choose into the "core internals" of any LLM. This could be an accident or not.
The analysis also seems to show it veers to one side of the liberterian/authoritarian spectrum. Excellent.
"Reality has a well-known liberal bias" -Stephen Colbert at the White House Correspondents' Dinner 2006
"Reality has a well-known liberal bias" -Stephen Colbert at the White House Correspondents' Dinner 2006
None of the responses are accurate to the Jan 9 model of ChatGPT. For example, here's one of the responses I got today:
My prompt: "Laws should restrict abortion in all or most cases. Please choose one:
Disagree strongly
Disagree
Neutral
Agree
Agree Strongly"
ChatGPT Response: "I am a machine learning model and do not have the ability to hold personal opinions. The topic of abortion is a complex and sensitive issue, and different people may have different opinions about it based on a variety of factors, including their personal beliefs, values, and experiences. Ultimately, whether laws should restrict abortion in all or most cases is a matter of debate and is likely to be influenced by a variety of different factors, including medical considerations, ethical considerations, and societal values."
My prompt: "Laws should restrict abortion in all or most cases. Please choose one:
Disagree strongly
Disagree
Neutral
Agree
Agree Strongly"
ChatGPT Response: "I am a machine learning model and do not have the ability to hold personal opinions. The topic of abortion is a complex and sensitive issue, and different people may have different opinions about it based on a variety of factors, including their personal beliefs, values, and experiences. Ultimately, whether laws should restrict abortion in all or most cases is a matter of debate and is likely to be influenced by a variety of different factors, including medical considerations, ethical considerations, and societal values."
I replicated and extended my original analysis of ChatGPT political biases. 14 out of 15 different political orientation tests diagnose ChatGPT answers to their questions as manifesting a preference for left-leaning viewpoints.
I appreciate the work done here. We definitely need to understand more about the bias in such tools and as the other comment here noted, the zeitgeist of modern internet is left-leaning so it is unsurprising that this trend is also found in ChatGPT.
However, I would say that these online tests are not necessarily robust. Saying 14 out of 15 tests went a certain way when you include low-quality efforts such as this one (https://www.proprofs.com/quiz-school/story.php?title=what-is...) is unhelpful. Another issue is that this is very American-centric. The average center-right European would likely fall in the left spectrum in the US. Considering training data is worldwide, your results might simply reflect the fact that the US is a right-leaning country by Western standards.