Analysis of whether IMDB ratings suffer recency bias(kyso.io)
kyso.io
Analysis of whether IMDB ratings suffer recency bias
https://kyso.io/KyleOS/movies
46 comments
It's well known that movie rankings do. This was key to the resolution of
https://en.wikipedia.org/wiki/Netflix_Prize
The time series of reviews of movies like
https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film)
and
https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Jus...
tend to have many early bad reviews by mainstream people who can't get past things that they find squicky and the lack of the usual "Hollywood Happy Ending" but that get better reviews from "true fans" later on who enjoy the fact that the directors tried to make something different even if it was flawed.
https://en.wikipedia.org/wiki/Netflix_Prize
The time series of reviews of movies like
https://en.wikipedia.org/wiki/Sucker_Punch_(2011_film)
and
https://en.wikipedia.org/wiki/Batman_v_Superman:_Dawn_of_Jus...
tend to have many early bad reviews by mainstream people who can't get past things that they find squicky and the lack of the usual "Hollywood Happy Ending" but that get better reviews from "true fans" later on who enjoy the fact that the directors tried to make something different even if it was flawed.
A quick Google shows a few sites that keep history of IMDB ratings. Unfortunately they seem to concentrate on movies actually in the IMDB 250
Some interesting ones:
Avatar - http://top250.info/movie/?0499549
Se7en - http://top250.info/movie/?0114369
Heat - http://top250.info/movie/?0113277
Star Wars: Episode VII - The Force Awakens http://top250.info/movie/?2488496
Terminator 2 - http://top250.info/movie/?0103064
Note: There seems to be a big shifts in movie ranks in May 2005 and July 2012, probably due to a change in the ranking formula.
Some interesting ones:
Avatar - http://top250.info/movie/?0499549
Se7en - http://top250.info/movie/?0114369
Heat - http://top250.info/movie/?0113277
Star Wars: Episode VII - The Force Awakens http://top250.info/movie/?2488496
Terminator 2 - http://top250.info/movie/?0103064
Note: There seems to be a big shifts in movie ranks in May 2005 and July 2012, probably due to a change in the ranking formula.
Interestingly, the opposite of what happened with Star Wars: The Last Jedi.
Casual fans and critics loved the subversion of genre tropes, story roles, character archetypes, and plot points.
"true fans" reviled that the direction align with their vision for the franchise, and it sounds like Episode 9 will be a return to the comfortable and familiar.
Casual fans and critics loved the subversion of genre tropes, story roles, character archetypes, and plot points.
"true fans" reviled that the direction align with their vision for the franchise, and it sounds like Episode 9 will be a return to the comfortable and familiar.
Recency bias came up a few days ago in a previous HN submission by the OP, and the OP also commented in: https://news.ycombinator.com/item?id=20099023
In that subthread, I posted a heat map data visualization I made of IMDB rating vs. release year, which is suspiciously similar to the one OP made: https://news.ycombinator.com/item?id=20099344
It's not plagiarism, but if the OP got the idea for checking for recency bias from the HN thread, they should explicitly credit that.
In that subthread, I posted a heat map data visualization I made of IMDB rating vs. release year, which is suspiciously similar to the one OP made: https://news.ycombinator.com/item?id=20099344
It's not plagiarism, but if the OP got the idea for checking for recency bias from the HN thread, they should explicitly credit that.
Hey - not OP but I wrote the article. There was a lot of discussion about recency bias in that thread, including the graph you shared and I decided to expand on the analysis. Note there are a few other graphs in my post. I did just draw this up quickly yesterday. But you're right, sorry - I should have cited your comment in my post. Adding it now.
Hi, there are what seem to me (not a statistician) to be a number of serious errors in the article. The most egregious is the first conclusion, which threatens to undermine the whole thing.
> There does not seem to be any strong connection between number of votes and a movie's IMDB rating.
Well, this just isn't true. I happen to know this because I've looked at this data before myself. The issue is that you're cutting off the graph at a maximum of 60k votes with no explanation or even pointing out that you're doing it! I'm sure this is just an honest mistake, but it cuts out basically every movie that is actually popular!
This slowly ate at me until it was enough to get me out of bed to redo this graph myself. I've uploaded my quick and dirty result in R here: https://i.imgur.com/TTuCFEL.png
As you can see from the graph, there's a direct and obvious link between the number of votes and the average rating. The reason I say this threatens to undermine the whole thing is that I also know from previous experience that more recent films tend to have a lot more ratings on average. This (naively) suggests to me that we should see some recency bias if only because more recent blockbusters get a much larger number of votes than others and the sort of people who vote on blockbusters have different standards for what makes a great movie. (Avengers: Endgame was briefly the #1 movie of all time on the top 250 list.)
I don't have time to plot all the different graphs you did, but perhaps you can recheck your results after fixing at least this issue, and get back to us?
> There does not seem to be any strong connection between number of votes and a movie's IMDB rating.
Well, this just isn't true. I happen to know this because I've looked at this data before myself. The issue is that you're cutting off the graph at a maximum of 60k votes with no explanation or even pointing out that you're doing it! I'm sure this is just an honest mistake, but it cuts out basically every movie that is actually popular!
This slowly ate at me until it was enough to get me out of bed to redo this graph myself. I've uploaded my quick and dirty result in R here: https://i.imgur.com/TTuCFEL.png
As you can see from the graph, there's a direct and obvious link between the number of votes and the average rating. The reason I say this threatens to undermine the whole thing is that I also know from previous experience that more recent films tend to have a lot more ratings on average. This (naively) suggests to me that we should see some recency bias if only because more recent blockbusters get a much larger number of votes than others and the sort of people who vote on blockbusters have different standards for what makes a great movie. (Avengers: Endgame was briefly the #1 movie of all time on the top 250 list.)
I don't have time to plot all the different graphs you did, but perhaps you can recheck your results after fixing at least this issue, and get back to us?
Follow up: I redid one of the graphs to satisfy my own interest in the question. Instead of averaging over all the movies ever released like the article did, I averaged over all the votes. This was an attempt to answer the question of whether the average viewing experience of a film from year x is better or worse than that of a film from year y.
I found that there has been a noticeable decline in the average rating of over half a point (out of 10) since ~1930 or so. https://i.imgur.com/bY9vPvk.png
I think this should be explained in a combination of two ways:
* History acts as a filter. If I choose to watch a movie from 1947 it's probably because a lot of people over the years have said it's good.
* To some extent, older movies may really be better than more recent ones.
I still suspect that the blockbuster effect means that the movies people are mostly to watch are going to receive higher ratings than the average film overall. And this is mostly because people rating blockbusters are less critical than the film-viewing community as a whole. So while there might be no "recency bias" of the kind this article was looking for, there might be blockbuster bias where in the last several decades studios have figured out how to capitalize on a less critical / cynical audience. (That hockey stick graph of 8-10 ratings in the article is certainly suggestive.)
I found that there has been a noticeable decline in the average rating of over half a point (out of 10) since ~1930 or so. https://i.imgur.com/bY9vPvk.png
I think this should be explained in a combination of two ways:
* History acts as a filter. If I choose to watch a movie from 1947 it's probably because a lot of people over the years have said it's good.
* To some extent, older movies may really be better than more recent ones.
I still suspect that the blockbuster effect means that the movies people are mostly to watch are going to receive higher ratings than the average film overall. And this is mostly because people rating blockbusters are less critical than the film-viewing community as a whole. So while there might be no "recency bias" of the kind this article was looking for, there might be blockbuster bias where in the last several decades studios have figured out how to capitalize on a less critical / cynical audience. (That hockey stick graph of 8-10 ratings in the article is certainly suggestive.)
Actually -- one more follow up for the zero people who will read this -- there's a simpler explanation, which other people who do these kinds of statistics should probably note.
The average rating between 1920 and 1980 fluctuated at about 7.5. Then the number of movies exploded, and the variability in quality of them exploded even more than that. Movies that were better than the 1980 average had their ratings compressed (you can only do a little better than 7.5 on IMDB). Movies that were worse were much less compressed (you can easily do much worse than a 7.5). So the average gets dragged down since there are now both more movies getting 0-4s and movies getting 8-10s.
I realized this after plotting the median, not the mean, and observing that it stays remarkably constant, and possibly even shows a little recency bias for the last 2 years.
The average rating between 1920 and 1980 fluctuated at about 7.5. Then the number of movies exploded, and the variability in quality of them exploded even more than that. Movies that were better than the 1980 average had their ratings compressed (you can only do a little better than 7.5 on IMDB). Movies that were worse were much less compressed (you can easily do much worse than a 7.5). So the average gets dragged down since there are now both more movies getting 0-4s and movies getting 8-10s.
I realized this after plotting the median, not the mean, and observing that it stays remarkably constant, and possibly even shows a little recency bias for the last 2 years.
I agree with you.
Does the opening sentence count? "I've read a lot recently about the recency bias of movie ratings, IMDB ratings in particular."
Does the opening sentence count? "I've read a lot recently about the recency bias of movie ratings, IMDB ratings in particular."
Doesn't this analysis rest on the assumption that movies are equally good each year? Maybe I suffer from nostalgia (or survivorship bias) but I'm not sure this is a good assumption.
The problem with past movies is you only see the good ones. I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies - because while both suffer from horrible plots usually modern movies at least have interesting visuals.
However for certain categories the past was most definitely better. Dramas being the main one. Without a lot of fancy effects the plot was the only thing you had to carry the movie. They also don't beat things over your head and can embrace subtlety in a way modern movies do not.
I also really liked the 70s trend of movies that didn't really have a satisfying ending, you just get to experience the lives of people who don't really improve. 5 easy pieces or The Last Detail fit that category very well.
However for certain categories the past was most definitely better. Dramas being the main one. Without a lot of fancy effects the plot was the only thing you had to carry the movie. They also don't beat things over your head and can embrace subtlety in a way modern movies do not.
I also really liked the 70s trend of movies that didn't really have a satisfying ending, you just get to experience the lives of people who don't really improve. 5 easy pieces or The Last Detail fit that category very well.
> I've spent some time watching the bottom barrel movies from the past and they seem even worse than Today's movies
That's because back then we didn't have the internet. No matter how bad a movie was, people would go see it for at least a week before the word got out how bad it was. Looking up movie times was a pain, you had to plan ahead and check the newspaper. If you went on a date Friday/Saturday night, the most likely thing to do was show up the movie theater after dinner and buy a ticket for the movie that started next, since you probably hadn't heard of any of them, or you would buy a ticket that you'd seen the most advertising for.
There were no ratings to check, no reviews to read, unless you happen to read them in the paper that morning.
Nowadays, they know they have to produce at least something minimally good, because word will get out very quickly that something is bad, and it's super easy for people to check rating when they arrive at the theater, or even before they arrive.
That's because back then we didn't have the internet. No matter how bad a movie was, people would go see it for at least a week before the word got out how bad it was. Looking up movie times was a pain, you had to plan ahead and check the newspaper. If you went on a date Friday/Saturday night, the most likely thing to do was show up the movie theater after dinner and buy a ticket for the movie that started next, since you probably hadn't heard of any of them, or you would buy a ticket that you'd seen the most advertising for.
There were no ratings to check, no reviews to read, unless you happen to read them in the paper that morning.
Nowadays, they know they have to produce at least something minimally good, because word will get out very quickly that something is bad, and it's super easy for people to check rating when they arrive at the theater, or even before they arrive.
So you want to say that Shutter Island is a bad example of drama? It's all just old-farts rants, nothing more.
Everything is evolving.
In old movies actors play often too dumb, just few of them have good examples of how to play a role.
Some "legendary" movies from the past would fail in theaters nowadays without their legendary reputation.
We have more movies right now, more choices, so our demands are higher - otherwise we will not have enough time to watch them all. Also, we have much more choices of how to watch them - age of digital distribution.
Higher demands give higher quality, movies evolve both in visuals and in acting art.
Everything is evolving.
In old movies actors play often too dumb, just few of them have good examples of how to play a role.
Some "legendary" movies from the past would fail in theaters nowadays without their legendary reputation.
We have more movies right now, more choices, so our demands are higher - otherwise we will not have enough time to watch them all. Also, we have much more choices of how to watch them - age of digital distribution.
Higher demands give higher quality, movies evolve both in visuals and in acting art.
" Higher demands give higher quality "
I would suggest an higher standard deviation on quality, meaning that you could reach very top notch standards (your point) meanwhile mass producing total flat works as in music and i presume, all of the arts (my view).
I would suggest an higher standard deviation on quality, meaning that you could reach very top notch standards (your point) meanwhile mass producing total flat works as in music and i presume, all of the arts (my view).
Lol, compare modern music with trash we had in 80s, 90s. Even nowadays' pop trash sounds significantly better than old hits made on synthesizers. Everything evolves, as I said. Maybe you just forgot low-quality movies of previous years, but they did exist, tones of them.
We only want to remember good things.
We only want to remember good things.
There were no meaning to mass produce and diffuse music in those decades like there are today. I'm not saying 80s and 90s music was perfect (why pick those decades?), i say the more the demand the more the money the more the highs and the lows.
The "quality" you're referring to, i don't know: you're talking about what features of a sound...fidelity, loudness, dynamic?
Anyway, truly the 80s were the years of trash music...but trash as in metal though.
The "quality" you're referring to, i don't know: you're talking about what features of a sound...fidelity, loudness, dynamic?
Anyway, truly the 80s were the years of trash music...but trash as in metal though.
Loudness, of course, man. What else is the measure of quality? Loudness, without any doubts. Of course I talk about loudness.
Good is relative. And you could easily rabbit hole on this. Do people have the same sense of quality? Have people changed their rating scale based on production/effects quality? Etc etc. Probably best to just take the subjective rating.
It seems pretty reasonable to assume that quality is independent of release year.
[deleted]
I feel like author is missing the point. Shouldn't the recency bias show up in recent votes (while the movie is "hot" and trendy) vs. later votes?
I think it's been interpreted as recent movies not recent votes.
Incidentally, in case the creator sees this, this site is borderline unusable for me on mobile Firefox. Text gets cut off in strange places, I can't zoom out properly to see all of the animations. "Request desktop site" seems to fix some things but break others - at least I can zoom all the way out but the text becomes hard to read.
Situation might be the same on Chrome (didn't check), but regardless it could use some work since it's pretty broken as is.
Situation might be the same on Chrome (didn't check), but regardless it could use some work since it's pretty broken as is.
Hi - founder of Kyso here, thanks for your feedback we will get to work fixing those issues now
Thanks! After playing with it a bit it's clear that part of the problem is that the charts are getting generated on the fly and that's very performance intensive. My brand new Android phone is just choking on the page, to the point I can actually feel the phone overheating. Perhaps static images would be an improvement for mobile devices, with the option to load the full data on request.
I was recently thinking about this as Chernobyl (which was excellent to be fair) quickly became the highest rated TV drama ever.
I mean, if you can suggest other similar shows I will be only glad.
A recent post on the same topic: https://news.ycombinator.com/item?id=20098900
Of course, there are plenty of examples of that.
Like the assassination of the NK leader movie, which at first received 10 star rating. Now look at it.
Sane goes with all the new blockbuster movies, you should usually wait awhile for the real ratings to show.
Sane goes with all the new blockbuster movies, you should usually wait awhile for the real ratings to show.
> Number of Movies By Ratings
I’m dying to see all of these as percentages rather than absolute numbers! I was even thinking that while scrolling down through, and then there the totals are right at the bottom. @KyleOS are you up for normalizing them?
I’m dying to see all of these as percentages rather than absolute numbers! I was even thinking that while scrolling down through, and then there the totals are right at the bottom. @KyleOS are you up for normalizing them?
I see from one of the charts that the average movie rating from 2019 is over .5 higher than the previous 5 years, in what appears to be a big jump. Is it possible that recency bias is only relevant on a shorter timescale? It looks like ratings are higher for movies that are a few months old, based on the data. I wonder if this is generally true, or if early 2019 happens to have higher rated movies.
Sure - for example, fans of a particular actor, director, or series are more likely to go see a movie soon after release in theaters as opposed to later via rental or streaming.
Fans are likely to rate higher than the general public, so over the first, lets say, 3-12 months, ratings often trend down.
Fans are likely to rate higher than the general public, so over the first, lets say, 3-12 months, ratings often trend down.
Fake reviews and ratings must be a factor. For big movie releases it is hard to find IMDB reviews that seem genuine. I assume good portion of the ratings are fake as well.
It must have an effect given how common it is.
It must have an effect given how common it is.
I wonder if recording ratings now and then every month or so would show a decline in ratings. IMDb must know already.
Great job! I'd love to see a similar analysis for Rotten Tomatoes since there's separate metrics for critics and audience, and I find RT ratings to be more extreme than IMDb. I'm often surprised to see how many bad movies get a rating of ~6 on IMDb.
Series tend to be rated higher than movies, probably because people that don't enjoy them stop watching and rating after one or a few episodes.
That effect may well be spilling over into movies, considering far more movies are now sequels.
That effect may well be spilling over into movies, considering far more movies are now sequels.
An alternative way of reading the same data: "Analysis of whether movies are getting better over time"
The problem with your formulation is that 'better' is perhaps the most subjective word there is. Define it in measurable terms, and perhaps you have a reading there. There are many factors to consider which may lead you to find things less clear-cut, for example here's one or two, with some sub-factors
- Modern audience Preference for familiar patterns in: (eg)
-- acting styles
-- music used
-- visual tricks and norms
-- special effects styles
-- acting styles
-- soundtrack loudness
-- colour and high resolution
-- High Concept movies
-- Fantasy and Comic-related films
-- modern movie themes and topics
- Internationalisation of movies, so that fans from abroad prefer modern films which are more universalised and less American-specific (see literature)
Right, I'm not asserting that my alternative title is the correct one, I'm lampooning the assertion that IMDB is "suffering from recency bias" as opposed to a more neutral claim like "ratings for more recent movies are higher."
Answering the question of _why_ they went up is a lot harder to prove.
Answering the question of _why_ they went up is a lot harder to prove.
What I find most amusing is ratings (and also awards/nominations) before a movie has been released.
In terms of shows, Sopranos has long been dethroned, possibly due to this effect. Then again, production values will only get better over time, so maybe the latest shows are of the best quality overall.
Best line:
Fucking Queeahs! (Paulie throws chair at his murder victim spirits)
Best line:
Fucking Queeahs! (Paulie throws chair at his murder victim spirits)
I've always noticed that when a show is released it gets WAY higher ratings in the first week or so of the release date. I've learned not to trust the ratings until at least a 2 weeks after a release.
It looks like this analysis focuses on the year a film was released. This seems like it would be a lot harder to determine because film quality changes (subjective) and the audience writing reviews is probably changing over time.