Some personal news(natesilver.substack.com)
natesilver.substack.com
Some personal news
https://natesilver.substack.com/p/some-personal-news
247 comments
He seems to have done a pretty good job of negotiating his contracts with respect to owning a lot of his model IP moving forward. Still needs to find a platform/pub moving forward of course but far from starting from scratch.
Quoting this here for the added context:
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney. The license term for these models expires with my contract this summer. I still own these models, and can license or sell them elsewhere.
This is one hell of an agreement to have with your employer. That was my next question given that most of FiveThirtyEight was Nate's own work, and he had very good insight to have an escape hatch.
I really hope to have the power to do this one day.
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney. The license term for these models expires with my contract this summer. I still own these models, and can license or sell them elsewhere.
This is one hell of an agreement to have with your employer. That was my next question given that most of FiveThirtyEight was Nate's own work, and he had very good insight to have an escape hatch.
I really hope to have the power to do this one day.
What interest would the Mouse have in numerical models? They want entertainment IP, not weird programs that they have no experience marketing or hiring people to run. They’re not just going to transfer it over to Pixar so the guys behind Cars 2 can predict the next election.
Disney wants the talent. It’s just not as valuable to them as keeping Nate happy
Disney wants the talent. It’s just not as valuable to them as keeping Nate happy
Disney is a multi industry conglomerate interested in understanding/manipulating public policy and political trends through numerical models in the same way you would be interested in your local politics and it's trends.
They're also not just after "entertainment IP". In fact I'd say they are one of the premier proprietary data analytics orgs with respect to park management, fast passing, queuing. Same with Market Research on film/animation, they've quite literally been doing it since it's been a thing.
They're also not just after "entertainment IP". In fact I'd say they are one of the premier proprietary data analytics orgs with respect to park management, fast passing, queuing. Same with Market Research on film/animation, they've quite literally been doing it since it's been a thing.
given that they own abc news and espn among others, they have a lot of interest
I wouldn't presume that most of it is Nate's own work. (I suspect that for most sports he is delegating but "owns" the output, he does seem to directly build most of the political stuff.)
Isn't this what the Skype guys pulled off with Microsoft too?
Isn't this what the Skype guys pulled off with Microsoft too?
Nate got his start in this world doing baseball forecasting (https://en.wikipedia.org/wiki/PECOTA), well before he delved into politics. Jay Boice definitely doing much of the heavy lifting these days, and deserves more personal recognition than he currently gets. But I wouldn't discount the amount of work Nate personally put in on the sports front.
I don't entirely disagree. We know from his public statements that he personally builds the political models. I think he is personally invested in the baseball modelling but we know it is a team effort and 538 is doing a lot. Nate is a media figure, he has to delegate.
There's some reporting that Disney execs didn't fully comprehend the terms he had when they decided to release him and now they're scrambling.
https://www.semafor.com/article/05/01/2023/abc-news-struggle...
https://www.semafor.com/article/05/01/2023/abc-news-struggle...
Wow... if that's true, it's some real amateur hour idiocy. As much as I like the brand, I'd trivially stick with Nate, especially if he somehow got the old band back together: Harry, Claire, etc.
I'm not sure how much I'd read into that. Maybe some Disney execs have seller's remorse (or never wanted to sell in the first place). But, at the end of the day, a lot of the "IP" is pretty much public in considerable detail. (That said, having some explicit ownership is a good way to avoid unnecessary haggling with lawyers.) And if Disney had actually wanted the full analytic and writing ability of FiveThirtyEight surely even an entertainment exec would realize you're giving that up if you let those people go.
Yeah, this is what stood out to me also. It’s likely a lot more than other people in his position could say.
i was struck by that as well. smart long game moves.
> There are also some complicated cases — for example, for some of the sports models, I own them, but Disney gets to keep a copy when I leave.
As others have mentioned, he's very wise to have negotiated contracts so he keeps some IP, but those sports models might be what generate the most revenue.
Are there precedents for this in the tech world, where somebody can legally bring IP with them to a new role but the old employer/competitor keeps an exact copy? Seems like an unusual arrangement that'd be hard to sell the upside to somebody competing with Disney/ESPN.
Also, that [poker bluff](https://www.youtube.com/watch?v=9cVrlVzoh48) was very entertaining to watch. Apparently Nate's quite the poker player (https://en.wikipedia.org/wiki/Nate_Silver#Economic_consultan...)
As others have mentioned, he's very wise to have negotiated contracts so he keeps some IP, but those sports models might be what generate the most revenue.
Are there precedents for this in the tech world, where somebody can legally bring IP with them to a new role but the old employer/competitor keeps an exact copy? Seems like an unusual arrangement that'd be hard to sell the upside to somebody competing with Disney/ESPN.
Also, that [poker bluff](https://www.youtube.com/watch?v=9cVrlVzoh48) was very entertaining to watch. Apparently Nate's quite the poker player (https://en.wikipedia.org/wiki/Nate_Silver#Economic_consultan...)
> Are there precedents for this in the tech world, where somebody can legally bring IP with them to a new role but the old employer/competitor keeps an exact copy?
This is actually the standard arrangement, depending on how you define "tech world". In California for example a Proprietary Information And Inventions Agreement will include a provision along the lines of:
> If, in the course of my employment with the Company, I incorporate a Prior Invention into a Company product, process or machine, the Company is hereby granted and shall have a nonexclusive, royalty-free, irrevocable, perpetual, worldwide license (with rights to sublicense through multiple tiers of sublicensees) to make, have made, modify, use and sell such Prior Invention. Notwithstanding the foregoing, I agree that I will not incorporate, or permit to be incorporated, Prior Inventions in any Company Inventions without the Company’s prior written consent.
Meaning anything you created before remains yours, even if, out of the kindness of your heart, you allow your employer to benefit from it.
What's unusual in Silver's case is maintaining that arrangement for a) the founder of a company, and b) through an acquisition.
This is actually the standard arrangement, depending on how you define "tech world". In California for example a Proprietary Information And Inventions Agreement will include a provision along the lines of:
> If, in the course of my employment with the Company, I incorporate a Prior Invention into a Company product, process or machine, the Company is hereby granted and shall have a nonexclusive, royalty-free, irrevocable, perpetual, worldwide license (with rights to sublicense through multiple tiers of sublicensees) to make, have made, modify, use and sell such Prior Invention. Notwithstanding the foregoing, I agree that I will not incorporate, or permit to be incorporated, Prior Inventions in any Company Inventions without the Company’s prior written consent.
Meaning anything you created before remains yours, even if, out of the kindness of your heart, you allow your employer to benefit from it.
What's unusual in Silver's case is maintaining that arrangement for a) the founder of a company, and b) through an acquisition.
How much value does the model actually have, in and of itself? It’s just a bunch of correlations, right? The value of the model is in the reporting that he does using it (articles, podcasts).
The competition would get value from having him host a show about it on their platform. Disney… I wonder if they even care about the model, or if they just took a license to it/current copy thing because, I dunno, maybe he offered it for really cheap.
I mean can you imagine, 538 but without the main character? It would be incredibly lame.
The competition would get value from having him host a show about it on their platform. Disney… I wonder if they even care about the model, or if they just took a license to it/current copy thing because, I dunno, maybe he offered it for really cheap.
I mean can you imagine, 538 but without the main character? It would be incredibly lame.
>> "I mean can you imagine, 538 but without the main character? It would be incredibly lame."
I assume Disney keeps 538's fursona, Fivey.
I assume Disney keeps 538's fursona, Fivey.
> It’s not just that I think these models have a lot of journalistic value
...do they, though?
I'm actually not so convinced in the journalistic value of FiveThirtyEight. It's something I've thought about before.
FiveThirtyEight's political forecasts absolutely provide a desirable service! Who doesn't want to know who will win an upcoming election?
But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy? I sometimes think they may actually be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
That's not to say Nate Silver is doing anything wrong by making FiveThirtyEight available, but he tends to talk about it as a sort of public good. I'm not sure whether that's accurate.
...do they, though?
I'm actually not so convinced in the journalistic value of FiveThirtyEight. It's something I've thought about before.
FiveThirtyEight's political forecasts absolutely provide a desirable service! Who doesn't want to know who will win an upcoming election?
But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy? I sometimes think they may actually be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
That's not to say Nate Silver is doing anything wrong by making FiveThirtyEight available, but he tends to talk about it as a sort of public good. I'm not sure whether that's accurate.
If anything, I suspect this detailed polling of making elections closer. Millions of voters aren’t reading 538. It’s a distinctly wonky thing to do. But people really into politics, the kind of people who volunteer for campaigns and make large contributions do pay attention and I think it helps rally their support behind realistic candidates for their party so that the candidates put forward for the general elections are as closely matched as possible. Well funded campaigns probably already had their own private polling and statisticians, but 538 helped make sure all highly invested individuals had access to decent data and could deploy their resources more effectively. If you keep that data out of grassroots hands, then you put the parities more firmly under the control of a very small party elite with all the experts.
> Millions of voters aren’t reading 538
True. But millions of them will hear a radio announcer, or TV person, or tiktok video or whatever else say something like "In the latest polling, <X> has pulled ahead of <Y> ..." where "the latest polls" will mean (among other things) 538.
I do believe that this has some effect (if small) on patterns of voting (particular by encouraging or discouraging turnout).
True. But millions of them will hear a radio announcer, or TV person, or tiktok video or whatever else say something like "In the latest polling, <X> has pulled ahead of <Y> ..." where "the latest polls" will mean (among other things) 538.
I do believe that this has some effect (if small) on patterns of voting (particular by encouraging or discouraging turnout).
Generally speaking, you don’t hear this because 538 is not a poll, it’s a political analysis brand and competitor (ABC news). Most news orgs or journalists quote the polls themselves.
And that will happen no matter what. Polls and prognostications will always exist and probably have always existed back to the first tribal chieftain election. It’s just that instead of getting totally inaccurate predictions that only serve a narrative, for a little while a few of us knew how to get a more unbiased prediction. Now even fewer people will know anything aside from their favorite outlet’s media bubble.
> where "the latest polls" will mean (among other things) 538.
No, it won’t; 538 doesn’t produce polls.
No, it won’t; 538 doesn’t produce polls.
I've heard numerous news organizations/corporations cite 538 as the source of their analysis.
538 does produce analysis. They do not produce polls - they consume polls.
None of which matters to people who hear "analysis of the electoral race" on some media.
I don't think dragonwriter was weighing in on your broader point, but being a little pedantic (and a lot correct) about what work is done by what people in our electoral coverage.
We have a word for this: horse race journalism.
It's generally not understood to have positive connotations. In part because there is indeed some evidence to suggest that your concerns about it harming the democratic process are valid. For example: https://www.journals.uchicago.edu/doi/10.1086/708682
It's generally not understood to have positive connotations. In part because there is indeed some evidence to suggest that your concerns about it harming the democratic process are valid. For example: https://www.journals.uchicago.edu/doi/10.1086/708682
In fact, horse race journalism has been observed to have such negative and self-fulling effects that some countries ban polling and/or electoral journalism for hours, days, or weeks before an election. Even in the US, it is illegal to do any "electioneering" within 100ft of a polling place.
On top of that, while 538 initially provided good and solid insights, it has recently been successfully gamed by right-wing players by standing up bogus polling organizations and getting their bogus results into the aggregator's (like 538) results. The goal of course was to make their candidates seem more popular, and the "reliability" scores did not work, as the aggregators results were significantly off in that direction.
Safe to ignore NS and any of his output (and more reliable). [0] https://en.wikipedia.org/wiki/Election_silence
On top of that, while 538 initially provided good and solid insights, it has recently been successfully gamed by right-wing players by standing up bogus polling organizations and getting their bogus results into the aggregator's (like 538) results. The goal of course was to make their candidates seem more popular, and the "reliability" scores did not work, as the aggregators results were significantly off in that direction.
Safe to ignore NS and any of his output (and more reliable). [0] https://en.wikipedia.org/wiki/Election_silence
538 actually did a terrific job accounting for those polling phenomena, finding that in spite of that narrative around right-leaning polls arising, polls that cycle were pretty spot on https://fivethirtyeight.com/features/2022-election-polling-a...
> On top of that, while 538 initially provided good and solid insights, it has recently been successfully gamed by right-wing players by standing up bogus polling organizations and getting their bogus results into the aggregator's (like 538) results. The goal of course was to make their candidates seem more popular, and the "reliability" scores did not work, as the aggregators results were significantly off in that direction.
[Citation needed]
Lots of people raised that concern at the time, predicting the effort would bias aggregators, but for 538 at least, I’ve seen no analysis indicating a significant swing to a right bias in 2020, and overall the 2020 results were quite accurate.
[Citation needed]
Lots of people raised that concern at the time, predicting the effort would bias aggregators, but for 538 at least, I’ve seen no analysis indicating a significant swing to a right bias in 2020, and overall the 2020 results were quite accurate.
>>[Citation needed]
I was watching it in real time. The effect was mostly in 2022 (they hadn't really gotten it going in 2020)
Here's the 538 headline before election day about how the Republicans were in very good shape, even potentially a blowout win [0]
In fact, it was the exact opposite.
Yes, there was a lot of statistical handwaving about how they were really right, but, not really. BTW, this was just the top result from a DDG search, and is only one of the things I remember.
[0] https://fivethirtyeight.com/features/2022-polling-error/
I was watching it in real time. The effect was mostly in 2022 (they hadn't really gotten it going in 2020)
Here's the 538 headline before election day about how the Republicans were in very good shape, even potentially a blowout win [0]
In fact, it was the exact opposite.
Yes, there was a lot of statistical handwaving about how they were really right, but, not really. BTW, this was just the top result from a DDG search, and is only one of the things I remember.
[0] https://fivethirtyeight.com/features/2022-polling-error/
FiveThirtyEight is beneficial, and it’s fairly simple as to why.
1. People want to know who is winning. It doesn’t matter whether you like it or not. People want to know, and thus there will always be a market for it.
2. Polling is imperfect and it’s a lot of work to understand how to interpret the various polls out or to forecast in lieu of polls. Political journalism before FiveThirtyEight was filled with even more crooks and charlatans coming up with nonsensical ways to forecast elections or interpret polls. There was a lot more noise in the system. FiveThirtyEight shut that stuff down to an incredible degree.
3. FiveThirtyEight has introduced more folks to probability theory than I think any other journalistic collective. What has always made them different is that they provide probabilities, not certainties.
1. People want to know who is winning. It doesn’t matter whether you like it or not. People want to know, and thus there will always be a market for it.
2. Polling is imperfect and it’s a lot of work to understand how to interpret the various polls out or to forecast in lieu of polls. Political journalism before FiveThirtyEight was filled with even more crooks and charlatans coming up with nonsensical ways to forecast elections or interpret polls. There was a lot more noise in the system. FiveThirtyEight shut that stuff down to an incredible degree.
3. FiveThirtyEight has introduced more folks to probability theory than I think any other journalistic collective. What has always made them different is that they provide probabilities, not certainties.
Your #1 is affirming the consequent.
My #1 is really attempt to say that OP’s question is a moot point.
As in, there is no way that polling or analysis of polls will ever be stopped, so why even debate its societal value?
It’s like saying “the world would be a better place if everyone started/stopped doing this one thing”. Okay?
As in, there is no way that polling or analysis of polls will ever be stopped, so why even debate its societal value?
It’s like saying “the world would be a better place if everyone started/stopped doing this one thing”. Okay?
Those are good, meaty questions.
In an election, the vote is the measure of popular support for a candidate/policy/platform. Democracy doesn't necessarily need polls.
I've found interesting some related essays by Jill Lepore, an American historian, on the advent of polling earlier in the 20th century:
- https://www.newyorker.com/magazine/2015/11/16/politics-and-t... - https://www.newyorker.com/magazine/2020/03/09/the-problems-i...
In an election, the vote is the measure of popular support for a candidate/policy/platform. Democracy doesn't necessarily need polls.
I've found interesting some related essays by Jill Lepore, an American historian, on the advent of polling earlier in the 20th century:
- https://www.newyorker.com/magazine/2015/11/16/politics-and-t... - https://www.newyorker.com/magazine/2020/03/09/the-problems-i...
Societal value != journalistic value. And in fact, even if there's a net negative societal value that doesn't mean it shouldn't be around.
Nate silver didn't exactly invent political polling either. That's doing a disservice to the enormous time and money invested in polling over several decades of elections (and the learnings when things didn't go correctly). Polling will be around regardless.
Nate silver didn't exactly invent political polling either. That's doing a disservice to the enormous time and money invested in polling over several decades of elections (and the learnings when things didn't go correctly). Polling will be around regardless.
> Societal value != journalistic value
That's fair. I was responding to ways I've heard Nate Silver talk about the model and polling in general in e.g. podcasts, which is why I've had the question floating around in my mind. But you're right, that's not quite what he wrote here.
> And in fact, even if there's a net negative societal value that doesn't mean it shouldn't be around.
To be clear: I don't mean to imply that FiveThirtyEight shouldn't exist. There's lots of things which serve a human desire but aren't necessarily a social good. Slot machines for example.
That's fair. I was responding to ways I've heard Nate Silver talk about the model and polling in general in e.g. podcasts, which is why I've had the question floating around in my mind. But you're right, that's not quite what he wrote here.
> And in fact, even if there's a net negative societal value that doesn't mean it shouldn't be around.
To be clear: I don't mean to imply that FiveThirtyEight shouldn't exist. There's lots of things which serve a human desire but aren't necessarily a social good. Slot machines for example.
The big value of polls is a backup check for discrepancies. A major difference between final polls and polls leading up to the election could be innocent (like unexpected demographics), but it could hint at voter suppression or even completely innocent issues with the voting systems. Paired with exit polling, there's enough information to find answers.
As opposed to horserace election coverage, which is the single biggest focus of all of American journalism for the year or so leading up to an election... which is of greater journalistic value?
Simply looking at 538 during election cycles makes it hard to argue they aren’t a horse race tracker (they literally started as a sanitized gambling odds site and Nate’s new book is about gambling).
They’re just the guy who went to the stables before to see who was eating the most hay.
They’re just the guy who went to the stables before to see who was eating the most hay.
538 absolutely is horserace coverage, but unlike basically all the other election coverage, it's rigorous, impartial, and intellectually honest.
[deleted]
Eh, eye of the beholder.
I honestly mostly follow the sports odds and I’d argue that Nate’s models are some of the most biased out there for that.
I honestly mostly follow the sports odds and I’d argue that Nate’s models are some of the most biased out there for that.
Which non-538 models consistently outperform 538's models?
EDIT: (cause I can't reply to the below comment) my question was what model beats 538, not which betting market beats 538. yeah, duh, the Vegas line beats 538, because it beats literally all publicly available models. It'd take a really disastrous gambling market failure for anything else to happen.
EDIT: (cause I can't reply to the below comment) my question was what model beats 538, not which betting market beats 538. yeah, duh, the Vegas line beats 538, because it beats literally all publicly available models. It'd take a really disastrous gambling market failure for anything else to happen.
Literally the Vegas line?
Nate put 2021/22 warriors at 0.5% to win, Vegas was +900.
Nate put 2020/21 bucks at 10% to win, Vegas was +550.
If you can’t even come close to beating the line, you’re not a good model. He’s just transparent about his variables.
> EDIT: (cause I can't reply to the below comment) my question was what model beats 538, not which betting market beats 538. yeah, duh, the Vegas line beats 538, because it beats literally all publicly available models. It'd take a really disastrous gambling
Lines aren’t built that way. They’re set by books to offset their risk. They’re based on a bunch of (usually drunk) wetware computers with huge biases (don’t believe me, look at any Dodgers line when they have an obviously bad pitcher matchup).
If your model is worse than that, then what’s the point of a model?
To the point of bookie models, those are based on how they think people will bet, not on who they think they will win. The point of books is to be outcome neutral.
Nate put 2021/22 warriors at 0.5% to win, Vegas was +900.
Nate put 2020/21 bucks at 10% to win, Vegas was +550.
If you can’t even come close to beating the line, you’re not a good model. He’s just transparent about his variables.
> EDIT: (cause I can't reply to the below comment) my question was what model beats 538, not which betting market beats 538. yeah, duh, the Vegas line beats 538, because it beats literally all publicly available models. It'd take a really disastrous gambling
Lines aren’t built that way. They’re set by books to offset their risk. They’re based on a bunch of (usually drunk) wetware computers with huge biases (don’t believe me, look at any Dodgers line when they have an obviously bad pitcher matchup).
If your model is worse than that, then what’s the point of a model?
To the point of bookie models, those are based on how they think people will bet, not on who they think they will win. The point of books is to be outcome neutral.
> Lines aren’t built that way. They’re set by books to offset their risk.
They're initially set by bookies based on their models, but then pretty quickly evolve to a market-based mechanism.
They're initially set by bookies based on their models, but then pretty quickly evolve to a market-based mechanism.
> Eh, eye of the beholder.
It's really not, which is why quantitative models have real value. You don't have to argue that he's biased, you just have to point to the evidence. But your whole comment is indistinguishable from someone with sour grapes who is simply casting aspersions on his track record without being willing to analyze it properly. At the very least, you ought to specifically identify what kind of biases his predictions have shown; he leaves himself wide open to being proven wrong, so it's only fair for you to do the same.
It's really not, which is why quantitative models have real value. You don't have to argue that he's biased, you just have to point to the evidence. But your whole comment is indistinguishable from someone with sour grapes who is simply casting aspersions on his track record without being willing to analyze it properly. At the very least, you ought to specifically identify what kind of biases his predictions have shown; he leaves himself wide open to being proven wrong, so it's only fair for you to do the same.
He regularly underperforms the Vegas line.
He makes bad models.
Nate put 2021/22 warriors at 0.5% to win, Vegas was +900. Nate put 2020/21 bucks at 10% to win, Vegas was +550. If you can’t even come close to beating the line, you’re not a good model. He’s just transparent about his variables.
Edit: To be clear I don’t gamble, I just follow sports and like data analysis, so this isn’t a bank account thing.
He makes bad models.
Nate put 2021/22 warriors at 0.5% to win, Vegas was +900. Nate put 2020/21 bucks at 10% to win, Vegas was +550. If you can’t even come close to beating the line, you’re not a good model. He’s just transparent about his variables.
Edit: To be clear I don’t gamble, I just follow sports and like data analysis, so this isn’t a bank account thing.
Horserace election coverage is the larger problem. But FiveThirtyEight is clearly a participant in that.
The models are one thing. It's the data that really matter imo. No other organization has created as thorough of a collection of polling firms, their polls, as well as their faults and past records as FTE.
Even if you're building your own model you'll likely be turning to FTE for the latest data.
In addition they're data driven journalism is often paired with amazing data sets that they open-source for free.
Even if you're building your own model you'll likely be turning to FTE for the latest data.
In addition they're data driven journalism is often paired with amazing data sets that they open-source for free.
It’s definitely an interesting discussion.
> who doesn't want to know who will an upcoming election?
In particular, politicians want to know. This to me is the interesting upshot. Polls are non binding, so the public could use them to signal to candidates how to tweak their platform.
I’m always curious if there’s any real good data saying turnout patterns change. The thing is, voting because you think your one vote might change the result, is already completely irrational. I always think of voting of being like when you go to a concert and buy the band’s CD even though you already have Spotify: it’s more about find a way to personally participate in an important life moment than practicality.
> who doesn't want to know who will an upcoming election?
In particular, politicians want to know. This to me is the interesting upshot. Polls are non binding, so the public could use them to signal to candidates how to tweak their platform.
I’m always curious if there’s any real good data saying turnout patterns change. The thing is, voting because you think your one vote might change the result, is already completely irrational. I always think of voting of being like when you go to a concert and buy the band’s CD even though you already have Spotify: it’s more about find a way to personally participate in an important life moment than practicality.
I think the major parties and other interested organizations will always have forecasts like these available internally, and will leak them (or lies about them) when it's to their advantage. Having a high-quality forecast directly available to the public can't hurt more than that.
> I sometimes think they may be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
This is an old tactic that is used to the hilt by both sides whenever it serves them, and media outlets already make forecasts, often without 538's scrupulous emphasis on how often they're wrong. I don't think 538 makes this worse.
> I sometimes think they may be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
This is an old tactic that is used to the hilt by both sides whenever it serves them, and media outlets already make forecasts, often without 538's scrupulous emphasis on how often they're wrong. I don't think 538 makes this worse.
> FiveThirtyEight's political forecasts absolutely provide a desirable service—who doesn't want to know who will an upcoming election?
> But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy?
Two points:
1. The model DO NOT tell you who will win the upcoming election.
2. It is absolutely beneficial to society and democracy, having a poll-free & rigorous-analysis-free democratic society means an environment where even the most apolitical citizen can be bamboozled into thinking that an election was "stolen" simply based on their close associates and news sources. Polling and aggregation, when executed with a modicum of accountability, prevents a devolution into a Soviet-style culture of paranoia particularly among the elites. It has tremendous value, that value will be backfilled by rumour and innuendo and that is very bad for society.
> But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy?
Two points:
1. The model DO NOT tell you who will win the upcoming election.
2. It is absolutely beneficial to society and democracy, having a poll-free & rigorous-analysis-free democratic society means an environment where even the most apolitical citizen can be bamboozled into thinking that an election was "stolen" simply based on their close associates and news sources. Polling and aggregation, when executed with a modicum of accountability, prevents a devolution into a Soviet-style culture of paranoia particularly among the elites. It has tremendous value, that value will be backfilled by rumour and innuendo and that is very bad for society.
You seem to be arguing about net social value, which is a distinct and much more complicated issue than journalistic value.
(That said, yes, I think the models clearly have net social value, too.)
(That said, yes, I think the models clearly have net social value, too.)
[deleted]
> But is it good for people to know who is more likely to win an election? Are these predictions beneficial to society or democracy? I sometimes think they may actually be harmful, decreasing voter turnout by convincing people (correctly or incorrectly) that the result is preordained.
For plurality voting (a.k.a. first-past-the-post[0]) with more than two candidates, voters have to cast their ballots strategically[1]. This is especially common in primary elections and non-partisan positions.
Unless all elections switch to a method that is not subject to the spoiler effect (like approval voting or instant runoff voting), voters need more information about which candidates are most likely to win, not less information. Sometimes candidate competitivness can be inferred from things like endorsements or fundraising numbers, but polls -- and aggregated models based on polls -- are an important tool.
[0] https://en.wikipedia.org/wiki/First-past-the-post_voting
[1] https://en.wikipedia.org/wiki/Strategic_voting
For plurality voting (a.k.a. first-past-the-post[0]) with more than two candidates, voters have to cast their ballots strategically[1]. This is especially common in primary elections and non-partisan positions.
Unless all elections switch to a method that is not subject to the spoiler effect (like approval voting or instant runoff voting), voters need more information about which candidates are most likely to win, not less information. Sometimes candidate competitivness can be inferred from things like endorsements or fundraising numbers, but polls -- and aggregated models based on polls -- are an important tool.
[0] https://en.wikipedia.org/wiki/First-past-the-post_voting
[1] https://en.wikipedia.org/wiki/Strategic_voting
It has been fairly useful to me as an observer and voter in understanding how the elections go down. Understanding what the chances are of each candidate and how certain voting groups can influence the election can (sometimes) reduce being blindsided by an election. It may reduce my impartiality as a voter, but it definitely helps me process politics as a citizen.
This is something I think about a lot - the actual value of polling in general and public opinion polling in specific.
A lot of public discourse is creeping on several layers of meta, to where we are constantly reacting to how people react to other reactions.
Sometimes I wish we would just let ideas stand on their face instead of constantly engaging in acts of national navel gazing.
A lot of public discourse is creeping on several layers of meta, to where we are constantly reacting to how people react to other reactions.
Sometimes I wish we would just let ideas stand on their face instead of constantly engaging in acts of national navel gazing.
Even if he result appears dead-even there is a cost. Framing the election in terms of who wins instead of which candidate is best increases the tendency of voters to vote for candidates who "have a chance," even though there is no rational reason to vote for candidate that is more likely to win.
There is a very real reason to vote for the candidate during the primary who is most likely to win the general election.
If your vote is the deciding vote (i.e. the candidate won by exactly 1 vote), please buy me a lottery ticket.
All votes are the deciding vote, in that case!
I would think forcing people to better appreciate how probabilities actually work is better than avoiding the topic entirely. Learning that 70% is not certain is important in lots of contexts. 30% probability events happen all the time…
Data illiteracy, not the existence of a forecast, is the core problem.
Data illiteracy, not the existence of a forecast, is the core problem.
The science is valid and they are a good aggregator of the various polls, it's perfectly legit and moderately beneficial.
Especially in a world of liars.
Especially in a world of liars.
The model is only one feature of the site, you realize right? Have you read anything else they put out besides the election forecast?
> FiveThirtyEight's political forecasts absolutely provide a desirable service! Who doesn't want to know who will win an upcoming election?
Does it actually work though? I don’t think they were able to predict the past 3 presidential elections, so not sure where they provide value.
Does it actually work though? I don’t think they were able to predict the past 3 presidential elections, so not sure where they provide value.
FiveThirtyEight has had astonishing accuracy since 2008. They don’t pick winners, they provide data analytics of polls and other factors. People may not appreciate probability theory but this is far more reality based than what passed for political analysis prior to 2008.
> it may even be harmful
It definitely is, I tried to make the same point a week ago :-)
https://news.ycombinator.com/item?id=35707786
It definitely is, I tried to make the same point a week ago :-)
https://news.ycombinator.com/item?id=35707786
The value is the hype not the journalism.
After watching this and the DPReview saga, there is only one takeaway:
Don't sell your company unless you are prepared to walk away and do something else - because at some point you're going to be forced to do so whether you want to or not.
Don't sell your company unless you are prepared to walk away and do something else - because at some point you're going to be forced to do so whether you want to or not.
Or do exactly what Nate Silver did – negotiate the contract in such a way that you still have control over your IP if they decide to terminate you or sunset your product. Of course chances are that you don't have the kind of leverage that he did, but still...worth a shot.
I agree completely. I wasn't thinking of addressing my comment to stars like that but more towards mere mortals like me.
Given that we're on HN, I probably should have thought that thru a bit more.
Either way, very glad that Nate was smart enough to license everything. Kudos to him for being a shrewd negotiator.
Given that we're on HN, I probably should have thought that thru a bit more.
Either way, very glad that Nate was smart enough to license everything. Kudos to him for being a shrewd negotiator.
Discussion here from last week when this was first "announced" (81 comments): https://news.ycombinator.com/item?id=35707372
Now that Disney owns the FiveThirtyEight trade name I want to see how it's integrated into the Star Wars franchise.
"538" can be hidden inside of "BB8", given the shape of the chars.
This is now canon for me.
> "In July 2013, ESPN (owned by Disney) acquired FiveThirtyEight, hiring Silver as editor-in-chief and a contributor for ESPN.com; the new publication launched on March 17, 2014."
> "BB-8 was first seen in the 88-second The Force Awakens teaser trailer released by Lucasfilm (owned by Disney) on November 28, 2014"
> "In July 2013, ESPN (owned by Disney) acquired FiveThirtyEight, hiring Silver as editor-in-chief and a contributor for ESPN.com; the new publication launched on March 17, 2014."
> "BB-8 was first seen in the 88-second The Force Awakens teaser trailer released by Lucasfilm (owned by Disney) on November 28, 2014"
this is the way.
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney. The license term for these models expires with my contract this summer. I still own these models, and can license or sell them elsewhere.
Sounds like he had some good leverage going into this deal. Why would Disney have agreed to a temporary license, as opposed to a purchase of this critical IP? Regardless, good for him.
Sounds like he had some good leverage going into this deal. Why would Disney have agreed to a temporary license, as opposed to a purchase of this critical IP? Regardless, good for him.
> Why would Disney have agreed to a temporary license, as opposed to a purchase of this critical IP?
It was probably a risk hedge from them.. he would have wanted a lot more for the full enchilada and they didn't know how it would perform in the future. And now his work is not hard to reproduce so it wasn't a bad call. Every political data team I have heard reporting on essentially concedes that whatever they do ends up more-or-less matching 538.. so it is hard to extract surplus value but also easy to hit that mark roughly.
I follow the NHL and there are a lot of models, they disagree in certain details but they largely agree and the "winner" at the end of any given season isn't winning by much.. even private data isn't a big boost.
It was probably a risk hedge from them.. he would have wanted a lot more for the full enchilada and they didn't know how it would perform in the future. And now his work is not hard to reproduce so it wasn't a bad call. Every political data team I have heard reporting on essentially concedes that whatever they do ends up more-or-less matching 538.. so it is hard to extract surplus value but also easy to hit that mark roughly.
I follow the NHL and there are a lot of models, they disagree in certain details but they largely agree and the "winner" at the end of any given season isn't winning by much.. even private data isn't a big boost.
And who knows how much of this Disney really cares about going forward outside of the Nate Silver Brand--and had they had wanted that they'd have kept Nate Silver.
Calculating the difference between the 538 brand and the Nate Silver brand at the time of acquisition was probably a bit of a toss-up too. Nate can move on to a substack that generates $2MM a year with 4 staff and be happy while Disney runs 538 at a reasonable margin and be happy and I guess everyone is a winner?
In the meteoric rise of 538, Nate could have easily walked into NYT or any other tech-forward media outlet and negotiated to retain the models. Disney would have lost out then.
Disney and WB Discovery seem to have totally lost any vision for their future.
538 has a very clear value proposition outside of exposing nerdy polling and forecasting breakdowns for a small group of wonks. It's the place a lot of other folks source for rigorous tracking+forecasting of elections, sports, etc. It's good.
Disney, however, seems to be putting out a lot of crappy content on their Disney+ platform that rates poorly and is expensive to produce. The Mandalorian + Andor are great but the other recent Star Wars IP is horrible and cost them a fortune.
538 has a very clear value proposition outside of exposing nerdy polling and forecasting breakdowns for a small group of wonks. It's the place a lot of other folks source for rigorous tracking+forecasting of elections, sports, etc. It's good.
Disney, however, seems to be putting out a lot of crappy content on their Disney+ platform that rates poorly and is expensive to produce. The Mandalorian + Andor are great but the other recent Star Wars IP is horrible and cost them a fortune.
This is why they just recently changed CEOs. You're going to see the ship changing direction over the next 2 years back to a more familiar course.
As a DIS shareholder, I'm livid. Losing the rights to use the best election forecasting models in the business a year before a Presidential election is monumental fuckup.
This is one screwup that can't be placed at the feet of their previous incompetent CEO.
This is one screwup that can't be placed at the feet of their previous incompetent CEO.
Do you really think it brought in that much money? I'm sure they had plenty of data when they made this decision.
In the grand scheme of Disney stuff, it probably didn’t make much money relatively speaking, but it seems weird to get rid of it right before the season where it’ll make any money.
Or after its 2016 performance they didn’t expect it to make that much so a year of salary was more than earnings.
538 was not wrong in 2016. They were actually one of the outlets that gave trump the best odds.
It’s not that they were wrong, they weren’t useful for what people want. What people expect and what is possible is out of sync and I think people misunderstood FiveThirtyEight for many years and thought it was an oracle. It’s not.
Was it really any better than NYTimes or CNN? I think it’s pretty unremarkable these days and I don’t think Disney is losing revenue or prestige.
> Losing the rights to use the best election forecasting models in the business a year before a Presidential election is monumental fuckup.
Here you go: Biden over Trump, 66% to 33%
I just saved Disney millions.
Here you go: Biden over Trump, 66% to 33%
I just saved Disney millions.
It feels like a waste to not just figure out how to have the founder buy back the domain name/trademarks.
A melancholic development. But in a way, kind of relieving.
I think Nate Silver and his team did an amazing job making statistical analysis interesting and transparent and trustworthy.
On the other hand, 538 was a weirdly sterile website to actually read. On the rest of the internet you have raging and bombastic news designed to elicit feelings and (most importantly) clicks. 538 tried their best to fit in this world, but you were still just reading data analysis about other news or topics.
I think Nate Silver and his team did an amazing job making statistical analysis interesting and transparent and trustworthy.
On the other hand, 538 was a weirdly sterile website to actually read. On the rest of the internet you have raging and bombastic news designed to elicit feelings and (most importantly) clicks. 538 tried their best to fit in this world, but you were still just reading data analysis about other news or topics.
I take it Disney owns Fivey Fox? That's kind of a bummer.
The last tweet from Fivey Fox's account, just after the second round of layoffs was announced, is ominous https://twitter.com/FiveyFox/status/1651634606272126995
> Much of FiveThirtyEight’s vital intellectual property — such as the election forecast models — is merely licensed to Disney.
This seems odd that Disney would buy FiveThirtyEight but not its IP. How common is this?
If I were buying a data journalism company known for its models, that seems like something important to purchase. This seems like a curious contract negotiation and seems like a mistake on Disney’s part. Or a future lawsuit if there’s a difference of opinions.
This seems odd that Disney would buy FiveThirtyEight but not its IP. How common is this?
If I were buying a data journalism company known for its models, that seems like something important to purchase. This seems like a curious contract negotiation and seems like a mistake on Disney’s part. Or a future lawsuit if there’s a difference of opinions.
Disney is a media company that was dabbling in data science because of its name recognition. It almost certainly didn't have the strategic vision to care much about something as nerdy as the intellectual property being used to generate that name recognition.
FiveThirtyEight made a big splash in its 2008 election debut. According to Wikipedia, there were numerous media outlets competing for its content in 2010. That likely gave Silver a lot of leverage. The same is probably true in 2014, when FiveThirtyEight moved from the New York Times to ESPN/Disney.
Interestingly, the deal with the NYT was that the entire blog was licensed. For the deal with Disney, it seems like Silver gave up a lot more, and only kept control of the models.
Interestingly, the deal with the NYT was that the entire blog was licensed. For the deal with Disney, it seems like Silver gave up a lot more, and only kept control of the models.
I mean, 538 was under the NYTimes.com domain before that, it's not like Nate had to sell, and when he agreed to move to ESPN/Disney, he surely had more of the bargaining power.
Before Disney/ABC get any -ideas-, might be a good chance to get our hands on at least their data[0]!
[0]: https://data.fivethirtyeight.com/
[0]: https://data.fivethirtyeight.com/
Was 538 journalism? I never followed them. Only on occasions of them being quoted with some type of forecast.
I don’t think anyone can answer this to your satisfaction without knowing your personal definition of journalism. Because I believe the question you’re really asking is, “Was 538 what I consider journalism?”
When I typed “define journalism” into Google, this came back:
> the activity or profession of writing for newspapers, magazines, or news websites or preparing news to be broadcast.
So yes, 538 was journalism by at least one definition of journalism.
Here’s one more definition:
> Journalism is the activity of gathering, assessing, creating, and presenting news and information. It is also the product of these activities
Another yes! We’re 2 for 2!
When I typed “define journalism” into Google, this came back:
> the activity or profession of writing for newspapers, magazines, or news websites or preparing news to be broadcast.
So yes, 538 was journalism by at least one definition of journalism.
Here’s one more definition:
> Journalism is the activity of gathering, assessing, creating, and presenting news and information. It is also the product of these activities
Another yes! We’re 2 for 2!
> Was 538 journalism?
Yes, it was, and still "is" until they take the website down: https://fivethirtyeight.com/
Yes, it was, and still "is" until they take the website down: https://fivethirtyeight.com/
All fact-based reporting on politics and current events is journalism. I'm sure some of your HN comments have been journalism.
538 is a bunch of opinionated statistics and bad editorial. Squarely in the center of journalism.
538 is a bunch of opinionated statistics and bad editorial. Squarely in the center of journalism.
i'm super curious about the book!
Nate, I'll work with you...
> NATE DOT SILVER DOT MEDIA AT GMAIL DOT COM
Artifacts of another age. This stuff won't work anymore.
Artifacts of another age. This stuff won't work anymore.
> > NATE DOT SILVER DOT MEDIA AT GMAIL DOT COM
> Artifacts of another age. This stuff won't work anymor
Its not a protection scheme, its repeating the content of the mailto link with emphasis.
(If it was a protection scheme, there wouldn’t be a mailto link.)
> Artifacts of another age. This stuff won't work anymor
Its not a protection scheme, its repeating the content of the mailto link with emphasis.
(If it was a protection scheme, there wouldn’t be a mailto link.)
I wondered if it's for accessibility reasons. Possibly a better experience for screen readers and such.
It comes right after a mailto link so I don't think it was intended to stop spam.
At least, I hope not.
At least, I hope not.
Presumably it's not super serious as there's a mailto link right before it
As someone who has written a lot of regexen, I’m surprised it ever worked.
I used to scrape craigslist for local offers. Many times I needed to be the first to call, so I wanted my script to get the numbers.
In about five minutes I came up with an algorithm that defeated most of the obfuscation tricks of the time:
Strip whitespace, convert "five" to 5, remove special characters, and look for ten consecutive digits in the body text. Maybe a couple of small tweaks after that like removing text between digits.
Most people, when they invent their little unique scheme, invent one that is already defeated by the algorithm above. At least for phone numbers.
Anyway, all that to say you're right, it never worked.
In about five minutes I came up with an algorithm that defeated most of the obfuscation tricks of the time:
Strip whitespace, convert "five" to 5, remove special characters, and look for ten consecutive digits in the body text. Maybe a couple of small tweaks after that like removing text between digits.
Most people, when they invent their little unique scheme, invent one that is already defeated by the algorithm above. At least for phone numbers.
Anyway, all that to say you're right, it never worked.
Spammers aren't that smart, they will absolutely go where the bar is at its absolutely lowest, if not missing entirely. Just changing your SSH server to operate a port that is not 22 is going to stop most if not all random login attempts.
I wouldn’t characterize that example as “not smart” to be honest. I’m not particularly smart, but if someone has gone through the trouble of changing their secure socket port from 22, I would realize there would be more work and time involved in finding the actual port, and then it would be a safe assumption that, if found, it would have significantly more security hardening than the average port 22 because clearly the admin knows enough to change ports. So I might even characterize it as smart to not go after ssh ports that are not 22.
The reason is different. When you need to scan 5 million IPs on 22 port is one thing, but 5 million IPs times 100 or 1000, then you simply run out of resources and since 99% of people use 22, why bother?
> As someone who has written a lot of regexen
There are at least \d of us!
There are at least \d of us!
More than that, there are \d+ of us! Or, since I’ve been writing Lua recently, %d+ of us.
Heh I almost edited it to \d+ but also I considered [0-9]{2,} and then I thought more on it and realized (?<!^)\d(?!$) is pretty open ended.
You need to upgrade your knowledge. Use only [2-9]. That's it.
There are at least 1 of us!
Looks weird :)
Looks weird :)
particularly odd when natesilver.net and natesilver.com are both domains that exist and presumably owned by him (for well known public figure trademark reasons), he can't just point the MX for them at gsuite and then set up an inbox?
That opens a surprisingly large landscape of possible attacks and failure modes. The jsit obvious subset being things that happen when your paid gsuite gets disabled (not paying, abuse report, etc), or the domain lapses or is taken over.
A basic free gmail account is very stable compared to almost any other option. L
A basic free gmail account is very stable compared to almost any other option. L
somebody of the public profile of Nate Silver presumably can hire IT/network people who know how to securely run authoritative DNS (or hire it out to a highly trusted third party as a service), so they can point the MX anywhere they want, and either do email in-house or hire it out to an email service specialist or big cloud service provider.
The risk of having the gsuite paid services shut off are about the same as any other high dollar value paying customer using gsuite or office365 as an email service.
The risk of having the gsuite paid services shut off are about the same as any other high dollar value paying customer using gsuite or office365 as an email service.
The question was why not, and the answer was because it's more risky. Why would Nate Silver want to do all the things you correctly listed as options when he can simply use the safer, more simple, free gmail option?
I really lost any semblance of respect for Silver after the 2016 election
Analysis after analysis about how Hillary was a sure thing. It was an absolute echo chamber and the election completely lifted the curtain for me
Analysis after analysis about how Hillary was a sure thing. It was an absolute echo chamber and the election completely lifted the curtain for me
He was criticized for giving trump too large a chance to win iirc
Here he spikes the football over it: https://fivethirtyeight.com/features/why-fivethirtyeight-gav...
Here he spikes the football over it: https://fivethirtyeight.com/features/why-fivethirtyeight-gav...
I think Nate Silver must be some kind of punching bag in right wing media circles, because some variation of this comment is posted basically every time 538 comes up on HackerNews.
This is more of an average person/mainstream media reading complaint. But they weren't paying attention in 2016 or else they would notice /he was the only person saying Hillary might not win/.
You're so 2016. Nate Silver is now a punching bag in left-liberal media circles.
He is the face of election prediction in America. All the other guys at NYT, WaPo, etc aren't as famous. Of course he's going to be the face you punch for them not predicting the election remotely right. (And don't give me this shit about 75% or whatever odds he gave being the rightest or whatever weasel nonsense, it was still plenty wrong and he still want to lmao "spike the football" really? Get out of here.).
How is saying "there's a one in three chance that such-and-such an event occurs" classified as "plenty wrong" in your view, if the event occurs?
If there is a 2/3 chance of a thing, and you say there's a 2/3 chance of the thing, you're doing your job right even though you "got it wrong" in hindsight 1/3 of the time. If you want to assess how someone is doing at prediction, you need to look at a lot of predictions and if they're giving probability estimates you want to apply a proper scoring rule. If you're going to make grand pronouncements based on a single close race being labeled the wrong side of close, get out of here.
> We also received a lot of criticism from Democratic partisans in the closing weeks of the campaign — more than we did from Trump supporters — because they thought we didn’t have Clinton as a heavy enough favorite. That’s unusual. We’ve forecasted enough races over the years to have taken criticism from almost every side. But in the past, it’s always been the trailing candidate’s supporters who gave us more grief.
Turns out when statistics say there is a 5% chance of something happening, there's about a 5% chance of it happening.
Also, for 2020, it turns out the actual vote split between the two US presidential candidates was pretty much exactly in line with polls.
Also, for 2020, it turns out the actual vote split between the two US presidential candidates was pretty much exactly in line with polls.
Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times?
The accuracy of the polls is another matter entirely. You could estimate the chances that your polling was wrong, given that the selection of the sample is a random variable, people lie, and intentions change, but that's not at all what the general public thinks the number means.
The accuracy of the polls is another matter entirely. You could estimate the chances that your polling was wrong, given that the selection of the sample is a random variable, people lie, and intentions change, but that's not at all what the general public thinks the number means.
> Your two paragraphs have no relationship. How it could possibly "turn out" that there's a 5% chance of Trump winning? Was the election run 100 times?
You can calibrate it by looking at their other election predictions and seeing how generally on target it is.
https://projects.fivethirtyeight.com/checking-our-work/
You can calibrate it by looking at their other election predictions and seeing how generally on target it is.
https://projects.fivethirtyeight.com/checking-our-work/
no, you can only "calibrate" his prediction abilities in general:
"given 100 predictions by Nate Silver, how many will be correct?"
that's what you just said.
In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
"given 100 predictions by Nate Silver, how many will be correct?"
that's what you just said.
In other words, if we forgot what actually happened, picked one past prediction out of a hat, asked "so was he right?" and then checked the results, he would be correct 95% of the time. That's what you're saying.
Correctness is actually a pretty bad evaluation because it reduces the outcome to a binary. There are better measures like a brier score that IIRC, they have pretty rigorously evaluated on their own predictions.
No, its saying...
Well, here:
https://projects.fivethirtyeight.com/checking-our-work/
Well, here:
https://projects.fivethirtyeight.com/checking-our-work/
OK, I read that.
It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why?
There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
It's telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article. Why?
There are many thousands of MLB games. There are a few thousand US House elections. There are only a handful of Presidential elections. It is not possible to measure their accuracy on those with any statistical rigor.
> OK, I read that.
Not very well.
> It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article.
All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
Not very well.
> It’s telling that he uses MLB games, rather than Presidential elections, which are not covered at all in that article.
All the types of predictions 538 does are covered in the Brier skills chart on the overall summary; there’s detailed analysis covering each type of prediction (with further available breakdowns for subtypes available once you’ve gone into the type) available from the dropdown at the top of the page.
OK, I missed the dropdown at the top. There is not "detailed analysis" for Presidential elections, although I guess you mean that each state is a data point.
All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most.
I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning:
All that graph shows is that their "probability" is roughly in the ballpark with huge errors in the middle of the range, which is probably the swing states we care about most.
I'll suggest one concrete case where their probability of a candidate winning a 2-person race has some exact meaning:
*You'll pick a number randomly from 1 to 100. If it's less than or equal to candidate A's "probability" of winning, then you bet $10,000 on A. Otherwise you bet $10,000 on B* (we need to put real skin in the game)
> There is not “detailed analysis” for Presidential elections
There is, in fact, detailed analysis for Presidential elections.
> although I guess you mean that each state is a data point.
No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point.
> Are you willing to abide by that rule? If not, you don’t really believe the probability.
Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
There is, in fact, detailed analysis for Presidential elections.
> although I guess you mean that each state is a data point.
No, each individual prediction of each of the 56 (enumerated upthread) distinct elections for one or more Presidential electors is a data point.
> Are you willing to abide by that rule? If not, you don’t really believe the probability.
Well, no, not being willing to abide by that rule means either not being rich enough that winning and losing $10,000 are roughly symmetric in utility or not being an SBF-style nutball and demanding a non-zero risk premium. As it turns out, I am in both categories.
what does this mean, "each individual prediction of each of the 56 (enumerated upthread) distinct elections"?
how many predictions are there for each election? does 538 issue more than one per election? Or how many data points are there?
as for the last paragraph: I guess you don't really believe in their probabilities, then, which was my whole point. I have no idea what you're talking about, re "non-zero risk premium."
how many predictions are there for each election? does 538 issue more than one per election? Or how many data points are there?
as for the last paragraph: I guess you don't really believe in their probabilities, then, which was my whole point. I have no idea what you're talking about, re "non-zero risk premium."
> how many predictions are there for each election?
One for each batch (which may be as small as one) of data (polls specifically in the polls-only model) added to the data; each takes into account current and past polls, with time.-based decay in weighting, distance from the election, and other factors. This includes polls for other contests in the same cycle, because the models accounts for correlation between them.
> does 538 issue more than one per election?
Sometimes multiple in a day for an election (especially the Presidential general election.)
> Or how many data points are there?
Well the scale for the bubbles on the Presidential election night calibration plot runs from 30-300 per bucket, and there’s 21 buckets.
> I have no idea what you're talking about, re "non-zero risk premium."
If you don't understand risk premiums and the asymmetric utility of nontrivial quantities of money, you have really no business talking about when a bet is reasonable.
One for each batch (which may be as small as one) of data (polls specifically in the polls-only model) added to the data; each takes into account current and past polls, with time.-based decay in weighting, distance from the election, and other factors. This includes polls for other contests in the same cycle, because the models accounts for correlation between them.
> does 538 issue more than one per election?
Sometimes multiple in a day for an election (especially the Presidential general election.)
> Or how many data points are there?
Well the scale for the bubbles on the Presidential election night calibration plot runs from 30-300 per bucket, and there’s 21 buckets.
> I have no idea what you're talking about, re "non-zero risk premium."
If you don't understand risk premiums and the asymmetric utility of nontrivial quantities of money, you have really no business talking about when a bet is reasonable.
> no, you can only "calibrate" his prediction abilities in general:
Yeah, but that's what we care about - the past has already happened so we only care about future events.
Yeah, but that's what we care about - the past has already happened so we only care about future events.
OK, so maybe we're agreed: you can bet on his abilities in one election. Let's say he's been right 95% of the time (I don't know if that's true) and we believe that's likely to continue, knowing nothing except "this is an election and Silver's predicted the result."
Then if he says "Hillary is likely to win" we can have 95% confidence he's right.
If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
Then if he says "Hillary is likely to win" we can have 95% confidence he's right.
If he says "Hillary has an 80% chance of winning" we ignore the 80, and just observe that it's more than 50.
It's a bit more flexible than that, though - rather than ignore 80% vs 70% and make it all up or down, you can let him predict a few different events in a row, add up the errors, and see how off they are.
Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
Or if you do want to review the past, you can look at the error for a category of elections or that entire year rather than his whole prediction career.
> predict a few different events in a row, add up the errors, and see how off they are.
I'm not seeing a formula there.
I'm not seeing a formula there.
How about here? https://en.wikipedia.org/wiki/Scoring_rule
that's the definition of a rule.
> One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell.
it still depends on many samples, or "over a long period" in your doc.
You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
> One could note the number of times that a 25% probability was quoted, over a long period, and compare this with the actual proportion of times that rain fell.
it still depends on many samples, or "over a long period" in your doc.
You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
> that's the definition of a rule.
And there are several example rules on the page.
> You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
That depends on what question you're asking. "How well calibrated are the electoral predictions that FiveThirtyEight makes?" is a sensible question with a lot of data points, seems to speak directly to the crowing about the one call being bad, and seems well suited to the application of a scoring rule for comparison between people making predictions about the same things.
And there are several example rules on the page.
> You can't escape the fact that there are only one or two samples, no matter how much math you throw around.
That depends on what question you're asking. "How well calibrated are the electoral predictions that FiveThirtyEight makes?" is a sensible question with a lot of data points, seems to speak directly to the crowing about the one call being bad, and seems well suited to the application of a scoring rule for comparison between people making predictions about the same things.
Thanks. It's reassuring to know that there still people who understand what probabilities actually mean, especially for one-off events.
Now that’s a funny one
With a straight face you’re going to say that in 2016 you saw a bunch of polls coming out showing trump winning?
Absolutely not. Nobody, and I mean nobody, gave him a snowflakes chance. Everyone including Silver mentioned “the greatest election landslide in history”, or some variant.
If there were polls showing trump winning they were hidden from ans downplayed in the msm. And to say anything else is a very poor lie.
With a straight face you’re going to say that in 2016 you saw a bunch of polls coming out showing trump winning?
Absolutely not. Nobody, and I mean nobody, gave him a snowflakes chance. Everyone including Silver mentioned “the greatest election landslide in history”, or some variant.
If there were polls showing trump winning they were hidden from ans downplayed in the msm. And to say anything else is a very poor lie.
There weren't polls showing Trump winning the popular vote. And he didn't.
https://projects.fivethirtyeight.com/2016-election-forecast/...
The last poll there from Nov 8 showed Clinton winning the popular vote by...wait for it...3.9 percent. She won the popular vote by...2.9 percent. National polls weren't far off at all.
The electoral college was decided by less than 50,000 votes overall. No one predicted the margins in those key states would be so small. It was a razor-thin win.
Statistics always have a margin of error and give probabilities. Polls didn't have to predict Trump winning. Instead, they give a probability of winning. FiveThirtyEight predicted a 28% chance of winning[2]. That's based on Monte Carlo simulations (that's where they run, e.g. 100 random elections and count how many times each outcome occurred). It actually doesn't matter if that estimate was low; it was high enough (1 in 4 chance) that the outcome wasn't insane. Even three days after the election, they already had an analysis of what they went wrong with polling[1]. I mean, you could read that or continue to trot out tired polarized arguments.
[1] https://fivethirtyeight.com/features/why-fivethirtyeight-gav... [2] https://projects.fivethirtyeight.com/2016-election-forecast/
https://projects.fivethirtyeight.com/2016-election-forecast/...
The last poll there from Nov 8 showed Clinton winning the popular vote by...wait for it...3.9 percent. She won the popular vote by...2.9 percent. National polls weren't far off at all.
The electoral college was decided by less than 50,000 votes overall. No one predicted the margins in those key states would be so small. It was a razor-thin win.
Statistics always have a margin of error and give probabilities. Polls didn't have to predict Trump winning. Instead, they give a probability of winning. FiveThirtyEight predicted a 28% chance of winning[2]. That's based on Monte Carlo simulations (that's where they run, e.g. 100 random elections and count how many times each outcome occurred). It actually doesn't matter if that estimate was low; it was high enough (1 in 4 chance) that the outcome wasn't insane. Even three days after the election, they already had an analysis of what they went wrong with polling[1]. I mean, you could read that or continue to trot out tired polarized arguments.
[1] https://fivethirtyeight.com/features/why-fivethirtyeight-gav... [2] https://projects.fivethirtyeight.com/2016-election-forecast/
I love how the popular vote is only ever mentioned when it’s to defend the idea that all these analyses “were actually right”
You’re not getting it. The analysis, the “28%” - those were wring. those were poor analyses
Yea they came back after and made up some bs about “oh we know what we did wrong now!”
When I was in school all the people who were cheating their way through math did the same thing. They couldn’t solve the problem, and the second you gave them the answer they magically “got it”
Fortune telling is a business of luck. Unless you can convince your followers that you’re right even when you’re clearly wrong
You’re not getting it. The analysis, the “28%” - those were wring. those were poor analyses
Yea they came back after and made up some bs about “oh we know what we did wrong now!”
When I was in school all the people who were cheating their way through math did the same thing. They couldn’t solve the problem, and the second you gave them the answer they magically “got it”
Fortune telling is a business of luck. Unless you can convince your followers that you’re right even when you’re clearly wrong
> Fortune telling is a business of luck.
It's more of a business of being connected to some forms of future reality. I can understand if people think nobody can do better than pure chance, but even so, I'm pretty sure the folks who believe statistics could give them an edge are quite misguided.
It's more of a business of being connected to some forms of future reality. I can understand if people think nobody can do better than pure chance, but even so, I'm pretty sure the folks who believe statistics could give them an edge are quite misguided.
This is a very strange take since it was Nate Silver who gave trump an order of magnitude higher chance of winning than everyone else. Around 30% chance, same for a batter getting a hit in baseball, which happens all the time.
I think a lot of people looked at his last prediction that Tuesday morning, observed that 70 > 30, and assumed that meant it was a lock. And then spent the last six years blaming Silver for their own innumeracy.
I find it fascinating that so many people here jump to the assumption that the average person doesn’t get that giving a 28% chance of victory doesn’t mean they can’t win
Clearly. Its not a difficult concept.
But those same people are wholly unable to see that all the other verbiage and propaganda that was behind/infused with Silver was, in no short terms, a demoralization campaign against the trump voters
And since they can’t see that all they can do is project their ego to say they aren’t wrong, its that the people who don’t agree with them don’t comprehend very basic statistics
Clearly. Its not a difficult concept.
But those same people are wholly unable to see that all the other verbiage and propaganda that was behind/infused with Silver was, in no short terms, a demoralization campaign against the trump voters
And since they can’t see that all they can do is project their ego to say they aren’t wrong, its that the people who don’t agree with them don’t comprehend very basic statistics
I mean he was getting crap for even suggesting trump had a chance. NYT had Hillary at like 99%
I don’t remember NYT. But I recall the other major poll site Real Clear Politics had an election map with a percentage chance for each state and they had Hillary at 99%.
Their mistake was that they treated each state as an independent event. Treating each state as a separate biased coin toss leads you to put Hillary at 99%.
Nate knew that these are not independent events and if republicans are being under polled in one state, it is more likely they’re being under polled in all of the states.
Their mistake was that they treated each state as an independent event. Treating each state as a separate biased coin toss leads you to put Hillary at 99%.
Nate knew that these are not independent events and if republicans are being under polled in one state, it is more likely they’re being under polled in all of the states.
1% isn’t a chance?
1% is a chance, but a proper scoring rule will penalize you more for being wrong about something you gave 1% than about something you gave 30%.
but NYT has, at times, been straight propaganda (like other obvious news sites for other propaganda), which i guess is your point.
for whom?
Well NYT themselves announced an editorial board overhaul a year or two ago because they said they had become too “left”, perhaps like Fox and WSJ were so far right? But now I’m not sure, since i subscribe and it seems balanced.
Wasn't that mainly due to the fact that people lie when they're asked if they'll vote for a highly distasteful candidate?
Fivethirtyeight had Trump at a 28.6% chance to win on the night[1], the web page is still up. I recall that being higher than any other polls I saw at the time. A 1 in 3.5 chance isn't that unlikely, it implies they're very close.
[1] https://projects.fivethirtyeight.com/2016-election-forecast
[1] https://projects.fivethirtyeight.com/2016-election-forecast
Percentage probabilities are meaningless for an event like this.
Can you run the 2016 election 1,000 times and count the times Hillary wins? Of course you can't.
Unlike estimating "what's the chance it rains tomorrow?" you can't even find 1,000 elections in the past with roughly similar conditions.
Unlike a baseball season, you don't have enough samples for the true probabilities to come out.
So, yes, despite all your downvotes, you're right: he pandered to the people who want a percentage number, despite the fact that it's meaningless and they will misunderstand it.
Can you run the 2016 election 1,000 times and count the times Hillary wins? Of course you can't.
Unlike estimating "what's the chance it rains tomorrow?" you can't even find 1,000 elections in the past with roughly similar conditions.
Unlike a baseball season, you don't have enough samples for the true probabilities to come out.
So, yes, despite all your downvotes, you're right: he pandered to the people who want a percentage number, despite the fact that it's meaningless and they will misunderstand it.
This is a lot like saying, "fractional exponents are meaningless, because you can't multiply something by itself half of a time." This intuitive model has broken down, but the problem is that the model wasn't sufficient to capture the complexity of the problem; it was a pedagogic stepping stone.
Similarly, probability is not the result of conducting repeated experiments (though the results of repeated experiments can be used to estimate probability). It's the measure of an event relative to the measure of the set of all events, eg, it's how much of the total volume in the space is taken up by a given event.
Similarly, probability is not the result of conducting repeated experiments (though the results of repeated experiments can be used to estimate probability). It's the measure of an event relative to the measure of the set of all events, eg, it's how much of the total volume in the space is taken up by a given event.
> This is a lot like saying, "fractional exponents are meaningless, ...
I don't even know what that means.
The second paragraph, I can't take issue with; the problem is "the set of all events" has cardinality 1.
I don't even know what that means.
The second paragraph, I can't take issue with; the problem is "the set of all events" has cardinality 1.
> I don't even know what that means.
If you have a question I'm happy to clarify. If a restatement is helpful, the problem is that this definition of probability is limited and fails to capture the complexity of the problem domain, not that the problem domain is meaningless.
You're misunderstanding the terminology, "events" refers to sets of potential outcomes (eg, when rolling a die, the outcomes are 1-6 and the event "even" is the set of 2,4, and 6).
(I don't know 538's actual methodology, consider the below an illustration rather than a literal description of 538's model.)
The outcomes here would be each way the electoral vote could possibly be allocated, based on the number of states and the idiosyncracies of each state constitution. The events would be the division of these outcomes by who wins the election in that instance. The model then seeks to attach a weight to each outcome based on how likely it is.
You can think of these weights as the volume of each outcome. We use some kind of model to assign them. The most straightforward model would be to repeat an experiment many times and then assume the rate at which the outcome appears in your sample is approximately the same as it is globally. Other times that isn't possible and you need to use more sophisticated modeling techniques. I'm not going to offer a defense of statistics and machine learning writ large, but I think looking around it should be pretty clear these things can work well even if there's a lot of art to getting it right.
You can imagine comparing the probabilities of each event by pouring their constituent outcomes into separate graduated cylinders so that you can compare their cumulative volume. If you divide their volume by the total volume of all outcomes, that's the probability of that event. Or rather, that's the estimate of the probability, given our model and the data we have.
This is more or less my drunk history version of measure theory and probability theory. I imagine I've gotten some details wrong, so I'd encourage anyone interested to check the subjects out for themselves. Here's some YouTube videos:
https://youtube.com/playlist?list=PLBh2i93oe2qvMVqAzsX1Kuv6-...
https://youtube.com/playlist?list=PLUl4u3cNGP60hI9ATjSFgLZpb...
https://youtube.com/@statquest
If you have a question I'm happy to clarify. If a restatement is helpful, the problem is that this definition of probability is limited and fails to capture the complexity of the problem domain, not that the problem domain is meaningless.
You're misunderstanding the terminology, "events" refers to sets of potential outcomes (eg, when rolling a die, the outcomes are 1-6 and the event "even" is the set of 2,4, and 6).
(I don't know 538's actual methodology, consider the below an illustration rather than a literal description of 538's model.)
The outcomes here would be each way the electoral vote could possibly be allocated, based on the number of states and the idiosyncracies of each state constitution. The events would be the division of these outcomes by who wins the election in that instance. The model then seeks to attach a weight to each outcome based on how likely it is.
You can think of these weights as the volume of each outcome. We use some kind of model to assign them. The most straightforward model would be to repeat an experiment many times and then assume the rate at which the outcome appears in your sample is approximately the same as it is globally. Other times that isn't possible and you need to use more sophisticated modeling techniques. I'm not going to offer a defense of statistics and machine learning writ large, but I think looking around it should be pretty clear these things can work well even if there's a lot of art to getting it right.
You can imagine comparing the probabilities of each event by pouring their constituent outcomes into separate graduated cylinders so that you can compare their cumulative volume. If you divide their volume by the total volume of all outcomes, that's the probability of that event. Or rather, that's the estimate of the probability, given our model and the data we have.
This is more or less my drunk history version of measure theory and probability theory. I imagine I've gotten some details wrong, so I'd encourage anyone interested to check the subjects out for themselves. Here's some YouTube videos:
https://youtube.com/playlist?list=PLBh2i93oe2qvMVqAzsX1Kuv6-...
https://youtube.com/playlist?list=PLUl4u3cNGP60hI9ATjSFgLZpb...
https://youtube.com/@statquest
> Percentage probabilities are meaningless for an event like this.
No, they aren't.
(They can’t be validated for a single event in isolation, but 538 predicts lots of election events, so their models can, generally, be assessed, and are historically quite accurate.)
No, they aren't.
(They can’t be validated for a single event in isolation, but 538 predicts lots of election events, so their models can, generally, be assessed, and are historically quite accurate.)
AlbertCory(1)
> Can you run the 2016 election 1,000 times and count the times Hillary wins?
I assume you mean "in real life", but otherwise this seems to be close to their actual methodology. You can see the outcome frequency charts under "What to expect from the Electoral College."[1]
[1] https://projects.fivethirtyeight.com/2016-election-forecast/...
I assume you mean "in real life", but otherwise this seems to be close to their actual methodology. You can see the outcome frequency charts under "What to expect from the Electoral College."[1]
[1] https://projects.fivethirtyeight.com/2016-election-forecast/...
> I assume you mean "in real life", but otherwise this seems to be close to their actual methodology
So, the "actual methodology" is not real life, but some abstraction (like, "assume no friction or air resistance")?
So, the "actual methodology" is not real life, but some abstraction (like, "assume no friction or air resistance")?
Well yes, I mean, all simulations are, right? You put as much real-life data in as you can, but everything's an abstraction unless you're the team from the Devs tv series.
The actual election that happened is not a simulation, at least not the same kind of simulation that was run by Nate.
That qualitative difference is fundamental. It suffers from the same problem when results of simulations are applied to the "real world". The quality of the simulation is paramount, and there's no way to determine whether the simulation is good enough besides assertions that the simulations are really sophisticated and Nate is very smart.
That qualitative difference is fundamental. It suffers from the same problem when results of simulations are applied to the "real world". The quality of the simulation is paramount, and there's no way to determine whether the simulation is good enough besides assertions that the simulations are really sophisticated and Nate is very smart.
People do misunderstand it, that is true. Including you, who thinks it is meaningless. Being imperfect is very different from useless or meaningless.
AlbertCory(1)
>Can you run the 2016 election 1,000 times and count the times Hillary wins? Of course you can't.
If you think that evaluating their model based on binary outcomes is the best way to do it, then, well, this argument makes sense.
If you want to evaluate the model based on all of the details available, then, well, it stops making nearly as much sense. What explanations were given for potential Trump victories? What information was available at the time? What information was available after? Do we have data around if those explanations did align with Trump's victory?
I've not done a deep dive on this - 538 is entertainment for me. But from what I remembered in 2016, what I read today in digging up some example articles for another comment, and looking at the actual outcomes... yes, those explanations were closely aligned with reality.
Humans have to come up with probabilities and make decisions based off of them all the time, even when we will never get a chance to get 1000 occurrences. These might not be provable in the same way as something we can run 1000 times is, but the idea that they are meaningless and without value is bizarre.
If you think that evaluating their model based on binary outcomes is the best way to do it, then, well, this argument makes sense.
If you want to evaluate the model based on all of the details available, then, well, it stops making nearly as much sense. What explanations were given for potential Trump victories? What information was available at the time? What information was available after? Do we have data around if those explanations did align with Trump's victory?
I've not done a deep dive on this - 538 is entertainment for me. But from what I remembered in 2016, what I read today in digging up some example articles for another comment, and looking at the actual outcomes... yes, those explanations were closely aligned with reality.
Humans have to come up with probabilities and make decisions based off of them all the time, even when we will never get a chance to get 1000 occurrences. These might not be provable in the same way as something we can run 1000 times is, but the idea that they are meaningless and without value is bizarre.
A prediction is not meaningless. An exact probability is.
Thank you. It's a non repeatable single event. Percentage probabilities is basically cover for the predictor and cope for the people buying their bs.
it's not surprising that so many people have trouble with this. Everyone studies card games, dice, and pulling a red ball out of a jar with N red balls and M white balls, because those are idealized problems.
Real life is harder.
Real life is harder.
What? At the time, people were lambasting him for saying Trump had a 30% chance, and I know people who think he was too smug after getting 2016 right. I mean, I know this is the internet and people can say whatever they want, but come on.
These are the people who got their feelings hurt over a culture war and lash out at Silver because he's become vaguely mainstream synonymous with political projections. Nobody with a serious position and serious attention on this would criticize Silver. It was the Buzzfeed types that gave Clinton a much higher chance.
No, literally everyone gave Clinton a much higher chance. I think FiveThirtyEight is mostly a clickbait circus, but they were the only serious prediction organization who rated Trump's chances as high as they did.
FiveThirtyEight had about a 1/3 chance of Trump winning, far higher than anyone else was modeling. And he was one of the few pundits openly talking about how it wasn't some insane longshot. I have no idea where this meme came from, but it's annoyingly persistent.
You mean >50% isn't a lock? /s
I remember this. Tough spot for NS. I remember thinking that 2:1 Hillary and three language around it, came across a useless hedge. In retrospect, I think he'd have gotten more respect if he shouted from the rafters that Hillary doesn't have a lock and Trump could really win. Then, he'd have been heralded as a genius. Strong forecasting, weak marketing.
I remember this. Tough spot for NS. I remember thinking that 2:1 Hillary and three language around it, came across a useless hedge. In retrospect, I think he'd have gotten more respect if he shouted from the rafters that Hillary doesn't have a lock and Trump could really win. Then, he'd have been heralded as a genius. Strong forecasting, weak marketing.
Maybe, but that's basically asking him to develop the opposite of the personality type that makes him abnormally good at forecasting.
I think both hardcore (D) and hardcore MAGA people have a lot to learn that wanting something to be true doesn't make it so.
> Analysis after analysis about how Hillary was a sure thing.
Really? I think I remember him giving Trump 30% on the night?
EDIT: Got it - some people struggle with incorporating uncertainty into their reasoning.
Really? I think I remember him giving Trump 30% on the night?
EDIT: Got it - some people struggle with incorporating uncertainty into their reasoning.
Correct. He was right about the 2016 presidential election.
What does it mean to be "right"?
A pollster claiming Trump:Hillary at 99:1, 50:50, or 33:66 could all claim to have been "right".
A pollster claiming Trump:Hillary at 99:1, 50:50, or 33:66 could all claim to have been "right".
For the individual prediction it's impossible to say, so you reward the people who got it right based on how likely they said it was, and you penalize the people who got it wrong based on how unlikely they said it was. If we apply a proper scoring rule, over time the you score better by giving better calibrated guesses.
I agree with this. But the original statement is poorly phrased, that "He was right." He was perhaps more right than others who gave Trump lower odds. But there's not an absolute right/wrong (unless someone gave 100:0 odds).
He seems to have been right that Trump's odds were higher because prediction error for different states wasn't independent as some other models were assuming.
I agree that we don't have a way of assessing whether the percentage given was "correct", but over a bunch of predictions we can keep track and see how he does when he gives various percentages, or compare against others with a proper scoring rule.
I don't make any claim that the odds he gave were right, but just that if they were we should still expect to see his "prediction" be wrong pretty often, so the fact that his prediction was wrong this time only counts so much against him (and to make up for that, only counts so much in is favor if he's right with low confidence).
I agree that we don't have a way of assessing whether the percentage given was "correct", but over a bunch of predictions we can keep track and see how he does when he gives various percentages, or compare against others with a proper scoring rule.
I don't make any claim that the odds he gave were right, but just that if they were we should still expect to see his "prediction" be wrong pretty often, so the fact that his prediction was wrong this time only counts so much against him (and to make up for that, only counts so much in is favor if he's right with low confidence).
You’re being downvoted by people who don’t have a good argument against you
To them Silver could have said Trump had a 1% chance of winning, and they would still be saying “yOu dOnT uNdErStaNd stAtIstIcS!!1”
Basically, a lot of people really really really wanted to believe Hillary winning was a sure thing. They ignored any and all data saying “maybe she won’t”
And now you can see the effect looking back - these people are still so salty about that bitter loss that they come in here and say that giving Trump a <30% chance of winning was absolutely the right call and if you don’t see that then you’re an idiot who can’t grasp statistics
Literally - half the comments responding to me was this vitriolic idiocy. They can’t think, they’re emotions have overridden that
That’s why they downvoted you for accurately pointing out that there’s no such thing as knowing that a statistical guess was “right”
They’re downvoting you because you upset them by making them accept reality. That they were wrong and the Hillary couldn’t win.
To them Silver could have said Trump had a 1% chance of winning, and they would still be saying “yOu dOnT uNdErStaNd stAtIstIcS!!1”
Basically, a lot of people really really really wanted to believe Hillary winning was a sure thing. They ignored any and all data saying “maybe she won’t”
And now you can see the effect looking back - these people are still so salty about that bitter loss that they come in here and say that giving Trump a <30% chance of winning was absolutely the right call and if you don’t see that then you’re an idiot who can’t grasp statistics
Literally - half the comments responding to me was this vitriolic idiocy. They can’t think, they’re emotions have overridden that
That’s why they downvoted you for accurately pointing out that there’s no such thing as knowing that a statistical guess was “right”
They’re downvoting you because you upset them by making them accept reality. That they were wrong and the Hillary couldn’t win.
Excellent troll, good sir, 10/10
Flip two coins. If they're both heads Trump wins.
That's approximately what Silver's team was saying. No idea where this "sure thing" idea of your's comes from.
That's approximately what Silver's team was saying. No idea where this "sure thing" idea of your's comes from.
I remember leading up to Election Day his predictions were giving her 66% and Trump 33% (IIRC). Which means if you roll a die and it comes up as 1 or 2 then Trump wins (which isn't all that unlikely)
Many people mistook these numbers to mean Clinton was going to get 66% of the vote and thus were expecting a landslide, but that is not what his numbers meant.
Now I wasn't reading any of the analysis he or his team was putting out, but the modeling seemed reasonable to me.
Many people mistook these numbers to mean Clinton was going to get 66% of the vote and thus were expecting a landslide, but that is not what his numbers meant.
Now I wasn't reading any of the analysis he or his team was putting out, but the modeling seemed reasonable to me.
[deleted]
I see this repeated all the time and I don't understand how people come to this conclusion.
538 consistently gave Trump a better chance than much of the mainstream media, with him showing a roughly 30% chance for much of the year, including right up at the election. At times it dropped as low as 15% or so, but... that's still not some insanely rare occurrence.
As for what they were actually saying? Silver and 538 never claimed it was a sure thing. Let's look at some actual articles:
https://fivethirtyeight.com/features/election-update-dont-ig...
"At the same time, it shouldn’t be hard to see how Clinton could lose. She’s up by about 3 percentage points nationally, and 3-point polling errors happen fairly often, including in the last two federal elections. Obama beat his polls by about 3 points in 2012, whereas Republicans beat their polls by 3 to 4 points in the 2014 midterms. If such an error were to favor Clinton, she could win in a borderline landslide. If the error favored Trump, however, she’d be in a dicey position, because the error is highly correlated across states.
There’s also reason to think a polling error is more likely than usual this year, because of the high number of undecided voters. In national polls, Clinton averages about 45 percent of the vote and Trump 42 percent; by comparison, Obama led Mitt Romney roughly 49-48 in national polls at the end of the 2012 campaign. That contributes significantly to uncertainty, since neither candidate has enough votes yet to have the election in the bag.
To be honest, I’m kind of confused as to why people think it’s heretical for our model to give Trump a 1-in-3 chance — which does make him a fairly significant underdog, after all. There are a lot of ways to build models, and there are lots of factors that a model based on public polling, like ours, doesn’t consider.3 But the public polls — specifically including the highest-quality public polls — show a tight race in which turnout and late-deciding voters will determine the difference between a clear Clinton win, a narrow Clinton win and Trump finding his way to 270 electoral votes."
https://fivethirtyeight.com/features/clinton-probably-finish... is about as bullish on Clinton as I can find from the archives, and even there Silver does not talk about how Hillary is a sure thing, and instead spend several paragraphs discussing reasons why Trump could still win and what those would look like. And as those things occurred, the model shifted back from being as bullish on Hillary, for the reasons Silver outlines.
I don't know why you would read a 70% chance of victory as being the same thing as a sure victory. I wouldn't want to bet my life on a 70% success rate!
538 consistently gave Trump a better chance than much of the mainstream media, with him showing a roughly 30% chance for much of the year, including right up at the election. At times it dropped as low as 15% or so, but... that's still not some insanely rare occurrence.
As for what they were actually saying? Silver and 538 never claimed it was a sure thing. Let's look at some actual articles:
https://fivethirtyeight.com/features/election-update-dont-ig...
"At the same time, it shouldn’t be hard to see how Clinton could lose. She’s up by about 3 percentage points nationally, and 3-point polling errors happen fairly often, including in the last two federal elections. Obama beat his polls by about 3 points in 2012, whereas Republicans beat their polls by 3 to 4 points in the 2014 midterms. If such an error were to favor Clinton, she could win in a borderline landslide. If the error favored Trump, however, she’d be in a dicey position, because the error is highly correlated across states.
There’s also reason to think a polling error is more likely than usual this year, because of the high number of undecided voters. In national polls, Clinton averages about 45 percent of the vote and Trump 42 percent; by comparison, Obama led Mitt Romney roughly 49-48 in national polls at the end of the 2012 campaign. That contributes significantly to uncertainty, since neither candidate has enough votes yet to have the election in the bag.
To be honest, I’m kind of confused as to why people think it’s heretical for our model to give Trump a 1-in-3 chance — which does make him a fairly significant underdog, after all. There are a lot of ways to build models, and there are lots of factors that a model based on public polling, like ours, doesn’t consider.3 But the public polls — specifically including the highest-quality public polls — show a tight race in which turnout and late-deciding voters will determine the difference between a clear Clinton win, a narrow Clinton win and Trump finding his way to 270 electoral votes."
https://fivethirtyeight.com/features/clinton-probably-finish... is about as bullish on Clinton as I can find from the archives, and even there Silver does not talk about how Hillary is a sure thing, and instead spend several paragraphs discussing reasons why Trump could still win and what those would look like. And as those things occurred, the model shifted back from being as bullish on Hillary, for the reasons Silver outlines.
I don't know why you would read a 70% chance of victory as being the same thing as a sure victory. I wouldn't want to bet my life on a 70% success rate!
Common problem with % statistics I find is they dissuade people from considering the actual occurrences of something.
70% chance of success is about a 1 out of 3 chance of failure.
70% chance of success is about a 1 out of 3 chance of failure.
100%. Never trusted him again.
You remember incorrectly.
https://www.vox.com/2016/11/3/13147678/nate-silver-fivethirt...
https://www.vox.com/2016/11/3/13147678/nate-silver-fivethirt...
I question this view. Why? He forecast, and whats more provided the error bounds which gave Trump a chance to win. Outside of a crystal ball, what else could be expected?
> I question this view. Why?
I think because what's the point of following the projections if at the end of the day they're indistinguishable from a coin toss? This applies to all polling. It's vaguely interesting but when (in Trump's case) a candidate can be consistently far behind for months and then still win because ¯\_(ツ)_/¯ that's how probability works, then what's the point of having followed the polls?
I think because what's the point of following the projections if at the end of the day they're indistinguishable from a coin toss? This applies to all polling. It's vaguely interesting but when (in Trump's case) a candidate can be consistently far behind for months and then still win because ¯\_(ツ)_/¯ that's how probability works, then what's the point of having followed the polls?
What? I'm struggling to find an analogy to help you understand what the projections actually mean, but also I'm asking "Do I care if this person is wrong on the internet?".
Maybe you need to compare it to a weather prediction. If you hear "30% chance of snow" where you're about to vacation, would you pack a winter jacket? What if it's 85%? If it's 0.5%? And if it's at 25-30% for months on end, that doesn't mean that "No chance it'll snow!".
It's not like a sports league where one person is behind for months and then suddenly wins it. Indeed what is the point of following the polls? For one thing people were probably addicted to the rollercoaster of "Oh shit my side's losing! " and then "Oh yeah my side's gaining!". At least for the campaigns themselves they should've been able to adjust their work in hopes of moving the needle (e.g. leaking stuff about the other side).
Maybe you need to compare it to a weather prediction. If you hear "30% chance of snow" where you're about to vacation, would you pack a winter jacket? What if it's 85%? If it's 0.5%? And if it's at 25-30% for months on end, that doesn't mean that "No chance it'll snow!".
It's not like a sports league where one person is behind for months and then suddenly wins it. Indeed what is the point of following the polls? For one thing people were probably addicted to the rollercoaster of "Oh shit my side's losing! " and then "Oh yeah my side's gaining!". At least for the campaigns themselves they should've been able to adjust their work in hopes of moving the needle (e.g. leaking stuff about the other side).
No, I understand what the probabilistic prediction means. The problem is that there is no way to tell if 538's prediction was any more accurate than if they'd said 50-50 (or any numbers whatsoever) because the presidential election is only every 4 years. We can't measure the predictive ability of his model at all. If Hillary won, we'd say "Yup, the model gave her 70%". When Trump won, we said "Yup, the model gave him 30%".
Correct. It's a forecast based on polling sentiment.
Who does it help? People who face regulatory uncertainty between the regimes. Mostly businesses, nonprofits, etc.
Who does it help? People who face regulatory uncertainty between the regimes. Mostly businesses, nonprofits, etc.
I get where you're coming from, but it's not strictly true. For instance if their model had given Hillary a 99.5% chance and Trump only 0.5%, then we could be pretty sure it wasn't a good model when Trump won.
You might say even then, how do we know the result didn't just fall into that 0.5% remaining chance? But polls can and do still give some useful idea of what the result might be.
You might say even then, how do we know the result didn't just fall into that 0.5% remaining chance? But polls can and do still give some useful idea of what the result might be.
In this case the candidates were very close. Fivethirtyeight's estimate had Hillary at 48.5% of the vote and Trump at 44.9% (and the rest on other canditates), only 3.6% apart.[1]
Usually, when things aren't quite so close, polls are a more useful gauge since they're more likely to be correct about who'll win as the gap widens.
[1] https://projects.fivethirtyeight.com/2016-election-forecast
Usually, when things aren't quite so close, polls are a more useful gauge since they're more likely to be correct about who'll win as the gap widens.
[1] https://projects.fivethirtyeight.com/2016-election-forecast
Why are you looking at the %vote, though? In that very link, 538 has Hillary as 71% chance of winning. That is not close at all.
I'm not arguing that something can ever be treated as a certainty, just that there's "no point" in following the polls since even a lopsided 71-28% prediction will go either way, and is indistinguishable from a 50-50 prediction on election night.
I'm not arguing that something can ever be treated as a certainty, just that there's "no point" in following the polls since even a lopsided 71-28% prediction will go either way, and is indistinguishable from a 50-50 prediction on election night.
What I mean is, the reason it's 71%-28% is that the actual % difference between predicted votes is very small. Once the estimated votes are say 10% apart, then you start getting a prediction that's more like 99%-1%, and the polls are much more likely to be telling you accurately who will actually win.
In this case the prediction is pretty close to just a 50/50 coin flip, but that's quite unusual and reflective of the chance legitimately being close to 50/50. Therefore I wouldn't say it invalidates polls in general.
In this case the prediction is pretty close to just a 50/50 coin flip, but that's quite unusual and reflective of the chance legitimately being close to 50/50. Therefore I wouldn't say it invalidates polls in general.
The whole point of a poll/stat prediction from the intuitive human standpoint is to get a good gauge on how likely outcomes are
Silver, as well as all other mainstream polls/predictions in the race had Hillary winning by “the greatest margin in presidential history”
You can try and whitewash it but we all remember that. We all remember how hopeless trump looked and how bulletproof hillary’s victory was
To say Silver and all the other polls weren’t framing this as a landslide victory for hillary and a humiliating defeat for trump is just living in a false alternate reality in one’s mind.
And judging by these responses that seems to be the more common state of mind for folks here to be in…
Silver, as well as all other mainstream polls/predictions in the race had Hillary winning by “the greatest margin in presidential history”
You can try and whitewash it but we all remember that. We all remember how hopeless trump looked and how bulletproof hillary’s victory was
To say Silver and all the other polls weren’t framing this as a landslide victory for hillary and a humiliating defeat for trump is just living in a false alternate reality in one’s mind.
And judging by these responses that seems to be the more common state of mind for folks here to be in…
> Silver, as well as all other mainstream polls/predictions in the race had Hillary winning by “the greatest margin in presidential history.” You can try and whitewash it but we all remember that
We don't need to remember though; we have the Internet Archive.
We can look back at Silver's prediction page on the morning of the election[1], where it's predicting Hillary winning by only 3.5% of the vote.
[1] https://web.archive.org/web/20161108000845/http://projects.f...
We don't need to remember though; we have the Internet Archive.
We can look back at Silver's prediction page on the morning of the election[1], where it's predicting Hillary winning by only 3.5% of the vote.
[1] https://web.archive.org/web/20161108000845/http://projects.f...
Yea, cherry picking is an excellent way to claim you’re right when reality doesn’t agree
Go back and get an aggregate sense of everything he said. Look at 90%+ if the articles he wrote saying what I said compared to the one or two that y’all want to keep referring to
For people discussing stats there’s a very strong inability to grok averages here…
Go back and get an aggregate sense of everything he said. Look at 90%+ if the articles he wrote saying what I said compared to the one or two that y’all want to keep referring to
For people discussing stats there’s a very strong inability to grok averages here…
I only linked that one because it's the main page where they posted their data.
Its easy to check what he was saying too, we can just look at the archive from the same day of Nate Silver's posts : https://web.archive.org/web/20161108082405/https://fivethirt...
They're all about how it's a very close, uncertain race. You might point out that the top one is "Clinton Gains...", but even that one is just talking about "Clinton’s projected margin of victory in the popular vote has increased to 3.5 percent from 2.9 percent."
I did try looking up your quote - "greatest margin in presidential history" - but the only Google result was your comment.
Its easy to check what he was saying too, we can just look at the archive from the same day of Nate Silver's posts : https://web.archive.org/web/20161108082405/https://fivethirt...
They're all about how it's a very close, uncertain race. You might point out that the top one is "Clinton Gains...", but even that one is just talking about "Clinton’s projected margin of victory in the popular vote has increased to 3.5 percent from 2.9 percent."
I did try looking up your quote - "greatest margin in presidential history" - but the only Google result was your comment.
I was there watching. Maybe Silver doesn’t understand how to communicate verbally, only through numbers designed for consumption by statisticians. The impression he gave in his comments, if not these exact words — Hillary in a landslide.
Communication is a skill. Like the skill of a great sportscaster like Scully. Election reporting is, I’m sure, a similar unique skill set. Silver doesn’t have it, and I never want to listen to him again.
Communication is a skill. Like the skill of a great sportscaster like Scully. Election reporting is, I’m sure, a similar unique skill set. Silver doesn’t have it, and I never want to listen to him again.