Nate Silver correctly called 50 out of 50 states(plus.google.com)
plus.google.com
Nate Silver correctly called 50 out of 50 states
https://plus.google.com/114613808538621741268/posts/LfoYJdWSK71
274 comments
Nice work.
The success, though, isn't his specific model. It's that Silver was able to market the idea that using statistical models is better than a table full of talking heads at predicting an outcome. And that a "dead heat" doesn't mean it's a coin flip. In hindsight, it seems almost ludicrous that it hasn't gained greater traction before.
He deserves a ton of credit for championing analytics into pop/political culture, but I'm sure we'll see many more models and a lot more statisticians during the next election, some of which will be better than Silver's models (if he's still doing them). I'm all for people asking to see the data instead of the media sound bites.
To keep this HN relevant: it was the marketing that drove his startup success. The product (the model) isn't quite perfect and an excellent substitute product was created by someone else in very short order (you). But he put together a great blog, grew out his brand, and eventually saw hockey stick growth.
The success, though, isn't his specific model. It's that Silver was able to market the idea that using statistical models is better than a table full of talking heads at predicting an outcome. And that a "dead heat" doesn't mean it's a coin flip. In hindsight, it seems almost ludicrous that it hasn't gained greater traction before.
He deserves a ton of credit for championing analytics into pop/political culture, but I'm sure we'll see many more models and a lot more statisticians during the next election, some of which will be better than Silver's models (if he's still doing them). I'm all for people asking to see the data instead of the media sound bites.
To keep this HN relevant: it was the marketing that drove his startup success. The product (the model) isn't quite perfect and an excellent substitute product was created by someone else in very short order (you). But he put together a great blog, grew out his brand, and eventually saw hockey stick growth.
Yep, I agree with this. I really built the model for fun, just to see how close it was possible to get with a few hours work.
A lot of forecasting problems have a structure like this one did:
- There is a lot of publically available data, and simple statistical models based on that data give pretty good predictions with very little effort.
- If you're an expert with a lot of time on your hands, you can put a lot of effort into improving the forecasts by 1-2%.
- In the end, it comes down to the effective business use of your model. Nate Silver had an excellent brand around his model.
In fact, I'd wager that he knows his model is more complex than it needs to be, but part of his brand is "rogue genius with a complex model that's far too difficult for mere mortals to understand".
A lot of forecasting problems have a structure like this one did:
- There is a lot of publically available data, and simple statistical models based on that data give pretty good predictions with very little effort.
- If you're an expert with a lot of time on your hands, you can put a lot of effort into improving the forecasts by 1-2%.
- In the end, it comes down to the effective business use of your model. Nate Silver had an excellent brand around his model.
In fact, I'd wager that he knows his model is more complex than it needs to be, but part of his brand is "rogue genius with a complex model that's far too difficult for mere mortals to understand".
While your project got the right result, I don't think it's mathematically justifiable to take an arithmetic mean of the published survey means without accounting for the standard deviation of each sample distribution.
Doing that would make your model more mathematically complicated, but it's no doubt one of the things that Nate Silver is doing.
But besides that, Silver isn't just trying to call the election the Monday before the election, he's trying to build a model that predicts the outcome as early as possible.
A lot of the stuff he's been throwing into it, like the 'economic factors' or 'convention bounce' adjustments are about keeping the model from going off into the weeds due to predictable fluctuations in public sentiment caused by ephemeral events.
If you compare the history on the right column of the site to the "now cast" tab, which excludes those adjustments, you'll notice that the former contains fewer and more modest fluctuations than the "now cast".
Doing that would make your model more mathematically complicated, but it's no doubt one of the things that Nate Silver is doing.
But besides that, Silver isn't just trying to call the election the Monday before the election, he's trying to build a model that predicts the outcome as early as possible.
A lot of the stuff he's been throwing into it, like the 'economic factors' or 'convention bounce' adjustments are about keeping the model from going off into the weeds due to predictable fluctuations in public sentiment caused by ephemeral events.
If you compare the history on the right column of the site to the "now cast" tab, which excludes those adjustments, you'll notice that the former contains fewer and more modest fluctuations than the "now cast".
You're right, it's not mathematically justifiable at all. The focus was on quick and dirty results over mathematical sophistication. Note that I described the adjustment for the number of polls as "a fudge"!
> Silver isn't just trying to call the election the Monday before the election, he's trying to build a model that predicts the outcome as early as possible.
That's true, but all the praise he's now getting is for correctly predicting the outcomes in every state - a prediction that he made less than 24 hours before the election took place (in fact, he changed his prediction for Florida from REP to DEM after I'd made my predictions, using up-to-the-minute polling data).
> Silver isn't just trying to call the election the Monday before the election, he's trying to build a model that predicts the outcome as early as possible.
That's true, but all the praise he's now getting is for correctly predicting the outcomes in every state - a prediction that he made less than 24 hours before the election took place (in fact, he changed his prediction for Florida from REP to DEM after I'd made my predictions, using up-to-the-minute polling data).
> In fact, I'd wager that he knows his model is more complex than it needs to be, but part of his brand is "rogue genius with a complex model that's far too difficult for mere mortals to understand".
Not what he said on The Colbert Report. "It's not hard math. You take averages then you count to 270."
Not what he said on The Colbert Report. "It's not hard math. You take averages then you count to 270."
part of his brand is "rogue genius with a complex model that's far too difficult for mere mortals to understand".
On the contrary, I think his brand is 'anyone who takes some time to study statistics & probability can work this out for themselves.' His key skill is demystifying the modeling and explaining that it's not the result of some secret sauce.
On the contrary, I think his brand is 'anyone who takes some time to study statistics & probability can work this out for themselves.' His key skill is demystifying the modeling and explaining that it's not the result of some secret sauce.
A slightly less cynical take on the last point is that it also lets him rebut many objections with, "nope, the model already takes that into account". By having a piece of the model devoted to most possible talking points: convention bounces, whether undecideds break for/against the incumbent, effect of the economy, etc., he can claim he's addressed those critics' points, even if the net effect of addressing them is close to nil.
And an addendum, that 1%-2% accuracy is crucial in close elections like this one, so it's more than worth putting the extra time and effort into getting.
You hit it on the head. It wasn't the idea itself, it's his marketing of it. This is how Levitt broke out of the pack with Freakonomics. It wasn't just being novel or counterintuitive, it was getting up on the roof an shouting about it.
So basically, Nate Silver is to politics what Billy Beane is to baseball?
Well, Nate Silver is kind of to politics what Nate Silver is to baseball. There had been forecasting systems prior to PECOTA, but it upped the bar.
More the Bill James of politics. (Bill James, created Sabermetrics, which are many of the models that Billy Beane utilizes)
Billy Beane is more a someone who utilizes new types of stats in player evaluation, then the person who creates the stats, I'm not sure who the "user" of Nate Silver's metrics are.
It is interesting to note that much of Nate's work stems from his work in Baseball.
http://en.wikipedia.org/wiki/Bill_James
Billy Beane is more a someone who utilizes new types of stats in player evaluation, then the person who creates the stats, I'm not sure who the "user" of Nate Silver's metrics are.
It is interesting to note that much of Nate's work stems from his work in Baseball.
http://en.wikipedia.org/wiki/Bill_James
I think Beane probably had a lot more institutional resistance to his methods. The pundit class is much stronger in sports, since commentators are the mediators of entire sports for most fans (i.e. the majority don't attend games).
And this is different from US politics, how?
Citizens are not subject to the rules of the sports they follow.
Commentators are mediators of politics for most people in the US.
You're making a category error.
No, I'm making a wry comment about the political apathy of most US citizens, and you are either not getting the joke or trolling by not getting the joke.
The risk of sarcasm is that you are taken seriously. You probably should have just deleted your comment rather than drag it out.
I agree to an extent. His product was clear much better than the existing alternatives - but he also made a brilliant move by choosing the NYT as his exclusive distribution outlet.
It's more than just marketing. It's largely about distribution.
It's more than just marketing. It's largely about distribution.
Silver's model isn't particularly advanced on election day. The further you get from election day, the more advanced his forecast is, accounting for various factors and ebbs and flows in polling data -- convention bounces, economic news, etc.
A big part of Silver's success has not been his math, but rather his excellent writing. He offers easy to consume analysis of the polls and makes a boring (to the masses) subject interesting. He can be portrayed as a stats geek when presented to the public, but really he's a great communicator who also knows his data.
A big part of Silver's success has not been his math, but rather his excellent writing. He offers easy to consume analysis of the polls and makes a boring (to the masses) subject interesting. He can be portrayed as a stats geek when presented to the public, but really he's a great communicator who also knows his data.
Is there an element in this of the fact that if you looked at with any degree of sophistication it wasn't actually as close as some made out?
Pundits and the media gain from the narrative that it was on a knife edge because it keeps people coming back for more, but in reality Romney never really picked up, let alone sustained, the sort of position in the swing states that would have led to a win. It was never comfortable for Obama and Romney wasn't out of it but the idea it was neck and neck was fairly dramatically over stated.
Not undermining what Silver did but it seems to me that the same pundits who spoke out against him made him look more impressive than it was by claiming the race was closer than it was.
Pundits and the media gain from the narrative that it was on a knife edge because it keeps people coming back for more, but in reality Romney never really picked up, let alone sustained, the sort of position in the swing states that would have led to a win. It was never comfortable for Obama and Romney wasn't out of it but the idea it was neck and neck was fairly dramatically over stated.
Not undermining what Silver did but it seems to me that the same pundits who spoke out against him made him look more impressive than it was by claiming the race was closer than it was.
A lot of the work silver doesn't isn't just aggregating the polls. He tries to take into account the probability that the polls are wrong.
According to a couple of his blog posts, going just by the most recent polls (and their margins of error) Obama had a 100% chance of winning -- but in the past, polls have sometimes been systematically misleading.
According to a couple of his blog posts, going just by the most recent polls (and their margins of error) Obama had a 100% chance of winning -- but in the past, polls have sometimes been systematically misleading.
Great. Now you can go on https://intrade.com/ , make some bets and clean up.
I already did.
With great statistical modelling comes great cash windfalls. You're right though, it's pretty straightforward to model it out so it's been baffling to see how many people have out and out claimed that the models are wrong.
Something I don't quite understand, but maybe someone here can help me with.
As I understand it, the poll averaging works because of the central limit theorem. But with so few data points, maybe a dozen polls at most in each state, and often half that, why does it still seem to work? I thought you'd need a few dozen data points at least.
What am I missing?
As I understand it, the poll averaging works because of the central limit theorem. But with so few data points, maybe a dozen polls at most in each state, and often half that, why does it still seem to work? I thought you'd need a few dozen data points at least.
What am I missing?
Let's take two examples - Ohio (a contested state with a lot of polls) and Alabama (only one poll since 1st August, but not contested).
In Ohio there were 44 polls in the month preceding the election, with a mean of 51.4% Obama, 48.6% Romney. If the confidence interval each poll is 4%, then the interval for a single poll is 4% / sqrt(44) = 0.6%, which is easily enough to make a confident forecast of an Obama victory in Ohio (my model was > 98% confident in its Ohio forecast).
In Alabama, the last poll was on Aug 16th, and it was 40% Obama, 60% Romney. Even with a margin of error of 4%, this state was clearly going to go to Romney (my model had 99.9% confidence in this forecast).
This pattern is repeated for almost every state - the states with few polls are not contested, and the hotly contested states are a focus for pollsters, so there are a lot of polls there. The only really difficult states were Florida, Colorado and North Carolina, where the candidates polled so similarly that even with 30-40 polls you didn't have sufficient sample size to make strong forecasts.
In Ohio there were 44 polls in the month preceding the election, with a mean of 51.4% Obama, 48.6% Romney. If the confidence interval each poll is 4%, then the interval for a single poll is 4% / sqrt(44) = 0.6%, which is easily enough to make a confident forecast of an Obama victory in Ohio (my model was > 98% confident in its Ohio forecast).
In Alabama, the last poll was on Aug 16th, and it was 40% Obama, 60% Romney. Even with a margin of error of 4%, this state was clearly going to go to Romney (my model had 99.9% confidence in this forecast).
This pattern is repeated for almost every state - the states with few polls are not contested, and the hotly contested states are a focus for pollsters, so there are a lot of polls there. The only really difficult states were Florida, Colorado and North Carolina, where the candidates polled so similarly that even with 30-40 polls you didn't have sufficient sample size to make strong forecasts.
It doesn't work like that. Polls are not independent. If poll A is out by a large amount, then it's much more likely that poll B is also out. As well, the confidence interval assumes that the sample was a fair sample, which is highly unlikely. It's very very difficult to get a true random sample of the voting populace. It's hard enough to get a random sample of the populace, and a random sample of those who vote is even harder.
That's why Nate Silver said that Romney had about a 10% chance of winning. Statistically it was much lower, but there was a very real chance of systematic polling errors.
That's why Nate Silver said that Romney had about a 10% chance of winning. Statistically it was much lower, but there was a very real chance of systematic polling errors.
Polls are not independent. If poll A is out by a large amount, then it's much more likely that poll B is also out.
That's a bias issue - I was only addressing sampling variation. If you wnat your model to address bias as well, of course you can build that in, given some priors on what the distribution of the bias is likely to be. I didn't bother in my model (which is why I was forecasting 98-99% chance of an Obama victory).
We could discuss potential poll bias as well, but I thought that was a bit too much for this short comment.
That's a bias issue - I was only addressing sampling variation. If you wnat your model to address bias as well, of course you can build that in, given some priors on what the distribution of the bias is likely to be. I didn't bother in my model (which is why I was forecasting 98-99% chance of an Obama victory).
We could discuss potential poll bias as well, but I thought that was a bit too much for this short comment.
Each poll's reported result is also an average.
I was keen to bet on Intrade, but the ToS and such are long and confusing, and it wasn't clear how easy it is to get one's money out after a win. What was your experience like?
I actually made more taking directional bets on BetFair, which generally had narrower spreads and more liquidity.
I made a some money buying Obama on InTrade and selling on BetFair, at a time when Obama was at 65% on one and 75% on the other, but I didn't start doing it until Monday because it was a lot of faff to set up the InTrade account.
I haven't tried to get my money out yet. I'll let you know how that goes.
I made a some money buying Obama on InTrade and selling on BetFair, at a time when Obama was at 65% on one and 75% on the other, but I didn't start doing it until Monday because it was a lot of faff to set up the InTrade account.
I haven't tried to get my money out yet. I'll let you know how that goes.
Props.
Putting just over $2000 on Obama several days ago would've gotten you $800.
I don't think that his writing gets enough credit -- explaining what is happening to generate the predictions is really useful. Pretty much everyone that builds models from the state level polls seems to have made more or less the same predictions, but I've found his discussion to be the most interesting and informative (I may have missed some other good sources though).
the hard work that silver did wasn't in being able to predict the election 2 days in advance, any middle schooler with a calculator could have done that this time around. the work that he did was in being able to forecast the results months in advance, adjusting for a broader variety of inputs, including historical research.
That's fair, but I'd point out two things:
1. Even though any middle schooler with a calculator could have predicted the election 2 days ago, it seems that not many of them did - at least, if you judge based on the odds for a Democrat win you could get on InTrade and BetFair 2 days ago (65% and 75% respectively).
2. One neat thing about my model is that, in principle you can run it in "historical" mode and see what its predictions would have been at any given point in the past. So I can see what probability of a DEM win I would have assigned 2 months ago, and see if that's significantly different from the forecast from a more complex model.
1. Even though any middle schooler with a calculator could have predicted the election 2 days ago, it seems that not many of them did - at least, if you judge based on the odds for a Democrat win you could get on InTrade and BetFair 2 days ago (65% and 75% respectively).
2. One neat thing about my model is that, in principle you can run it in "historical" mode and see what its predictions would have been at any given point in the past. So I can see what probability of a DEM win I would have assigned 2 months ago, and see if that's significantly different from the forecast from a more complex model.
Yes
But what people are saying is not "Nate Silver's predictions 1 month before are spot on". They are saying "his predictions ON THE DAY OF ELECTION were spot on".
Which is not that hard to do. Does not mean Nate Silver is a hack. Means people are not measuring his success appropriately.
#NateSilver2016
But what people are saying is not "Nate Silver's predictions 1 month before are spot on". They are saying "his predictions ON THE DAY OF ELECTION were spot on".
Which is not that hard to do. Does not mean Nate Silver is a hack. Means people are not measuring his success appropriately.
#NateSilver2016
It's one thing to hold the bottle, quite another to aim the funnel from 10ft away.
Yes it's quite easy to write off the simplicity after the fact. Kudos.
I'd be interested to see how your model works for Senate races. That seems to be the bigger challenge.
House races seem to be far to data sparse to be reliable.
House races seem to be far to data sparse to be reliable.
Maybe I'll get around to adding a forecast for the senate/house as well. Although any forecasts I make would be somewhat less compelling now that the results are out...
Nate Silver's calculations are certainly more sophisticated that a simple time-weighted average of legitimate polling numbers, but are they more accurate?
His baseball projection system, PECOTA, was extremely complex, but barely, barely outperformed a simple 3-year weighted average with an age component (Marcel), and some years was worse. Other projection systems that didn't take player comps into account were better overall than PECOTA (CHONE being the best, before the creator was hired away by a team).
His baseball projection system, PECOTA, was extremely complex, but barely, barely outperformed a simple 3-year weighted average with an age component (Marcel), and some years was worse. Other projection systems that didn't take player comps into account were better overall than PECOTA (CHONE being the best, before the creator was hired away by a team).
The difference? Sample size. In baseball, everyone has the same data. It's all the data available. There's no sampling going on; it's a projection from past performance. Which, in many ways, is way more difficult.
His advantage in 538 is that he is using very sound statistical methods to combine multiple polls to increase the effective sample size and shrink confidence intervals. The complexities there are all related to increasing the individual pollster's historic performance and simulating the actual election.
His advantage in 538 is that he is using very sound statistical methods to combine multiple polls to increase the effective sample size and shrink confidence intervals. The complexities there are all related to increasing the individual pollster's historic performance and simulating the actual election.
I agree with that, but that's part of why I'm not convinced of his overall model's necessity: I think he gets the vast majority of his predictive power simply by being a good poll aggregator, who lays a reasonable statistical model over the polls.
His model has a bunch of other stuff too, though: estimates of how current jobs numbers effect the election, "state fundamentals" based on demographic variables fit to historical election data, etc. I'm not sure these things are adding much predictive power, especially given that they're generally fit to sparse, heterogeneous, and noisy data sources (e.g. the 11 Presidential elections 1968-2008).
On the other hand, one difficulty in evaluating it is that a good portion of the work his model is doing is not aimed at giving the point estimate, but the distribution and probabilities, to give an accurate probabilistic picture of the current state of the election at any given point, integrating various available data and historical trends. The goal, which is sensible, is to take a more rigorous approach to numbers you hear thrown around like, "when an incumbent is up by x% y weeks before the election, they win z% of the time". But even now in retrospect we can't easily evaluate how good those distributions were: we can see how well his point estimate fared, but with only one instance of how the election turned out, can't evaluate his overall picture's accuracy.
His model has a bunch of other stuff too, though: estimates of how current jobs numbers effect the election, "state fundamentals" based on demographic variables fit to historical election data, etc. I'm not sure these things are adding much predictive power, especially given that they're generally fit to sparse, heterogeneous, and noisy data sources (e.g. the 11 Presidential elections 1968-2008).
On the other hand, one difficulty in evaluating it is that a good portion of the work his model is doing is not aimed at giving the point estimate, but the distribution and probabilities, to give an accurate probabilistic picture of the current state of the election at any given point, integrating various available data and historical trends. The goal, which is sensible, is to take a more rigorous approach to numbers you hear thrown around like, "when an incumbent is up by x% y weeks before the election, they win z% of the time". But even now in retrospect we can't easily evaluate how good those distributions were: we can see how well his point estimate fared, but with only one instance of how the election turned out, can't evaluate his overall picture's accuracy.
I'm not sure his model adds much in itself, but...
I think he adds tremendous value in his write-ups. He explains what he's doing in pretty good detail, and provides some insight into what he believes it means. If you're seriously into it, you'll find places you disagree with him (same as it works in any other field when people are into it), but even if you're just casually interested you'll come away with some additional understanding. Or at the very least some additional questions of your own.
And he manages to write it in such a way that general audiences can pick up some of it, and come away with the impression that there's a lot more there. Not saying it's not a true impression, just that the fact that they get the impression at all is what matters.
I think he adds tremendous value in his write-ups. He explains what he's doing in pretty good detail, and provides some insight into what he believes it means. If you're seriously into it, you'll find places you disagree with him (same as it works in any other field when people are into it), but even if you're just casually interested you'll come away with some additional understanding. Or at the very least some additional questions of your own.
And he manages to write it in such a way that general audiences can pick up some of it, and come away with the impression that there's a lot more there. Not saying it's not a true impression, just that the fact that they get the impression at all is what matters.
I believe Nate Silver is, by far, the best popularizer of statistical thinking in the U.S.
> to combine multiple polls to increase the effective sample size and shrink confidence intervals
How does this work? I was under the impression that to shrink the confidence interval by half you need 4x as many sample points, but there must be more sophisticated statistical methods at work here.
How does this work? I was under the impression that to shrink the confidence interval by half you need 4x as many sample points, but there must be more sophisticated statistical methods at work here.
His models have a memory (increase sample size by using a handful of day or week old data) and aggregate as many as 5 to 8 independent polls per state plus as many as a dozen national polls that factor into his simulations. That's easily four to sixteen times the sample size.
You can improve effective sample size by eliminating extraneous variables. One thing that Nate does is take advantage of implicit biases (for example, PPP polls lean slightly left)
PPP seems to have predicted 332 EV and correctly picked the winner in every state they polled. They didn't poll some of the obvious states...
And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%.
I haven't looked into Nate Silver's numbers, and certainly there would still be a reason for supporting him even if Romney had won tonight, but I think what most people are missing is that it's relatively straightforward to get pretty close to accurate, but incredibly difficult if not impossible to be completely accurate.
The predictions, as they get better, are going to asymptotically approach the actual results.
I haven't looked into Nate Silver's numbers, and certainly there would still be a reason for supporting him even if Romney had won tonight, but I think what most people are missing is that it's relatively straightforward to get pretty close to accurate, but incredibly difficult if not impossible to be completely accurate.
The predictions, as they get better, are going to asymptotically approach the actual results.
And Netflix was willing to pay $1m to someone who could beat their own algorithm by 10%
Yet at the end they decided not to implement it http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-...
Yet at the end they decided not to implement it http://www.forbes.com/sites/ryanholiday/2012/04/16/what-the-...
Ugh I hate it when people make this claim or link this article. If you follow one more article link deep, they say implemented some of the earlier code that came out of it, code that provided the real business value consistently.
Do you know what caused a team to win?
They Realized (1) people rate movies higher on specific days ( why would you want to implement that into your estimator? And (2) they realized that all the movies that Netflix was asking about we're movies the user was willing to rate, and because users tended to rate movies they like, this biased the results, so basically, they didn't implement the hacking the test
Do you know what caused a team to win?
They Realized (1) people rate movies higher on specific days ( why would you want to implement that into your estimator? And (2) they realized that all the movies that Netflix was asking about we're movies the user was willing to rate, and because users tended to rate movies they like, this biased the results, so basically, they didn't implement the hacking the test
They may not have implemented it, but they now know a) what's possible, b) one possible way to achieve the goal and c) how much effort is involved. I think the article misses the point as research often yields an answer that isn't a winning business proposition (at least at the moment). I'd love to know whether there were any actions taken based on the contest.
True, but other articles have said the main reason is that their business changed. The contest was based on DVD rentals, now they emphasize streaming, and that changes viewing patterns...people skip around, sample several movies before settling on one, watch their favorite parts, etc.
And that they took some of the learnings during the contest and saw some decent gains.
For those interested in why and how, there's also http://blog.echen.me/2011/10/24/winning-the-netflix-prize-a-...
They didn't implement it in the end? Doofuses (or doofii).
I think I speak for everyone when I say thank god politics is not like baseball :)
But seriously, congratulations to Nate Silver. I think this vindicates his method, at least for America. He was badly wrong with the last British election, whether because of bad polling data or an insufficient understanding of the system, so I was apprehensive going into this one. But it seems like, for America at least, he's cracked it. I think it's inevitable considering the absurd quantity of polling data available that someone would eventually.
But seriously, congratulations to Nate Silver. I think this vindicates his method, at least for America. He was badly wrong with the last British election, whether because of bad polling data or an insufficient understanding of the system, so I was apprehensive going into this one. But it seems like, for America at least, he's cracked it. I think it's inevitable considering the absurd quantity of polling data available that someone would eventually.
It shows how important it is to have experience and know your domain when you build a model and how understanding your subject matter is more important than cleverness.
Mr. Silver is one of the most clever people we have on the public stage, but that doesn't mean he has a silver bullet for everything. Rather, it enables him to work in a field that he knows very well and knock it out of the park.
Mr. Silver is one of the most clever people we have on the public stage, but that doesn't mean he has a silver bullet for everything. Rather, it enables him to work in a field that he knows very well and knock it out of the park.
2010 British Election (Number of seats)
Silver predicted: Conservative 312, Labour 204, Libdem 103 Actual result : Conservative 307, Labour 258, Libdem 57
Silver predicted: Conservative 312, Labour 204, Libdem 103 Actual result : Conservative 307, Labour 258, Libdem 57
This may have been because a) Clegg's performance in the first debate led to a lot of polls where LD support was high, but ultimately this support melted away by the election and the history bias kept polls wrong until the last minute, or b) because there were lots of Polly Toynbees telling pollsters "I'm voting lib dem" to make sure they were in with a chance in the coverage, but actually intending to vote Labour (she actually advocated this tactic of lying to pollsters in the Guardian.)
Either way, it shows that statistical averages of what people say they'll do isn't always a good indicator of what they'll actually do when push comes to shove.
Disclaimer: as a (now rather embarrassed) Lib Dem, I'd quite like these predictions to be more accurate so I don't get my hopes up again.
Either way, it shows that statistical averages of what people say they'll do isn't always a good indicator of what they'll actually do when push comes to shove.
Disclaimer: as a (now rather embarrassed) Lib Dem, I'd quite like these predictions to be more accurate so I don't get my hopes up again.
I think your explanation is reasonable, but that's no excuse.
His US model makes a lot of allowances for things like how self-proclaimed independents are likely to vote.
It didn't work as well in the UK though.
His US model makes a lot of allowances for things like how self-proclaimed independents are likely to vote.
It didn't work as well in the UK though.
I have no idea about the political culture in the UK. Is it possible that he just didn't have as much data to work with? In other words, a lot fewer relevant polls to feed into a model?
we don't generally have very many regional polls, and certainly not constituency polls, that are available for aggregators (an internal poll in a key marginal, perhaps.) There's a lot less money spent on UK elections.
Indeed, I don't disagree. I'm just saying I think it may well be impossible to be completely accurate, or even remotely accurate, in crazy elections.
103 for the libdems! that's one of the funniests things i've seen for ages.
how did he get it so wrong?
how did he get it so wrong?
In reading his analysis of the election here, he repeatedly mentions that having only two candidates makes things much more predictable.
In any case, we don't know if he got it "wrong" unless we know what % chance he gave this outcome. (He tries to account for the likelihood that polls are misleadingly biased, and that would correlate across multiple seats.)
In any case, we don't know if he got it "wrong" unless we know what % chance he gave this outcome. (He tries to account for the likelihood that polls are misleadingly biased, and that would correlate across multiple seats.)
Because lots of people lie to pollsters or change their mind based on the last thing they read in the paper?
>> I think I speak for everyone when I say thank god politics is not like baseball :)
politics is not nearly as important. :-)
politics is not nearly as important. :-)
Revenge of the Nerd. Sweet, sweet revenge :)
wasn't he badly wrong in the 2010 midterms? I'm not convinced this means he cracked anything. RCP uses simple average of polls, and they got 50/51 correct with Florida being the only wrong if it goes dem (which it looks like but hasn't been called).
Wes Colley, who runs one of the college football BCS ranking models, also did an electoral model. His model is much simpler, essentially taking the median poll result in each state.
http://www.colleyrankings.com/election2012/
Their predictions were basically the same as Sliver's, missing only on Florida (they manually broke a tie in the model on Virginia so count that however you'd like).
http://www.colleyrankings.com/election2012/
Their predictions were basically the same as Sliver's, missing only on Florida (they manually broke a tie in the model on Virginia so count that however you'd like).
If you take all the simple weighted average with an X component and then selectively choose X after the fact, you can beat a lot of things.
I've been reading Fooled by randomness , so i would ask , what proportion of his forecasting record can be attributed to just plain luck ?
It's looking like the Real Clear Politics Poll average is going to be 49/50.
It was 48/50 in 2008 and 49/50 in 2004.
It was 48/50 in 2008 and 49/50 in 2004.
Silver's model does seem unjustifiably complicated. For a simple model that managed to predict the electoral college see
http://election.princeton.edu
Are evaluating baseball players and poll results even similar? One has a binary result, the other a spectrum. One is physically and mentally extremely difficult, the other a simple vote.
Maybe he got better.
So, Nate Silver seems to deserve the even higher profile he'll have after this election, but his model is pretty explicitly not predicting the outcomes of any individual states. They're giving estimated probabilities of the various outcomes, and the (presumptive) outcome that happened was given a probability of ~20%, which was more likely than any other particular outcome, but a far cry from "calling the states."
The big difference is that if this were to happen over and over again ("this" meaning that the most likely predicted outcome actually materializes), that would be (weak) evidence against his model -- if the model's right, he should "correctly call" the states only about 20% of the time. No more and no less.
But good for him, because he's had a lot of unjustified criticism for the last month or so, so he's due some (slightly) unjustified praise.
The big difference is that if this were to happen over and over again ("this" meaning that the most likely predicted outcome actually materializes), that would be (weak) evidence against his model -- if the model's right, he should "correctly call" the states only about 20% of the time. No more and no less.
But good for him, because he's had a lot of unjustified criticism for the last month or so, so he's due some (slightly) unjustified praise.
Huh?
http://fivethirtyeight.blogs.nytimes.com/
"State-by-State Probabilities"
http://fivethirtyeight.blogs.nytimes.com/
"State-by-State Probabilities"
Take Florida, for example. He predicted a 50.3% chance of an Obama win. If Romney actually wins it, he's only a little bit wrong - counting him as 0/1 on Florida in this case wouldn't be accurate. If Obama wins it, he's only a little bit right - counting him as 1/1 on Florida here wouldn't be accurate.
It's hard to quantify just how close he was. I give him props because his prediction was that Florida was very close and that turned out to be true no matter which way the state actually ends up going in the final result.
Now take Virginia, which was one of the later states to be called. He had Obama at 79.4% chance to win. But it turned out to be reasonably close -- which makes me wonder if we really had enough data to justify such a large %. It's a highly polled state, so maybe we did and we just came somewhat close to hitting that 20% of the time where all the polling was inaccurate. Or maybe he had Obama's chances too high.
Another way to judge a prediction model like this is to credit him with .794/1 for Virginia, rather that 1/1. For Florida he gets .503 if Obama wins, and .497 if Romney wins. That doesn't capture it perfectly, either, but at least it doesn't give him 0/1 in Florida if Romney wins even though he successfully predicted that the state was super close.
It's hard to quantify just how close he was. I give him props because his prediction was that Florida was very close and that turned out to be true no matter which way the state actually ends up going in the final result.
Now take Virginia, which was one of the later states to be called. He had Obama at 79.4% chance to win. But it turned out to be reasonably close -- which makes me wonder if we really had enough data to justify such a large %. It's a highly polled state, so maybe we did and we just came somewhat close to hitting that 20% of the time where all the polling was inaccurate. Or maybe he had Obama's chances too high.
Another way to judge a prediction model like this is to credit him with .794/1 for Virginia, rather that 1/1. For Florida he gets .503 if Obama wins, and .497 if Romney wins. That doesn't capture it perfectly, either, but at least it doesn't give him 0/1 in Florida if Romney wins even though he successfully predicted that the state was super close.
You are definitely confusing prediction confidence with winning margin.
Nate Silver's model gave Obama a 79% chance to win Virginia, and predicted that the vote share would be Obama 50.7% - 48.7% Romney. He predicted that the election would be close, but with high confidence would favor Obama.
The actual vote share was Obama 50.8% - 47.8% Romney, so if anything Silver's model predicted a narrower outcome than actually happened. Given the available evidence, the model might have underestimated Obama's probability of winning.
Nate Silver's model gave Obama a 79% chance to win Virginia, and predicted that the vote share would be Obama 50.7% - 48.7% Romney. He predicted that the election would be close, but with high confidence would favor Obama.
The actual vote share was Obama 50.8% - 47.8% Romney, so if anything Silver's model predicted a narrower outcome than actually happened. Given the available evidence, the model might have underestimated Obama's probability of winning.
I am definitely not confusing the two. My entire point is you can't look at the colors of Nate's map and the colors of the results map and say he got 50/50 because Nate's map is explicitly predicting some probability of a win in each state. They may be color coded but his prediction in a given state wasn't "Obama wins" or "Romney wins" but was instead "there is an x% chance that Obama wins.". My point relies entirely on my understanding that what was predicted in the big map is a percentage chance, not a result and not a vote share.
I may be communicating poorly but I feel I made that distinction clear in my post where I say "he predicted a 50.3% chance of an Obama win", "he had Obama at 79.4% chance to win", "makes me wonder if we really had enough data to justify such a large %". That last one is ambiguous but in the context it means "such a large % chance to win."
I don't know how to better clarify my post other than to restate that I am confident I am not confusing the two.
My point was simply that when a longshot comes in or is close to coming in, it's possible you were wrong about it being a longshot. It doesn't mean you were, but it's the first place to look for continued analysis.
I didn't know until just a few moments ago that Nate also had his vote share predictions on the blog. I've just compared all of those to the actual results and found only one state to be outside of his given margin of error.
http://pastebin.com/0RB5GRjQ
I may be communicating poorly but I feel I made that distinction clear in my post where I say "he predicted a 50.3% chance of an Obama win", "he had Obama at 79.4% chance to win", "makes me wonder if we really had enough data to justify such a large %". That last one is ambiguous but in the context it means "such a large % chance to win."
I don't know how to better clarify my post other than to restate that I am confident I am not confusing the two.
My point was simply that when a longshot comes in or is close to coming in, it's possible you were wrong about it being a longshot. It doesn't mean you were, but it's the first place to look for continued analysis.
I didn't know until just a few moments ago that Nate also had his vote share predictions on the blog. I've just compared all of those to the actual results and found only one state to be outside of his given margin of error.
http://pastebin.com/0RB5GRjQ
The confusion is probably due to the other commentators assuming you knew about Nate's vote share predictions when you didn't.
You don't think you are confusing "the two", since you thought that Nate only had one set of numbers in the first place.
The other commentators see you using vote share results to question winner probabilities, and assume that you're confusing Nate's winner probabilities with Nate's vote share predictions, since it would make more sense to compare vote share results with the vote share predictions.
You don't think you are confusing "the two", since you thought that Nate only had one set of numbers in the first place.
The other commentators see you using vote share results to question winner probabilities, and assume that you're confusing Nate's winner probabilities with Nate's vote share predictions, since it would make more sense to compare vote share results with the vote share predictions.
I think you're confusing probability of winning with winning margin. His 79.4% number is his estimate of the probability that Obama would win Virginia by any margin. I don't think it's correct to say he's "a little bit right" -- either his prediction is correct, or it isn't. The only way I know to gauge the accuracy of his actual probabilities would be to run the election multiple times -- but there might be a more statistically advanced technique I'm not aware of.
I'm not confusing the two - I'm saying the fact that Florida was so close indicates Nate may have been doing about as good as possible with the data available in Florida, while the fact that Virginia was so close might indicate that Nate might not have done as well as possible with the data there. Neither case is at all conclusive.
It's possible the quality of the data was just different in the two states, due to random factors in sampling or random factors on election day.
And you are right, we couldn't know without running it multiple times with everything the same except these random factors.
It's possible the quality of the data was just different in the two states, due to random factors in sampling or random factors on election day.
And you are right, we couldn't know without running it multiple times with everything the same except these random factors.
Whether the race was close has nothing to do with how likely one side was to win. If there are 100 voters and I know with absolute certainty that 51 of them are die-hard republicans and 49 of them are die-hard democrats, I'm probably justified in saying that the republican candidate has some large probability of winning, 90% or something depending on actuarial probability that some of the republicans die etc. But that's a 90% probability of winning by a 51/49 margin. Alternately if there are 10 republicans, 10 democrats, and 80 independents I may have no idea how the independents will vote so AFAIK it's equally likely that either candidate wins. The margin of victory has nothing to do with the probability of the outcome.
You could reasonably conclude someone has 80% chance to win by using a large amount of slightly lopsided data, or you could reasonably conclude the same thing with a smaller amount of highly lopsided data.
As such there is a relationship involving the probability of the outcome, the quantity of the polling data, the quality of the polling data, and the margins of the polling data.
At the time I wasn't aware that Nate had published his predicted margins. So, seeing he had a high probability for Obama in Virginia, which turned out to be close, I concluded that one of three things was true:
1. Virginia was very heavily polled and there was just a ton of good and consistent data, justifying a prediction of 80%. OR 2. Virginia had a normal amount of good and consistent data, but it was all very lopsided. OR 3. There was an error with Nate's model or with the data.
Had I known that I could have easily accessed his predicted margins I could have easily seen that it was 1. But without knowing that, I was ruling out 2 because it turned out the state was very close.
My point was that it's not easy to just look at Nate's map and score it against the result map. That's a very shallow and fairly uninteresting way to look at it.
My secondary point was that even if you understand that he had predicted some percentage chance to win in each state, it's not even easy to look at that percentage chance and score it against the result map, because, given a prediction like "80% chance Obama wins", there are many different ways he could have arrived at his conclusion, and it would be more accurate to inspect the method by which he arrived at the conclusion compared to the margins in the state.
I've put up a text file comparing the two here: http://pastebin.com/0RB5GRjQ and it turns out the results were within Nate's given interval in 49 of 50 states. It's not the shallow and misleading "50/50" you get by just comparing the color of Nate's map with the results map -- but I think it's much more interesting, more accurate, and more impressive.
As such there is a relationship involving the probability of the outcome, the quantity of the polling data, the quality of the polling data, and the margins of the polling data.
At the time I wasn't aware that Nate had published his predicted margins. So, seeing he had a high probability for Obama in Virginia, which turned out to be close, I concluded that one of three things was true:
1. Virginia was very heavily polled and there was just a ton of good and consistent data, justifying a prediction of 80%. OR 2. Virginia had a normal amount of good and consistent data, but it was all very lopsided. OR 3. There was an error with Nate's model or with the data.
Had I known that I could have easily accessed his predicted margins I could have easily seen that it was 1. But without knowing that, I was ruling out 2 because it turned out the state was very close.
My point was that it's not easy to just look at Nate's map and score it against the result map. That's a very shallow and fairly uninteresting way to look at it.
My secondary point was that even if you understand that he had predicted some percentage chance to win in each state, it's not even easy to look at that percentage chance and score it against the result map, because, given a prediction like "80% chance Obama wins", there are many different ways he could have arrived at his conclusion, and it would be more accurate to inspect the method by which he arrived at the conclusion compared to the margins in the state.
I've put up a text file comparing the two here: http://pastebin.com/0RB5GRjQ and it turns out the results were within Nate's given interval in 49 of 50 states. It's not the shallow and misleading "50/50" you get by just comparing the color of Nate's map with the results map -- but I think it's much more interesting, more accurate, and more impressive.
It's not clear whether you mean that link as a rejoinder or as a supporting argument for the parent. My take is that it supports the parent.
> It's hard to quantify just how close he was.
It is hard to do with the final, "chance of winning" numbers, but it is much easier with his share of the vote predictions, and you can check those predictions for all states that way, even the ones where the winner was certain.
Say Nate predicts that candidate A will get 80% of the votes in a particular state, with a margin of error of 2%. Then, after the results are out, if that candidate got 81% of the votes, then it was a good prediction, if they got 90% of the votes, it was a bad prediction.
To quantify that, you want to compare the actual error with the predicted error. There are better ways than this, but I'll make one up on the spot: score = 1 / (actual error percentage points * predicted error percentage points). In the first case, the score would be 1 / (1 * 2) = 0.5. In the second, 1 / (10 * 2) = 0.05. The bigger the score, the better the prediction. (This isn't a great model, since it rates a prediction with a large margin of error which happened to be bang on higher than a roughly right prediction with a narrow predicted error, but this is the general idea.)
It is hard to do with the final, "chance of winning" numbers, but it is much easier with his share of the vote predictions, and you can check those predictions for all states that way, even the ones where the winner was certain.
Say Nate predicts that candidate A will get 80% of the votes in a particular state, with a margin of error of 2%. Then, after the results are out, if that candidate got 81% of the votes, then it was a good prediction, if they got 90% of the votes, it was a bad prediction.
To quantify that, you want to compare the actual error with the predicted error. There are better ways than this, but I'll make one up on the spot: score = 1 / (actual error percentage points * predicted error percentage points). In the first case, the score would be 1 / (1 * 2) = 0.5. In the second, 1 / (10 * 2) = 0.05. The bigger the score, the better the prediction. (This isn't a great model, since it rates a prediction with a large margin of error which happened to be bang on higher than a roughly right prediction with a narrow predicted error, but this is the general idea.)
Excellent point. In that case he only had one state outside his given margin of error in West Virginia.
I didn't do any error analysis aside from asking if it was within his margin of error, and all but one were. Hawaii was also exactly on the line.
http://pastebin.com/0RB5GRjQ
I didn't do any error analysis aside from asking if it was within his margin of error, and all but one were. Hawaii was also exactly on the line.
http://pastebin.com/0RB5GRjQ
[deleted]
If Silver said there was a 60% chance that each of 10 states would go to Obama, and they all went to Obama, by Silver's own standards he messed up. Only 6 of them should have gone to Obama.
Since he had so many that were much less than 99%, saying "he got them all right" means he didn't get it right by his own definition.
Since he had so many that were much less than 99%, saying "he got them all right" means he didn't get it right by his own definition.
Given 9 swing states, that gives 512 different possibilities. Nate ranked these possibilities. The actual result (assuming Florida ends up democratic) was his top pick, with a 20% probability. (Unfortunately, this site no longer appears to be available in its previous form).
In 2008, he only got 49/50 of the states right. Yet pundits still call this a victory for Nate.
To judge the accuracy of silver's prognostications, take the top 5 or so of his "most probable scenarios", which added to gether give a probability way above 50%. Did one of these scenarios manifest? Yes, his top scenario manifested. His second most probable scenario included a Republican Florida, which still might happen.
So yes, Nate nailed it.
In 2008, he only got 49/50 of the states right. Yet pundits still call this a victory for Nate.
To judge the accuracy of silver's prognostications, take the top 5 or so of his "most probable scenarios", which added to gether give a probability way above 50%. Did one of these scenarios manifest? Yes, his top scenario manifested. His second most probable scenario included a Republican Florida, which still might happen.
So yes, Nate nailed it.
Like I said, I'm happy for him to get credit for nailing it. But I'm pretty sure that conclusion's based on misunderstanding his forecasts. Unless I missed it somewhere, he's never said "VA is going to Obama, IA's going to Obama, NC's going to Romney, etc." If he has, please provide a link.
For analogy, let's say I have a biased coin. I tell you that I'm pretty sure it comes up heads 75% of the time. Then we flip it and it comes up heads. Did I "nail it?" No. The coin coming up heads is stronger evidence in favor my "model" (P[heads] = 0.75) than if it had come up tails, but it's pretty far from conclusive. The logic doesn't change if I tell you that the probability of heads is 90%, 99.9%, or whatever. (but, if I say it's 99.9% and it comes up tails, I'll concede that the model's wrong. If the prediction is extreme enough, a single observation can invalidate the model).
To extend the analogy, we only know if I have a good model for this biased coin if we flip it a bunch of times and roughly 3/4 of the outcomes are heads. If we can't do that, because the coin flip is a one-off event (stretching the analogy quite a bit, but whatever), we still could tell if I'm good at modeling coin flips: if you give me a bunch of different coins, I give you a probability of heads for each one, and then we see whether the proportion of heads after we flip each one is close to the average of the probabilities I gave (i.e., let B(i) be 1 if the ith coin is heads, 0 if it's tails. Then n^{-1/2} (B(1) - Prob[coin 1 is heads] + B(2) - Prob[coin 2 is heads] + ... + B(n) - Prob[coin n is heads]) obeys a CLT and -- if the probabilities are right -- it becomes normal with mean zero and known variance as n gets large).
If I tell you that each probability is .75 or some other number greater than 0.5 but less than 1, and all the coins always come up heads than I'm not "nailing it." But, I want to emphasize, it's premature to say that that's what's going on with Silver's predictions. And from what I've read of his blog, he completely understands this and explains it well.
edit: minor change for clarity in the first and third paragraphs.
For analogy, let's say I have a biased coin. I tell you that I'm pretty sure it comes up heads 75% of the time. Then we flip it and it comes up heads. Did I "nail it?" No. The coin coming up heads is stronger evidence in favor my "model" (P[heads] = 0.75) than if it had come up tails, but it's pretty far from conclusive. The logic doesn't change if I tell you that the probability of heads is 90%, 99.9%, or whatever. (but, if I say it's 99.9% and it comes up tails, I'll concede that the model's wrong. If the prediction is extreme enough, a single observation can invalidate the model).
To extend the analogy, we only know if I have a good model for this biased coin if we flip it a bunch of times and roughly 3/4 of the outcomes are heads. If we can't do that, because the coin flip is a one-off event (stretching the analogy quite a bit, but whatever), we still could tell if I'm good at modeling coin flips: if you give me a bunch of different coins, I give you a probability of heads for each one, and then we see whether the proportion of heads after we flip each one is close to the average of the probabilities I gave (i.e., let B(i) be 1 if the ith coin is heads, 0 if it's tails. Then n^{-1/2} (B(1) - Prob[coin 1 is heads] + B(2) - Prob[coin 2 is heads] + ... + B(n) - Prob[coin n is heads]) obeys a CLT and -- if the probabilities are right -- it becomes normal with mean zero and known variance as n gets large).
If I tell you that each probability is .75 or some other number greater than 0.5 but less than 1, and all the coins always come up heads than I'm not "nailing it." But, I want to emphasize, it's premature to say that that's what's going on with Silver's predictions. And from what I've read of his blog, he completely understands this and explains it well.
edit: minor change for clarity in the first and third paragraphs.
I think there's a minor error in the analogy of "state results" to "coin flip".
In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting.
This doesn't entirely invalidate your point (yes, if you "re-ran" the election repeatedly and he "nailed it" everytime we might conclude his probabilities were incorrect), but it does explain why if 10 "60% Obaba" states vote we might not necessarily expect 6 of them to come up for Obama...
In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting.
This doesn't entirely invalidate your point (yes, if you "re-ran" the election repeatedly and he "nailed it" everytime we might conclude his probabilities were incorrect), but it does explain why if 10 "60% Obaba" states vote we might not necessarily expect 6 of them to come up for Obama...
> In the coin flip scenario each throw of the biased coins are independent. How one lands does not effect the other. The same is likely not true for how states end-up voting.
For sure. If I have a point, it's that there's not enough information in any election to claim that he "nailed it." Since the outcome he said was most likely seems to have occurred, it definitely supports his model and his approach. But you'd need to see a longer track record than exists to be sure. (think of each coin flip as being a separate national election).
Independence is a little bit of a red herring, though. There are LLNs and CLTs that allow for weak dependence between the observations, and any strong/systematic dependence between the states belongs in the model (and I think is in Silver's model, but I could be misremembering). And we would expect that 6 of 10 "60% Obama" states would come up for Obama. The interdependence is going to affect the variance but not the mean. So we should expect to see (say) 8 or more of 10 going to Obama happen more frequently than a naive binomial(.6, 10) distribution would predict.
For sure. If I have a point, it's that there's not enough information in any election to claim that he "nailed it." Since the outcome he said was most likely seems to have occurred, it definitely supports his model and his approach. But you'd need to see a longer track record than exists to be sure. (think of each coin flip as being a separate national election).
Independence is a little bit of a red herring, though. There are LLNs and CLTs that allow for weak dependence between the observations, and any strong/systematic dependence between the states belongs in the model (and I think is in Silver's model, but I could be misremembering). And we would expect that 6 of 10 "60% Obama" states would come up for Obama. The interdependence is going to affect the variance but not the mean. So we should expect to see (say) 8 or more of 10 going to Obama happen more frequently than a naive binomial(.6, 10) distribution would predict.
No, he never said "VA is going to Obama, ...". He said "the probability of VA going to Obama and IA going to Obama and NC going to Romney ..." is X%.
This used to be at the "Paths to Victory" on the NYT, but they've been collapsed now.
We can say that somebody "nails" a prediction through both specificity and repetition. Nate had a modest amount of both. 50/50 and 49/50 are pretty specific. He did that well in 2008 and 2012, and similarly in 2010.
This used to be at the "Paths to Victory" on the NYT, but they've been collapsed now.
We can say that somebody "nails" a prediction through both specificity and repetition. Nate had a modest amount of both. 50/50 and 49/50 are pretty specific. He did that well in 2008 and 2012, and similarly in 2010.
The OP said, "Nate Silver correctly predicted every single state" which is mistaken. His model is not designed to make predictions like "50/50" and "49/50" and he never claims to make those predictions, so I honestly don't know what you mean by, 'we can say that somebody "nails" a prediction through both specificity and repetition. Nate had a modest amount of both.'
For every state he listed which direction he thought it would go, and an associated probability.
Every state went the direction that he thought it would go.
Every state went the direction that he thought it would go.
No, the uncertainty is correlated. Each individual state might have a 60% chance for Obama but all of them will tend to vote the same way.
You are assuming that they are all independent events. What if in the calculation to get to 60% the main risk factor was a late national swing rather than a local polling error?
Do any of them equal 1? Saying that >50% probability means that the model picks the state for Obama would be fine as a classification rule, but he's trying to estimate the probabilities, not forecast who will win.
The ~20% comes from the electoral vote distribution, where you can back out the mix of states that gives 332 Electoral college votes.
The ~20% comes from the electoral vote distribution, where you can back out the mix of states that gives 332 Electoral college votes.
I've got to say that I've been a cynical admirer of Silver since the last election.
Why cynical? Because I began to notice throughout the Republican nomination process that his observations varied wildly on which Republican candidate presented great values on intrade. I think over time he gave a nod to every candidate, save Michelle Bachmann, Tim Pawlenty, and John Huntsmann. From the outset, Romney was the favorite to win, and not surprisingly, he won. But throughout the nomination process where much less was known, many of his projections proved to be wrong. So despite all of the contrarian opinion on which candidates may pull it out against popular opinion (or perception) these faded away until Romney was the clear favorite, Silver predicted him to win, and eventually he carried the nomination.
Once in the Presidential election, Obama was the favorite. Incumbent Presidents rarely lose, and Mitt Romney was not exactly a compelling candidate.
All that said, he's clearly a brilliant dude, and I give him a lot of respect for the accuracy of his predictions. I just haven't found them to be hugely useful for any practical purposes. I guess people tend to be enamored by the things they don't understand (statistical analysis), and I don't think there's anything wrong with that.
Why cynical? Because I began to notice throughout the Republican nomination process that his observations varied wildly on which Republican candidate presented great values on intrade. I think over time he gave a nod to every candidate, save Michelle Bachmann, Tim Pawlenty, and John Huntsmann. From the outset, Romney was the favorite to win, and not surprisingly, he won. But throughout the nomination process where much less was known, many of his projections proved to be wrong. So despite all of the contrarian opinion on which candidates may pull it out against popular opinion (or perception) these faded away until Romney was the clear favorite, Silver predicted him to win, and eventually he carried the nomination.
Once in the Presidential election, Obama was the favorite. Incumbent Presidents rarely lose, and Mitt Romney was not exactly a compelling candidate.
All that said, he's clearly a brilliant dude, and I give him a lot of respect for the accuracy of his predictions. I just haven't found them to be hugely useful for any practical purposes. I guess people tend to be enamored by the things they don't understand (statistical analysis), and I don't think there's anything wrong with that.
I'm not from or in the US but can I ask a slightly off-topic question?
Is Romney really the best the Republican's can offer? It seemed from here (UK) that he was not a compelling candidate from the start but most the others in the nomination process were massively flawed in some way or other and/or their political views were just too far from the mainstream to really fly.
Romney might still have won if Obama had made a mess of something in the last days before the election as a couple of percent swing would have changed everything but it just seems like the Republicans should be able to do better.
Two questions:
1) Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)
2) If not is there a structural problem in the Republican party that means extreme views (or massive personal cash) is required to reach the position to become a candidate and it somehow weeds out the compelling candidates?
Is Romney really the best the Republican's can offer? It seemed from here (UK) that he was not a compelling candidate from the start but most the others in the nomination process were massively flawed in some way or other and/or their political views were just too far from the mainstream to really fly.
Romney might still have won if Obama had made a mess of something in the last days before the election as a couple of percent swing would have changed everything but it just seems like the Republicans should be able to do better.
Two questions:
1) Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)
2) If not is there a structural problem in the Republican party that means extreme views (or massive personal cash) is required to reach the position to become a candidate and it somehow weeds out the compelling candidates?
As a liberal American, my thesis is that the Republican party is extremely fractured between fiscal conservatives (who are largely centrist or even liberal on social issues) and the straight-up social conservatives. Which means that finding a single individual who can coherently represent all of them is basically impossible.
So enter Romney, who seemed willing to be mercurial - literally adopting different positions in front of different audiences and scoffing whenever anybody pointed it out. But that cynical strategy was necessary to have any viability whatsoever in such a diverse party.
So enter Romney, who seemed willing to be mercurial - literally adopting different positions in front of different audiences and scoffing whenever anybody pointed it out. But that cynical strategy was necessary to have any viability whatsoever in such a diverse party.
>"Which means that finding a single individual who can coherently represent all of them is basically impossible."
And by the time the Republican nomination was over, the strength of the social conservatives had dragged the rhetoric so far right that it was difficult for Romney to get back to the centre for the election.
And by the time the Republican nomination was over, the strength of the social conservatives had dragged the rhetoric so far right that it was difficult for Romney to get back to the centre for the election.
These two comments very completely and eloquently sum up the problem. The divide between social and fiscal conservatism is something that gives the republican party the biggest disadvantage.
I should also mention that the big-L Libertarians, as instantiated in the U.S., are not the fiscally-conservative socially-liberal party that centrists would even entertain. They're far too radical to be at all politically viable (abolish Fed and Education, open borders, legalize drugs, etc.).
[not american] my impression is that it is the latter - that it is hard to appeal consistently to both the party (to become the chosen candidate) and to the nation. hence the accusations of flip-flopping and the arguments from "reasonable republicans" that romney would actually have been quite good even though he had to say various odd things.
i wonder if they will attempt to change how candidates are selected.
[above i have tried to be impartial; a more personal take is that the republican party has become accustomed to lying to its supporters and is now trapped with the results. to have to rely on people trusting that their candidate is "bluffing" does not seem healthy.]
---
more on topic: the results reporting really disappointed me. the take-away from silver's work is not that this is very complex. it's that the electoral college system plus most people voting the same way over time means that only a few results are uncertain. so focussing on those places allows you to predict the election. similarly, reporting all the other (known) data is irrelevant. it's not news, and it's not exciting.
yet on election night we saw every result being treated equally. there was no acknowledgement of the reality. it looked exactly like an entertainment show.
was there any presentation of the results, anywhere, that ignored most of the data and focussed only on the swing states? that spent the rest of the time analyzing what had happened in the light of the clearly expected, accurately predicted result?
i wonder if they will attempt to change how candidates are selected.
[above i have tried to be impartial; a more personal take is that the republican party has become accustomed to lying to its supporters and is now trapped with the results. to have to rely on people trusting that their candidate is "bluffing" does not seem healthy.]
---
more on topic: the results reporting really disappointed me. the take-away from silver's work is not that this is very complex. it's that the electoral college system plus most people voting the same way over time means that only a few results are uncertain. so focussing on those places allows you to predict the election. similarly, reporting all the other (known) data is irrelevant. it's not news, and it's not exciting.
yet on election night we saw every result being treated equally. there was no acknowledgement of the reality. it looked exactly like an entertainment show.
was there any presentation of the results, anywhere, that ignored most of the data and focussed only on the swing states? that spent the rest of the time analyzing what had happened in the light of the clearly expected, accurately predicted result?
I watched PBS's coverage and I would say they did. They showed every state very briefly but talked about nothing but the battlegrounds.
thanks. i admit i didn't look there (on the web; can't get direct tv).
Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)
Yes. In this election John Huntsman would have had a good chance of winning the election by carrying the moderates and picking up a few Democrat voters (although he probably would never have got enough Republican voters). In many ways Rick Perry was a better candidate than Romney (in terms of who could win the election), but he screwed up in the debate and I think the US is a little wary of another Texas Republican.
If not is there a structural problem in the Republican party that means extreme views (or massive personal cash) is required to reach the position to become a candidate and it somehow weeds out the compelling candidates?
Sort of. To win the primary you need to carry either the Christian conservatives or the business wing of the Republican party. In this election the Christian conservatives vote was split many ways which left Romney.
Romney's big problem was that he didn't appeal to Republicans enough for them to vote for him. He was sort of a Republican version of John Kerry - someone they thought moderates could vote for (which was kind of, somewhat true - he wasn't 100% insane anyway), but he didn't really appeal to his own party that much.
Yes. In this election John Huntsman would have had a good chance of winning the election by carrying the moderates and picking up a few Democrat voters (although he probably would never have got enough Republican voters). In many ways Rick Perry was a better candidate than Romney (in terms of who could win the election), but he screwed up in the debate and I think the US is a little wary of another Texas Republican.
If not is there a structural problem in the Republican party that means extreme views (or massive personal cash) is required to reach the position to become a candidate and it somehow weeds out the compelling candidates?
Sort of. To win the primary you need to carry either the Christian conservatives or the business wing of the Republican party. In this election the Christian conservatives vote was split many ways which left Romney.
Romney's big problem was that he didn't appeal to Republicans enough for them to vote for him. He was sort of a Republican version of John Kerry - someone they thought moderates could vote for (which was kind of, somewhat true - he wasn't 100% insane anyway), but he didn't really appeal to his own party that much.
I'll agree on Huntsman nationally, but he never stood a chance in the Republican party. He's too close to center (like pre-2008 McCain) and he didn't satisfy the minority, but influential, extreme right.
There are plenty of good Republican, slightly-right leaning legislators - senators, representatives, governors - but the party has been hijacked by the extreme right.
As much as they like to talk about being the party of Reagan, Reagan would be chastised now for working with a Democrat Congress. Reagan was good friends with Tip O'Neil (Speaker of the House, from Massachusetts, liberal) and nowadays that wouldn't sit well with the party.
My father, my father-in-law, and my mother-in-law were all Republicans. After Bush II they all voted Democrat. They'd return if there was some sanity.
There are plenty of good Republican, slightly-right leaning legislators - senators, representatives, governors - but the party has been hijacked by the extreme right.
As much as they like to talk about being the party of Reagan, Reagan would be chastised now for working with a Democrat Congress. Reagan was good friends with Tip O'Neil (Speaker of the House, from Massachusetts, liberal) and nowadays that wouldn't sit well with the party.
My father, my father-in-law, and my mother-in-law were all Republicans. After Bush II they all voted Democrat. They'd return if there was some sanity.
Better candidates may have decided to wait out this time - battling against a incumbent is always harder.
Plus whoever won would still have to continue cleaning up Bush's mess while not ticking off the Teabaggers too much - an impossible task.
There are a couple of halfway non-insane Republicans who are sharpening their blades for 2106 - I'd say Mitch Daniels and Chris Christie are likely contenders who have some credibility. Well, Daniels doesn't much with me - I'm from Indiana and I can't say he's been a stellar governor.
My opinion from the start is that the saner Republicans wisely decided to sit this one out and let the crazies blow off some steam. Even Palin sat it out - she's morally atrocious but, I think, no dummy. We'll see her in 2016 for sure.
There are a couple of halfway non-insane Republicans who are sharpening their blades for 2106 - I'd say Mitch Daniels and Chris Christie are likely contenders who have some credibility. Well, Daniels doesn't much with me - I'm from Indiana and I can't say he's been a stellar governor.
My opinion from the start is that the saner Republicans wisely decided to sit this one out and let the crazies blow off some steam. Even Palin sat it out - she's morally atrocious but, I think, no dummy. We'll see her in 2016 for sure.
Can we just stop with the terms libtards and teabaggers? It's childish and disrepectful.
I don't much care for the term teabagger, but it is their own coinage. see here: http://theweek.com/article/index/202620/the-evolution-of-the... and think back to the many pictures of people suspending tea bags from hats etc., until they became aware of the slang meaning of the term. While they've tried to drop it and call themselves Tea Partiers, they've shown little inclination to temper their criticisms of others with civility or moderation, so far. Perhaps this electoral defeat will prompt some soul-searching.
Dude, I'm from Indiana and it's what they call themselves. I know what the term means to you people from cities, but come on, is that even real? Why?
I think within recent context, this is no longer the case. I'm not a member of the tea party movement, but I'm pretty sure it's seen as a derogatory term these days.
> 1) Do the Republicans have people who would make compelling candidates in the Presidential election? (Who?)
Lots. Governor of New Jersey Chris Christie was often mentioned. The candidates who stepped in the ring for the Republican party were second stringers who saw their chance. The A Players were either not available or too smart to step in the ring. (The economy was wrecked under the Republican watch and Obama was in the process of fixing it.)
Lots. Governor of New Jersey Chris Christie was often mentioned. The candidates who stepped in the ring for the Republican party were second stringers who saw their chance. The A Players were either not available or too smart to step in the ring. (The economy was wrecked under the Republican watch and Obama was in the process of fixing it.)
The GOP fielded a horrible panel of candidates; all were very flawed and Romney was the most likely out of any of them to win against the President. They also hamstrung each other during the primary campaign in ways too numerous to list. This gave the Democratic Party a huge advantage since the President didn't have to undergo much scrutiny during this period. Incumbency is really hard to overcome.
1. There are numerous candidates who might prove compelling in 2016. I don't think Christie belongs in that group since he's being viewed as a pariah after the tropical storm. Rubio would satisfy the more conservative elements plus potentially bring in the Latino vote.
2. I don't think the GOP has huge structural issues, but I'll grant that the Democratic Party is much more unified.
The GOP also has the nickname as the stupid party. Things like Todd Akin's comments (and Mourdock's) tainted the entire party. Had they repudiated both immediately and stuck with it even in the face of losing those two Senate seats (which they lost anyways) they would have done better in attracting women voters. Instead, the comments reinforced the Democratic meme that the GOP hated women.
1. There are numerous candidates who might prove compelling in 2016. I don't think Christie belongs in that group since he's being viewed as a pariah after the tropical storm. Rubio would satisfy the more conservative elements plus potentially bring in the Latino vote.
2. I don't think the GOP has huge structural issues, but I'll grant that the Democratic Party is much more unified.
The GOP also has the nickname as the stupid party. Things like Todd Akin's comments (and Mourdock's) tainted the entire party. Had they repudiated both immediately and stuck with it even in the face of losing those two Senate seats (which they lost anyways) they would have done better in attracting women voters. Instead, the comments reinforced the Democratic meme that the GOP hated women.
The Akin fiasco was modern Republican egotism and political incompetence in a nutshell. To start with the primary: Steelman is a serial officeholder too scared to challenge the incumbent Democrat Nixon for governor, who ran basically because she thought it was her turn in the Senate. Brunner is a rich dude who enjoys buying things. Akin is an "evangelical" Christian who visits lots of different churches. Also there were various cranks.
If there were a core group of republican leaders in this state (rather than just a bunch of tired old emeriti angling for lobbyist gigs), they would have recognized that the evangelical vote was going to Akin. It's not as though he's an unknown quantity; he's been representing the same dozy safely-gerrymandered suburbs of St. Louis for decades. So they would know what a tool he is, and they would have known they had to get all the moderates voting for the same candidate in order not to be saddled with Akin in the general election.
Which, since they have no way of influencing Brunner, would mean that Steelman would have been urged to step aside. She could have stepped into some committee appointment, or kept lobbying, or whatever, but she would have been taken care of by the establishment. Next time around, it could have been her "turn" again.
Nothing like that occurred of course. Perhaps they thought McCaskill could have been beaten by anyone, and they wanted to prove it by nominating the dumbest guy they could find. If so they've been listening to Rush too much. McCaskill is not the pwog extremist they'd like to paint her as. As she bragged in her ads, she got the perfectly moderate score of "50" among Senators. If she had been running in 2010, Akin might have had a chance even with his rude comments, but in the quadrennial general election you can't just take the piss. Obama has coattails in St. Louis and Kansas City.
McCaskill isn't as dumb as we all thought either. She actually ran ads during the Republican primary supporting Akin as the "True Conservative Choice". She did so openly, and all the Bible-thumpers thought she was trying to trick them with reverse psychology. I guess her checkerboard has more dimensions than theirs.
And then within a week after the primary Akin spouted all that voodoo-illiterate-snake-handler let-me-tell-you-about-legitimate-rape nonsense. Why on earth is any Republican candidate ever allowed to answer a question about rape? Why do they ever utter anything besides "rape is terrible, and I'm against it 100%"? There are no upsides to answering that question!
At that point the national Republican machine noticed what a hash of things their chums in Missouri had made. I'm sure they tried to buy off Akin, much as the state folks should have tried to buy off Steelman earlier. In Akin's case, the offer would have had to have been much sweeter, not least because he's much less rational: a national post in a potential Romney administration plus the promise of really sweet lobby dollars for him (they could have hit up Brunner for some of this) while he stayed in the House and an office on K Street after he left it. I'm not sure if they didn't offer a big enough bribe, or if Akin was too dumb and prideful (pride is ever the evangelicals' favorite deadly sin) not to take it, but the deal didn't get done.
It didn't help that all the local Republican bigwigs, even those not up for reelection, immediately jumped on the airwaves to deplore Akin as soon as they possibly could. You can trick, buy, or even (rarely) reason with an evangelical, but you can't bully one. As soon as the rustics in the Ozarks heard that fancypants Danforth was trying to overrule the voters' choice for nominee, the phones in every political office in the state lit up. At that point it was too easy for Akin to fool himself. After all, he's a believer.
So, this is the Republican Party. I still vote for them occasionally, and if I lived in a Blue state I probably would do so more often. Mostly I'm really bummed Johnson only got 1% this time around.
If there were a core group of republican leaders in this state (rather than just a bunch of tired old emeriti angling for lobbyist gigs), they would have recognized that the evangelical vote was going to Akin. It's not as though he's an unknown quantity; he's been representing the same dozy safely-gerrymandered suburbs of St. Louis for decades. So they would know what a tool he is, and they would have known they had to get all the moderates voting for the same candidate in order not to be saddled with Akin in the general election.
Which, since they have no way of influencing Brunner, would mean that Steelman would have been urged to step aside. She could have stepped into some committee appointment, or kept lobbying, or whatever, but she would have been taken care of by the establishment. Next time around, it could have been her "turn" again.
Nothing like that occurred of course. Perhaps they thought McCaskill could have been beaten by anyone, and they wanted to prove it by nominating the dumbest guy they could find. If so they've been listening to Rush too much. McCaskill is not the pwog extremist they'd like to paint her as. As she bragged in her ads, she got the perfectly moderate score of "50" among Senators. If she had been running in 2010, Akin might have had a chance even with his rude comments, but in the quadrennial general election you can't just take the piss. Obama has coattails in St. Louis and Kansas City.
McCaskill isn't as dumb as we all thought either. She actually ran ads during the Republican primary supporting Akin as the "True Conservative Choice". She did so openly, and all the Bible-thumpers thought she was trying to trick them with reverse psychology. I guess her checkerboard has more dimensions than theirs.
And then within a week after the primary Akin spouted all that voodoo-illiterate-snake-handler let-me-tell-you-about-legitimate-rape nonsense. Why on earth is any Republican candidate ever allowed to answer a question about rape? Why do they ever utter anything besides "rape is terrible, and I'm against it 100%"? There are no upsides to answering that question!
At that point the national Republican machine noticed what a hash of things their chums in Missouri had made. I'm sure they tried to buy off Akin, much as the state folks should have tried to buy off Steelman earlier. In Akin's case, the offer would have had to have been much sweeter, not least because he's much less rational: a national post in a potential Romney administration plus the promise of really sweet lobby dollars for him (they could have hit up Brunner for some of this) while he stayed in the House and an office on K Street after he left it. I'm not sure if they didn't offer a big enough bribe, or if Akin was too dumb and prideful (pride is ever the evangelicals' favorite deadly sin) not to take it, but the deal didn't get done.
It didn't help that all the local Republican bigwigs, even those not up for reelection, immediately jumped on the airwaves to deplore Akin as soon as they possibly could. You can trick, buy, or even (rarely) reason with an evangelical, but you can't bully one. As soon as the rustics in the Ozarks heard that fancypants Danforth was trying to overrule the voters' choice for nominee, the phones in every political office in the state lit up. At that point it was too easy for Akin to fool himself. After all, he's a believer.
So, this is the Republican Party. I still vote for them occasionally, and if I lived in a Blue state I probably would do so more often. Mostly I'm really bummed Johnson only got 1% this time around.
> Mostly I'm really bummed Johnson only got 1% this time around.
Third party candidates don't stand a chance in a FPTP system, because a vote for Johnson is, in practical terms, a vote against Romney (in swing states, that is; the idiocy of the electoral college ensures that it really doesn't matter in states like MO). You either need to have a proportional voting system for Congress so that third parties can gain a foothold in the public conscious, or you need IRV, à la Australia, so people don't feel like their vote's being wasted.
Third party candidates don't stand a chance in a FPTP system, because a vote for Johnson is, in practical terms, a vote against Romney (in swing states, that is; the idiocy of the electoral college ensures that it really doesn't matter in states like MO). You either need to have a proportional voting system for Congress so that third parties can gain a foothold in the public conscious, or you need IRV, à la Australia, so people don't feel like their vote's being wasted.
"don't stand a chance"
I didn't actually expect Johnson to win. The majority of voters don't support peace and freedom at this time, so I didn't expect them to vote for those things this time. However, I had hoped that a better showing by Johnson would prompt more voters to at least consider the possibility of peace and freedom, and that that consideration could inform their votes next time.
"the idiocy of the electoral college"
One might think the case for this proposition is obvious, but it's not. The electoral college effectively prevents having a recount in every state and every precinct any time the popular vote is close. Only when the electoral count is close and the popular vote of some state is also close will we have a recount, with all of its inherent confusion and lawsuits. I'm glad we have an institution that prevents recounts in most elections. What other institution would you propose, or do you see no problem with recounts?
I didn't actually expect Johnson to win. The majority of voters don't support peace and freedom at this time, so I didn't expect them to vote for those things this time. However, I had hoped that a better showing by Johnson would prompt more voters to at least consider the possibility of peace and freedom, and that that consideration could inform their votes next time.
"the idiocy of the electoral college"
One might think the case for this proposition is obvious, but it's not. The electoral college effectively prevents having a recount in every state and every precinct any time the popular vote is close. Only when the electoral count is close and the popular vote of some state is also close will we have a recount, with all of its inherent confusion and lawsuits. I'm glad we have an institution that prevents recounts in most elections. What other institution would you propose, or do you see no problem with recounts?
At least he was weeded out in the election. For all the faults of our system, it does a decent if imperfect job of trying to keep the loonies out.
1) No. Just look at the first RNC primary debate videos. These are the Republicans rocket scientists.
2) Right pundits are already arguing that Romney wasn't extreme enough. This is not a condition they can get out of. All future candidates will have to meet socialcon and Koch requirements before even thinking about swing or blue states.
2) Right pundits are already arguing that Romney wasn't extreme enough. This is not a condition they can get out of. All future candidates will have to meet socialcon and Koch requirements before even thinking about swing or blue states.
In the last election, the Republicans ran John McCain†, who is actually pretty much dead center in the context of US politics. It seems to me they ran Romney this time because 1) Romney has pretty strong economic credentials and Obama hasn't been outstanding in that department, and 2) going centrist didn't help last time. Also, like others said, the best candidates are probably waiting until next time, because Obama hasn't messed up badly enough for his supporters to turn on him en masse, so this year's candidate was likely to be a sacrificial lamb.
† McCain's running mate, Sarah Palin, was nutjob who seemed to be there purely to appeal to the demographics he didn't, but the man himself has historically been pretty even-handed.
† McCain's running mate, Sarah Palin, was nutjob who seemed to be there purely to appeal to the demographics he didn't, but the man himself has historically been pretty even-handed.
I, in general, like McCain, but even though I am slightly right of center, I was disgusted by the things he had to do and say to get the nomination. During primary season he played the part of far right fairly well.
I thought Romney was an excellent candidate, but his campaign was rather hamstrung by the Republican party.
Based on media and social media I have seen (a very distorting view I know) there has been little sign of enthusiasm such as yours except at his campaign rallies. From the Republican supporters there seemed to be much more negativity about Obama than positivity for Romney.
Do you think your view is widely shared? Can you say what you like about him?
Do you think your view is widely shared? Can you say what you like about him?
>I just haven't found them to be hugely useful for any practical purposes.
His models are dry, boring even. But they're based in some semblance of data. When compared to the vast majority of the pundit class, and political journalism in general, his model is a breath of fresh air if you want to understand the state of the campaigns beyond what some pundit heard from "a source" or some journalist asked of a random campaign worker.
His models are dry, boring even. But they're based in some semblance of data. When compared to the vast majority of the pundit class, and political journalism in general, his model is a breath of fresh air if you want to understand the state of the campaigns beyond what some pundit heard from "a source" or some journalist asked of a random campaign worker.
He makes mention of that in his book. He uses the example of online poker players, which he used to be. He doesn't consider himself great at that either, just better than the hordes of rank idiots that played during the online poker bubble before the feds shut him down. One they were chased away by the various lawsuits, he lost a great deal more of games.
He also says that effort/optimization falls under the Pareto principle, where the last 20% of your optimizations is about 80% of your work, so it's a lot harder to get from being pretty good to world class. But any model that takes into account prior data will get you a hell of a lot of the way there.
He also says that effort/optimization falls under the Pareto principle, where the last 20% of your optimizations is about 80% of your work, so it's a lot harder to get from being pretty good to world class. But any model that takes into account prior data will get you a hell of a lot of the way there.
There were a couple notable projections that he was considerably off on, particularly some Santorum primary numbers late in the game. But, even someone with a projected 98% chance of winning should lose 2% of the time. In the aggregate, Silver's projections are excellent. I would like to see something like a Brier score for his projections, and compare it to various polling organizations, and other analytical pundits (do any notable ones exist?) to see who has done the best. On a related note, it would be nice to have a place where people could publicly lock in projections, and develop a record. Maybe that would help put the Joe Scarboroughs of the world out of work.
If it helps, http://marginoferror.org/2012/11/08/aggregating-the-aggregat... reports a RMSE for just his state-level margin of victory.
But as part of a project I was aggregating predictions as well, and for his state wins I got 0.02833, for his Senate-level predictions I got a Brier of 0.04484545, and for his Presidency win & Senate wins & state wins, I got a Brier of 0.03710118.
Hope this helps!
But as part of a project I was aggregating predictions as well, and for his state wins I got 0.02833, for his Senate-level predictions I got a Brier of 0.04484545, and for his Presidency win & Senate wins & state wins, I got a Brier of 0.03710118.
Hope this helps!
The project, specifically, was http://appliedrationality.org/2012/11/09/was-nate-silver-the... / http://news.ycombinator.com/item?id=4760649
Nate Silver claims to be a statistician, not a clairvoyant.
While there is a sense of inevitability of Romney's nomination, I think it is obvious that the Republican voters agree with you that he is not a compelling candidate and were desperately looking for alternatives, and as each candidate wilted in the spotlight, they kept moving on to the next, and polling numbers reflected that, and Nate Silver was merely reporting that shift.
While there is a sense of inevitability of Romney's nomination, I think it is obvious that the Republican voters agree with you that he is not a compelling candidate and were desperately looking for alternatives, and as each candidate wilted in the spotlight, they kept moving on to the next, and polling numbers reflected that, and Nate Silver was merely reporting that shift.
So is that at all at odds with what I said?
I'm saying that I haven't found his predictions to be useful because they have not correctly predicted anything that I myself didn't predict with the same information without any statistical analysis. Sure, he predicted it with 3% greater accuracy. So what?
Sadly what makes most elections predictable is how shitty the candidates are on one side.
To be fair I don't think that a statistical analysis will ever be useful in an election unless the election is absolutely a dead heat. Most elections for the past several years (with the obvious exception of 2000) have been fairly predictable from the outset.
I'm saying that I haven't found his predictions to be useful because they have not correctly predicted anything that I myself didn't predict with the same information without any statistical analysis. Sure, he predicted it with 3% greater accuracy. So what?
Sadly what makes most elections predictable is how shitty the candidates are on one side.
To be fair I don't think that a statistical analysis will ever be useful in an election unless the election is absolutely a dead heat. Most elections for the past several years (with the obvious exception of 2000) have been fairly predictable from the outset.
In the primaries there are more candidates and fewer polls so you get sparser measurements of the state of a given candidate's appeal. So maybe it's natural for the predictions to have more variation at that stage.
Hard cold numbers won again.
The venom directed at Nate Silver before the election was astonishing. I guess the pundits had sensed their careers of predication were numbered and fought back hard, but Nate was proved right again. Kudos.
The venom directed at Nate Silver before the election was astonishing. I guess the pundits had sensed their careers of predication were numbered and fought back hard, but Nate was proved right again. Kudos.
Perhaps this will lead to a greater interest in statistics and data in general politics and less in pundit opinions. Haha, who am I kidding...
There will always be a market for uninformed jabbering. Pundits wouldn't worry if they looked at their own record.
Is there a "pundit prediction tracker" anywhere? Somewhere that tracks the predictions made by talking heads and shows how often they're right or wrong?
> uninformed jabbering.
And for informed jabbering. It's frustrating that analysis is being replaced with opinion, especially when it's clear that opinion driven prediction is garbage.
> uninformed jabbering.
And for informed jabbering. It's frustrating that analysis is being replaced with opinion, especially when it's clear that opinion driven prediction is garbage.
I found a couple of things that look useful at a glance (I googled for pundit prediction tracker):
http://www.pundittracker.com/
http://nymag.com/daily/intel/2012/11/exhaustive-collection-o...
I remember hearing about Pundit Tracker before it launched, but I didn't look into it after that.
http://www.pundittracker.com/
http://nymag.com/daily/intel/2012/11/exhaustive-collection-o...
I remember hearing about Pundit Tracker before it launched, but I didn't look into it after that.
I've been a huge fan of Nate Silver since forever, and I must say, I really regret his "acquirement" by NY Times. When it was his own website, it was a much more user-friendly experience.
For instance, though I have an NYT account, every other page view I get redirected to a login prompt if I don't press Esc fast enough (this page has expired? WTF?). It also made him more of an independent, though he is (falsely) discounted on many counts by other parts of media, his affiliation with NYT is yet another facet for them to criticize.
For instance, though I have an NYT account, every other page view I get redirected to a login prompt if I don't press Esc fast enough (this page has expired? WTF?). It also made him more of an independent, though he is (falsely) discounted on many counts by other parts of media, his affiliation with NYT is yet another facet for them to criticize.
Disagree with this. NYT brought a much better user experience and much nicer data presentation, and leant him a feeling of legitimacy too. There's a reason the NYT has the reputation it does.
I disagree. Here's an archive from post-2008 election: http://web.archive.org/web/20081106113055/http://www.fivethi...
The old page contains more information yet looks friendlier. The nytimes style is more streamlined, but it became too sanitized. The content became sanitized too; Nate Silver could be more himself when it was his blog. He gave personal accounts, he was more honest, etc.
The old page contains more information yet looks friendlier. The nytimes style is more streamlined, but it became too sanitized. The content became sanitized too; Nate Silver could be more himself when it was his blog. He gave personal accounts, he was more honest, etc.
I think the answer here is that it was better for Nate to join NYT- I imagine he was compensated well for the move. We can complain about it, but.. well, it isn't in our hands.
I pay for an NYTimes subscription because they provide content above and beyond the background noise of the top social sharing websites like Reddit.
I'd probably pay for 538 alone depending on the price asked.
I'd probably pay for 538 alone depending on the price asked.
Try browsing the NYT incognito - it seems to avoid the login prompts.
It's just a lease, he can leave next year (and take the 538 brand with him).
Doesn't that destroy the allegations of voting fraud in the US? (Not a US-citizen here)
If there was widespread fraud with the voting machines as alleged elsewhere, the voting outcome should have diverged widely from the predictions. The predictions are so close to the actual outcome, and because from what I understand the predictions are mostly based on polling, there couldn't have been much fraud, or am I wrong?
Flippant edit: Assuming of course that Nate Silver wasn't in on the fraud and didn't adjust his predictions accordingly.
If there was widespread fraud with the voting machines as alleged elsewhere, the voting outcome should have diverged widely from the predictions. The predictions are so close to the actual outcome, and because from what I understand the predictions are mostly based on polling, there couldn't have been much fraud, or am I wrong?
Flippant edit: Assuming of course that Nate Silver wasn't in on the fraud and didn't adjust his predictions accordingly.
Not on it's own, no. At most, is shows that fraud wasn't widespread enough (or equally widespread on both sides) to skew the election away from the will of the people.
Also, fraud is much more effective in smaller races - of which there are lot, many of them too small to have as good poll coverage as the presidential one - so if there is fraud, those races are probably where to look for it.
Also, fraud is much more effective in smaller races - of which there are lot, many of them too small to have as good poll coverage as the presidential one - so if there is fraud, those races are probably where to look for it.
In 2004 there was evidence of voting fraud in Ohio -- the exit polls weren't matching the results (http://whatreallyhappened.com/WRHARTICLES/2004votefraud_ohio...).
Bush won Ohio, which helped him win the election by the narrowest of margins. Ohio used Diebold electronic voting machines, and Diebold's CEO Walden W. O'Dell was a long-time Bush loyalist. Later it was confirmed that Diebold voting machines could be hacked remotely (http://news.ycombinator.com/item?id=3045086).
So yes, being able to create an independent, accurate model will add another layer of verification and will hopefully help keep voting fraud in check.
Bush won Ohio, which helped him win the election by the narrowest of margins. Ohio used Diebold electronic voting machines, and Diebold's CEO Walden W. O'Dell was a long-time Bush loyalist. Later it was confirmed that Diebold voting machines could be hacked remotely (http://news.ycombinator.com/item?id=3045086).
So yes, being able to create an independent, accurate model will add another layer of verification and will hopefully help keep voting fraud in check.
Alternatively, it just suggests that people who engage in voter fraud tend to suck at it.
Nate Silver seems like a pretty smart guy, wouldn't it be better if he spent his time doing something more productive? Predicting the outcome of an election may have practical applications in gambling or for a hedge fund, and yes he gets mad publicity and attention from women sure, but aren't there more useful ways to apply statistics?
EDIT: Haldean, your point is excellent and cancels out mine absolutely. Good thinking, you are entirely right. I would add that his attention and publicity itself should increase the credibility of rational analysis in the news, and that alone would be a great accomplishment.
EDIT: Haldean, your point is excellent and cancels out mine absolutely. Good thinking, you are entirely right. I would add that his attention and publicity itself should increase the credibility of rational analysis in the news, and that alone would be a great accomplishment.
He has utterly discredited the horse race scam of big media political coverage. That is a significant contribution, because such coverage has been so influential in managing public opinion and how people think about politics.
He did it first and more dramatically in the 2008 primaries, but this time feels more like the watershed.
It's not all Nate's doing, of course; this is a long term trend and many people are working on it.
He did it first and more dramatically in the 2008 primaries, but this time feels more like the watershed.
It's not all Nate's doing, of course; this is a long term trend and many people are working on it.
Imagine, too, a scenario where the results deviated significantly from his predictions. In the past, the public, guided by the media, might have been satisfied shrugging it off as "god's will" or "people are just unpredictable" or even "pshaw, more conspiracy theories!" Now I like to believe there will be more pressure to investigate and rationally explain the source of that deviation. Are dead people really voting in Chicago? Were voting machines hacked in Cuyahoga County? I feel like Silver's counter-narrative -- and more critically, his methods -- have been an important contribution to a fair and open democratic process.
The idea that this was "a close race", "too close to call", was a narrative pushed by the media to make it more artificially exciting to drive viewers.
Pundits hate him for it because it shows how content-free, and ideological/ opinion-driven their arguments are rather than sound judgement based on available data.
Pundits hate him for it because it shows how content-free, and ideological/ opinion-driven their arguments are rather than sound judgement based on available data.
Now here's a bet I'm willing to take.
The horse race coverage of the next political race will be just as strong as it was this time. This isn't the triumph of data over narrative, it's just data showing it can be more predictive.
Narrative still has more mass popularity, and it always will.
The horse race coverage of the next political race will be just as strong as it was this time. This isn't the triumph of data over narrative, it's just data showing it can be more predictive.
Narrative still has more mass popularity, and it always will.
Hmm, I'm not sure how my point got turned into a claim that narrative will henceforth have less mass popularity than data. I'm talking about a shift among elites. Political media are going to be forced to adapt, in a direction I think most people here would consider a good one. If they don't, they will look ridiculous, which will weaken them even faster; appearing serious and important is their trump card, after all.
But tell me how you propose to measure "just as strong" and I might take your bet :)
But tell me how you propose to measure "just as strong" and I might take your bet :)
You've got to read the answers above concerning the UK 2008 election... Just for the record about 'horse race' 'discredited' and 'big media'...
This is beginning to look to me like 'big stats'.
This is beginning to look to me like 'big stats'.
I looked for and read what I think are the comments you mean. He got the 2008 British election wrong, yes? I'm inclined to cut him some slack about that. The two systems are so different that domain expertise in one might not translate well to the other, and it would be a classic geek mistake to underestimate the differences at first.
Instead of thinking of him as a statistician who works for a newspaper, think of him as a statistically-minded journalist. I wish _more_ people did what he did: apply scientific reasoning to national and global issues.
To his credit, he was wise enough to publish a book at just the time when people want to hear from him most. The downside of being the king of political prediction is that you're only popular once every four years. But you're really really popular, and he seems to be aware of and capitalizing on that.
It's really every two years since he was one of the first that showed Republicans picking up big gains in the US House and governorships in 2010.
There is room in this world for every corner of productivity. He's doing some very good work; other people are doing very good work in exactly the other fields you're talking about. One is not inherently better than the other, since it takes many different outcomes and works to create this thing we call society.
In fact, you could say that the maximal way to be productive might be to be diversely productive, since it produces the widest array of new ideas and conclusions, and tests every nook and cranny of possibility. Surely we have enough people in the world and in the country to provide for that diversity.
In fact, you could say that the maximal way to be productive might be to be diversely productive, since it produces the widest array of new ideas and conclusions, and tests every nook and cranny of possibility. Surely we have enough people in the world and in the country to provide for that diversity.
Psh I think it's awesome and he's making statistics hawt right now. Gotta find your niche in something to make a name for yourself, anyway.
> attention from women
Ha, well, that's got to be annoying for him: he's gay.
Ha, well, that's got to be annoying for him: he's gay.
He began his career predicting baseball scores, so I guess that's more useful?
Ahem, he would be a warlock.
Proprietor of that site here.
Nope, “witch” can apply to men just as well as women and always has. In fact, you’ll find several dictionaries citing Malory’s “Le Morte d’Arthur” (1485) as an early usage, where Merlin is referred to by some as a “wytche”.
It’s also the standard academic term across gender. If memory serves, it’s used this way in the groundbreaking study “Witchcraft, Oracles and Magic Among the Azande” by E. E. Pritchard, which set the standard for the anthropology of witchcraft.
It’s also used for a man in the television show “Futurama”, for a more contemporary colloquial reference.
Nope, “witch” can apply to men just as well as women and always has. In fact, you’ll find several dictionaries citing Malory’s “Le Morte d’Arthur” (1485) as an early usage, where Merlin is referred to by some as a “wytche”.
It’s also the standard academic term across gender. If memory serves, it’s used this way in the groundbreaking study “Witchcraft, Oracles and Magic Among the Azande” by E. E. Pritchard, which set the standard for the anthropology of witchcraft.
It’s also used for a man in the television show “Futurama”, for a more contemporary colloquial reference.
He should have the right to self-identify as a witch if he wants. You have no right to take that away from him.
Self identification is a ridiculous concept if it's not based on fact. They put people in the nut house for proclaiming they're a farm animal or Teddy Roosevelt.
Transphobia in the software industry? Shocking
Phobia? Hardly. You don't call an ftp client and ssh client do you? Likewise you would and should never call a man a woman, unless you're tying to insult him and vice versa. It's called logic and not some ridiculous concept of accepting a mental disorder as something to be celebrated.
"it's called logic"
Actually, It's called not knowing what the fuck you're talking about
The logical thing to do would be to educate yourself.
P.S. trans people's brains objectively show non expected neural activation. And this is just one of the many ways for people to understand trans. It's a real thing and you need to EDUCATE YOURSELF instead of acting like bill oreilly. Ignorant asshole.
Actually, It's called not knowing what the fuck you're talking about
The logical thing to do would be to educate yourself.
P.S. trans people's brains objectively show non expected neural activation. And this is just one of the many ways for people to understand trans. It's a real thing and you need to EDUCATE YOURSELF instead of acting like bill oreilly. Ignorant asshole.
It's what's called a defect such as when a system doesn't perform as expected. Sorry. No amount of emotionally supported bullshit is going to change facts.
You came out from under the bridge for that?
No. Males are witches as well, Warlock translates to oath breaker, and is not considered with the pagan community (at least the parts with which I have interacted) as a male term for witch.
Or witch doctor. And he'd be in good company.
http://www.businessinsider.com/a-kenyan-witch-doctor-predict...
http://www.businessinsider.com/a-kenyan-witch-doctor-predict...
Or wizard, if we're in the Harry Potter universe.
Keep in mind, however, that we could not have expected Nate Silver's own model to predict 50/50 states. Given that Nate predicted that there was a 50.3% chance that Florida would go to Obama, the only difference between this being a story of predicting 50/50 states and this being a story of missing at least one state is a coin toss. The reasons his model turned out to be 100% accurate is sheer luck.
Doesn't that mean he undercalibrated some of his predictions?
If you say 50 events should happen with a 60% probability, and they all happen, shouldn't you have upped your confidence?
That said, since most his naysayers were saying he was OVERconfident at the time, kudos to him for proving them wrong.
If you say 50 events should happen with a 60% probability, and they all happen, shouldn't you have upped your confidence?
That said, since most his naysayers were saying he was OVERconfident at the time, kudos to him for proving them wrong.
If the predictions are independent, then I agree with you. I can think of reason why they would not be independent (trying to account for non-random polling samples, for example).
They aren't independent. The developer of the 512 paths to the White House lamented this in getting conditional probabilities into the path choices. The simulations take into account the national popular vote polls and similar-state demographics (if one state goes one way, it's more likely that a similar one will, too).
Look at his EV histogram; 332 EVs had the highest probability of occurrence at nearly 20%. While there are other ways to get to 332, I'd imagine most of that percentage is from a map like last night. That's clearly not the partial probability of each state multiplied together.
Look at his EV histogram; 332 EVs had the highest probability of occurrence at nearly 20%. While there are other ways to get to 332, I'd imagine most of that percentage is from a map like last night. That's clearly not the partial probability of each state multiplied together.
Such extrapolations are really tough -- keep in mind, for each state we only have results with a sample size of 1.
[deleted]
Nate's Prediction and actual results side by side http://twitter.com/cosentino/status/266042007758200832/photo...
I think it's realatively easy, for a smart guy with a solid understandings of statistics to come up with a reasonable model (weight polls by sample size, come up with probability distribution for each state, run x Monte Carlo simulations and take the average of the result...).
But a thing is coming up with a good model, another is to defend it publicly putting your reputation on the line, while it would have been a much safer bet simply to say 'it's too close to call', as many so-called experts did.
Full credit to Nate, he made math cool for a lot of people.
But a thing is coming up with a good model, another is to defend it publicly putting your reputation on the line, while it would have been a much safer bet simply to say 'it's too close to call', as many so-called experts did.
Full credit to Nate, he made math cool for a lot of people.
Even with an incredibly exact model, to actually get ALL 50 right really does require a lot of luck in addition to skill. Even if he's 98% accurate for each state (which seems amazingly high to me), that's still only a 1 in 3 chance to get all 50 right.
He really only got 7/7 right. The other 43 states were not really in contention.
Once California has its results in, I'm guessing his prediction on the popular vote will make it 51/51.
He predicted that Obama would get 50.8% of the popular vote.
Currently Obama stands at 50.3% of the popular vote. However the majority of the currently uncounted ballots are in Oregon and Washington. Therefore Obama's share is likely to go up a smidge.
Still getting the popular vote to within 0.5% is pretty good. Especially considering that many national polls going in were consistently calling for a Romney win there.
Currently Obama stands at 50.3% of the popular vote. However the majority of the currently uncounted ballots are in Oregon and Washington. Therefore Obama's share is likely to go up a smidge.
Still getting the popular vote to within 0.5% is pretty good. Especially considering that many national polls going in were consistently calling for a Romney win there.
[deleted]
He was down by 200k a few minutes ago, now he's only down by 110k. California has about 20 million more votes to count, if Obama takes them 55/45, should be good enough to take the popular vote.
You're forgetting washington DC
It's a little much to claim that this guy called it. The minor detail of actually conducting the polling was done by others.
The point isn't that Silver is some superhuman guessing machine. The point is that Silver explicitly did nothing particularly special over applying sound statistics over publicly available data, came out with very different predictions than the traditional media did, took a lot of flak from them for it, and turned out to be exactly correct in every meaningful way.
538 is a statement against the traditional election coverage, in effect saying that math is better than pseudo-objective talking head bullshit. From his blog:
Nevertheless, these arguments are potentially more intellectually coherent than the ones that propose that the leader in the race is “too close to call.” It isn’t. If the state polls are right, then Mr. Obama will win the Electoral College. If you can’t acknowledge that after a day when Mr. Obama leads 19 out of 20 swing-state polls, then you should abandon the pretense that your goal is to inform rather than entertain the public.
http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...
This is why the more math-literate portions of the internet have erupted with cheers for Silver.
538 is a statement against the traditional election coverage, in effect saying that math is better than pseudo-objective talking head bullshit. From his blog:
Nevertheless, these arguments are potentially more intellectually coherent than the ones that propose that the leader in the race is “too close to call.” It isn’t. If the state polls are right, then Mr. Obama will win the Electoral College. If you can’t acknowledge that after a day when Mr. Obama leads 19 out of 20 swing-state polls, then you should abandon the pretense that your goal is to inform rather than entertain the public.
http://fivethirtyeight.blogs.nytimes.com/2012/11/03/nov-2-fo...
This is why the more math-literate portions of the internet have erupted with cheers for Silver.
and turned out to be exactly correct in every meaningful way.
Silver himself wouldn't call his predictions correct.
He was not making thumbs-up v thumbs-down predictions. When he said "60% chance of Obama winning this state," it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong.
Silver himself wouldn't call his predictions correct.
He was not making thumbs-up v thumbs-down predictions. When he said "60% chance of Obama winning this state," it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong.
> it means that 6 times out of 10 Obama would win and 4 times out of 10 Romney would win. If Obama won all 10 times, by his own account he would be wrong.
That's a very frequentist perspective. I think the Bayesian interpretation is a little more sensible. Given a prior estimate, updated with the information we have, it is logical to assume that Obama has a better chance of winning.
That is, Nate Silver's 60% doesn't mean that if the election in a given state were repeated, we'd see different results four times out of 10, but rather that his information in making the prediction was incomplete.
That's a very frequentist perspective. I think the Bayesian interpretation is a little more sensible. Given a prior estimate, updated with the information we have, it is logical to assume that Obama has a better chance of winning.
That is, Nate Silver's 60% doesn't mean that if the election in a given state were repeated, we'd see different results four times out of 10, but rather that his information in making the prediction was incomplete.
Nate Silver himself said (in a radio interview that I cannot locate despite much Googling :< ) that if he said something would happen 60% of the time and it was 10 out of 10 that he his figure was wrong and should have said 100%.
But you can't run the same election 10 times. Without hearing the interview, I'm going to guess he's talking about multiple states, all with 60% chance of winning, which puts us in a different place entirely.
I guess I'm not disagreeing with the main point of your post. Just your use of the phrase "something would happen 60% of the time", as being oddly frequentist, when a Bayesian perspective is more appropriate here.
I guess I'm not disagreeing with the main point of your post. Just your use of the phrase "something would happen 60% of the time", as being oddly frequentist, when a Bayesian perspective is more appropriate here.
> But you can't run the same election 10 times.
That makes assessment difficult, but is irrelevant for interpretation. If something happens 60% of the time, it happens with the same probability as drawing a red ball from an urn with 6 red balls and 4 green balls, whether it happens once or several times.
Trying to determine the quality of a model using some observed data is inherently a frequentist exercise -- a dyed-in-the-wool Bayesian would take the election results, use them to update his or her posterior distributions, and carry on happily. (I know that no one would actually do this; no one who analyzes data is really a "pure" Bayesian or a "pure" frequentist).
That makes assessment difficult, but is irrelevant for interpretation. If something happens 60% of the time, it happens with the same probability as drawing a red ball from an urn with 6 red balls and 4 green balls, whether it happens once or several times.
Trying to determine the quality of a model using some observed data is inherently a frequentist exercise -- a dyed-in-the-wool Bayesian would take the election results, use them to update his or her posterior distributions, and carry on happily. (I know that no one would actually do this; no one who analyzes data is really a "pure" Bayesian or a "pure" frequentist).
You're all over this thread with the same comment, and you're wrong.
When an outcome has 60% likelihood of occurring, then it should occur 6 out of 10 times. But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time.
In this case, you describe 10 separate events, each with 60% chance that a certain outcome occurs. So we have 10 separate outcomes that occurred 1 out of 1 times.
When an outcome has 60% likelihood of occurring, then it should occur 6 out of 10 times. But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time.
In this case, you describe 10 separate events, each with 60% chance that a certain outcome occurs. So we have 10 separate outcomes that occurred 1 out of 1 times.
But the election only happens once. The prediction isn't "wrong" when the outcome occurs 1 out of 1 time.
It's not a single event. Nate Silver made many many predictions about last night. And none[1] of them were binary up-down events. If he did make binary predictions, we could claim he got them "all right," but Silver was never doing that in the first place. Nor am I saying he should have.
Try this: if Romney had won, would it mean that Nate Silver was wrong? Should we say, like Silver's critic Dylan Byers did, "it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance"?
No, we shouldn't. Because Silver didn't say "Obama would win." He said "Obama has a ~86% chance of winning." Silver gave a 14% chance of finding himself in a universe where Romney would win despite what his data was telling him. Silver made essentially no prediction about Florida -- and we shouldn't push him into making an up/down decision if he doesn't think he can. But everyone wants to force him into the binary prediction box.
EDIT: took out snark
[1] Okay, he was 100% on some states like New York, but so was everyone else, so those aren't interesting to talk about.
It's not a single event. Nate Silver made many many predictions about last night. And none[1] of them were binary up-down events. If he did make binary predictions, we could claim he got them "all right," but Silver was never doing that in the first place. Nor am I saying he should have.
Try this: if Romney had won, would it mean that Nate Silver was wrong? Should we say, like Silver's critic Dylan Byers did, "it’s difficult to see how people can continue to put faith in the predictions of someone who has never given that candidate anything higher than a 41 percent chance"?
No, we shouldn't. Because Silver didn't say "Obama would win." He said "Obama has a ~86% chance of winning." Silver gave a 14% chance of finding himself in a universe where Romney would win despite what his data was telling him. Silver made essentially no prediction about Florida -- and we shouldn't push him into making an up/down decision if he doesn't think he can. But everyone wants to force him into the binary prediction box.
EDIT: took out snark
[1] Okay, he was 100% on some states like New York, but so was everyone else, so those aren't interesting to talk about.
Ha! I've been thinking about this pretty hard and talking to a colleague at work, and changed my mind. You're right... he's not making predictions, he's making forecasts. And you can't say that a forecast was "right" or "wrong" based on which side of the forecast any one result landed.
Assuming the n isn't just too small to say anything, the only two reasonable conclusions to choose between are a) in this set of events, the outcome in EVERY state happened to be from more likely side of the forecast, which together is unlikely, or b) Nate's forecasts are inaccurate, being skewed much more to the middle than they ought to be.
Also, "all over this thread" was just meant to imply that I noticed the same comment from you more than once. Sorry for not being more specific.
Assuming the n isn't just too small to say anything, the only two reasonable conclusions to choose between are a) in this set of events, the outcome in EVERY state happened to be from more likely side of the forecast, which together is unlikely, or b) Nate's forecasts are inaccurate, being skewed much more to the middle than they ought to be.
Also, "all over this thread" was just meant to imply that I noticed the same comment from you more than once. Sorry for not being more specific.
Ok then, mea culpa. I guess I take it for granted that anyone actually pays attention to talking heads. IMO, they have been completely useless since the OJ trial.
I was looking at the Intrade prediction market a couple of days ago.. I think they got everything correct too.
Most markets got the winner right, which isn't a surprise, but they didn't get the margin right. I found Obama to win 310-330 electoral college votes at $8 on Betfair yesterday (and I put it on, and it looks like I won).
The betting markets did believe that Obama would win, but there was a lot of money on the race being closer and not as wide as Silver was predicting. This is probably because the media narrative of the past two weeks was that the race is close.
There is a rumor that some Republican donors were shortening the odds in the margin betting in order to perpetuate the line that the race is closer. I don't know how true that is, but it would be possible to move the market with a few large bets.
The betting markets did believe that Obama would win, but there was a lot of money on the race being closer and not as wide as Silver was predicting. This is probably because the media narrative of the past two weeks was that the race is close.
There is a rumor that some Republican donors were shortening the odds in the margin betting in order to perpetuate the line that the race is closer. I don't know how true that is, but it would be possible to move the market with a few large bets.
Where did you hear that rumor? I have all but concluded that on my own based on the bizarrely low prices on obama shares on intrade over the past week. It would be very interesting to hear something more than speculation.
I was hanging out and talking to other people who are interested in betting markets online, we were just speculating. We mostly talk about sports betting but the election was such a big topic in betting that you couldn't avoid it and I ended up getting sucked into it.
I should have looked into it further, this has since been posted:
http://news.ycombinator.com/item?id=4756229
I don't have much experience with Intrade, but there must be a way to dump data from the site and investigate the orders and buys being made.
As that post says, putting down $1M to keep the odds more in line with a narrative of the contest being close is small change in the context of political spending today.
I had two people tell me that rumor. One heard it via friends on Wall St and the other read it in another forum, IIRC.
There must be something to explain the spread between Intrade and Betfair, that the market was being manipulated by those with a vested interest makes sense.
I should have looked into it further, this has since been posted:
http://news.ycombinator.com/item?id=4756229
I don't have much experience with Intrade, but there must be a way to dump data from the site and investigate the orders and buys being made.
As that post says, putting down $1M to keep the odds more in line with a narrative of the contest being close is small change in the context of political spending today.
I had two people tell me that rumor. One heard it via friends on Wall St and the other read it in another forum, IIRC.
There must be something to explain the spread between Intrade and Betfair, that the market was being manipulated by those with a vested interest makes sense.
Nope. Florida was predicted as going for Romney up until results starting coming in: http://www.intrade.com/v4/markets/contract/?contractId=74573...
Presumably Silver's projection is "priced in" to Intrade.
[deleted]
I wonder if having such good predictions is ultimately harmful to the democratic process? If I were only voting on the presidential election, I would not have bothered going to the polls because my state was solidly on one side and the chance of my vote being pivotal was basically zero. But some of our local elections were less certain, so I went for those.
As the prediction models get better, get applied to local elections, and get more publicity, I wonder what it will do to people's incentive to vote? I'm much more likely to go out and cast my vote when the outcome is uncertain and I think I have a chance of being the pivotal vote. But if super accurate forecasts tell me my vote won't matter, then maybe I won't bother standing in line for an hour.
As the prediction models get better, get applied to local elections, and get more publicity, I wonder what it will do to people's incentive to vote? I'm much more likely to go out and cast my vote when the outcome is uncertain and I think I have a chance of being the pivotal vote. But if super accurate forecasts tell me my vote won't matter, then maybe I won't bother standing in line for an hour.
When the incentive of the people changes, then the outcome will change as well.
I wonder how much money he could be (is?) making in prediction markets like Intrade.
It's hard to make really significant money on InTrade. I don't think you could make a living at it, for example - there's not enough liquidity.
There were some fun opportunities to arbitrage InTrade against BetFair this election, if you could be bothered with the faff of setting up an account on both of them. InTrade had Obama at 66-66% for a while, whereas BetFair had Obama at 79-80%. You could buy on InTrade and hedge on BetFair, and make a guaranteed profit (even after t-costs).
There were some fun opportunities to arbitrage InTrade against BetFair this election, if you could be bothered with the faff of setting up an account on both of them. InTrade had Obama at 66-66% for a while, whereas BetFair had Obama at 79-80%. You could buy on InTrade and hedge on BetFair, and make a guaranteed profit (even after t-costs).
The presidential market on InTrade was quite liquid. $300k of volume with a bid-ask spread of like 0.2%.
Do you know why there was such a persistent gap for arbitrage between InTrade and BetFair? I don't know about BetFair's vig and rules, but InTrade has only a couple of small fixed transaction costs. ($5/mo to have an account and $10 to pull your money out, I think.) Seems like it should have been eaten up.
Do you know why there was such a persistent gap for arbitrage between InTrade and BetFair? I don't know about BetFair's vig and rules, but InTrade has only a couple of small fixed transaction costs. ($5/mo to have an account and $10 to pull your money out, I think.) Seems like it should have been eaten up.
[deleted]
This remains to be seen. As of this writing, Florida and Virginia have yet to be called.
Here's VA http://electionresults.virginia.gov/resultsSW.aspx?type=PRE&...
Reporting at 100%. 50%/48% in favor of Obama.
Silver affirms that public opinion polling is accurate - as long as you know how to read the data.
Not to diminish Silver's work - I'm just saying the data he uses needs to be accurate in order for his models to be accurate.
Not to diminish Silver's work - I'm just saying the data he uses needs to be accurate in order for his models to be accurate.
Why do I have to enter my google profile information to open this?
It's just a brief G+ post with two links, but if you want to see what it's talking1bout just go here instead: http://fivethirtyeight.blogs.nytimes.com/
quick image reference on Nate's predictions versus actual results: http://cl.ly/image/0744432U0S0L
I'd like to see Nate run the numbers which would point to districts where there were statistical anomalies on the lookout for election funny business.
I'd love for him to go to past elections and see how well our political voting system works...
You might like the economic sub discipline of "Social Choice Theory"
I'd like to see the comparison for -90,-60,-30,-15 days from election too.
Let's be honest though, the number of likely outcomes was low enough that simply picking the correct one isn't too impressive.
Edit: I meant to say that the prediction itself was unspectacular, however I respect the meticulous-ness of his process.
Edit: I meant to say that the prediction itself was unspectacular, however I respect the meticulous-ness of his process.
I don't know what you mean by this.
Are you referring to "Obama wins" as a potential coin flip? Because Silver's prediction was more nuanced than that.
Are you talking about his predictions for the 50 states, which represented a potential (albeit naive) 2^50 combinations? Because he called some very close states, including a Florida election that he forecasted as an Obama victory by hundredths of a percent.
Or are you just trolling?
Are you referring to "Obama wins" as a potential coin flip? Because Silver's prediction was more nuanced than that.
Are you talking about his predictions for the 50 states, which represented a potential (albeit naive) 2^50 combinations? Because he called some very close states, including a Florida election that he forecasted as an Obama victory by hundredths of a percent.
Or are you just trolling?
The only highly uncertain states were Ohio, Virginia, and Florida. (Based on the time before results started coming in). So really there were 9 safe predictions, having picked one of them does not make the OP's story spectacular.
You clearly were not following the polls very closely.
An average of the polls going into the election put 2 of the three that you name as "highly uncertain" at likely polling within 3%. According to the average of polling data at http://www.electoral-vote.com/evp2012/Pres/Maps/Nov06.html all of Colorado, Iowa, Wisconsin, and North Carolina should also be on your list of "highly uncertain states". So now you have 128 potentially reasonable combinations by your criteria.
(Admittedly the one that happened is in the top 4 outcomes. But a naive analysis would not have seen that Florida would be a photo finish while North Carolina would not. However Nate's analysis clearly did see that.)
An average of the polls going into the election put 2 of the three that you name as "highly uncertain" at likely polling within 3%. According to the average of polling data at http://www.electoral-vote.com/evp2012/Pres/Maps/Nov06.html all of Colorado, Iowa, Wisconsin, and North Carolina should also be on your list of "highly uncertain states". So now you have 128 potentially reasonable combinations by your criteria.
(Admittedly the one that happened is in the top 4 outcomes. But a naive analysis would not have seen that Florida would be a photo finish while North Carolina would not. However Nate's analysis clearly did see that.)
2^3 == 8
So there's the pundit response or the data response.
The pundit response: "this election is a toss-up!" In which case we'll just reject your assertion outright.
No one really buys the pundit response, though! Okay, the data response to your assertion: pollster.com kind of rejects this if you scroll right on the top row of charts.[0] Colorado, Iowa, Nevada, New Hampshire, and North Carolina all polled closely, bumping us to 2^8 = 256 possible outcomes.
This ignores the absurd accuracy of Silver's predictions. My favorite is his prediction for Florida:
https://twitter.com/fivethirtyeight/status/26583576993401241...
Florida sat at 49.8% to 49.4% for the majority of the night until it recently opened up to a chasm-sized 49.9/49.2 for Obama. Silver had it at 49.79/49.72 for Obama.
So...yeah. If you look at his predictions it's not just monkeys on a typewriter. His results are incredibly accurate from a discrete or a continuous point of view.
[0] http://elections.huffingtonpost.com/pollster#2012-presidenti...
So there's the pundit response or the data response.
The pundit response: "this election is a toss-up!" In which case we'll just reject your assertion outright.
No one really buys the pundit response, though! Okay, the data response to your assertion: pollster.com kind of rejects this if you scroll right on the top row of charts.[0] Colorado, Iowa, Nevada, New Hampshire, and North Carolina all polled closely, bumping us to 2^8 = 256 possible outcomes.
This ignores the absurd accuracy of Silver's predictions. My favorite is his prediction for Florida:
https://twitter.com/fivethirtyeight/status/26583576993401241...
Florida sat at 49.8% to 49.4% for the majority of the night until it recently opened up to a chasm-sized 49.9/49.2 for Obama. Silver had it at 49.79/49.72 for Obama.
So...yeah. If you look at his predictions it's not just monkeys on a typewriter. His results are incredibly accurate from a discrete or a continuous point of view.
[0] http://elections.huffingtonpost.com/pollster#2012-presidenti...
And yet no serious publications were willing to admit this info. I think the biggest value of fivethirtyeight has been to cut through all the partisan and non-committal predictions coming from the media and rest of the web.
In this particular election, so far there have been zero surprises. In other words, an idiot who simply read the polls could have made the correct prediction simply by being unclever.
This election is not a validation of Silver's models because this particular election did not need a model.
This election is not a validation of Silver's models because this particular election did not need a model.
What the hell are you talking about? If you actually placed bets by nate silver you could make quite a lot of money.
The number of outcomes is very high on a parlay bet like this.
The number of outcomes is very high on a parlay bet like this.
What I mean is that if you put 100 random people and a room, and explain the likely outcomes to them, its reasonable to expect that each of those outcomes is chose at least once.
There was not anything wildly unexpected that happened, and therefore his predictions being correct seems unspectacular.
There was not anything wildly unexpected that happened, and therefore his predictions being correct seems unspectacular.
Gaining traction with unspectacular predictions in the spectacle of politics is spectacular.
He has consistently predicted election results throughout his career based on some of the most extensive polling analysis of anyone out there. Alleging that he just made a couple lucky guesses is ridiculous.
But he won as compared to all the pure poll averages. One presumes that weighting of polls is the biggest factor in his model now over others, so that sounds pretty good to me.
The number of possible outcomes is not the same as the probabilities of those outcomes. If I have a weighted coin that shows heads 90% of the time, there are two possible outcomes, but I'm happy to bet you $1 heads each time we flip the coin.
One successful or unsuccessful outcome is not evidence that the model works or doesn't work, but this is how probabilities work.
One successful or unsuccessful outcome is not evidence that the model works or doesn't work, but this is how probabilities work.
what is the technique he is using? is it something like a generalized linear mixed model? (i.e. like using R's lme4 module)?
Quite impressive. Prediction in any form is difficult; more so when there are a million people involved.
Actually, I believe when there are a million people involved it's actually easier. As long as you have access to good data that is.
As long as you have access to good data that is.
I hope you mean "having more data doesn't lead to better prediction automatically", because, used correctly, it always gives more accurate predictions (e.g. prediction intervals[1] are smaller, one can make better determinations about the distribution of the underlying population, etc).
[1]: https://en.wikipedia.org/wiki/Prediction_interval
[1]: https://en.wikipedia.org/wiki/Prediction_interval
Let me me correct my statement. More data does not necessarily lead to better prediction. I must add that it is not always about usage as well. In some cases, the data is insufficient to make any kind of prediction.
The only data that is insufficient to make a prediction is either no data at all, or data suspected to be wrong (e.g. faulty observation equipment).
Even given a single sample, one can make a prediction of what the next observation will be: the same value. Once you get more samples and more knowledge about the subject, you can bring stronger statistical tools to bear to get error estimates, prediction intervals, improved models, etc.
Even given a single sample, one can make a prediction of what the next observation will be: the same value. Once you get more samples and more knowledge about the subject, you can bring stronger statistical tools to bear to get error estimates, prediction intervals, improved models, etc.
> Once you get more samples and more knowledge about the subject, you can bring stronger statistical tools to bear to get error estimates, prediction intervals, improved models, etc
Like I said earlier, increase in data does not always lead to improved models.
Like I said earlier, increase in data does not always lead to improved models.
If you have enough data, you have a census and then there is no model, just reality...
In polls like this, the more data the better as long as it is unbiased (i.e. as long as it is DATA).
In polls like this, the more data the better as long as it is unbiased (i.e. as long as it is DATA).
It is difficult to make predictions, especially about the future.
-- Yogi Berra
-- Yogi Berra
Wow: http://www.larry.denenberg.com/predictions.html
"Who first said "It is difficult to make predictions, especially about the future (or one of its many variants)?"
"Who first said "It is difficult to make predictions, especially about the future (or one of its many variants)?"
Including Alaska and Hawai'i, with zero precincts reporting!
</sarcasm>
</sarcasm>
If he's wrong on AK and HI, I will eat my hat, a moose, and a rainbow.
I built a model in a couple of hours on Sunday afternoon, which simply takes all the most recent polling data, takes an average, does a quick fudge to adjust for the number of polls, and then runs 10,000 simulations to get a probability for each state. The source is on Github:
and the predictions are in this gist:
The result? My model gets 50/51 correct if Florida eventually goes DEM (which looks likely) or 51/51 correct if Florida goes REP.
--
Edit: full disclosure - with all data up to 6 Nov 20120 it predicts Colorado to be a toss-up, and I manually broke the tie in favour of Democrats, based on earlier models favouring them in that state.