You totally ignored the point. Overfitting to previous results doesn’t account for the fact that the facts on the ground are changing. I mean this year alone we have a once in a century pandemic, can you fit that to any other election?
That's why you test predictions. Silver says 10% is enough uncertainty, Taleb says it's not. From the article: "Premise 1: If you give a probability, you must be willing to wager on it". Unless they're willing to publish and make a friendly bet on their predictions, it's just bloviating.
Every day/year/election is unique, but only to a certain magnitude. Some elections have had active wars, some health crises, some criminal scandals, some terrorist attacks, etc. In the end, the polls attempt to account for those uncertainties and historically that's been a much better predictor than a coin flip.
For example, Washington DC has never voted Republican since it won electoral votes in 1962. Polls have Biden winning 80-90% of the vote there. Even Taleb would agree there's a near 100% chance of Biden winning there, pandemic or not. Silver's methodology is just extrapolating that out to each state. When it gets to some tipping point states it's closer to 60/40 Biden or 70/30.
If Taleb wants to create his own forecast, it would be interesting to track it's performance over time. I have a hard time believing that model closer to a coin-flip is going to outperform weighted polling data and historical precedent over multiple predictions.
Right, but the original point was that Biden's lead was likely large enough in enough swing states to overcome a 2016-style polling bias, which looks like has occurred in some states. That's why it was a 10% chance for Trump, because he had to not just have a large polling error go his way, but also in enough swing states to total 270.
A landslide Biden victory was as probable as a narrow Biden victory. And both were more likely than a narrow Trump victory.
But that’s simply not true. Because Florida and Texas alone were so far off, 10% odds was definitely “way” wrong. Nate’s model predicted that if Florida went Trump the odds were at 33% or so. Texas moreso, etc. Since Florida went incredibly strongly Trump, we can say with certainty he was completely wrong on the 10% chance. That’s the point: he was way overconfident his past model would fit, it should have been closer to a slightly advantage, which would read something like 60/40 at best.
> Since Florida went incredibly strongly Trump, we can say with certainty he was completely wrong on the 10% chance.
The model generated 40k scenarios, and the scenario you're describing is one of them. The most extreme scenarios have Trump winning all swing states by a few %, which doesn't look likely. So what's playing out is not wildly outside the predictions by any means.
In other words, if a 10% chance happens, it doesn't mean the 10% prediction was wrong.
The only way to prove a prediction was wrong is to bet against it over time with your own predictions. As Silver has shown there's a huge appetite for election / sports predictions. Anyone able to beat him over time would have enormous income potential.
For example, if you thought Trump had at least a 40% chance to win, you'd have a great betting opportunity in the market that had Trump in the 25-33% range. You could have bought in then and sold when Trump's chance peaked at about 50-60%. If you arbitrage those mistakes you've identified in the market, overtime you could become very wealthy.
Until then, it's just pundits pontificating after the results are known without putting any money or reputation on the line beforehand, similar to a casual sports fan late in the 4th quarter: "of course the 49ers were going to blow their lead -- I just knew it!"
His probability of Florida being won by that much was basically 0. But it was actually almost the opposite, in the real world the probability of Biden winning was almost 0.
The polls were so far off systematically in one direction - it doesn’t mean he was right about the 10% chance, because that would mean there was equally likely a chance they were all off in the other direction, which is obviously just wrong. Polling missed huge groups of voters opinions - it was just wrong.
We have numbers now that show that Trump voters also aren’t forthcoming on their vote. Probably on purpose as an effect of last election and being spiteful towards pollsters in general. I actually had this as a strong prior, so my model predicted this would be close, but of course if you “just go by the polls” you’re essentially trusting a flawed system that is gameable. Anyone who trusts the polls is essentially a fool next time, they have been shown to be incorrect now twice in large amounts and I wouldn’t doubt that the “meme” the right has started to purposely deceive them continues even after Trump.
You are absolutely wrong on your 49ers analogy and the 10% covering this. Florida disproves it, as do the systemic nature of being off. If they weren’t off systematically, you’d have expected the polls to get Florida wrong one direction but other states wrong in a different direction. That would prove the model, but if every single state was multiple points off all in the same direction, you don’t get to call margin of error. Your bell curve was shifted one direction, it wasn’t a case of the curve being right but the dice rolled in the tail.
Yes, some states had big polling errors, while others like GA and AZ were reasonably accurate. But the model doesn't care if the polls are off in FL by 0.1% or 10% because it's a winner take all scenario. So while large polling errors are surprising, they have the same exact impact as a small polling error in a close race, which isn't surprising. But people like to cherry pick and say "how did you get X so wrong?" rather than "how did you get X/10 so right?" In fact, Trump might only end up winning 3 "upsets": FL, NC and ME-2, but they were all under 2% difference in the polls, so no one is shocked to see Trump win there. WI and MI would have been much bigger upsets, but that didn't end up happening and a 20k vote win is as good as a 2M vote win in the EC.
The 10% odds come from the path to 270. Here are the 9 states / districts that polled within 2% pre-election (considered toss ups): NE-2, AZ, FL, NC, ME-2, GA, OH, IA, TX. Even if Trump won all of those, he wouldn't get to 270. Using 50/50 odds for all 9 toss ups, Trump only gets a clean sweep 11% of the time. He would then still need another state like PA, WI or MI to get to 270. That's why Silver said that Biden's chances are so high, specifically because Biden could overcome a 2016-level polling error in every toss-up state and still win. In fact we saw 2016-level polling errors in some battleground states, but not all.
Another way to view it, is you have a 1/6 (16%) chance of rolling a 1 on a 6-sided die. Even if all 53 states and districts were projected at 84% confidence, and rolling a 1 meant "upset" and rolling 2-6 meant "polls were accurate", you'd end up with 8-9 states resulting in upsets! 84% confidence still means a lot of surprises. Their final FL prediction was only 69% confidence. Also, confidence intervals aren't linear, meaning that a 95% confidence anticipates half as many upsets as a 90% confidence interval. That +5% means -50% upsets, so 69 or 84% confidence is really not as sure a thing as it might sound.
So, yes it's worth re-examining why polling was very accurate in AZ & GA but very off in TX & FL (specifically Latino's in Miami). But a 10% chance was not crazy IMO given how easy it was for Biden to overcome even huge polling / vote disparities in multiple states and still win.
Again, those who disagree are free to get great odds in the betting markets.
Bringing in betting markets has nothing to do with the point, but you are right on one thing: if you had bet along with Nate’s odds in a simulation you’d lose money in a Monte Carlo simulator based on current results.
The end story is the polls weren’t a reliable measure of the vote. Their predictive ability was low, and even playing by the rules of Nate’s model you have to admit he was off by some 30% at minimum.
Nate’s model has the average PA poll at Biden +6 (!!) where the final result looks to be under 1 (and could go either way). It’s simply impossible to argue that was an accurate model.
> Nate’s model has the average PA poll at Biden +6 (!!) where the final result looks to be under 1 (and could go either way). It’s simply impossible to argue that was an accurate model.
538's PA indicator ended at Biden +4.7%. But the overall 538 model doesn't care if a candidate wins PA by +0.7% or +4.7%. The 538 model is designed to do 1 thing: predict the EC winner. It uses state-wide predictions as indicators, but you can't judge a model based on the performance of a single low-level indicator, you judge it by it's final prediction, which is Biden 90% / Trump 10% to win the EC.
But yes, the lower you go in the model the higher variance you'll see:
Level 1 -> EC Winner (538's focus -- could end up 100% correct)
Level 2 -> State Winners (low variance -- could end up 90-95% correct)
Level 3 -> State Polling Averages (moderate variance & could be 75-90% within margin of error)
Level 4 -> Individual Polls (high variance & could be 60-80% within margin of error)
If you click on any individual state (like PA: https://projects.fivethirtyeight.com/2020-election-forecast/...), you can see the state-wide vote projections. Biden was projected to earn between 49 and 55% of PA's vote with Trump expected to get between 45 and 50%. The final outcome looks certain to fall within those expected ranges. Again, the margin really doesn't matter, just that the idea of Trump getting above 50% and Biden getting below 50% seemed pretty difficult, hence Trump's low (but not impossible) 16% chance of carrying the state.
I like the way their "Winding Path to Victory" chart (https://projects.fivethirtyeight.com/2020-election-forecast) explains their model. It's basically saying: "Trump's path to 270 likely goes through PA, NE-2, AZ, FL, NC, ME-2, GA, OH, IA & TX, where he has a fighting chance of winning any of those, but a very low chance at winning all of them." On the flip side, "Biden's path to 270 goes through Wisconsin, Michigan, Nevada and Pennsylvania, where he's likely to win all of them". That seems like a pretty reasonable EC prediction to me.
If Silver was promoting the model as being able to accurately predict state vote percentages, I'd agree that it's underwhelming. But when you average all of that variability and uncertainty, you can get a pretty reasonable EC winner prediction IMO, at least better than others I've seen that had Biden closer to 95 or 98% to win the EC, or betting markets that had him as low as 60%.
If every single range it’s at at or below the lower bound for one candidate and at or just past the upper bound for the other, systematically, that’s the definition of a flawed model. You seem to breeze past every point I make, I suppose there’s no educating the unwilling so I’ll leave this thread. The bell curve was systemically off, the polls were systemically off. They actually missed the bounds in many races entirely. The betting markets were much more accurate.
> If every single range it’s at at or below the lower bound for one candidate and at or just past the upper bound for the other, systematically, that’s the definition of a flawed model.
Yes, I agree that individual low-level polls for 2 elections in a row have underrepresented Trump's actual support. But, 538's model addresses exactly your point: low-level indicators like state polling can potentially be systematically flawed and biased. By simulating those unreliable low-level indicators through 40k scenarios, the scenario where Biden systematically underperforms biased pre-election polls in key swing states but Biden still wins (as we're seeing) ends up being a very reasonable outcome and the large reason why Biden was favored overall. Trump needed to outperform the polls in 7 states to win, and that just didn't happen.
And the polls weren't off dramatically in every single race. Here are some of 538's last predictions in key states:
AZ: 50.7% Biden / 48.1% Trump
NE-2: 51% Biden / 47.8% Trump
GA: 50.1% Biden / 49.2% Trump
PA: 52% Biden / 47.3% Trump
NC: 50.5% Biden / 48.8% Trump
Each poll has a margin of error, usually +/-2.5% or more. So, 2-3% swings on Election day are completely normal. NC was off by a tiny margin and enough to flip the state to Trump, while the others were also off by tiny margins, but not enough to flip the state.
Sizable polling swings are not uncommon and historically have gone in both directions, but the model was more concerned about just how narrow Trump's path to 270 was, more so than a routine or even historical polling error.
> The betting markets were much more accurate.
Everyone who bet on Trump at a 30/40% chance to win is about to lose their bet. I definitely wouldn't have taken those odds for Trump to run the table in 7 straight must-win swing states all polling as a coin-flip and within the margin of error. In a completely random scenario Trump would have only had a 14% chance of pulling that off.
“The polls are not supposed to be off by that much, which is why we said that the polls messed up, and our forecast failed (in some part) by not correcting for these errors”
You have a series of predictions that happens incredibly rarely (4 year interval). They are within a close range of accuracy. Your model is working.
Then, one year, the model is off by a full std dev or two.
How will it perform 4 years later? How do you correct for it if you don’t even have a high prior on why it was off last time? How many effects do you just guess and try and model for/against and can you find any data to use for it that’s actually reliable.
How many unknown number of forces is your team arbitrarily deciding were acting on it last time vs now based entirely on their personal histories, combined.
Models can be useful no doubt, whether they work 6/10 times or 9/10 - it entirely depends on the use. But if your model is right 8/10 times and the misses were the last 2? It’s time to rattle it and see what’s broken.
The takeaway: until someone makes a model that proves it can do a few elections in a row (not retroactively) who will really care about the model?
I don't think Taleb or other's have put predictions on paper for nearly as long.
For this year's model Nate Silver has been doing a post-mortem on complaints similar to yours on Twitter:
> We have years and years of well-calibrated forecasts. (If anything, underdogs have won a bit less than they're supposed to, although not to a degree that's statistically significant.) We know what we're doing with these probabilities.
> So I'm admittedly kind of prickly about criticism after elections that both our model and our broader reporting handled quite robustly, which only happened because we put so much work into them. Especially given that the topline outcome + 48/50 states were "called" correctly.
> The reason Biden's win probability was ~90% is precisely because he could withstand a fairly large polling error when Clinton couldn't, which is exactly what happened. Indeed, our model assumes polls are fairly error-prone.
> One slightly unrelated point, but the fact that Biden could lose any 2 states from among the GA/WI/AZ/PA group and still win the election makes his win a bit more robust (and a bit less "close") than if he had to sweep those exact states.
> Especially given that those states are in different parts of the country (South/Midwest/West/Northeast). We've seen in the past few elections largely regionally-driven polling errors. Polls badly underestimated Trump in WI, but only modestly in AZ and PA and not at all in GA.
> This [polling error] averages out to Trump +3.3. Probably pretty similar in national polls, where we'll most likely wind up with an error in the 3.5- or 4-point range I think.
> One of the questions I'm more intrigued by is whether polls missed Trump support among lower-propensity Hispanic (and perhaps also Black) voters. These voters can be hard to get on the phone and/or may be screened out by likely voter models.
It seems like less a case of "shy" voters (which you would expect to see in all states - or at least all blue-leaning states) and perhaps more an instance of pollsters missing sizeable minority voting blocks in certain states, which would influence some state polling more than others.
Of course Nate Silver thinks Nate Silver is right... can you find another prominent statician not employed by him that’s defending his model? I linked to Gelman, a legend in the field and he’s critical of himself and trusting polls, he admits he was wrong for this election.
And here’s Nate admitting he’s wrong the day of (if your model moves from 90/10 to 50/50 with one state change, maybe you shouldn’t be predicting 90/10):
> Look up "unbiased estimator" in your statistics textbook. Now relate that concept to the fact that the vast majority of outcomes were on one side of the sample mean.
