“The polls are not supposed to be off by that much, which is why we said that the polls messed up, and our forecast failed (in some part) by not correcting for these errors”
You have a series of predictions that happens incredibly rarely (4 year interval). They are within a close range of accuracy. Your model is working.
Then, one year, the model is off by a full std dev or two.
How will it perform 4 years later? How do you correct for it if you don’t even have a high prior on why it was off last time? How many effects do you just guess and try and model for/against and can you find any data to use for it that’s actually reliable.
How many unknown number of forces is your team arbitrarily deciding were acting on it last time vs now based entirely on their personal histories, combined.
Models can be useful no doubt, whether they work 6/10 times or 9/10 - it entirely depends on the use. But if your model is right 8/10 times and the misses were the last 2? It’s time to rattle it and see what’s broken.
The takeaway: until someone makes a model that proves it can do a few elections in a row (not retroactively) who will really care about the model?
I don't think Taleb or other's have put predictions on paper for nearly as long.
For this year's model Nate Silver has been doing a post-mortem on complaints similar to yours on Twitter:
> We have years and years of well-calibrated forecasts. (If anything, underdogs have won a bit less than they're supposed to, although not to a degree that's statistically significant.) We know what we're doing with these probabilities.
> So I'm admittedly kind of prickly about criticism after elections that both our model and our broader reporting handled quite robustly, which only happened because we put so much work into them. Especially given that the topline outcome + 48/50 states were "called" correctly.
> The reason Biden's win probability was ~90% is precisely because he could withstand a fairly large polling error when Clinton couldn't, which is exactly what happened. Indeed, our model assumes polls are fairly error-prone.
> One slightly unrelated point, but the fact that Biden could lose any 2 states from among the GA/WI/AZ/PA group and still win the election makes his win a bit more robust (and a bit less "close") than if he had to sweep those exact states.
> Especially given that those states are in different parts of the country (South/Midwest/West/Northeast). We've seen in the past few elections largely regionally-driven polling errors. Polls badly underestimated Trump in WI, but only modestly in AZ and PA and not at all in GA.
> This [polling error] averages out to Trump +3.3. Probably pretty similar in national polls, where we'll most likely wind up with an error in the 3.5- or 4-point range I think.
> One of the questions I'm more intrigued by is whether polls missed Trump support among lower-propensity Hispanic (and perhaps also Black) voters. These voters can be hard to get on the phone and/or may be screened out by likely voter models.
It seems like less a case of "shy" voters (which you would expect to see in all states - or at least all blue-leaning states) and perhaps more an instance of pollsters missing sizeable minority voting blocks in certain states, which would influence some state polling more than others.
Of course Nate Silver thinks Nate Silver is right... can you find another prominent statician not employed by him that’s defending his model? I linked to Gelman, a legend in the field and he’s critical of himself and trusting polls, he admits he was wrong for this election.
And here’s Nate admitting he’s wrong the day of (if your model moves from 90/10 to 50/50 with one state change, maybe you shouldn’t be predicting 90/10):
> Look up "unbiased estimator" in your statistics textbook. Now relate that concept to the fact that the vast majority of outcomes were on one side of the sample mean.
https://statmodeling.stat.columbia.edu/2020/11/06/comparing-...
You have a series of predictions that happens incredibly rarely (4 year interval). They are within a close range of accuracy. Your model is working.
Then, one year, the model is off by a full std dev or two.
How will it perform 4 years later? How do you correct for it if you don’t even have a high prior on why it was off last time? How many effects do you just guess and try and model for/against and can you find any data to use for it that’s actually reliable.
How many unknown number of forces is your team arbitrarily deciding were acting on it last time vs now based entirely on their personal histories, combined.
Models can be useful no doubt, whether they work 6/10 times or 9/10 - it entirely depends on the use. But if your model is right 8/10 times and the misses were the last 2? It’s time to rattle it and see what’s broken.
The takeaway: until someone makes a model that proves it can do a few elections in a row (not retroactively) who will really care about the model?