I don't know much about Taleb, but one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out. Then when they don't work out, instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models. His sports models are -especially- bad; no better than a coin flip in some instances. I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego, which always makes me take him with a grain of salt. When it comes to politics his former work with Obama's campaign and DailyKos also make me question if he can be truly unbiased in his predictions, but that's a separate issue.
Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. See: https://mashable.com/2012/11/07/nate-silver-wins/
He's not perfect (for instance, I think calling the 2016 model too certain early on is valid and I think some other people have basically equivalently-good models, but are less well known), but if confidence is smugness, then he has a right to be.
> Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for.
Well, this certainly isn't how you'd go about evaluating model quality. Calling which candidate a state will vote for is not a good task; it's too discrete to be a good evaluator. For the same reason, no political model produces this output. You predict the vote share given to each candidate -- a nice continuous variable -- you don't predict who will win. Calling the state is a gimmick on top of that (since victory is a function of vote share, it's easy to degrade your model's output into victory predictions) which serves only to make the statistics worse.
Nate's probabilities are based on the electoral college, not the popular vote, so calling which way individual states goes is by definition necessary for a good evaluation.
Consider states like Maine who specifically allocate their electors by vote share rather than in a block to the winner.
Again, calling individual states is neither necessary for a good evaluation nor even helpful to it. It is a further processing step that runs on top of a finished prediction. If you have a vote share distribution, you can convert that into victor probabilities with basic algebra. You cannot operate in the reverse direction, and you cannot develop a good model that directly produces victor probabilities as its output.
To be perfectly honest, I think the Obama elections were a unique point in politics where models like Nate's worked a lot better. Realistically there weren't many up for grabs states, which means you could guess 8-12 states right and run the table. A lot has a evolved since then, especially in the social media landscape and the accelerated death of answering phones, which live polls rely so heavily on. Maybe Silver will hit this election out of the park, I don't know, but I would err on the side that he gets a number of states wrong.
It’s worth pointing out that Nate was mad at this result because it meant his implied uncertainty for each state was wrong. It meant his model was sandbagged to some degree.
What is your criteria for saying "his models don't work out"? If he says "candidate A has a 90% chance of winning", and then they don't win, that doesn't mean his model was wrong. In the same vain, if the 90% candidate won, it also doesn't mean his model was correct either.
You have to look at a lot of his predictions, and see if the percentages match up.
When you say his models don't work out, are you saying you have done that analysis and found that his percentages don't match the results? If so, I would love to see that analysis.
Put another way, if trump wins...again, do you take that as part of the expected outcomes or evidence your distribution is wrong?
If he said something was a 10% likelihood, and it happened three times in a row, would you still think the prior is that the event was actually 10%, or that the estimate of a 10% likelihood was off?
Pretty sure those are all based on sports or at least baseball. Question isn’t can silver predict baseball. We’ve know that since his time at baseball prospectus in 2007!
> he's very smug about this models but they often enough don't work out.
Taking all of the Presidential forecasts together, there's too few to really generalize about that. Taking the down allot results though, the odds have been pretty close to what he's forecast.
So I'm not sure what your basis is for saying they “often enough don't work out”.
> His sports models are -especially- bad
Maybe, but his political models are especially good.
> I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego
He's never really come off that way to me, not that whether or not he has a big ego has any bearing on the quality of his models.
I frankly don't care if someone is smug or not; I want to know who builds the best models. If Silver's models are often wrong, who would you recommend we look into instead?
Silver might have the best models, but he isn't infallible, which a sizable number of his followers seem to believe...and I think that only feeds into his ego and is probably damaging to his models.
He recently tweeted that the only way Trump can win is through polling error or if Trump steals the election. Do you think that sounds like an unbiased person? His model can't be the reason why he was wrong, it had to be the input data! Imagine saying that to your boss.
His model is based off of point data. Ultimately what his model encodes is the polls+a degree of uncertainty of the polls are completely wrong. If you looked at just the polls, there was no way for trump to win, even assuming a polling error of the same variety as 2016, which appears to be about the situation we are in now.
So no, there's nothing wrong with his tweet. Otherwise, you should be able to explain how trump could win without a polling error. Which states that trump was polling behind by 5-7 points he'd win, and how he's win them without a polling error.
> one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out.
What does it mean for a probabilistic prediction to "not work out"? If I tell you that the odds of flipping 2 heads in a row with a fair coin is only 25%, and it happens, did my model "not work out"?
> instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models.
I really don't intend this in any sort of rude way, but I think you might be interpreting his response as dismissive because it's accurate...
It's difficult to say a model "doesn't work out" based on a single event, especially when it puts a significant probability on the less-likely option. For example, 538's 2016 forecast gave Trump a roughly 1 in 3 chance of winning: that's _higher_ than the odds of flipping 2 heads in a row.
Measuring the quality of a model is more complicated: one way would be applying the model to the same outcome multiple times, but this isn't possible for single-event forecasts. Another way to see how calibrated your predictions are is to aggregate multiple predictions and see how often they line up with reality: an event predicted to occur with X% probability should occur X% of the time.
Lucky for us, 538 has done exactly this analysis[1]! Naturally, it's internal, so take it with a grain of salt, but it looks like their predictions are fairly well-calibrated.
