I'm not sure how to verify your comment since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there and can be found here: https://www.natesilver.net/p/pollster-ratings-silver-bulleti...
It actually explicitly looks for statistical evidence of "herding" (e.g. not publishing poll results that might go against the grain) and penalized those pollsters.
In both rating systems, polls that had a long history of going against the grain and being correct, like Ann Seltzer's Iowa poll, were weight very heavily. Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Polls that go heavily against the grain but DON'T have a polling history simply aren't weighted heavily in general.
Here's how 538 explains how they factor in bias into their grading:
> Think about this another way. If most polls in a race overestimate the Democratic candidate by 10 points in a given election, but Pollster C's surveys overestimate Republicans by 5, there may be something off about the way Pollster C does its polls even if its accuracy is higher. We wouldn't necessarily expect it to keep outperforming other pollsters in subsequent elections since the direction of polling bias bounces around unpredictably from election to election.
> since 538 was cut by ABC a month or 2 ago. But Nate Silver's pollster rating methodology is pretty much the same as 538's was during his tenure there
Nate took his model with him. After he left, 538 rolled a new model.
> Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career).
> Over a year ago I advised the Register I would not renew when my 2024 contract expired with the latest election poll as I transition to other ventures and opportunities.
> Would I have liked to make this announcement after a final poll aligned with Election Day results? Of course. It’s ironic that it’s just the opposite. I am proud of the work I’ve done for the Register, for the Detroit Free Press, for the Indianapolis Star, for Bloomberg News and for other public and private organizations interested in elections. They were great clients and were happy with my work.
You can of course choose to interpret this if you wish as her conveniently inventing a narrative after-the-fact, but that seems unlikely. The polling industry doesn't work that way, and it's particularly unlikely that the Register would have cut ties with her over a single poll after her impressive track record for nearly forty years.
Since there's no real evidence that's what happened, the most reasonable conclusion is that, as she said, she decided to retire after spending four decades doing the same thing.
The idea is not that the Register cut ties with her, but that her personal safety was at risk and she was getting brutally harassed by millions of people and the POTUS-elect. See my other comment for details.
Yeah, after reading the NYT article[0], I'll amend my earlier statement "since there's no real evidence that's what happened," since that is some pretty strong evidence that something happened (albeit more complicated than simply "her poll was wrong, so she lost her job").
Huh? It’s evidence-based speculation. Of course it’s conceivable that a 68-year-old would have less concerns about putting her career on the line. I’m trying to understand how you think a conversation on HN should operate
Seems like an absurd claim given it was less than 2 weeks out from the election and polls that show a side losing can often have a motivating effect on a base
Truth is that voter behavior changes more radically than pollsters admit from election to election. We can statistically model direction of "error" but we can't statistically model direction of "bias". A grade A pollster like Ann Selzer could have the perfect formula one election but that formula could be a total miss the next
The important thing that forecasters like Silver point out is that we need those pollsters to stick around and use the SAME methodology. Even if gets less useful at predicting the outcome, having a consistent signal with the same methodology can still be of immense statistical importance. And it's extremely important that we don't scare away pollsters from publishing "outlier" results. Doing so only encourages herding which is a growing problem in polling
you're being downvoted but you're actually pretty much right. She got a lot of backlash from the right accusing her poll of being politically motivated. To the point where they pressured institutions to force her out
You’re delusional if you think the media kowtows to him. I’m not even going to post sources, a quick search will show even left leaning sites saying the overwhelming reporting is negative about him.
Selzer announced her retirement November 17th in the op-ed below, which was after the election results were known. All the news I saw about her retirement is dated the same day, so that appears to be the announcement. In the article, she claims to have informed some people privately in 2023 that she would retire after the election.
This November 18 article - from the next day - says the police had warned her that her safety was at risk. It doesn't say when that warning happened, but it seems likely it was not in the 24 hours since announcing her retirement.
It’s more of a systemic failure than a single mistake. If you’re a business that releases one major product version every year or two that accounts for your entire revenue, and you completely blow the product on the last release, there’s a good chance you’re not getting another chance if you’re a small-medium business. Compare her firm to any number of smaller gaming and entertainment studios.
Selzer wasn’t gallup or some other big player that was continuously releasing a wide range of evolving polls, but a once-an-election shop that was multiple standard deviations away from reality in the wrong direction, on the biggest stage in history. Who would contract Selzer ever again?
Presumably anyone who is familiar with her overall body of work and the concept of an outlier.
Pollsters are increasingly herding. Perhaps after this election all unexpected results will be suppressed or adjusted, which seems to be what you're advocating for.
It actually explicitly looks for statistical evidence of "herding" (e.g. not publishing poll results that might go against the grain) and penalized those pollsters.
In both rating systems, polls that had a long history of going against the grain and being correct, like Ann Seltzer's Iowa poll, were weight very heavily. Seltzer went heavily against the grain 3 elections in a row and was almost completely correct the first 2 times. This year she was off by a massive margin (ultimately costing her her career). Polls that go heavily against the grain but DON'T have a polling history simply aren't weighted heavily in general.