The edge is almost never the choice of model architecture. It's very common knowledge now that deep neural networks and gradient boosted machines are incredibly effective at different problem classes, and that ensembling almost always marginally boosts performance.
The edge winning teams have varies from competition to competition. It includes
- robust cross-validation strategies
- robust feature selection strategies
- creative feature engineering
- finding uniquely valuable external datasets that improve performance
- robustly controlling for distributional differences between train and test sets
- (and unfortunately on occasion) information leakage
Very interesting. Tangentially - it surprised me how ubiquitous gradient boosting was in these competitions, since the algorithm was never mentioned in my stats/ML coursework. Especially seeing how well they perform.
I'm particularly interested in your fourth point - uniquely valuable external datasets. I hadn't even thought about that being a factor.
Are we as a community learning anything noteworthy about the particular composition of those approaches, e.g. the list of stuff at the bottom of your comment, from the contest winners?
If yes, then why don't many, many participants snap to doing things exactly like past winners did them (not just same model architectures, but same cross-validation strategies, same inclusion of good outside data sets, etc.) -- thus making the differences between entries almost indistinguishable very fast, and meaning that winning vs. losing would probably just come down to random noise, thus contest performance probably shouldn't be related to whether someone is good in that field or would be good for a certain job?
If no, then what purpose do the contests serve? Perhaps just having fun, which is fine, but clearly not advancing knowledge of how to design good general model pipelines -- thus contest performance really shouldn't be related to whether someone is good in the field (that is, the researchers or whomever is figuring out general new knowledge about the best model pipelines would be, but not necessarily contest winners), and contest performance again wouldn't be very related to, say, whether someone would be good at a job in this area.
I don't point this out to suggest the contests aren't valuable. On the contrary, I think they are fun, interesting, and very valuable.
But I have always been skeptical that, even from first principles, contests like this cannot be useful for finding "the best" engineers to hire. Either (a) everyone snap-copies what the winners do, so either everyone's a good employee or none of them are, or (b) you don't learn anything over time from the general types of things that winners repeatedly do differently or better than losers, and so being with the "winner" category is not systematically different than being in the "loser" category.
The only third option I see is that we do steadily learn new systematic principles that emerge from what winner groups happen to do, but for someone reason, none of the other contestants choose to copy or recycle those ideas. Then, whatever property it is that makes winners winners would be related to discovering new effective ways of doing this, and not just luck-of-the-noise.
I think things other than choice of algorithm are harder to copy. For instance, finding good outside sets sounds a lot easier than it is. But I do see your point. For me these competitions are more a fun educational tool where I can practice the model-building process. I agree that they probably aren't useful for finding the best engineers... although there are companies who see things differently.
