Kaggle

Kaggle is an online service that hosts data science and machine learning competitions. TechCrunch reported on March 8, 2017 that Google confirmed it was acquiring Kaggle, a move announced at the Google Cloud Next conference in San Francisco; the terms were not disclosed.

How a competition ranks participants

According to the abstract of Roelofs et al. (2019), participants in these competitions repeatedly evaluate their progress against a holdout set that forms the basis of a public ranking available throughout the competition, while performance on a separate test set used only once determines the final ranking. The split limits how far a participant can fit the final test data by repeated submissions, since the final test data is used only once.

What the meta-analysis found

Roelofs et al. analyzed over one hundred Kaggle competitions. By comparing the public ranking with the final ranking, they assessed how much participants had adapted to the holdout set during a competition. They report little evidence of substantial overfitting and describe this as speaking to the robustness of the holdout method across data domains, loss functions, model classes, and human analysts.

That finding is about overfitting to the holdout set. It is not a statement that a model that wins a competition will perform the same way after deployment.

What a leaderboard score leaves out

• A fixed task. The dataset, the metric, and the test distribution are set by the organizers. A real system faces data that changes and a metric that may differ from the competition’s.
• Costs and constraints. A score does not include latency, compute cost, maintainability, or the effort of keeping the model current.
• Noise. A final score is an estimate from a finite test set. Rank differences smaller than the uncertainty in the score do not show that one model is better; Confidence Intervals and Comparing Models: Uncertainty and Significance discuss this.
• Splits that match use. How training and evaluation data are separated should match how the model will be used; see Cross-Validation (CV).

A competition is best read as a fixed benchmark. It can show how methods compare on one well-specified problem, and it leaves the rest of the evaluation to the application.

References:

• TechCrunch, Google confirms its acquisition of data science community Kaggle (March 8, 2017)
• Roelofs et al., A Meta-Analysis of Overfitting in Machine Learning (NeurIPS 2019)