The Emerging Science of Machine Learning Benchmarks

The first comprehensive introduction to benchmarking, the engine behind progress in AI

In machine learning, researchers split their data into training and test sets, let model builders compete on the test set, and call it a benchmark. Statistical tradition prescribed locking test sets in a vault, but machine learning practitioners shared them freely. Benchmarking shouldn’t have worked, but it did, and the machine learning community never figured out the science behind it. How did benchmarkin...