One-vs-Rest and One-vs-One Strategies

One-vs-rest (OvR, also called one-vs-all) and one-vs-one (OvO) are the two standard ways to extend binary classifiers to multiclass problems. Given K classes, OvR trains K separate binary classifiers: each learns to separate a single class from all the others combined. At prediction time, all K classifiers score the input, and the class whose classifier gives the highest confidence wins. OvO instead trains a classifier for every pair of classes, giving K(K-1)/2 classifiers total. Each pairwise classifier only sees examples from its two classes during training, so its decision boundary can be simpler and its training set smaller. At prediction time, every pairwise classifier votes for one of its two classes, and the class with the most votes is chosen, with ties broken by summed confidence scores. The tradeoffs are the main reason the choice matters. OvR trains fewer classifiers but each is trained on an imbalanced dataset (one class vs. everyone else), which can hurt performance when classes are unevenly sized. OvO trains many more classifiers for large K, but each pairwise classifier sees only two classes, so its training set is often smaller (though class counts within a pair can still differ), which is why OvO is the traditional default for kernel support vector machines, where training time scales poorly with dataset size. A common misconception is that one strategy is universally better; in practice the choice depends on the algorithm's cost function and dataset size, and libraries pick sensible defaults per algorithm.