Hinge Loss

For binary classification, encode labels as y in {-1, +1} and let f(x) be the model's decision score. The signed margin is y times f(x). Hinge loss is max(0, 1 - y f(x)). A correctly classified example with score in the correct direction can still incur loss if it lies within the unit margin. A point on or beyond the margin has zero hinge loss. A misclassified point has a negative signed margin and receives a penalty greater than one. This loss encourages a classifier to separate classes with a margin, not merely to assign correct training labels. In a linear support vector machine, the objective balances margin width against the sum of hinge losses, with a regularization parameter controlling the tradeoff. A stronger penalty for violations can fit training data more tightly; allowing more violation can favor a wider, simpler boundary. Kernel methods change the feature representation while retaining the margin-based objective. Consider a hypothetical example with label +1 and score 0.4. Its margin is 0.4 and its loss is 0.6. If another positive example has score 1.3, its loss is zero. If a negative-labeled case has score 0.5, its signed margin is -0.5 and its loss equals 1.5. These values show why classification correctness alone does not determine hinge loss. Logistic loss is also convex and margin-sensitive, but unlike hinge loss it continues to assign a positive penalty to correctly classified examples, even far beyond the margin. Its probabilistic form connects naturally to log-likelihood. Hinge loss is not itself a calibrated probability estimate. SVMs can be paired with probability calibration when probabilities are needed, using an appropriate validation process. Scale features, tune regularization on training folds, and evaluate error costs on representative data. Hinge loss does not define an ideal decision threshold for every application; business consequences may require a separate threshold or class-weighting decision.