Gaussian Mixture Models

A finite GMM models a density as a sum of K component densities weighted by mixing proportions. Each component is Gaussian with its own mean and a covariance structure selected by the model. The weights are nonnegative and sum to one. For an observation, Bayes' rule yields a responsibility: the posterior probability that each component generated that point. A hard cluster label can be created by choosing the largest responsibility, but doing so discards uncertainty. GMMs can represent elliptical clusters, and full covariance matrices capture relationships among features within each component. Diagonal covariance assumes no within-component feature covariance, tied covariance shares one general matrix across components, and spherical covariance uses a scalar variance per component. More flexible covariance models require more parameters and can overfit when data are limited. Scaling and feature units matter because covariance is measured in the input space. EM is commonly used to estimate GMM parameters. It alternates responsibilities and weighted parameter updates. Because the likelihood is nonconvex, initialization can affect the solution. Multiple restarts reduce dependence on one starting point but do not prove a global optimum. Covariance regularization can stabilize near-singular estimates; it is a modeling or numerical setting that should be reported. K-means can be viewed under restrictive assumptions as related to spherical, equal-size Gaussian clusters with hard assignments, but practical objectives differ: k-means minimizes squared distances and GMM maximizes likelihood. GMMs offer density estimates and soft assignments, but do not automatically discover the true number of meaningful groups. Use criteria such as BIC as one model-selection aid and check held-out likelihood, stability and domain usefulness. Mixture components are mathematical parts of a fitted density and may not correspond to distinct real-world populations.