āļāđāļāđāļāļāļģāđāļāļ°āļāļģāļāđāļāđāļ
āļĢāļ°āļāļāđāļŠāļāļāļāļĨāļāļĢāļąāļ§ AI āđāļĨāļ°āļāļģāļŦāļāļāđāļ§āļĨāļēāļāļąāđāļ§
āļāļļāļāļŠāļēāļŦāļāļĢāļĢāļĄ
āļāļđāđāļĄāļ·āļāļāļ·āđāļāļāļēāļ
The Lottery Ticket Hypothesis proposes that some large, randomly initialized neural networks contain sparse subnetworks that can train in isolation from their matched initial weights to reach accuracy comparable to the dense model.
āļāļąāļāļāļģāđāļāļ°āļāļģāļāļĩāđāļāļĢāļĢāļĒāļēāļĒāđāļāļĒ AI · ~2 min
The original experiments support this in tested settings, rather than guaranteeing a compact ticket for every model or task.
Jonathan Frankle and Michael Carbin introduced the Lottery Ticket Hypothesis in 2018. In experiments on fully connected and convolutional networks for MNIST and CIFAR-10, they found sparse subnetworks that, when trained in isolation from their original initialization, reached test accuracy comparable to the dense network in a similar number of iterations. Their reported tickets were less than 10â20% of the original model size in those settings; this is an empirical result, not a general compression guarantee. Their iterative magnitude pruning procedure trains a network, removes weights with small magnitudes, restores surviving weights to their initial values, and repeats. Later work studied rewinding to an early training checkpoint. Randomly reinitializing a found mask can reduce the benefit observed in the original experiments. The broader suggestion that dense overparameterization helps optimization discover these sparse subnetworks is a conjecture about the mechanism, not a separate result established by the original experiments.
Testing whether a sparse subnetwork can retain a dense modelâs accuracy on a specified dataset and architecture
Comparing iterative pruning and weight rewinding with training a sparse model from a fresh random initialization
Studying how discovered tickets transfer between related datasets or model architectures
Evaluating whether sparse models reduce storage or inference cost on a target device
Lottery-ticket research asks when sparse subnetworks can be found cheaply, whether they transfer across tasks, and how pruning affects training and inference costs. A separate strong lottery-ticket line studies whether a sufficiently overparameterized random network contains a subnetwork that approximates a target network without further weight training; this is a distinct theoretical claim and not a guarantee about every practical model.
āļāļāļŦāļĄāļēāļĒāļāđāļēāļ§āļāļĢāļĩ
āđāļĢāļ·āđāļāļāļĢāļēāļ§ AI āļāļĩāđāļāļĢāļ§āļāļŠāļāļāđāļĨāđāļ§ 3 āđāļĢāļ·āđāļāļāļāļļāļāļ§āļąāļāļāļģāļāļēāļ āđāļāđāļē, āđāļāļĩāļĒāļāđāļāđāļāļ āļēāļĐāļēāļāļąāļāļāļĪāļĐāļāļĢāļĢāļĄāļāļē āļāļĢāļĩāļāļĨāļāļāđāļ āđāļĄāđāļĄāļĩāđāļāļĐāļāļē
āļāļĩāđāļĄāļĨāļŦāļāļķāđāļāļāļāļąāļāļāļļāļāļ§āļąāļāļāļģāļāļēāļ āļĒāļāđāļĨāļīāļāļāļēāļĢāļŠāļĄāļąāļāļĢāđāļāļāļĨāļīāļāđāļāļĩāļĒāļ§ āđāļĢāļēāđāļĄāđāđāļāļĒāļāļēāļĒāļŦāļĢāļ·āļāđāļāļĢāđāļāļĩāđāļāļĒāļđāđāļāļĩāđāļĄāļĨāļāļāļāļāļļāļ
āļāļāļŠāļāļāļāļąāļ§āđāļāļ
āļāđāļāđāļŠāļāļāđāļāļ°āļāļąāļāļāļĩāļŠāļģāļŦāļĢāļąāļāļāļļāļāļāļģāļāļāļ āđāļĨāļ°āđāļāļĢāļąāļāļĢāļāļāļāļĩāđāļŠāļēāļĄāļēāļĢāļāđāļāļĢāđāđāļāđāļāļĢāđāļāļĄ ID āļāļĩāđāļāļĢāļ§āļāļŠāļāļāđāļāđāđāļĄāļ·āđāļāļāļļāļāļāđāļēāļāļŦāļĨāļąāļāļŠāļđāļāļĢ
āļŠāļāļąāļāļŠāļāļļāļāļāļēāļĢāļĻāļķāļāļĐāļē AI āļāļĢāļĩ AI Understanding āđāļāđāļāļāļāļāđāļāļĢāđāļĄāđāđāļŠāļ§āļāļŦāļēāļāļĨāļāļģāđāļĢ 501(c)(3) â āđāļĄāđāļĄāļĩāđāļāļĐāļāļē āđāļĄāđāļĄāļĩāļāļģāđāļāļāđāļāļīāļ, āļāļĨāļāļāđāļ āļāļĢāļīāļāļēāļ
The Lottery Ticket Hypothesis proposes that some large, randomly initialized neural networks contain sparse subnetworks that can train in isolation from their matched initial weights to reach accuracy comparable to the dense model. The original experiments support this in tested settings, rather than guaranteeing a compact ticket for every model or task.
āļāļąāđāļ§āļāļĩāđāļāļāļ°āļāļ·āļāđāļāļĢāļ·āļāļāđāļēāļĒāļĒāđāļāļĒāļāļāļēāļāđāļĨāđāļāļāļķāđāļāđāļĄāļ·āđāļāļāļķāļāđāļĒāļāļāļēāļāļāđāļģāļŦāļāļąāļāđāļĢāļīāđāļĄāļāđāļāđāļāļĩāļĒāļ§āļāļąāļ āļāļ°āđāļāđāļāļ§āļēāļĄāđāļĄāđāļāļĒāļģāļāļĩāđāđāļāļĩāļĒāļāđāļāđāļāļąāļāđāļāļĢāļ·āļāļāđāļēāļĒāļŦāļāļēāđāļāđāļ
The original method paired a discovered pruning mask with the surviving weights reset to their original initialization; later work also studied early-checkpoint rewinding.
Jonathan Frankle āđāļĨāļ° Michael Carbin āđāļāļ°āļāļģāļŠāļĄāļĄāļāļīāļāļēāļāđāļāļĢāļēāļĒāļāļēāļāļāļāļ MIT āļāļĩ 2018
The original procedure iteratively pruned low-magnitude weights and reset the surviving weights to their original initialization before retraining; early-checkpoint rewinding was studied later.
The original experiments identify winning tickets in tested settings; the claim that dense overparameterization helps the search for them is a conjecture about why the phenomenon occurs.
āđāļĢāļĩāļĒāļāļĢāļđāđāļāđāļāđāļ
āļāļģāđāļāļ°āļāļģāđāļāļīāđāļĄāđāļāļīāļĄāļāļĩāđāđāļĨāļ·āļāļāļŠāļģāļŦāļĢāļąāļāļŦāļąāļ§āļāđāļāļāļĩāđ
āļāđāļāđāļāļāļģāđāļāļ°āļāļģāļāđāļāđāļ
āļĢāļ°āļāļāđāļŠāļāļāļāļĨāļāļĢāļąāļ§ AI āđāļĨāļ°āļāļģāļŦāļāļāđāļ§āļĨāļēāļāļąāđāļ§
āļāļļāļāļŠāļēāļŦāļāļĢāļĢāļĄ