深度学习

Deep learning is a way of building software that learns a task from examples. Its core object is an artificial neural network: a large collection of simple arithmetic units, loosely inspired by brain cells, arranged in layers. Each unit multiplies its inputs by adjustable numbers called weights, adds the results, and passes the total through a simple nonlinear function. The word deep refers to the number of layers between the raw input, such as the pixels of a photo, and the final answer, such as the label cat. In their 2015 review in Nature, Yann LeCun, Yoshua Bengio and Geoffrey Hinton describe the central idea as learning representations with multiple levels of abstraction. In an image model, early layers tend to respond to edges, middle layers to combinations of edges and textures, and later layers to parts of objects. Nobody programs those detectors. They emerge during training.

Training follows the same loop for almost every deep network. First, gather many examples with known answers, called labels. Second, run an example through the network to get a prediction; this is the forward pass. Third, score the prediction with a loss function, a formula that produces a large number when the prediction is far from the label and a small one when it is close. Fourth, use backpropagation to work out, for every weight, how much a tiny change would raise or lower the loss. That quantity is called the gradient. Fifth, nudge every weight a small step in the direction that lowers the loss, a method called gradient descent. Sixth, repeat over millions of examples, usually in small groups called batches. When training ends, the weights are frozen and the network is used on new inputs, which is called inference.

Backpropagation is the step that makes depth practical. David Rumelhart, Geoffrey Hinton and Ronald Williams described it in Nature in October 1986: the procedure repeatedly adjusts connection weights to shrink the gap between the network's actual output and the desired output. Their key observation was that hidden units, the ones that are neither input nor output, come to represent useful features of the task on their own. The 2015 Nature review explains why that matters. Conventional machine learning needed careful engineering and domain expertise to design a feature extractor before any learning could happen. Deep learning moves that work into the training process.

Different layer designs suit different data. Convolutional networks slide the same small filter across every position of an image, so a pattern learned in one corner is recognized anywhere. Recurrent networks read a sequence one step at a time and carry a memory forward, which suited speech and text for many years. Transformers, introduced by Ashish Vaswani and colleagues at Google in June 2017, replace that step-by-step reading with attention, a mechanism that lets every position in a sequence look directly at every other position. Because attention can be computed for all positions at once, the original Transformer trained to a new English-to-French translation record in 3.5 days on eight GPUs. Residual connections, introduced by Kaiming He and colleagues in December 2015, let a layer's input skip ahead and be added to its output. That simple shortcut made networks of 152 layers trainable, which the authors describe as eight times deeper than the earlier VGG networks.

The modern boom has a clear starting point. In 2012, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered AlexNet in the ImageNet image classification contest. The network had 60 million weights and 650,000 units, and it was trained for about 90 passes over 1.2 million labeled images, which took five to six days on two NVIDIA GTX 580 graphics cards with 3 GB of memory each. Its winning top-5 error of 15.3% compared with 26.2% for the next entry. The authors wrote that their results could likely improve simply by waiting for faster GPUs and bigger datasets, and the following decade largely followed that recipe: more data, more compute, and better architectures such as ResNet in 2015 and the Transformer in 2017. By 2024, according to the Stanford AI Index 2025, nearly 90% of notable new AI models came from industry rather than universities, up from 60% in 2023, a sign of how expensive frontier work has become.

It helps to place deep learning among its alternatives. Rule-based software follows instructions a person wrote, which is predictable and easy to audit but breaks on messy inputs like handwriting or accents. Classical machine learning, such as logistic regression or decision trees, learns from data but usually relies on features a person chose in advance. Deep learning learns the features too, which is why it dominates tasks with raw, unstructured inputs: images, audio, video and free text. For small tables of numbers, such as a spreadsheet of a few thousand loan records, simpler methods are cheaper to train and easier to explain, and they remain a sensible first try. Deep learning is also distinct from generative AI: generative systems are one use of deep networks, not a separate technology.

The method has real limits. Deep networks need large amounts of data, and they can memorize their training examples instead of learning general patterns, a failure called overfitting. The AlexNet authors reported substantial overfitting until they added dropout, a technique that randomly switches off half of the units in certain layers during each training step. A trained network's knowledge is spread across millions or billions of weights, so it is hard to say why it produced a particular answer. Networks can latch onto accidental shortcuts in the data, such as a background that happens to appear with one label, and they can fail without warning on inputs unlike anything in training. They also give confident answers when they are wrong unless they are specifically calibrated. None of these problems is solved by adding more layers.

The practical requirements depend heavily on scale. A small image classifier can be trained on a single consumer graphics card in hours, and most practitioners start from a network someone else already trained and adjust it for their task, which is called fine-tuning. Frontier models are a different world. Epoch AI estimates that training compute for frontier language models has grown about 5 times per year since 2020 and that frontier training costs have grown about 3.5 times per year, and it puts the up-front cost of a 1-gigawatt AI data center at roughly 38 billion dollars. Running models has become far cheaper, though: the Stanford AI Index 2025 reports that the price of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024, while hardware costs fell about 30% per year and energy efficiency improved about 40% per year.