概述
它是現代圖像識別、語音識別、翻譯和聊天機器人背後的技術,因為各層從示例中學習自己的特徵,而不是依賴手工編寫的規則或特徵。
重點摘要
- 深度學習訓練多層神經網絡,每一層從資料中學習自己的特徵,而不是使用手寫規則。
- 訓練重複一個循環:預測、使用損失函數測量誤差、使用反向傳播計算梯度,並微調每個權重以減少誤差。
- 反向傳播於 1986 年在《自然》雜誌上進行了描述,但該領域在 2012 年開始起飛,當時 AlexNet 將 ImageNet top-5 錯誤率從亞軍的 26.2% 降低到了 15.3%。
- 卷積網路適合影像,循環網路曾經在序列中占主導地位,而 2017 年推出的 Transformer 則可以同時處理整個序列。
- 深度網路需要大量數據,難以解釋,容易出現過度擬合和捷徑學習,並且在不熟悉的輸入上可能會出錯。
- Epoch AI 估計,自 2020 年以來,前緣訓練計算每年增長約 5 倍,而固定性能水準的成本卻大幅下降。
深入探討
Deep learning is a way of building software that learns a task from examples. Its core object is an artificial neural network: a large collection of simple arithmetic units, loosely inspired by brain cells, arranged in layers. Each unit multiplies its inputs by adjustable numbers called weights, adds the results, and passes the total through a simple nonlinear function. The word deep refers to the number of layers between the raw input, such as the pixels of a photo, and the final answer, such as the label cat. In their 2015 review in Nature, Yann LeCun, Yoshua Bengio and Geoffrey Hinton describe the central idea as learning representations with multiple levels of abstraction. In an image model, early layers tend to respond to edges, middle layers to combinations of edges and textures, and later layers to parts of objects. Nobody programs those detectors. They emerge during training.
Training follows the same loop for almost every deep network. First, gather many examples with known answers, called labels. Second, run an example through the network to get a prediction; this is the forward pass. Third, score the prediction with a loss function, a formula that produces a large number when the prediction is far from the label and a small one when it is close. Fourth, use backpropagation to work out, for every weight, how much a tiny change would raise or lower the loss. That quantity is called the gradient. Fifth, nudge every weight a small step in the direction that lowers the loss, a method called gradient descent. Sixth, repeat over millions of examples, usually in small groups called batches. When training ends, the weights are frozen and the network is used on new inputs, which is called inference.
Backpropagation is the step that makes depth practical. David Rumelhart, Geoffrey Hinton and Ronald Williams described it in Nature in October 1986: the procedure repeatedly adjusts connection weights to shrink the gap between the network's actual output and the desired output. Their key observation was that hidden units, the ones that are neither input nor output, come to represent useful features of the task on their own. The 2015 Nature review explains why that matters. Conventional machine learning needed careful engineering and domain expertise to design a feature extractor before any learning could happen. Deep learning moves that work into the training process.
Different layer designs suit different data. Convolutional networks slide the same small filter across every position of an image, so a pattern learned in one corner is recognized anywhere. Recurrent networks read a sequence one step at a time and carry a memory forward, which suited speech and text for many years. Transformers, introduced by Ashish Vaswani and colleagues at Google in June 2017, replace that step-by-step reading with attention, a mechanism that lets every position in a sequence look directly at every other position. Because attention can be computed for all positions at once, the original Transformer trained to a new English-to-French translation record in 3.5 days on eight GPUs. Residual connections, introduced by Kaiming He and colleagues in December 2015, let a layer's input skip ahead and be added to its output. That simple shortcut made networks of 152 layers trainable, which the authors describe as eight times deeper than the earlier VGG networks.
The modern boom has a clear starting point. In 2012, Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton entered AlexNet in the ImageNet image classification contest. The network had 60 million weights and 650,000 units, and it was trained for about 90 passes over 1.2 million labeled images, which took five to six days on two NVIDIA GTX 580 graphics cards with 3 GB of memory each. Its winning top-5 error of 15.3% compared with 26.2% for the next entry. The authors wrote that their results could likely improve simply by waiting for faster GPUs and bigger datasets, and the following decade largely followed that recipe: more data, more compute, and better architectures such as ResNet in 2015 and the Transformer in 2017. By 2024, according to the Stanford AI Index 2025, nearly 90% of notable new AI models came from industry rather than universities, up from 60% in 2023, a sign of how expensive frontier work has become.
It helps to place deep learning among its alternatives. Rule-based software follows instructions a person wrote, which is predictable and easy to audit but breaks on messy inputs like handwriting or accents. Classical machine learning, such as logistic regression or decision trees, learns from data but usually relies on features a person chose in advance. Deep learning learns the features too, which is why it dominates tasks with raw, unstructured inputs: images, audio, video and free text. For small tables of numbers, such as a spreadsheet of a few thousand loan records, simpler methods are cheaper to train and easier to explain, and they remain a sensible first try. Deep learning is also distinct from generative AI: generative systems are one use of deep networks, not a separate technology.
The method has real limits. Deep networks need large amounts of data, and they can memorize their training examples instead of learning general patterns, a failure called overfitting. The AlexNet authors reported substantial overfitting until they added dropout, a technique that randomly switches off half of the units in certain layers during each training step. A trained network's knowledge is spread across millions or billions of weights, so it is hard to say why it produced a particular answer. Networks can latch onto accidental shortcuts in the data, such as a background that happens to appear with one label, and they can fail without warning on inputs unlike anything in training. They also give confident answers when they are wrong unless they are specifically calibrated. None of these problems is solved by adding more layers.
The practical requirements depend heavily on scale. A small image classifier can be trained on a single consumer graphics card in hours, and most practitioners start from a network someone else already trained and adjust it for their task, which is called fine-tuning. Frontier models are a different world. Epoch AI estimates that training compute for frontier language models has grown about 5 times per year since 2020 and that frontier training costs have grown about 3.5 times per year, and it puts the up-front cost of a 1-gigawatt AI data center at roughly 38 billion dollars. Running models has become far cheaper, though: the Stanford AI Index 2025 reports that the price of GPT-3.5-level performance fell more than 280-fold between November 2022 and October 2024, while hardware costs fell about 30% per year and energy efficiency improved about 40% per year.
04工作範例
手動追蹤一個梯度下降
採用盡可能最小的網路:預測 y = w * x 的一個權重 w。從 w = 0.5 開始。
使用訓練範例:輸入 x = 2 和正確答案 3。前向傳播預測 0.5 * 2 = 1。
使用平方誤差損失(預測 - 答案)平方對預測進行評分,得出 (1 - 3) 平方 = 4。
使用鍊式法則計算梯度:dLoss/dw = 2 * (預測 - 答案) * x = 2 * (-2) * 2 = -8。
使用學習率 0.05 更新權重:w := 0.5 - 0.05 * (-8) = 0.9。
再次運行前向傳遞:0.9 * 2 = 1.8,因此損失現在為 (1.8 - 3) 平方 = 1.44。
重複同樣的四個步驟幾次,並記錄每一步後的體重和損失。
它顯示了什麼
一步後損失從 4 下降到 1.44,並不斷縮小,w 接近 1.5,即 2 到 3 的精確映射值。真正的深度網路正是執行這種計算,一次僅針對數百萬或數十億的權重,並透過反向傳播提供每個梯度。
現實世界的實施
影像辨識:AlexNet 是一個具有 5 個卷積層和 3 個全連接層的網絡,它以 15.3% 的 top-5 測試誤差贏得了 ILSVRC-2012 競賽,而第二名的得分為 26.2%。
非常深的視覺模型:Microsoft Research 的 2015 年 ResNet 論文使用快捷連接訓練了高達 152 層的網絡,並在 ILSVRC 2015 分類任務中以 3.57% 的錯誤率獲得第一名。
機器翻譯:2017 年 Transformer 在 WMT 2014 英語到德語基準測試中達到了 28.4 BLEU,比之前的最佳成績高出了 2 BLEU 以上,並在英語到法語測試中創下了 41.8 BLEU 的單一模型記錄。
語音、科學和醫學:LeCun、Bengio 和 Hinton 在 2015 年《自然》雜誌上發表的評論認為,深度網絡在語音識別、物體檢測、藥物發現和基因組學等手工設計功能停滯不前的領域取得了巨大進展。
深度學習的未來
幾個懸而未決的問題影響著目前的研究。首先是效率。如果前緣訓練計算繼續以 Epoch AI 自 2020 年以來測得的每年 5 倍的速度增長,所需的能源和資本將急劇增加; Epoch 指出,到 2025 年,最大的人工智慧資料中心的電力容量已經接近 1 吉瓦。這種壓力推動了更小的模型、更好的訓練方案和更便宜的晶片的發展,斯坦福人工智慧指數 2025 記錄的硬體成本每年下降約 30%。二是競爭對手之間的趨同。 AI Index 2025發現,在某些基準上,排名第一的模型和排名第十的模型之間的分數差距從11.9%縮小到5.4%,領先的開放權重模型和封閉模型之間的差距從8%縮小到1.7%。現在的優勢更多來自數據、計算或工程還是一個懸而未決的問題。三是理解。深度網路的工作效果仍然比任何人都可以完全解釋的要好,可解釋性研究試圖映射各個單元和電路的計算內容。對陌生輸入的可靠性、誠實的不確定性估計以及對捷徑學習的抵制仍然沒有得到解決。簡單地擴展當前架構是否會繼續產生收益,或者是否需要新的想法,正在積極爭論而不是解決。
付諸實踐
為什麼這很重要
- 更明確的決策. 它可以幫助您將清晰的技術聲明與行銷語言分開。
- 成本與預算. 在花費金錢或時間之前,您可以提出更好的實施問題。
- 團隊與工作流程. 具有共同理解的團隊可以做出更好的產品、政策和學習決策。
留意
- 不同的團隊可能會以不同的方式使用相同術語,因此請儘早定義範圍。
- 基準測試可能看起來很強大,但實際效能卻參差不齊。
- 忽視數據品質和評估計劃通常會產生脆弱的結果。
後續步驟
- 從您需要的結果的簡單語言定義開始。
- 在測試之前選擇一種成功指標和一種失敗條件。
- 使用代表性資料運行小型試點,而不是完善的演示集。
- 記錄深度學習在哪些方面有幫助以及在哪些方面更簡單的方法更好。
資料來源與延伸閱讀
- Nature (LeCun, Bengio and Hinton, 2015)深度學習
- Nature (Rumelhart, Hinton and Williams, 1986)透過反向傳播誤差來學習表示
- NeurIPS 2012使用深度卷積神經網路進行 ImageNet 分類
- arXiv (Microsoft Research, 2015)用於影像辨識的深度殘差學習
- arXiv (Google, 2017)您所需要的就是關注
- Stanford Institute for Human-Centered AI2025年人工智慧指數報告
- Epoch AI機器學習趨勢
不斷探索
常見問題
什麼是深度學習?
深度學習是機器學習的一個分支,它訓練具有許多堆疊層的人工神經網絡,每個層將其輸入轉換為稍微更有用的表示。它是現代圖像識別、語音識別、翻譯和聊天機器人背後的技術,因為各層從示例中學習自己的特徵,而不是依賴手工編寫的規則或特徵。
深度學習模型中的「深度」描述什麼?
在本指南中,深度是指多層網路結構;它沒有建立類似人類的理解或表現。
網路有 2 個輸入、3 個和 2 個單元的隱藏層以及 1 個輸出。對於每個隱藏/輸出單元的偏差,有多少個權重和偏差?
這些層貢獻 (2×3+3)=9、(3×2+2)=8 和 (2×1+1)=3,總共 20 個。
為什麼在隱藏層之間要使用非線性活化?
該指南解釋說,線性運算的堆疊仍然是線性的;非線性活化讓網路表示非線性關係。
在推理過程中,訓練後的模型參數通常會發生什麼?
使用訓練好的模型進行推理;訓練是更新參數的過程。
模型非常適合其訓練範例,但在可比較的保留資料上表現不佳。這說明了什麼規律呢?
過度擬合是指較強的訓練表現與較弱的新資料表現結合。