Neural Networks

Layers transform the input into an output, and training adjusts those parameters to improve performance on a chosen objective.

Key takeaways

• Weights and biases are learned parameters; activation functions transform intermediate results.
• Backpropagation calculates gradients used by an optimizer.
• An internal activation is not automatically a probability or an explanation.

A basic artificial neuron combines input values using weights, adds a bias, and applies an activation function. The weights control how strongly each input contributes. The bias shifts the result. A nonlinear activation lets layers represent relationships that a stack of purely linear operations could not. For example, the ReLU activation returns zero for a negative input and leaves a positive input unchanged. Networks can use different activations in different layers. An output layer is chosen to suit the task: a numeric prediction is not interpreted in the same way as scores for possible categories. During training, a loss function compares the output with the desired result. Backpropagation uses the chain rule to calculate how parameters affect the loss. An optimizer then uses that information to update parameters. Backpropagation computes gradients; it is not a guarantee that the model will find the best possible solution or generalize well. The brain analogy is limited. Artificial neurons are mathematical abstractions, and a successful network is not evidence of a human-like mind. A larger network can model complicated relationships, but it can also cost more to run, fit irrelevant patterns, or fail when conditions change. Compare it with a simpler baseline and test on examples outside the training data.

1. Use two inputs, 0.8 and 0.5, with weights 0.6 and -0.4 and a bias of 0.1.
2. The weighted sum is (0.8 × 0.6) + (0.5 × -0.4) + 0.1 = 0.38.
3. ReLU returns 0.38. If the second input changes to 1.5, the sum becomes -0.02 and ReLU returns 0.

This illustrative calculation is one transformation inside a network. The value 0.38 is an activation, not a 38% confidence claim.