Why Backprop Goes Backward
- The usual explanation of backpropagation (Rumelhart et al., 1986), the algorithm used to train neural networks, is that it is propagating errors for each node backwards.
- But when I first learned about the algorithm, I had a question that I could not find answered directly: why does it have to go backwards?
- A neural network is just a composite function, and we know how to compute the derivatives of composite functions using the chain rule.
Unverified
- The usual explanation of backpropagation (Rumelhart et al., 1986), the algorithm used to train neural networks, is that it is propagating errors for each node backwards.
- But when I first learned about the algorithm, I had a question that I could not find answered directly: why does it have to go backwards?
- A neural network is just a composite function, and we know how to compute the derivatives of composite functions using the chain rule.
Sources: Gregorygundersen