The Softmax function and its derivative
Eli Bendersky provides a thorough mathematical breakdown of the softmax function, its Jacobian derivative, numerical stability considerations, and its application with cross-entropy loss in machine learning.
read it A clear, first-principles walkthrough of vector calculus, Jacobians, and numerical stability for neural network layers that rewards deep reading.
summary
This article explains the softmax function, its probabilistic interpretation, and how to compute its vector derivative using vector calculus and the multivariate chain rule. It details the step-by-step mathematical derivation of both the standalone softmax function and a fully-connected softmax layer combined with cross-entropy loss. Additionally, it addresses practical implementation concerns like numerical stability to prevent floating-point overflows and NaNs.
key points
- Softmax converts an N-dimensional real vector into a probability distribution summing to 1.0, acting as a soft version of the maximum function.
- Computing the derivative requires the Jacobian matrix rather than a standard gradient because softmax maps vectors to vectors.
- Subtracting the maximum input value from the vector elements prevents exponential overflow and resulting NaNs during implementation.
- The multivariate chain rule allows systematic derivation of complex compositions like a fully-connected layer followed by softmax and cross-entropy loss.
who should read it
Machine learning practitioners and software engineers looking for a rigorous, first-principles understanding of neural network math and derivatives.
Generated by an LLM from the article and thread. Check the sources before relying on it.