Gradient points in the direction of steepest ascent

Statement

Let $U \subseteq \mathbb{R}^n$ be open, let $f \colon U \to \mathbb{R}$ be differentiable at $a \in U$, and suppose that $\nabla f(a) \ne 0$. Then for each unit vector $u \in \mathbb{R}^n$, the directional derivative satisfies $-\|\nabla f(a)\| \le D_u f(a) \le \|\nabla f(a)\|$. The upper bound is attained if and only if $u = \nabla f(a) / \|\nabla f(a)\|$, and the lower bound if and only if $u = -\nabla f(a) / \|\nabla f(a)\|$.

Remarks

The proof combines $D_u f(a) = \nabla f(a) \cdot u$, from the statement that the derivative gives the directional derivatives, with the Cauchy–Schwarz inequality $|w \cdot u| \le \|w\| \, \|u\|$ and its case of equality. So the gradient gives the direction of fastest increase, and its norm is the fastest rate of increase per unit of distance. The method of gradient descent uses the opposite direction $-\nabla f(a)$ to look for a minimum of $f$.

Explanations

L1 human

If you stand on a hillside and want to climb as fast as possible, walk in the direction of the gradient. If you walk at right angles to the gradient, your height does not change at first: you follow a contour line.