Functions of several variables

The last chapter left us on a mountainside with a simple question: which way should you go to climb as fast as possible? All our tools, the derivative, the integral, differential equations, are designed for functions of one number. But the height on a slope depends on two: how far you’ve walked east and how far north. A step east may lead up, a step north down, and about a step north-east you can’t say anything until you’ve done the sums.

There are more such functions around us than functions of one variable. The temperature in a room depends on three coordinates and on time. The price of a plane ticket depends on the departure date, the return date and the number of seats already sold. The error of a neural network depends on millions of parameters at once. We’ll set off on an expedition across terrain with just two coordinates: that’s enough to see everything important, and the formulas then carry over to any number of variables almost unchanged. We’ll take a map, a magnifier and a compass. By the end of the trip they’ll have found us peaks and passes, the highest point of a trail, the way down in fog, and even the area under a curve that has no antiderivative.

A map instead of a mountain

Choose an origin on the terrain, point the $x$-axis east and the $y$-axis north. Each point $(x, y)$ has a height $z = f(x, y)$.

It’s hard to draw a surface on paper so that anything can be measured from the drawing. Map-makers found a way out long ago: on a topographic map they draw contour lines, lines along which the height is the same, for example 100, 200 and 300 metres above sea level.

We’ve already met level curves in chapter 11: the profit $300x + 400y$ there was a linear function, and its level curves were parallel lines. On a real mountain they bend. Let’s start with a simple hill, $f(x, y) = 6 - x^2 - 2y^2$: distances in kilometres, height in hundreds of metres. Its level curves $x^2 + 2y^2 = 6 - c$ are nested ellipses, stretched from west to east. Where the contours crowd together, the slope is steep; where they spread apart, it’s gentle. At the same distance from the summit, the slope to the north is twice as steep here as to the east.

The water level is a handy way to see all the level curves one after another. Lowering the water over an archipelago, you’ll notice a pattern: the number of islands doesn’t change just anywhere. A new island appears when a summit comes out of the water, and two islands merge into one when the water drops below the pass between them. Summits and passes are special points of the map, and we’ll come back to them.

Can two level curves of the same function, $f = 1$ and $f = 2$, cross?

No, never Yes, at a pass Yes, at a summit

At a crossing point the function would equal both $1$ and $2$ at once, but a function has one value at each point. Contours of different heights on a map never touch or cross.

Slices

The simplest way to measure steepness is to walk due east. Then $y$ doesn’t change, and the height becomes a function of $x$ alone. We’ve cut the mountain with the vertical plane $y = b$ and are looking at the profile of the cut, an ordinary graph of a function of one variable. We’ve known how to find its slope since chapter 26.

We shift only the first coordinate, that is, we step due east. The second coordinate is frozen: we stay in the plane of the cut, $y = b$. The length of the step. In the limit we get the slope of the profile: how many times faster than $x$ the height changes. Example: for the hill $f = 6 - x^2 - 2y^2$ with $y$ frozen, the term $-2y^2$ is a constant, so $f_x = -2x$; likewise $f_y = -4y$. At the point $(1, 1)$ we get $f_x = -2$ and $f_y = -4$: to the east the slope goes down, and to the north down twice as steeply.

Partial derivatives are easy to compute: treat all the variables but one as numbers and differentiate by the old rules. Their meaning is quite practical. The volume of a tin can, $V(r, h) = \pi r^2 h$, depends on the radius and the height, and $\frac{\partial V}{\partial r} = 2\pi rh$, $\frac{\partial V}{\partial h} = \pi r^2$. For a can with $r = 3$ cm and $h = 10$ cm these are $60\pi \approx 188$ and $9\pi \approx 28$ cubic centimetres per centimetre. Add a millimetre to the radius and the volume grows by about $19$ cm³, while a millimetre on the height gives less than $3$ cm³: the radius is almost seven times more “expensive”.

Find $\frac{\partial f}{\partial y}$ for $f(x, y) = x^3y^2 - 4xy + y$.

Treat $x$ as a number. Then $x^3y^2$ is $x^3$ times $y^2$, and its derivative with respect to $y$ is $2x^3y$; the derivative of $-4xy$ is $-4x$; of $y$, one. Altogether $\frac{\partial f}{\partial y} = 2x^3y - 4x + 1$.

The magnifier

Under the microscope of chapter 26 a smooth graph became a straight line. With a surface it’s the same: the higher the magnification, the more the piece of mountain around a point looks like a plane. This plane passes through the tangents to both slices, and a plane is determined uniquely by two intersecting lines.

The height at the point itself: the plane passes through the point of the surface. The slope to the east. At $y = b$ the formula turns into the tangent to the blue slice. The slope to the north: at $x = a$ we get the tangent to the orange slice. Example: for the hill at the point $(1, 1)$ we get $z = 3 - 2(x - 1) - 4(y - 1)$. At the point $(1.1, 0.9)$, a hundred metres to the east and to the south, the plane gives $3 - 0.2 + 0.4 = 3.2$, and the hill itself $6 - 1.21 - 1.62 = 3.17$. The error is $0.03$. A shift half as big, to the point $(1.05, 0.95)$, gives an error four times smaller, $0.0075$: it falls faster than the shift itself.

The formula for the plane makes sense whenever both partial derivatives exist. But the “fan” shows that the plane may then mean nothing. We need a requirement that speaks about all directions at once.

If a function is differentiable, the numbers $A$ and $B$ have to be the partial derivatives: put $k = 0$, divide by $h$ and let $h$ tend to zero, and you get $A = f_x(a, b)$; likewise $B = f_y(a, b)$. The “fan” isn’t differentiable at zero: along the diagonal the error of the plane $z = 0$ is $\frac12$ and doesn’t get smaller. The cone $\sqrt{x^2 + y^2}$ doesn’t even have partial derivatives at its tip: the slopes to the left and right are $-1$ and $1$. How can you tell a function is differentiable without checking every direction? Continuity of the partial derivatives is enough.

The idea: go from point to point not directly but by a staircase, one coordinate at a time. On each step the theorem for functions of one variable works.

Take a point $B = (a + h, b + k)$ so close to $A$ that the whole picture lies in the neighbourhood from the statement, and an intermediate point $C = (a, b + k)$. The change in height splits into two steps: $f(B) - f(A) = \big(f(C) - f(A)\big) + \big(f(B) - f(C)\big)$. On the segment $CB$ only $x$ changes. The function $x \mapsto f(x, b + k)$ is differentiable on it, because $f_x$ exists there. By Lagrange’s mean value theorem (chapter 27) there’s a point $M_1$ on the segment $CB$ at which $f(B) - f(C) = f_x(M_1)\,h$. On the segment $AC$ only $y$ changes, and the same theorem for the function $y \mapsto f(a, y)$ gives a point $M_2$ on it at which $f(C) - f(A) = f_y(M_2)\,k$. Both points $M_1$ and $M_2$ lie no further than $|AB| = \sqrt{h^2 + k^2}$ from $A$ and tend to $A$ together with $B$. By continuity $f_x(M_1) = f_x(A) + \varepsilon_1$ and $f_y(M_2) = f_y(A) + \varepsilon_2$, where $\varepsilon_1, \varepsilon_2 \to 0$. Add the steps: $f(B) - f(A) = f_x(A)h + f_y(A)k + \varepsilon_1h + \varepsilon_2k$. The remainder is at most $(|\varepsilon_1| + |\varepsilon_2|)\sqrt{h^2 + k^2}$ in absolute value, since $|h|$ and $|k|$ are at most $|AB|$. That’s exactly the definition of differentiability.

Polynomials, exponentials, sines and their combinations have continuous partial derivatives wherever they’re defined, so they have a tangent plane at every point. The “fan” and the cone are exceptions put together on purpose.

The compass

A hiker on a slope doesn’t have to walk only east or north. Suppose they walk in the direction of the unit vector $\mathbf u = (\cos\varphi, \sin\varphi)$; vectors, as in the university chapters that follow, are written in bold. How fast does the height change?

We won’t have to compute it from the definition for each direction. Everything is settled by a rule that links the change of a function to motion along any curve.

For the line $\gamma(t) = P + t\mathbf u$ the velocity is $\mathbf u = (\cos\varphi, \sin\varphi)$, so $D_{\mathbf u}f = f_x\cos\varphi + f_y\sin\varphi$. On the right is the dot product from chapter 22, of the vector $\mathbf u$ and the vector of the two partial derivatives. That vector deserves a name.

The gradient $\nabla f = (f_x, f_y)$: a single vector recording the slopes in all directions at once. The unit vector of the direction the hiker is walking in. The angle between the direction and the gradient. The formula holds for a differentiable function; for the “fan” at zero it fails. Example: the hill $f = 6 - x^2 - 2y^2$ at the point $(1, 1)$, $\nabla f = (-2, -4)$. Walk straight towards the summit, $\mathbf u = \frac{1}{\sqrt2}(-1, -1)$: $D_{\mathbf u}f = \frac{2 + 4}{\sqrt2} = 3\sqrt2 \approx 4.24$. But along the gradient the climb is $|\nabla f| = \sqrt{20} \approx 4.47$, steeper, although the gradient doesn’t point at the summit.

The idea: the dot product with a unit vector is a projection, and a projection is no longer than the vector itself.

By the chain rule $D_{\mathbf u}f = \nabla f \cdot \mathbf u$. From $P$ draw the gradient $\overrightarrow{PG} = \nabla f(P)$ and the unit vector $\mathbf u$. Drop a perpendicular from $G$ to the line along $\mathbf u$ and call its foot $Q$. The dot product with a unit vector is the projection (chapter 22), so $D_{\mathbf u}f$ is the signed length of $PQ$: plus if $Q$ lies on the side of $P$ that $\mathbf u$ points to. $PQ$ is a leg of the right triangle $PQG$ with hypotenuse $PG = |\nabla f|$, and a leg is no longer than the hypotenuse. So $|D_{\mathbf u}f| \le |\nabla f|$, with equality only when $Q = G$, that is, when $\mathbf u$ points along $\nabla f$ (a climb of $|\nabla f|$) or against it (a descent of $-|\nabla f|$). The angle $PQG$ is a right angle, so all the points $Q$ lie on the circle with diameter $PG$: that’s the compass “rose”. $D_{\mathbf u}f = 0$ when $Q$ coincides with $P$, that is, $\mathbf u \perp \nabla f$: exactly two directions. Along a level curve the height is constant: $f(\gamma(t)) = c$, and by the chain rule $\nabla f(P) \cdot \gamma'(0) = 0$. The tangent to the level curve runs along $\gamma'(0)$, and it’s perpendicular to the gradient.

The path of steepest ascent always runs along the gradient and so crosses the contours at right angles. On our hill, from the point $(1, 1)$ it doesn’t lead straight to the summit. The condition “the velocity points along $\nabla f = (-2x, -4y)$” gives the equation $\frac{dy}{dx} = \frac{2y}{x}$, and its solution by the methods of the last chapter is the parabola $y = x^2$. Streams flow the same way, only against the gradient. That’s why rivers on maps cross the contours at right angles.

A hiker stands on the slope of the elliptical hill $f = 6 - x^2 - 2y^2$ at the point $(1, 1)$. Where does the gradient point?

Straight at the summit Perpendicular to the contour through the hiker Along the contour

The gradient is always perpendicular to the level curve. It points at the summit only when the contours are circles around it.

The quadratic forms from chapter 38 give an instructive example. For $Q(x, y) = ax^2 + 2bxy + cy^2 = \langle \mathbf x, S\mathbf x\rangle$ with the symmetric matrix $S = \left(\begin{smallmatrix} a & b \\ b & c \end{smallmatrix}\right)$ the gradient is $(2ax + 2by, 2bx + 2cy) = 2S\mathbf x$. So the ends of the axes of the ellipse $Q = \text{const}$, the points where the normal $2S\mathbf x$ runs along the radius, lie on the eigenvectors of $S$.

Find the gradient of the function $5x^2 + 4xy + 2y^2$ at the point $(1, 1)$.

$f_x = 10x + 4y$, $f_y = 4x + 4y$; at the point $(1, 1)$ we get $\nabla f = (14, 8)$. Check with the matrix: $S = \left(\begin{smallmatrix} 5 & 2 \\ 2 & 2 \end{smallmatrix}\right)$, $S\mathbf x = (7, 4)$, and $2S\mathbf x = (14, 8)$.

Peaks and passes

At a summit the compass is lost: there’s nowhere higher to go. This can be stated precisely.

The converse is false, and mountains know this well. Take $f(x, y) = x^3 - 3x - y^2$. The equations $f_x = 3x^2 - 3 = 0$ and $f_y = -2y = 0$ give two stationary points: $(-1, 0)$ and $(1, 0)$. At the first $f = 2$, and it’s a summit. At the second $f = -2$, but it’s neither a summit nor a pit: along the $x$-axis the point $x = 1$ is the bottom of a hollow (the function $x^3 - 3x$ has a minimum there), while across it, in the direction of the $y$-axis, it’s a ridge ($-2 - y^2$ has a maximum). Mountain roads are laid through exactly such points.

To tell summits, pits and saddles apart, we need the next term of the approximation after the plane: the second derivatives. Write $f_{xx} = \frac{\partial^2 f}{\partial x^2}$, $f_{yy}$ and the mixed $f_{xy} = \frac{\partial}{\partial y}\frac{\partial f}{\partial x}$. Near a stationary point $P$ the surface looks like the graph of the quadratic form $\frac12(Ah^2 + 2Bhk + Ck^2)$ with coefficients $A = f_{xx}(P)$, $B = f_{xy}(P)$, $C = f_{yy}(P)$, and a single number decides everything.

The bend of the profile along the $x$-axis: negative at a summit, positive at a pit. The bend along the $y$-axis. The mixed derivative: how “twisted” the surface is, like a twisted sheet of paper. It can turn two bends of the same sign into a saddle. Example: for $x^3 - 3x - y^2$ we have $f_{xx} = 6x$, $f_{yy} = -2$, $f_{xy} = 0$. At the point $(-1, 0)$: $\Delta = (-6)(-2) - 0 = 12 > 0$ and $f_{xx} < 0$, a maximum. At the point $(1, 0)$: $\Delta = 6 \cdot (-2) = -12 < 0$, a saddle.

The idea: reduce everything to a function of one variable along a segment, and find the sign of the quadratic form by completing the square, as in chapter 10.

Take a shift $(h, k)$ and the function $\varphi(t) = f(P + t(h, k))$. Applying the chain rule twice, we get $\varphi'(t) = f_xh + f_yk$ and $\varphi''(t) = f_{xx}h^2 + 2f_{xy}hk + f_{yy}k^2$ (all the derivatives at the point $P + t(h, k)$; the equation $f_{xy} = f_{yx}$ is proved below, in the section on mixed derivatives). By Taylor’s formula with the Lagrange form of the remainder (chapter 30) $\varphi(1) = \varphi(0) + \varphi'(0) + \frac12\varphi''(\theta)$ for some $0 < \theta < 1$. At a stationary point $\varphi'(0) = 0$, so $f(P + (h, k)) - f(P) = \frac12 Q_M(h, k)$, where $Q_M(h, k) = A_Mh^2 + 2B_Mhk + C_Mk^2$ is the form with the second derivatives at the point $M = P + \theta(h, k)$. Complete the square: $A\,Q(h, k) = (Ah + Bk)^2 + (AC - B^2)k^2$. If $\Delta > 0$, then $A \ne 0$, and the right side is positive for $(h, k) \ne (0, 0)$: for $k \ne 0$ because of the second term, for $k = 0$ because of the first. So $Q$ has the sign of $A$ everywhere except zero; its level curves are ellipses. The second derivatives are continuous, so the numbers $A_M$ and $\Delta_M = A_MC_M - B_M^2$ keep the signs of $A$ and $\Delta$ for $M$ close to $P$. So for all small enough non-zero shifts the difference $f(P + (h, k)) - f(P) = \frac12Q_M(h, k)$ has the sign of $A$: for $A > 0$ the point $P$ is a strict minimum, for $A < 0$ a maximum. Let $\Delta < 0$. If $A \ne 0$, then $Q(1, 0) = A$ and $Q(-B, A) = A(AC - B^2) = A\Delta$ are numbers of opposite signs. If $A = 0$, then $B \ne 0$ (otherwise $\Delta = 0$), and $Q(t, 1) = 2Bt + C$ changes sign along with $t$. In both cases there are vectors $\mathbf v$ and $\mathbf w$ with $Q(\mathbf v) > 0 > Q(\mathbf w)$. For small $s$ the form $Q_M(s\mathbf v) = s^2Q_M(\mathbf v)$ keeps the sign of $Q(\mathbf v)$ by continuity, and likewise for $\mathbf w$. So near $P$ there are points both higher and lower than it: a saddle.

The case $\Delta = 0$ really is hopeless for the test: for $x^4 + y^4$, for $x^4 - y^4$ and for the “monkey saddle” $x^3 - 3xy^2$ all the second derivatives at zero vanish, yet the behaviour differs: a minimum, an ordinary saddle, and a saddle with three hollows, for two legs and a tail. You can look at it in the chapter’s first widget.

Why mixed derivatives don’t depend on the order

If the mixed derivatives $f_{xy}$ and $f_{yx}$ exist near the point $(a, b)$ and are continuous at it, then $f_{xy}(a, b) = f_{yx}(a, b)$.

Let’s compute one quantity in two ways. Let $D = f(a + h, b + k) - f(a + h, b) - f(a, b + k) + f(a, b)$ be the sum of the heights at the corners of a rectangle, with a plus at two opposite corners and a minus at the other two. Put $u(x) = f(x, b + k) - f(x, b)$; then $D = u(a + h) - u(a)$. By Lagrange’s theorem $D = u'(\xi)h = \big(f_x(\xi, b + k) - f_x(\xi, b)\big)h$ for some $\xi$ between $a$ and $a + h$. The difference in brackets is the increment of the function $y \mapsto f_x(\xi, y)$, and by the same theorem it equals $f_{xy}(\xi, \eta)k$. So $D = f_{xy}(\xi, \eta)hk$ for some point $(\xi, \eta)$ of the rectangle. If we start from the function $v(y) = f(a + h, y) - f(a, y)$, then $D = v(b + k) - v(b)$, and in the same way we get $D = f_{yx}(\xi', \eta')hk$. So $f_{xy}(\xi, \eta) = f_{yx}(\xi', \eta')$. As $h, k \to 0$ both points tend to $(a, b)$, and by continuity $f_{xy}(a, b) = f_{yx}(a, b)$.

Euler and Clairaut were already using the equation in the eighteenth century, and a rigorous proof for continuous derivatives was published by Hermann Schwarz in 1873. Without continuity the theorem is false: for the function $\frac{xy(x^2 - y^2)}{x^2 + y^2}$, $f(0, 0) = 0$, at zero $f_{xy} = -1$ while $f_{yx} = 1$.

Find the stationary point of the function $f(x, y) = x^2 + xy + y^2 - 3x$. Is it a summit, a pit or a saddle?

$f_x = 2x + y - 3 = 0$, $f_y = x + 2y = 0$. From the second equation $x = -2y$; substitute into the first: $-3y = 3$, $y = -1$, $x = 2$. The second derivatives: $A = 2$, $B = 1$, $C = 2$, $\Delta = 4 - 1 = 3 > 0$ and $A > 0$, so it’s a minimum, $f(2, -1) = 4 - 2 + 1 - 6 = -3$.

The trail

The expedition won’t make it to the summit: the route follows a trail that skirts round the mountain. Which point of the trail is the highest? This is a problem about a constrained extremum: the maximum of $f(x, y)$ is sought not over the whole map but only among points satisfying the condition $g(x, y) = c$, the equation of the trail.

Look at the map. Where the trail crosses a contour, a step one way leads higher and the other way lower, so such a point can’t be the highest. The highest point of the trail is where the trail doesn’t cross a contour but touches it. Touching curves share a tangent, and gradients are perpendicular to their level curves. So at that point the gradients of $f$ and $g$ lie on one line.

The gradient of the function we’re maximising or minimising. The gradient of the constraint function; it’s perpendicular to the trail $g = c$ and must be non-zero. The number $\lambda$ is the Lagrange multiplier. Three unknowns $x$, $y$, $\lambda$ and three equations: two for the coordinates and the constraint itself. Example: the largest value of $x + 2y$ on the circle $x^2 + y^2 = 5$. The equations $1 = 2\lambda x$ and $2 = 2\lambda y$ give $y = 2x$, then $5x^2 = 5$ and $x = \pm 1$. At the point $(1, 2)$ the value $5$ is the maximum, at the point $(-1, -2)$ the value $-5$ is the minimum; there $\lambda = \pm\frac12$.

The idea: split $\nabla f$ into a part along the curve and a part across it. The part along the curve has to vanish.

Along the curve $g(\gamma(t)) = c$, and by the chain rule $\nabla g(P) \cdot \gamma'(0) = 0$: the gradient of $g$ is perpendicular to the curve, that is, it points along the normal to it. Split $\nabla f(P)$ into two parts: one along the tangent, which points like $\gamma'(0)$, and one along the normal. The height along the curve, $\varphi(t) = f(\gamma(t))$, has an extremum at $t = 0$, so $\varphi'(0) = 0$ by Fermat’s theorem. By the chain rule $\varphi'(0) = \nabla f(P) \cdot \gamma'(0)$: the tangential part of $\nabla f$ times $|\gamma'(0)|$. If it were non-zero, a shift along the curve in its direction would raise $f$, and in the opposite direction lower it. So the tangential part is zero, and $\nabla f(P)$ lies on the normal, on the same line as the non-zero vector $\nabla g(P)$. Hence $\nabla f(P) = \lambda\nabla g(P)$. On the map this means the level curve of $f$ touches the curve $g = c$.

We solved the tin can problem in chapter 27: we expressed the height through the radius and looked for the minimum of a function of one variable. Lagrange multipliers manage without that substitution. The surface area is $S = 2\pi r^2 + 2\pi rh$, and the volume $V = \pi r^2h$ is given. The equations $\nabla S = \lambda\nabla V$ in the coordinates $r$ and $h$:

$$4\pi r + 2\pi h = \lambda \cdot 2\pi rh, \qquad 2\pi r = \lambda\pi r^2.$$

From the second, $\lambda = \frac2r$; substitute into the first: $4\pi r + 2\pi h = 4\pi h$, that is, $h = 2r$. The height equals the diameter, whatever the volume. For a one-litre can $r = \sqrt[3]{500/\pi} \approx 5.42$ cm, $h \approx 10.84$ cm, and it takes about $554$ cm² of tinplate.

Descent in fog

While it’s light, summits and pits can be found on the map by solving the equation $\nabla f = \mathbf 0$. But such equations often can’t be solved explicitly, and there can be millions of variables. Then you do what a hiker does in fog: look at the slope underfoot and step where it drops away most steeply, that is, against the gradient.

Where we are now: the point after $k$ steps. The step size, a positive number. Too small a step and we walk for years; too big and we jump over the bottom. The gradient at the current point. The minus in front turns us downhill; on a steep slope the step is longer, near the bottom shorter. Example: $f = \frac12(x^2 + 10y^2)$, $\nabla f = (x, 10y)$, start at $(2, 1)$, $\eta = 0.1$. The first step: $(2, 1) - 0.1 \cdot (2, 10) = (1.8, 0)$. The $y$-coordinate vanished at once, while $x$ will then shrink by only $10\,\%$ per step.

The method was proposed by Cauchy in 1847 for solving systems of equations. Today, with various improvements, it’s used to train neural networks: the height is the model’s error, and the coordinates are its parameters. What this looks like in practice is shown in the chapter Walking downhill of the Sprout course, where a small language model learns by gradient descent.

The ravine shows descent’s main trouble. Across the ravine the slopes are steep, and a step suited to them is too small for the gentle floor: the hiker trudges along the ravine for a long time. Make the step longer and the hiker starts jumping from slope to slope; a bit more and the descent blows up. Where the boundary lies can be said precisely.

The idea: in such a ravine the coordinates don’t interfere with each other, and each one turns into a geometric progression.

The gradient is $\nabla f = (ax, by)$, so a descent step takes $(x, y)$ to $\big((1 - \eta a)x, (1 - \eta b)y\big)$: each coordinate is multiplied by its own number. After $k$ steps $x_k = (1 - \eta a)^kx_0$ and $y_k = (1 - \eta b)^ky_0$: geometric progressions (chapter 13). A progression with a non-zero first term tends to zero if and only if the absolute value of its ratio is less than one. In the picture the step is small, both ratios are between $0$ and $1$, and the point creeps towards the bottom without jumping over it. The condition $|1 - \eta b| < 1$ is equivalent to $0 < \eta b < 2$. If $1 < \eta b < 2$, the ratio $1 - \eta b$ is negative: the $y$-coordinate changes sign at every step, and the hiker jumps from one side of the ravine to the other, but still gets closer to the bottom. If $\eta b > 2$, then $|1 - \eta b| > 1$, and for $y_0 \ne 0$ the coordinate $y_k$ grows in absolute value: the descent blows up; at $\eta b = 2$ it jumps between $y_0$ and $-y_0$. The same goes for $x$ and $a$. So convergence from every point requires $\eta a < 2$ and $\eta b < 2$, that is, $\eta < \frac{2}{\max(a, b)}$, and this is enough. The step size is limited by the steepest direction.

And the speed is set by the gentlest. With the best step, $\eta = \frac{2}{a + b}$, both coordinates shrink by a factor of at least $\frac{\kappa - 1}{\kappa + 1}$ per step, where $\kappa = \frac{\max(a, b)}{\min(a, b)}$ is the condition number. At $\kappa = 100$ that’s $0.98$: to cut the error a millionfold takes about $690$ steps. Near a non-degenerate minimum a smooth function looks like a quadratic form, and the numbers $a$ and $b$ are the eigenvalues of its Hessian, so close to the bottom the conclusion holds for real mountains too. That’s why plain descent is hardly ever used to train neural networks: momentum is added, and the step is adjusted for each coordinate.

How much earth is in the mountain

The expedition’s geologist needs the volume of the mountain: how much earth lies over a patch of the map. We assembled the area under the graph of a function of one variable from narrow rectangles (chapter 28). Here we’ll assemble volume from columns. Divide a rectangle $R$ on the map into cells, stand a column over each with height $f$ at some point of the cell, and add up the volumes.

There’s no need to compute such limits directly: a double integral reduces to two ordinary ones. Cut the mountain with the planes $x = \text{const}$, find the area of each cross-section and add up the cross-sections: this is Cavalieri’s principle from chapter 20, turned into a theorem.

The idea: add up the same columns in a different order, first within strips, then the strips with each other.

Divide $R$ into $n \times n$ cells with sides $\Delta x$ and $\Delta y$ and take the sum $\sum_{i,j}f(x_i, y_j)\,\Delta x\,\Delta y$, where the $x_i$ are the midpoints of the columns of cells and the $y_j$ the midpoints of the rows. The double integral is the limit of these sums as $n \to \infty$. First add up the columns of one strip $x = x_i$. We get $\Delta x \cdot \sum_j f(x_i, y_j)\,\Delta y$, and the sum over $j$ is a Riemann sum for the integral $A(x_i) = \int_c^d f(x_i, y)\,dy$, the area of the cross-section of the mountain by the plane $x = x_i$. A continuous function on a rectangle is uniformly continuous (chapter 53): for any $\varepsilon > 0$, with a fine enough grid, $|f(x_i, y) - f(x_i, y_j)| < \varepsilon$ for all $y$ in the $j$th row, in all the strips at once. Then $\left|\sum_j f(x_i, y_j)\Delta y - A(x_i)\right| \le \sum_j\int |f(x_i, y_j) - f(x_i, y)|\,dy \le \varepsilon(d - c)$, where the integral is taken over the $j$th row: the columns of a strip hardly differ from its profile. So the whole sum differs from $\sum_i A(x_i)\,\Delta x$ by at most $\varepsilon(d - c)(b - a)$. The function $A$ is continuous: by the same uniform continuity $|A(x) - A(x')| \le \varepsilon(d - c)$ for nearby $x$ and $x'$. So $\sum_i A(x_i)\Delta x \to \int_a^b A(x)\,dx$. Letting $n \to \infty$ and then $\varepsilon \to 0$, we get $\iint_R f\,dA = \int_a^b A(x)\,dx$. Adding up the columns by the strips $y = y_j$ gives the other order of integration.

Compute $\iint_R (x + y^2)\,dA$ over the rectangle $R = [0, 2] \times [0, 1]$.

The inner integral, with respect to $y$: $\int_0^1(x + y^2)\,dy = x + \frac13$. The outer one: $\int_0^2\left(x + \frac13\right)dx = 2 + \frac23 = \frac83$.

Round mountains are easier to cut not into cells but into rings and sectors.

A little piece of a ring between the radii $r$ and $r + \Delta r$ and the angles $\varphi$ and $\varphi + \Delta\varphi$ has area exactly $\frac{\Delta\varphi}{2}\big((r + \Delta r)^2 - r^2\big) = \left(r + \frac{\Delta r}{2}\right)\Delta r\,\Delta\varphi$. It’s almost a rectangle with sides $\Delta r$ and $r\,\Delta\varphi$, and the further from the centre, the wider it is. Hence the factor $r$ in the formula.

The same function, written in terms of distance and angle. The area of a small sector of a ring is roughly $r\,\Delta r\,\Delta\varphi$: sectors far from the centre have a longer arc. Forgetting this factor is the most common mistake. The limits of the angle; for a whole disc or ring, from $0$ to $2\pi$. Regions of the form $r_1 \le r \le r_2$, $\alpha \le \varphi \le \beta$ are written like this. Example: the volume under the dome $z = 1 - x^2 - y^2$ over the unit disc is $\int_0^{2\pi}\int_0^1(1 - r^2)\,r\,dr\,d\varphi = 2\pi\left(\frac12 - \frac14\right) = \frac{\pi}{2}$, exactly half the cylinder with the same base and height. Archimedes already knew this result about the paraboloid.

The bell

The curve $y = e^{-x^2}$ is the famous bell from which the normal distribution grows (chapter 48). The area under it is finite, but you won’t find an antiderivative: in the 1830s Liouville proved that it can’t be written in terms of elementary functions, and the integral solver honestly gives up on it. And yet the area is known exactly.

The trick is to move to the plane: the square of the integral is the volume under the round bell $e^{-(x^2 + y^2)}$, and round solids are conveniently cut into rings.

Let $I_R = \int_{-R}^R e^{-x^2}\,dx$. By Fubini’s theorem $I_R^2 = \int_{-R}^R e^{-x^2}\left(\int_{-R}^R e^{-y^2}\,dy\right)dx = \iint_{K_R} e^{-(x^2 + y^2)}\,dA$, where $K_R = [-R, R]^2$ is a square. The integrand is positive, and the disc of radius $R$ lies inside the square, which itself lies inside the disc of radius $R\sqrt2$. So $J(R) \le I_R^2 \le J(R\sqrt2)$, where $J(\rho)$ is the integral over the disc of radius $\rho$. The integral over a disc is computed in rings. The ring between the radii $r$ and $r + \Delta r$ has area $\pi(r + \Delta r)^2 - \pi r^2 = 2\pi r\,\Delta r + \pi\,\Delta r^2$, and cut and unrolled it’s almost a strip $2\pi r \times \Delta r$. On the ring $e^{-r^2}$ is squeezed between $e^{-(r + \Delta r)^2}$ and $e^{-r^2}$, so $J(\rho)$ is squeezed between two sums that converge to the same number as $\Delta r \to 0$: $\int_0^\rho 2\pi r\,e^{-r^2}\,dr = \left[-\pi e^{-r^2}\right]_0^\rho = \pi\left(1 - e^{-\rho^2}\right)$. We’ve got $\pi\left(1 - e^{-R^2}\right) \le I_R^2 \le \pi\left(1 - e^{-2R^2}\right)$. As $R \to \infty$ both bounds tend to $\pi$, so $I_R^2 \to \pi$. The integral is positive, and $\int_{-\infty}^{+\infty}e^{-x^2}\,dx = \lim\limits_{R \to \infty} I_R = \sqrt\pi$.

The substitution $x = \frac{t}{\sqrt2}$ turns this formula into $\int_{-\infty}^{+\infty}e^{-t^2/2}\,dt = \sqrt{2\pi}$. Hence the factor $\frac{1}{\sqrt{2\pi}}$ in the formula for the normal distribution: with it the whole area under the bell equals one, as the sum of all probabilities should.

Rest stop

Before heading down into the valley, let’s check the kit. The trainer has three levels: partial derivatives; the gradient, the directional derivative and the tangent plane; stationary points and their type.

Where next

The expedition is on its way back, and at the rest stop someone gets out a violin. Sound is air pressure changing over time: a function of just one variable, but what a complicated one. A violin and a flute playing the same note A at 440 hertz draw completely different curves, and the ear tells them apart unerringly. The ear breaks sound down into pure tones, sine waves of different frequencies. Can any periodic signal be broken down like that? And how can you tell from a curve which sine waves it’s made of and how loud each one sounds? The answer comes from Fourier series.