First Order Approximation

I have yet to see any problem, however complicated, which, when looked at in the right way did not become still more complicated, Paul Anderson.

First Order Approximation

Definition. The affine function given by $\tilde{f}(x) = f(a) + \nabla f(a)^T(x-a)$ is a first order approximation of a real-valued function at a point x = a ∈ int(dom(f)) where:

• f(a) is the value of the function f evaluated at the point a, representing the “height” of the function at that particular point.
• $\nabla f(a)$ is the gradient of the function f evaluated at the point a. It is a vector pointing in the direction of the greatest rate of increase of the function at that point. For a real-valued function of multiple variables, the gradient is a vector of partial derivatives: $\nabla f(a) = (\frac{\partial f}{\partial x_1}(a), \frac{\partial f}{\partial x_2}(a), \cdots, \frac{\partial f}{\partial x_n}(a))$
• $\nabla f(a)^T$ is the transpose of the gradient vector, turning it into a column vector, necessary for the matrix multiplication that follows.
• (x - a) is the difference between the point x (where we want to approximate the function) and the point a (where we’re basing our approximation). It’s a vector that represents the displacement from a to x.
• $\nabla f(a)^T(x-a)$ is the dot product between the transposed gradient and the displacement vector, representing the directional derivative of f at a in the direction of (x -a). It tells us how much the function is changing as we move from a to x along a straight line.
• Finally, $f(a) + \nabla f(a)^T(x-a)$ start with the function’s value at a (f(a)) and add the change predicted by the gradient ($\nabla f(a)^T(x-a)$). This gives us a linear approximation of the function’s value at x.

  The first-order approximation assumes that the function behaves linearly near a. This is a good approximation if x is very close to a. However, as x moves further away from a, the curvature of the function typically becomes more significant, and the linear approximation becomes less accurate.

Theorem. Let $\mathbb{f}:\mathbb{R}^n \to \mathbb{R}$ be a real valued function defined on an open set $\mathbb{S} = dom(\mathbf{f})$. If $\mathbb{f}$ is continuously differentiable on its domain, then the following statement holds true, $\lim_{d \to 0} \frac{f(x+d)-f(x)-\nabla f(x)^Td}{||d||} = 0, \forall x \in \mathbb{S}$. In words, this first order approximation accuracy theorem states that for a differentiable function f at an arbitrary point x in its domain, the first order linear approximation provided by the gradient $\nabla f(x) = \biggr(\begin{smallmatrix}\frac{\partial f}{\partial x_1}\\\\ \frac{\partial f}{\partial x_2}\\\\ \cdots \\\\ \frac{\partial f}{\partial x_n}\end{smallmatrix}\biggl), f(x) + \nabla f(x)^Td$, becomes increasingly accurate as the displacement d from x approaches zero.

As the displacement d becomes smaller and smaller, the difference between the actual function value $\mathbb{f}(x+d)$ and its linear approximation $\mathbb{f}(x) + \nabla \mathbb{f}(x)^Td$ becomes negligible compared to the magnitude of the displacement ∣∣d∣∣. In other words, the linear approximation becomes increasingly accurate as we zoom in closer to the point x.

This theorem assumes that the function f is differentiable at the point x (f is continuously differentiable on its domain). If f is not differentiable at x, the limit in the theorem may not even exist or may not be zero. Mathematically, this idea can also be express as $\mathbb{f}(x) = \mathbb{f}(a)+ \nabla \mathbb{f}(a)^T(x-a) + o(||x-a||), a \in \mathbb{S}, \frac{o(t)}{t} \to 0 \text{ as } t \to 0⁺$