Newton iteration

Newton's approach starts from an initial value ${\tt x}_0$ and refines this value using the assumption that $f$ is locally linear. A first order approximation of $f({\tt x}_0+\Delta)$ yields:

$$f({\tt x}_0+\Delta) = f({\tt x}_0) + {\bf J}\Delta$$ (E13)

with ${\bf J}$ the Jacobian matrix and $\Delta$ a small displacement. Under these assumptions minimizing ${\tt\hat{e}} = {\tt\hat{e}}_0 - {\bf J}\Delta$ can be solved through linear least-squares. A simple derivation yields

$${\bf J}^\top{\bf J}\Delta = {\bf J}^\top {\tt\hat{e}} \enspace .$$ (E14)

This equation is called the normal equation. The solution to the problem is found by starting from an initial solution and refining it based on successive iterations

$${\tt x}_{i+1} = {\tt x}_i + \Delta_i$$ (E15)

with $\Delta_i$ the solution of the normal equation 5.14 evaluated at ${\tt x}_i$. One hopes that this algorithm will converge to the desired solution, but it could also end up in a local minimum or not converge at all. This depends a lot on the initial value ${\tt x}_0$.