So far, we’ve spent a lot of time trying to solve systems of the form
We’ve asked whether there is a solution and, if so, how many solutions there are. Now we want to answer a different question: Suppose there isn’t a solution. Among all vectors in the column space of , which one is closest to ? In other words, how close can get to ?
This question leads to least squares, an idea with applications throughout science, engineering, and machine learning. Let’s start with a problem that makes the question concrete.
A line that cannot pass through every point¶
Suppose we collect some measurements and want to describe their relationship using a line
The slope and intercept let us summarize a trend and predict new values. But measurements are often noisy, and a line is often only an approximation to the relationship. What should we do when no line passes through all the points exactly?
For a small example, consider these four points:
| Input | Measured output |
|---|---|
| 3 | 5 |
| 2 | 2 |
| 7 | 7 |
| 6 | 8 |
No single line passes through all four points. We still want a useful line, so we need to decide what it means for one line to fit these points better than another.

Turning the data into equations¶
We’ll write our model as
Here, is the input, is the intercept, and is the slope. We’ve changed the letters so that can mean the vector of unknown coefficients and can mean the vector of measured outputs.
Passing through all four points would require
These equations have the form , where
In statistics, is called the design matrix. Each row corresponds to one measurement. Each column corresponds to one coefficient: the first column records how the intercept contributes to each prediction, and the second records how the slope contributes. The inputs need not be sorted; each row of must stay matched to the corresponding entry of .
Even though there is no exact solution, the matrix makes predictions for any choice of coefficients:
Our goal is to make this vector of predicted outputs as close as possible to the vector of measured outputs, .
Measuring the error¶
For any proposed coefficients , define the error vector to be
Each entry is a measured output minus its predicted output. A positive error means the model predicts too low; a negative error means it predicts too high.
For example, the line has coefficient vector
Its predictions and error vector are
The second and third points lie on the line. The line predicts 2 too low at the first and fourth points.
The length measures the distance between the predicted-output vector and the measured-output vector. It is usually easier to work with its square:
Minimizing the length and minimizing its square give the same answer, since lengths are nonnegative. Squaring also makes the contributions from positive and negative errors count toward the error instead of canceling each other.

The errors in the data plot are vertical differences, measured at the given input values. We are not measuring the shortest perpendicular distances from the plotted points to the fitted line.
Later, we will use a perpendicular projection to find the best fit. That projection takes place in a different space: the space of output vectors. Keeping these two pictures separate will help us interpret the geometry.
The column space picture¶
Recall from Chapter 5.1 that is the set of all possible outputs of . In our example,
The two columns are independent, so their span is a two-dimensional plane through the origin in . Every vector in this plane is a vector of predictions produced by some line .
Our measured-output vector is
It lies outside that plane, because the original system has no solution. Changing and moves within the plane, but can never make equal to .
We can now restate the fitting problem geometrically:
Which vector in is closest to ?
The data plot contains four points in . The column-space picture concerns one measured-output vector in , with one coordinate for each measurement. A fitted line in the data plot corresponds to one predicted-output vector in .
In Chapter 7.2, we’ll use a schematic picture to understand the projection, without needing to draw all four coordinate directions.
The least squares problem¶
The same question makes sense for any matrix and vector of compatible shapes.
The name comes from the squared error length. If the entries of are , then
We are finding coefficients that make this sum of squares least.
There are three different objects to keep track of:
is the vector of best-fitting coefficients, with entries.
is the vector of predicted outputs, with entries.
is the error vector, also with entries.
If has an exact solution, its squared error length is 0, the smallest possible value. Least squares includes exact solving as a special case.
Why this question appears so often¶
Fitting a line to four points is a small example of a much broader problem: we have measurements we want to explain, and a model with coefficients we can choose. In practice, we may have thousands or millions of measurements and only a few unknown coefficients.
Calibrating a sensor: We record a sensor’s output at several known temperatures. A fitted line can describe how its output changes with temperature, even when repeated measurements vary.
Estimating motion: We measure an object’s position at several times. If its velocity is approximately constant, a fitted line estimates its initial position and velocity.
Predicting from several measurements: A model might use a home’s size, age, and number of bedrooms to predict its price. Each coefficient describes the contribution of one input, and each observed home contributes an equation.
These problems often have more measurements than coefficients. We call a system with more equations than unknowns overdetermined. Such a system can be consistent, but noisy measurements often make an exact fit impossible.
Least squares gives us a precise way to find the best approximation under a squared-error measure. More complicated machine learning models build on the same pattern: choose a model, measure how its predictions differ from the data, and choose coefficients or parameters that reduce that error. Many use squared error, though other error measures are useful too.
We already know the geometric idea that will solve our problem. In Chapter 2.3, the closest vector on a line was the orthogonal projection. Next, we’ll extend that idea from a line to the column space of a matrix.