3.2: Matrix-vector multiplication

Introduction¶

We’ll start by showing you an example of how to do the calculation. Consider

A=\begin{bmatrix}4&-3&5\\-2&6&7\end{bmatrix},\qquad \vec v=\begin{bmatrix}{\color{#3d81f6}2}\\{\color{orange}-1}\\{\color{#d81a60}3}\end{bmatrix}.

To multiply $A$ by $\vec v$ , multiply corresponding entries in the first row and the vector, then add. Repeat with the second row:

A\vec v =\begin{bmatrix}4({\color{#3d81f6}2})+(-3)({\color{orange}-1})+5({\color{#d81a60}3})\\(-2)({\color{#3d81f6}2})+6({\color{orange}-1})+7({\color{#d81a60}3})\end{bmatrix} =\begin{bmatrix}26\\11\end{bmatrix}.

What did we just do? There are two interpretations.

Interpretation 1: Dot products of $\vec v$ with each row of $A$ ¶

The operations that led us to 26 and 11 look like dot products. Indeed, those values came from dot products, but only if we treat each row as a column vector; you can’t take a dot product between a row vector and a column vector. The two dot products we computed were

\begin{bmatrix}4\\-3\\5\end{bmatrix}\cdot\begin{bmatrix}2\\-1\\3\end{bmatrix}=26, \qquad \begin{bmatrix}-2\\6\\7\end{bmatrix}\cdot\begin{bmatrix}2\\-1\\3\end{bmatrix}=11.

To take these dot products, we needed the rows of $A$ to have the same number of entries as $\vec v$ . Each row of $A$ had 3 entries, because $A$ had 3 columns. This tells us our golden rule:

There are $d$ entries in each row, so the input needs $d$ entries. There are $n$ rows, so the output has $n$ entries. Our first example matrix $A$ takes a vector in $\mathbb{R}^3$ ( $\begin{bmatrix} 2 \\ -1 \\ 3 \end{bmatrix}$ ) and turns it into a vector in $\mathbb R^2$ ( $\begin{bmatrix} 26 \\ 11 \end{bmatrix}$ ).

Interpretation 2: Linear combination of the columns of $A$ ¶

Let’s look at the computation one more time.

A=\begin{bmatrix}4&-3&5\\-2&6&7\end{bmatrix},\qquad \vec v=\begin{bmatrix}{\color{#3d81f6}2}\\{\color{orange}-1}\\{\color{#d81a60}3}\end{bmatrix}

A\vec v =\begin{bmatrix}4({\color{#3d81f6}2})+(-3)({\color{orange}-1})+5({\color{#d81a60}3})\\(-2)({\color{#3d81f6}2})+6({\color{orange}-1})+7({\color{#d81a60}3})\end{bmatrix} =\begin{bmatrix}26\\11\end{bmatrix}.

Let $\vec a_1,\vec a_2,\vec a_3$ denote the columns of $A$ .

In each row, the first entry is multiplied by ${\color{#3d81f6}2}$ , the second by ${\color{orange}-1}$ , and the third by ${\color{#d81a60}3}$ . Group those contributions vertically:

\begin{aligned} A\vec v &={\color{#3d81f6}2}\underbrace{\begin{bmatrix}4\\-2\end{bmatrix}}_{\vec a_1} +({\color{orange}-1})\underbrace{\begin{bmatrix}-3\\6\end{bmatrix}}_{\vec a_2} +{\color{#d81a60}3}\underbrace{\begin{bmatrix}5\\7\end{bmatrix}}_{\vec a_3}\\ &={\color{#3d81f6}\begin{bmatrix}8\\-4\end{bmatrix}} +{\color{orange}\begin{bmatrix}3\\-6\end{bmatrix}} +{\color{#d81a60}\begin{bmatrix}15\\21\end{bmatrix}} =\begin{bmatrix}26\\11\end{bmatrix}. \end{aligned}

In other words, $A\vec v={\color{#3d81f6}2}\vec a_1+({\color{orange}-1})\vec a_2+{\color{#d81a60}3}\vec a_3$ .

So $A\vec v$ is a linear combination of the columns of $A$ ! This is an incredibly important interpretation of the operation. The entries of $\vec v$ are the coefficients.

Click through the figure below to see both interpretations of the same calculation.

import html
import warnings
from pathlib import Path
from IPython.display import HTML, display
_notes_root = next(p for p in [Path.cwd(), *Path.cwd().parents] if (p / 'myst.yml').exists())
_widget = (_notes_root / 'ch03/multiplication-widget.html').read_text()
with warnings.catch_warnings():
    warnings.filterwarnings('ignore', message='Consider using IPython.display.IFrame instead')
    display(HTML('<iframe title="Scroll through a matrix-vector product: rows and columns" style="display:block;width:100%;height:690px;border:1px solid #cbd5e1;border-radius:6px;background:white" srcdoc="' + html.escape(_widget, quote=True) + '"></iframe>'))

Loading...

Another example¶

Let’s consider another example:

B=\begin{bmatrix}3&-4&8\\-5&2&-3\\6&7&4\end{bmatrix}.

The product of $B$ with $\vec u=\begin{bmatrix}{\color{#3d81f6}5}\\{\color{orange}1}\\{\color{#d81a60}0}\end{bmatrix}$ is

B\vec u=\begin{bmatrix} 3({\color{#3d81f6}5})+(-4)({\color{orange}1})+8({\color{#d81a60}0})\\ (-5)({\color{#3d81f6}5})+2({\color{orange}1})+(-3)({\color{#d81a60}0})\\ 6({\color{#3d81f6}5})+7({\color{orange}1})+4({\color{#d81a60}0}) \end{bmatrix}=\begin{bmatrix}11\\-23\\37\end{bmatrix}.

As before, write each row of $B$ as a column vector in $\mathbb R^3$ . Its dot product with $\vec u$ gives the corresponding entry of $B\vec u$ .

We can also use a linear combination of the columns:

\begin{aligned} B\vec u &={\color{#3d81f6}5}\begin{bmatrix}3\\-5\\6\end{bmatrix} +{\color{orange}1}\begin{bmatrix}-4\\2\\7\end{bmatrix} +{\color{#d81a60}0}\begin{bmatrix}8\\-3\\4\end{bmatrix}\\ &={\color{#3d81f6}\begin{bmatrix}15\\-25\\30\end{bmatrix}} +{\color{orange}\begin{bmatrix}-4\\2\\7\end{bmatrix}} +{\color{#d81a60}\begin{bmatrix}0\\0\\0\end{bmatrix}} =\begin{bmatrix}11\\-23\\37\end{bmatrix}. \end{aligned}

The zero in $\vec u$ means the third column contributes nothing to this product.

Properties¶

For an $n\times d$ matrix $A$ , vectors $\vec u,\vec v\in\mathbb R^d$ , and scalars $s,t$ :

Distributivity over vector addition: $A(\vec u+\vec v)=A\vec u+A\vec v$ .
Compatibility with scaling: $A(t\vec v)=t(A\vec v)$ .
Linearity: $A(s\vec u+t\vec v)=sA\vec u+tA\vec v$ , combining the first two properties.
Distributivity over matrix addition: $(A+B)\vec v=A\vec v+B\vec v$ when $B$ is also $n\times d$ .

You can check these by expanding the sums in each entry. In particular, $A\vec 0=\vec 0$ : every matrix sends the zero input to the zero output.

Using linearity¶

Suppose you already know that

A\vec u=\begin{bmatrix}6\\12\end{bmatrix},\qquad A\vec v=\begin{bmatrix}-15/2\\-6\end{bmatrix}.

We can find $A(2\vec u-\vec v)$ directly from these outputs. By linearity,

A(2\vec u-\vec v)=2A\vec u-A\vec v =2\begin{bmatrix}6\\12\end{bmatrix} -\begin{bmatrix}-15/2\\-6\end{bmatrix} =\begin{bmatrix}39/2\\30\end{bmatrix}.

The two known outputs are enough; we do not need the entries of $A$ , $\vec u$ , or $\vec v$ .

Order matters!¶

$A\vec v$ is different from $\vec vA$ . In fact, the latter is usually not defined! If $A$ has shape $n\times d$ and $\vec v$ has shape $d\times1$ , then

\underbrace{A}_{n\times d}\;\underbrace{\vec v}_{d\times1} =\underbrace{A\vec v}_{n\times1}.

Reversing the order gives

\underbrace{\vec v}_{d\times1}\;\underbrace{A}_{n\times d}.

Now the inner numbers are 1 and $n$ , so this product is only defined when $n=1$ . For our first example, $A\vec v$ has shapes $(2\times3)(3\times1)$ and is defined, while $\vec vA$ has shapes $(3\times1)(2\times3)$ and is not.

Even in the special case $n=1$ , the two orders do different things: $A\vec v$ has shape $1\times1$ , while $\vec vA$ has shape $d\times d$ .

Matrices as operations¶

When we multiply $A$ by $\vec v$ , what comes out is a new vector, $A\vec v$ . We should think of the process of multiplying $\vec v$ by $A$ as transforming $\vec v$ from one vector to another.

\underbrace{\vec v}_{\text{input}} \;\longrightarrow\;\boxed{\text{Multiply by }A}\;\longrightarrow\; \underbrace{A\vec v}_{\text{output}}.

Let’s try and understand the various ways a matrix $A$ can transform a vector $\vec v$ . If $A$ has shape $n\times d$ , the input has shape $d\times1$ and the output has shape $n\times1$ . The input and output don’t even need to have the same number of entries!

For example, consider the $2\times5$ matrix

A=\begin{bmatrix}1&2&0&-1&3\\0&-1&2&1&1\end{bmatrix}.

Multiplication by $A$ takes a vector in $\mathbb R^5$ and produces a vector in $\mathbb R^2$ . Here is one input and its output:

\underbrace{\begin{bmatrix}1\\2\\3\\4\\5\end{bmatrix}}_{\vec v\,\in\,\mathbb R^5} \;\longrightarrow\;\boxed{\text{Multiply by }A}\;\longrightarrow\; \underbrace{\begin{bmatrix}16\\13\end{bmatrix}}_{A\vec v\,\in\,\mathbb R^2}.

The two rows of $A$ give the two output entries:

A\vec v=\begin{bmatrix}1(1)+2(2)+0(3)-1(4)+3(5)\\0(1)-1(2)+2(3)+1(4)+1(5)\end{bmatrix} =\begin{bmatrix}16\\13\end{bmatrix}.

The input has shape $5\times1$ , and the output has shape $2\times1$ .

For the next five examples, we’ll use $2\times2$ matrices. Then both the input and output have shape $2\times1$ , so both are vectors in $\mathbb R^2$ . This lets us draw the transformation on a coordinate plane and see what it does.

We’ll use the same two input vectors in each example:

{\color{orange}\vec u=\begin{bmatrix}-4\\6\end{bmatrix}},\qquad {\color{#3d81f6}\vec v=\begin{bmatrix}5\\-3\end{bmatrix}}.

In each pair of graphs, the inputs appear on the left and the outputs appear on the right, with the same axes and scale. Orange identifies $\vec u$ and $A\vec u$ ; blue identifies $\vec v$ and $A\vec v$ . We’ll also multiply $A$ by an arbitrary input $\begin{bmatrix}a\\b\end{bmatrix}$ to understand what happens to every vector.

import sys
import plotly.graph_objects as go
import plotly.io as pio
# Kaleido 1.0 does not accept Plotly 7's optional HTTP headers.
if hasattr(pio.defaults, 'headers'):
    pio.defaults.headers = {}
from IPython.display import Image
if str(_notes_root) not in sys.path:
    sys.path.insert(0, str(_notes_root))
from plot_style import style_plotly

def transformation_plot(points=None, names=None, limit=16):
    fig = style_plotly(go.Figure())
    colors = ['orange', '#3d81f6']
    if points is not None:
        for point, name, color in zip(points, names, colors):
            fig.add_trace(go.Scatter(x=[0,point[0]], y=[0,point[1]], mode='lines', line=dict(color=color,width=3), showlegend=False))
            fig.add_annotation(x=point[0],y=point[1],ax=0,ay=0,axref='x',ayref='y',showarrow=True,arrowhead=3,arrowsize=1.2,arrowwidth=2,arrowcolor=color,text='')
            coords = r'\\'.join(str(x) for x in point)
            label = '$'+name+r'=\begin{bmatrix}'+coords+r'\end{bmatrix}$'
            fig.add_annotation(x=point[0],y=point[1],text=label,showarrow=False,xshift=(-50 if point[0] > 0.65*limit else (42 if point[0]>=0 else -45)),yshift=24,font=dict(color=color,size=18),bgcolor='rgba(255,255,255,0.85)')
    fig.update_layout(width=520,height=500,margin=dict(l=42,r=20,t=25,b=42),font=dict(family='Palatino',size=16))
    fig.update_xaxes(range=[-limit,limit],dtick=5 if limit==30 else 4,title='First coordinate',zeroline=True,zerolinewidth=1.5)
    fig.update_yaxes(range=[-limit,limit],dtick=5 if limit==30 else 4,title='Second coordinate',scaleanchor='x',scaleratio=1,zeroline=True,zerolinewidth=1.5)
    return fig

def save_plot(filename, points=None, names=None, limit=16):
    png=transformation_plot(points,names,limit).to_image(format='png',scale=6)
    (_notes_root/'ch03'/filename).write_bytes(png)
    return png
save_plot('transformation-blank.png')
save_plot('transformation-blank-wide.png',limit=30)
save_plot('transformation-before.png',[[-4,6],[5,-3]],[r'\vec u',r'\vec v'])
_ = save_plot('transformation-before-wide.png',[[-4,6],[5,-3]],[r'\vec u',r'\vec v'],limit=30)

import base64

def display_pair(filename, points, names, limit=16):
    after = save_plot(filename, points, names, limit)
    before_name = 'transformation-before-wide.png' if limit == 30 else 'transformation-before.png'
    before = (_notes_root / 'ch03' / before_name).read_bytes()
    panels = []
    for caption, png in [('Before: inputs', before), ('After: outputs', after)]:
        encoded = base64.b64encode(png).decode('ascii')
        panels.append('<figure style="margin:0;min-width:0;text-align:center">'
                      '<img alt="' + caption + '; orange represents u, blue represents v" '
                      'src="data:image/png;base64,' + encoded + '" '
                      'style="display:block;width:100%;height:auto;margin:0" />'
                      '<figcaption style="font:17px Palatino,serif;margin-top:6px">' + caption + '</figcaption></figure>')
    display(HTML('<div style="display:grid;grid-template-columns:minmax(0,1fr) minmax(0,1fr);gap:16px;width:100%;max-width:720px;margin:16px auto;background:white;color:black;padding:10px;box-sizing:border-box">' + ''.join(panels) + '</div>'))

Example 1: The identity matrix¶

A=\begin{bmatrix}1&0\\0&1\end{bmatrix},\qquad\vec u=\begin{bmatrix}-4\\6\end{bmatrix},\qquad\vec v=\begin{bmatrix}5\\-3\end{bmatrix}.

Multiplying by each input gives

A\vec u=\begin{bmatrix}1(-4)+0(6)\\0(-4)+1(6)\end{bmatrix}=\begin{bmatrix}-4\\6\end{bmatrix},

A\vec v=\begin{bmatrix}1(5)+0(-3)\\0(5)+1(-3)\end{bmatrix}=\begin{bmatrix}5\\-3\end{bmatrix}.

Loading...

This matrix leaves every vector unchanged.

A\begin{bmatrix}a\\b\end{bmatrix}=\begin{bmatrix}a\\b\end{bmatrix}.

Example 2: A diagonal matrix¶

A=\begin{bmatrix}-3/2&0\\0&2\end{bmatrix},\qquad\vec u=\begin{bmatrix}-4\\6\end{bmatrix},\qquad\vec v=\begin{bmatrix}5\\-3\end{bmatrix}.

Multiplying by each input gives

A\vec u=\begin{bmatrix}(-3/2)(-4)+0(6)\\0(-4)+2(6)\end{bmatrix}=\begin{bmatrix}6\\12\end{bmatrix},

A\vec v=\begin{bmatrix}(-3/2)(5)+0(-3)\\0(5)+2(-3)\end{bmatrix}=\begin{bmatrix}-15/2\\-6\end{bmatrix}.

Loading...

The key here is that a diagonal matrix transforms each component of the input vector independently. This matrix multiplies the first component by $-3/2$ and the second by 2. Each output component depends only on the corresponding input component; the components do not get mixed together.

A\begin{bmatrix}a\\b\end{bmatrix}=\begin{bmatrix}-3a/2\\2b\end{bmatrix}.

Example 3: A permutation matrix¶

A=\begin{bmatrix}0&1\\1&0\end{bmatrix},\qquad\vec u=\begin{bmatrix}-4\\6\end{bmatrix},\qquad\vec v=\begin{bmatrix}5\\-3\end{bmatrix}.

Multiplying by each input gives

A\vec u=\begin{bmatrix}0(-4)+1(6)\\1(-4)+0(6)\end{bmatrix}=\begin{bmatrix}6\\-4\end{bmatrix},

A\vec v=\begin{bmatrix}0(5)+1(-3)\\1(5)+0(-3)\end{bmatrix}=\begin{bmatrix}-3\\5\end{bmatrix}.

Loading...

The two coordinates swap. Geometrically, this reflects vectors across the line with equation $x_2=x_1$ .

A\begin{bmatrix}a\\b\end{bmatrix}=\begin{bmatrix}b\\a\end{bmatrix}.

Example 4: Sending vectors onto a line¶

A=\begin{bmatrix}3&6\\-2&-4\end{bmatrix},\qquad\vec u=\begin{bmatrix}-4\\6\end{bmatrix},\qquad\vec v=\begin{bmatrix}5\\-3\end{bmatrix}.

Multiplying by each input gives

A\vec u=\begin{bmatrix}3(-4)+6(6)\\(-2)(-4)+(-4)(6)\end{bmatrix}=\begin{bmatrix}24\\-16\end{bmatrix},

A\vec v=\begin{bmatrix}3(5)+6(-3)\\(-2)(5)+(-4)(-3)\end{bmatrix}=\begin{bmatrix}-3\\2\end{bmatrix}.

Loading...

Even though the input vectors $\vec u$ and $\vec v$ do not lie on the same line through the origin, the output vectors $A\vec u$ and $A\vec v$ do! Multiplication by this matrix sends them onto the same line. Every output is a multiple of $\begin{bmatrix}3\\-2\end{bmatrix}$ .

A\begin{bmatrix}a\\b\end{bmatrix}=\begin{bmatrix}3a+6b\\-2a-4b\end{bmatrix}=(a+2b)\begin{bmatrix}3\\-2\end{bmatrix}.

Example 5: A rotation matrix¶

A=\begin{bmatrix}3/5&-4/5\\4/5&3/5\end{bmatrix},\qquad\vec u=\begin{bmatrix}-4\\6\end{bmatrix},\qquad\vec v=\begin{bmatrix}5\\-3\end{bmatrix}.

Multiplying by each input gives

A\vec u=\begin{bmatrix}(3/5)(-4)-(4/5)(6)\\(4/5)(-4)+(3/5)(6)\end{bmatrix}=\begin{bmatrix}-36/5\\2/5\end{bmatrix},

A\vec v=\begin{bmatrix}(3/5)(5)-(4/5)(-3)\\(4/5)(5)+(3/5)(-3)\end{bmatrix}=\begin{bmatrix}27/5\\11/5\end{bmatrix}.

Loading...

This matrix rotates every vector counterclockwise by the same angle, approximately $53.1^\circ$ , while preserving its length.

A\begin{bmatrix}a\\b\end{bmatrix}=\begin{bmatrix}(3a-4b)/5\\(4a+3b)/5\end{bmatrix}.

The angle $\theta$ satisfies $\cos\theta=3/5$ and $\sin\theta=4/5$ .

You’ll explore this more in Lab 6.

Comparing the five operations¶

Example	Matrix	Operation
1	$\begin{bmatrix}1&0\\0&1\end{bmatrix}$	Identity: leaves every vector unchanged.
2	$\begin{bmatrix}-3/2&0\\0&2\end{bmatrix}$	Diagonal: scales the first coordinate by $-3/2$ and the second by 2.
3	$\begin{bmatrix}0&1\\1&0\end{bmatrix}$	Permutation: swaps the coordinates, reflecting across $x_2=x_1$ .
4	$\begin{bmatrix}3&6\\-2&-4\end{bmatrix}$	Sends every vector onto the line $2x_1+3x_2=0$ .
5	$\begin{bmatrix}3/5&-4/5\\4/5&3/5\end{bmatrix}$	Rotation: turns every vector counterclockwise by about $53.1^\circ$ , preserving its length.

Moving forward¶

There are two questions we will explore over the coming weeks.

Given $A$ and $\vec v$ , what is $\vec b=A\vec v$ ? Compute the output and describe the operation that $A$ applies to $\vec v$ . The five examples above illustrate some possible operations: leaving vectors unchanged, scaling coordinates, swapping coordinates, sending vectors onto a line, and rotating vectors.
Given $A$ and $\vec b$ , can we find $\vec v$ such that $A\vec v=\vec b$ ? This asks us to work backward from a desired output.

The second question need not have an answer. For the matrix in Example 4,

A\begin{bmatrix}a\\b\end{bmatrix}=(a+2b)\begin{bmatrix}3\\-2\end{bmatrix}.

The target $\begin{bmatrix}24\\-16\end{bmatrix}$ is reachable: the input $\begin{bmatrix}-4\\6\end{bmatrix}$ produces it. But $\begin{bmatrix}24\\-15\end{bmatrix}$ is unreachable, because every output satisfies $2x+3y=0$ . No input can make $A\vec v$ leave that line.

Even when an input exists, it need not be unique: $\begin{bmatrix}8\\0\end{bmatrix}$ also produces $\begin{bmatrix}24\\-16\end{bmatrix}$ .

Recap: Two interpretations of the same product¶

There are two ways to understand $A\vec v$ : dot products with the rows, written as column vectors, and a linear combination of the columns. They always give the same result. Let’s see both with a new example:

A=\begin{bmatrix}2&-1\\3&4\\-2&5\end{bmatrix},\qquad \vec v=\begin{bmatrix}3\\-2\end{bmatrix}.

The shapes are $(3\times2)(2\times1)$ , so the output has shape $3\times1$ .

Dot products with the rows¶

Write each row as a column vector and take its dot product with $\vec v$ :

\begin{bmatrix}2\\-1\end{bmatrix}\cdot\begin{bmatrix}3\\-2\end{bmatrix}=8,\qquad \begin{bmatrix}3\\4\end{bmatrix}\cdot\begin{bmatrix}3\\-2\end{bmatrix}=1,\qquad \begin{bmatrix}-2\\5\end{bmatrix}\cdot\begin{bmatrix}3\\-2\end{bmatrix}=-16.

These three numbers are the entries of the output:

A\vec v=\begin{bmatrix}2(3)+(-1)(-2)\\3(3)+4(-2)\\(-2)(3)+5(-2)\end{bmatrix} =\begin{bmatrix}8\\1\\-16\end{bmatrix}.

A linear combination of the columns¶

The entries of $\vec v$ tell us to multiply the first column by 3 and the second column by -2, then add:

A\vec v=3\begin{bmatrix}2\\3\\-2\end{bmatrix} +(-2)\begin{bmatrix}-1\\4\\5\end{bmatrix} =\begin{bmatrix}6\\9\\-6\end{bmatrix}+\begin{bmatrix}2\\-8\\-10\end{bmatrix} =\begin{bmatrix}8\\1\\-16\end{bmatrix}.

The row interpretation computes the output one entry at a time. The column interpretation builds the output as a sum of vectors. Both describe the same operation!

Introduction¶

Interpretation 1: Dot products of v⃗\vec vv with each row of AAA¶

Interpretation 2: Linear combination of the columns of AAA¶

Another example¶

Properties¶

Using linearity¶

Order matters!¶

Matrices as operations¶

Example 1: The identity matrix¶

Example 2: A diagonal matrix¶

Example 3: A permutation matrix¶

Example 4: Sending vectors onto a line¶

Example 5: A rotation matrix¶

Comparing the five operations¶

Moving forward¶

Recap: Two interpretations of the same product¶

Dot products with the rows¶

A linear combination of the columns¶

Interpretation 1: Dot products of $\vec v$ with each row of $A$ ¶

Interpretation 2: Linear combination of the columns of $A$ ¶