1.3 The Dot Product: Algebraic and Geometric Views

We have learned how to add vectors and scale them. Now, let's explore one of the most powerful operations in all of linear algebra: the Dot Product.

Instead of combining two vectors to create a new vector, the dot product multiplies two vectors to produce a single number (a scalar).

To master the dot product, we look at it from two different angles: algebraically (how we compute it) and geometrically (how we visualize it).

1. The Algebraic View: How to Calculate It

Algebraically, the dot product is calculated by multiplying the matching components of two vectors and adding the results.

$\mathbf{v} \cdot \mathbf{w} = v_1 w_1 + v_2 w_2 + \dots + v_n w_n$

• Example: If $\mathbf{v} = \begin{bmatrix} 4 \\ 1 \end{bmatrix}$ and $\mathbf{w} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}$, then: $\mathbf{v} \cdot \mathbf{w} = (4 \times 2) + (1 \times 3) = 8 + 3 = 11$

Transpose Notation ($\mathbf{v}^T \mathbf{w}$)

In machine learning literature, you will rarely see the dot symbol ($\cdot$) used. Instead, researchers use transpose notation:

$\mathbf{v}^T \mathbf{w}$

Here is why:

• In matrix algebra, vectors are assumed to be column matrices (size $n \times 1$).
• You cannot multiply two $n \times 1$ matrices directly.
• By transposing the first vector ($\mathbf{v}^T$), we turn it into a row matrix of size $1 \times n$.
• We can then multiply a $1 \times n$ row matrix by an $n \times 1$ column matrix, resulting in a single $1 \times 1$ number:

$\mathbf{v}^T \mathbf{w} = \begin{bmatrix} v_1 & v_2 & \dots & v_n \end{bmatrix} \begin{bmatrix} w_1 \\ w_2 \\ \vdots \\ w_n \end{bmatrix} = v_1 w_1 + v_2 w_2 + \dots + v_n w_n$

Writing the Norm as a Dot Product

An extremely useful trick in machine learning is re-writing the squared $L_2$ norm of a vector as a dot product of the vector with itself:

$\|\mathbf{v}\|_2^2 = \mathbf{v} \cdot \mathbf{v} = \mathbf{v}^T \mathbf{v}$

This allows us to simplify complex error equations when we compute gradients for optimization algorithms like Gradient Descent.

2. The Geometric View: What It Measures

Geometrically, the dot product measures alignment and direction between two vectors.

$\mathbf{v} \cdot \mathbf{w} = \|\mathbf{v}\|_2 \|\mathbf{w}\|_2 \cos(\theta)$

Where $\theta$ is the angle between the two vectors. Depending on the direction the vectors point, the dot product falls into three distinct categories:

    Acute (dot > 0)          Perpendicular (dot = 0)        Obtuse (dot < 0)
        ▲                         ▲                              ▲
       ╱  w                       │                              │
      ╱                           │ w                            │
     ┼─────►                      ┼─────►                  ◄─────┼
      v                            v                       w      v

1. Acute Angle ($\theta < 90^\circ$): The vectors point in similar directions. The dot product is positive ($>0$).
2. Obtuse Angle ($\theta > 90^\circ$): The vectors point in opposite directions. The dot product is negative ($<0$).
3. Perpendicular / Orthogonal ($\theta = 90^\circ$): The vectors are completely independent and perpendicular. The dot product is exactly zero ($=0$).

3. Hands-on Experiment

Interact with the visualizer below. Change the coordinates of Vector A (blue) and Vector B (green). Observe how the dot product value shifts from positive to negative as the angle sweeps past $90^\circ$.

4. Machine Learning Connection: Cosine Similarity

In machine learning, we often want to know if two objects are similar. For example:

• In a movie recommendation engine: Is user A's taste similar to user B's taste?
• In search engines: Is this document similar to the search query?

To do this, we represent items as vectors and calculate the Cosine Similarity:

$\cos(\theta) = \frac{\mathbf{v} \cdot \mathbf{w}}{\|\mathbf{v}\|_2 \|\mathbf{w}\|_2}$

By dividing the dot product by the lengths of the vectors, we isolate the angle $\theta$.

• A cosine similarity of $1$ means the vectors point in the exact same direction (perfect match).
• A similarity of $0$ means they are perpendicular (unrelated).
• A similarity of $-1$ means they are completely opposite.

This allows us to measure similarity based purely on content profile (direction) rather than scale (magnitude).

5. Check Your Understanding