Linear algebra, practically
Lesson 5 of 9
5Linear algebra, practically
The big pictureOnly three ideas follow you to the end of this roadmap: a vector as meaning, a dot product as similarity, a matrix as a transform. No theorems, only what you will actually use.
What the picture shows: Documents a and b are about networks and training, so the angle between them is small, while c is about cooking and stands almost at right angles to a. Cosine similarity reads that angle and ignores the length of the arrow.
Map of this part
The concepts in order, click any one to jump to it:
Two ways into the same idea
A vector is a list of numbers describing something: a person's height, weight and age, say. If each word is described by a vector, we can measure how alike two words are by comparing their vectors. A matrix is a machine that transforms a vector: rotates it, stretches it, or moves it somewhere else. That is all you need to understand models.
A vector is a point in a space whose dimensions are features. The dot product a @ b measures alignment, and divided by both lengths it becomes cosine similarity. Multiplying a matrix by a vector is a linear map, which is what every Linear layer in a neural network does: y = W @ x + b.
Before You Start
🧱Arrays
You know shape and elementwise operations from the arrays lesson.
📐School geometry
You remember right angles, and that a length comes from the square root of a sum of squares.
The Core Idea
Three ideas in order: a vector as meaning, then the dot product and cosine similarity as a measure of likeness, then a matrix as a transform.
1A vector as meaning
A vector is a list of numbers standing for one thing. Here each document becomes the counts of four words in it.
Look at the picture: document a says "network" twice and "training" three times, and never mentions cooking.
Each number measures one dimension of meaning, so the document becomes a point in a space. Nearness in that space means similarity, and it is where every vector search you build will work.
2The dot product: multiply and add
Multiply the matching entries in each position, then add the products. Look at the picture: the product for each word between documents a and b.
The number grows when the two vectors point the same way and drops to zero when they share nothing. This one operation is the heart of a dense layer and of attention.
3Cosine similarity: divide out the length
The raw dot product has a flaw: a long document has bigger numbers, so it seems to resemble everything. The fix is to divide by both lengths, so you compare direction alone.
import numpy as np
vocab = ["شبكة", "تدريب", "بيانات", "طبخ"]
doc_a = np.array([2, 3, 1, 0])
doc_b = np.array([1, 2, 2, 0])
doc_c = np.array([0, 0, 1, 4])
def cosine(u, v):
return float(u @ v) / (float(np.linalg.norm(u)) * float(np.linalg.norm(v)))
print(int(doc_a @ doc_b))
print(f"a.b = {cosine(doc_a, doc_b):.3f}")
print(f"a.c = {cosine(doc_a, doc_c):.3f}")
print(f"b.c = {cosine(doc_b, doc_c):.3f}")
The first two documents are about networks and training, so their similarity is 0.891; the third is about cooking, so its similarity with the first is 0.065, near zero. No model was involved, only four numbers and the angle between them. This is the core that all of retrieval is built on.
4A matrix as a transform
A matrix takes a vector and returns another: it rotates it, stretches it, or both. Look at the picture: the point [2, 1] before and after each transform.
import numpy as np rotate = np.array([[0.0, -1.0], [1.0, 0.0]]) stretch = np.array([[3.0, 0.0], [0.0, 1.0]]) point = np.array([2.0, 1.0]) print([round(float(v), 1) for v in rotate @ point]) print([round(float(v), 1) for v in stretch @ point]) print([round(float(v), 1) for v in (stretch @ rotate) @ point]) print([round(float(v), 1) for v in (rotate @ stretch) @ point]) print(round(float(np.linalg.norm(point)), 3))
Look at the third and fourth lines: stretch @ rotate gave [-3.0, 2.0] while rotate @ stretch gave [-1.0, 6.0]. The same two transforms in a different order give a different answer, which is why layer order in a network is not cosmetic.
Every layer in a network is a matrix turning one vector into another, and training adjusts its numbers until the transform is useful. Because order changes the result, layer order is not cosmetic.
How It Works, Step by Step: how cosine similarity between two documents is computed
Represent each document by how often each vocabulary word appears, so doc_a becomes [2, 3, 1, 0].
Compute the dot product: multiply matching entries and add, so doc_a @ doc_b gives 10.
Compute each vector's length with np.linalg.norm, the square root of the sum of its squared entries.
Divide the dot product by the product of the two lengths. Counts are never negative, so the result here lies between 0 and 1: 0.891 for a and b.
Worked example
Try it without code
Myth vs Reality
| Myth | Reality |
|---|---|
| A bigger dot product means more similar. | A longer vector gives a bigger product with everything, which is why the longest document tops every search. Divide by both lengths first. |
| The order of a matrix product does not matter, as with numbers. | In the lesson stretch @ rotate gave [-3.0, 2.0] while rotate @ stretch gave [-1.0, 6.0]. Matrix multiplication does not commute. |
| You need all of linear algebra before you start. | Three ideas carry you through this roadmap: a vector as meaning, a dot product as similarity, a matrix as a transform. |
Try it yourself
You are building document search and notice that your longest document ranks top for every query. What is the likely cause?
Without normalisation a long document has larger numbers in every dimension, so its dot product with almost any query is larger. Dividing by the norm, that is cosine similarity, removes the length effect and compares direction only. Whole retrieval systems ship with this bug until someone measures them.
Quick review cards
Check Your Understanding
two vectors have cosine similarity 0. What does that mean in practice?
They are orthogonal and share no dimension that is non-zero in both. In a word representation that means two documents whose vocabularies never intersect. Note that zero means "unrelated", not "opposite"; opposition is not measured this way at all.
What is the length of the vector [2.0, 1.0], and why?
2.236, because it is the square root of 2² + 1², the square root of 5, which is what the last line of the transform example printed.
Resources for this lesson
Grouped by type. Start with the official docs for precision, or an article or video for the explanation.
Official docs1
Done with this lesson?
Sign in to save your progress.