Cosine Similarity

Cosine similarity compares the directions of two nonzero vectors expressed in the same coordinate system. Its numerator is the dot product, the sum of corresponding coordinate products. Dividing by both Euclidean lengths removes their overall magnitudes:

\[ \cos(x, y) = \frac{x \cdot y}{\lVert x \rVert_2 \, \lVert y \rVert_2} \]

It ranges from −1 to 1: 1 means the same direction, 0 means orthogonal directions, and −1 means opposite directions. Nonnegative features restrict it to 0 through 1. A zero vector has no defined angle. In retrieval and recommendation, the usefulness of this geometric comparison depends on what the coordinates represent; 0.8 is not an 80% probability that two items mean the same thing.

How normalization connects cosine and Euclidean distance

For L2-normalized vectors the two are related by an exact identity, \( \lVert x - y \rVert^2 = 2(1 - \cos(x,y)) \), so they cannot disagree about an ordering.

Take x = (1, 0) and y = (3, 4). Their dot product is 3 and their lengths are 1 and 5, so cosine similarity is 3/5 = 0.6. After normalization, y becomes (0.6, 0.8). Its squared distance from x is (1 − 0.6)² + (0 − 0.8)² = 0.8, matching 2(1 − 0.6). Generally, expanding the square gives x·x + y·y − 2x·y; for unit vectors the first two terms are both one.

Because 2(1 − c) decreases as cosine c increases, exact cosine and Euclidean search on normalized vectors give the same ranking, with the same ties. This does not mean their numerical scores are equal, nor that independently configured approximate indexes must return identical candidates. Without normalization, magnitude affects Euclidean distance: with query (1, 0), candidate (10, 0) has cosine 1 but Euclidean distance 9, while (1, 1) has cosine about 0.707 and distance 1.

Scale invariance is a decision, not a free property

Discarding magnitude is the feature most often cited for cosine. It is also a deliberate loss of information, and whether the loss is acceptable is domain-specific.

A term-count vector (3, 2, 1, 1, 0) and its fortyfold multiple (120, 80, 40, 40, 0) have cosine 1. They represent seven and 280 counted words with identical proportions. Positive rescaling multiplies both the dot product and the norm by the same factor, which cancels. If absolute frequency or length matters to the task, that information must be handled separately rather than inferred from cosine.

Scaling individual coordinates is different from scaling the whole vector. Consider two raw feature vectors (1, 100) and (0, 100), where the second coordinate is price and the first describes content. Their cosine is 100/√10001 ≈ 0.99995. Expressing price in hundreds changes them to (1, 1) and (0, 1), whose cosine is about 0.707. Normalizing each complete vector does not undo that change of coordinate weights. Choose units or feature-block weights for the intended similarity; standardizing every coordinate is not automatically correct for an embedding trained with a particular geometry.

A threshold depends on the representation and population

A retrieval threshold such as 0.7 selects a geometric region, but its relevance depends on the embedding model and the query/candidate population. The following NumPy experiment isolates dimension using independent, isotropic Gaussian vectors. Rows are independent pairs; each row is normalized implicitly by dividing its dot product by its lengths. This is a random-direction baseline, not a semantic embedding benchmark.

import numpy as np

rng = np.random.default_rng(404)
n = 4000

print("%8s %16s %14s %16s %18s"
      % ("d", "mean cosine", "sd", "1/sqrt(d)", "99th percentile"))
for d in (2, 8, 32, 128, 512, 2048):
    A = rng.normal(size=(n, d))
    B = rng.normal(size=(n, d))
    c = np.einsum("ij,ij->i", A, B) / (np.linalg.norm(A, axis=1) * np.linalg.norm(B, axis=1))
    print("%8d %16.4f %14.4f %16.4f %18.4f"
          % (d, c.mean(), c.std(ddof=1), 1 / np.sqrt(d), np.quantile(c, 0.99)))
print()
print("%8s %26s %26s" % ("d", "cos 0.30 is this many sd", "cos 0.60 is this many sd"))
for d in (2, 8, 32, 128, 512, 2048):
    A = rng.normal(size=(n, d))
    B = rng.normal(size=(n, d))
    c = np.einsum("ij,ij->i", A, B) / (np.linalg.norm(A, axis=1) * np.linalg.norm(B, axis=1))
    s = c.std(ddof=1)
    print("%8d %26.1f %26.1f" % (d, 0.30 / s, 0.60 / s))
print()
print("Both vectors are independent isotropic Gaussian draws; the cosine")
print("distribution has mean zero, not a zero value for every pair.")
#        d      mean cosine             sd        1/sqrt(d)    99th percentile
#        2          -0.0115         0.7071           0.7071             0.9992
#        8           0.0009         0.3574           0.3536             0.7547
#       32           0.0013         0.1780           0.1768             0.4156
#      128           0.0002         0.0890           0.0884             0.2052
#      512           0.0014         0.0443           0.0442             0.1039
#     2048          -0.0001         0.0220           0.0221             0.0496
#
#        d   cos 0.30 is this many sd   cos 0.60 is this many sd
#        2                        0.4                        0.8
#        8                        0.9                        1.7
#       32                        1.7                        3.3
#      128                        3.4                        6.7
#      512                        6.8                       13.6
#     2048                       13.6                       27.2
#
# Both vectors are independent isotropic Gaussian draws; the cosine
# distribution has mean zero, not a zero value for every pair.

For independent isotropic zero-mean Gaussian vectors, directions are uniform on the sphere. The distribution of their cosine has mean zero and standard deviation 1/sqrt(d); independence alone is insufficient. To see the standard deviation, fix one of the two directions; rotational symmetry means the choice does not matter, so take it along the first coordinate axis. The cosine is then the first coordinate of the other vector divided by that vector’s length. Across the d coordinates these squared ratios have identical distributions and sum to one, so each has expected value 1/d, and the mean is zero because flipping a coordinate’s sign preserves the distribution. Each pair’s cosine is its actual geometric similarity, not an estimate of a hidden true zero. The simulated 99th percentiles, 0.7547 at d = 8 and 0.0496 at d = 2,048, describe this particular random-direction baseline, not semantic relevance.

A dimension change does not by itself determine the new threshold: learned vectors need not follow this isotropic distribution. Even a model change that preserves dimension can alter score distributions. Re-evaluate the threshold on representative relevant and irrelevant pairs when the representation or retrieval population changes. Standard deviations from an unrelated-pair baseline do not substitute for measuring retrieval precision and recall.

L2-normalize nonzero vectors before using dot products as cosine scores. Cosine is unchanged by positive rescaling; zero vectors have no defined angle and need an explicit handling rule. Angular distance arccos(cos(x,y)) is a metric on the unit sphere, or on directions modulo positive scale. It is not a metric on all nonzero vectors because x and 2x have zero angular distance despite being distinct. The quantity 1−cos generally fails the triangle inequality.