Recommender Systems

A recommender selects and orders items for a user in a particular context. A shopping site might recommend available products that ship to the user’s region; a music service might include tracks the listener has already played. The eligible candidates, information available at the time, and objective are part of the problem. This article compares simple recommendation models and shows how evaluation choices change the conclusions drawn from them.

What interaction data can tell a model

Explicit feedback includes ratings or stated preferences. Implicit feedback includes clicks, purchases, and viewing time. A click indicates an interaction under some exposure and context; the absence of a click does not establish dislike. The user may never have seen the item. A user–item matrix records one row per user and one column per item, with entries representing the chosen feedback signal.

Collaborative filtering uses interaction patterns across users and items. Item–item methods recommend items associated with those a user has already consumed. Matrix factorization represents users and items with shorter vectors and scores their compatibility, often with a dot product. Content-based methods instead use attributes such as genre, text, or product specifications to relate items to a user’s interests. Hybrid systems combine these sources.

A popularity baseline ranks items by aggregate interaction counts. It needs those counts to be estimated, but no personalized user vector. Context-specific popularity, such as trending items in a region, can be a stronger baseline than global counts. Whether personalization is worth its added cost depends on the measured benefit and operational objective, not a universal accuracy margin.

Three simple scoring models

The experiment generates binary interactions from latent user tastes and an item-specific popularity term. The generated U and V vectors determine interaction probabilities; the fitted models receive only the training interaction matrix, not those hidden vectors. The generator has no timestamps, exposure policy, stock constraints, or repeated interactions. Hiding a random observed interaction per eligible user therefore tests reconstruction in a static synthetic matrix. It is not a simulation of predicting the next purchase. Users with fewer than five original interactions would be excluded from evaluation; the printed count shows whether that filter removes anyone here.

The SVD baseline approximates the binary matrix with rank 32. In this uncentered reconstruction, unobserved zeros contribute to the approximation as zeros. This is a simple low-rank baseline, not a confidence-weighted implicit-feedback objective or a pairwise ranking model. Its reconstructed scores are not probabilities. If a user vector is [1, 2] and an item vector is [3, 0.5], their dot-product score is 4; the components are learned coordinates, not necessarily named interests.

Item–item cosine similarity normalizes each item’s column of interactions, then takes column dot products. Items consumed by overlapping users become similar. For example, columns [1, 1, 0] and [1, 1, 1] have similarity 2/√6, about 0.8165. Multiplying the interaction matrix by the similarity matrix sums similarities to each user’s observed items. Setting the similarity diagonal to zero removes self-similarity. All three methods below exclude already observed training items from recommendations, which defines a new-item task.

Run the Python blocks in order with NumPy, SciPy, and scikit-learn installed. The dense score arrays are convenient for 4,000 users and 1,200 items. At larger scale, avoid materializing every user–item score or a dense item–item similarity matrix: their sizes grow with users × items and items² respectively.

import numpy as np
from scipy.sparse import csr_matrix
from sklearn.decomposition import TruncatedSVD

rng = np.random.default_rng(0)
n_users, n_items, k = 4000, 1200, 8
U = rng.normal(size=(n_users, k)); V = rng.normal(size=(n_items, k))
pop = rng.normal(0, 2.2, n_items)                    # a strong popularity skew
logits = (U @ V.T) * 0.55 + pop[None, :] - 5.2
R = (rng.random((n_users, n_items)) < 1 / (1 + np.exp(-logits))).astype(np.int8)

# leave-one-out: hide one interaction per user
held, Rtr = {}, R.copy()
for u in range(n_users):
    items = np.flatnonzero(R[u])
    if len(items) < 5: continue
    h = int(rng.choice(items)); held[u] = h; Rtr[u, h] = 0
users = sorted(held)

counts = Rtr.sum(0).astype(float)                    # popularity baseline
svd = TruncatedSVD(32, random_state=0)
mf_all = svd.fit_transform(csr_matrix(Rtr.astype(float))) @ svd.components_
Rn = Rtr / (np.linalg.norm(Rtr, axis=0, keepdims=True) + 1e-9)
S = Rn.T @ Rn; np.fill_diagonal(S, 0)
ii_all = Rtr @ S
models = {"popularity": np.tile(counts, (n_users, 1)),
          "SVD-32": mf_all, "item-item": ii_all}
print("evaluated users", len(users), "of", n_users)
# evaluated users 4000 of 4000

Define the ranking metric and its candidates together

Each evaluated user has exactly one held-out relevant item. Normalized discounted cumulative gain (NDCG) measures gain weighted by position relative to an ideal ordering. With the target’s one-based rank r, NDCG@10 is 1/log₂(r+1) when r ≤ 10, and zero otherwise. The ideal discounted gain is one in this special case. A target at rank 1 scores 1, rank 3 scores 0.5, and rank 11 scores 0. With multiple or graded relevant items, sum their discounted gains and divide by the corresponding ideal ordering. Here the final score averages users equally.

Hit rate@10 would only record whether the target appears, while NDCG also rewards an earlier position. Neither tells us whether an unobserved item would actually satisfy the user. The evaluation treats non-held-out, non-training items as distractors under its incomplete labels. Calling them sampled negatives describes that protocol, not verified negative preferences.

The full protocol ranks against every item not observed for that user in training. The alternatives include the same target with at most 100 other eligible items. Uniform sampling gives pool items equal chances; the popularity-weighted sampler uses training count + 1 as weights, so even zero-count items have positive weight. Candidates are drawn without replacement and shared across models. Equal scores are ordered by ascending item ID in every protocol, preventing candidate-array order from deciding ties.

def ordered_items(score, candidates):
    candidates = np.asarray(candidates, dtype=int)
    return candidates[np.lexsort((candidates, -score[candidates]))]

def make_candidates(mode="full", n_neg=100, seed=1):
    if mode not in ("full", "uniform", "popular"):
        raise ValueError("Unknown candidate protocol")
    if n_neg < 0:
        raise ValueError("n_neg must be nonnegative")
    g = np.random.default_rng(seed)
    result = []
    for u in users:
        available = np.flatnonzero(Rtr[u] == 0)
        h = held[u]
        pool = available[available != h]
        size = min(n_neg, len(pool))
        if mode == "full" or size == len(pool):
            result.append(available)
            continue
        weights = counts[pool] + 1.0 if mode == "popular" else None
        sampled = g.choice(pool, size, replace=False,
                           p=None if weights is None else weights/weights.sum())
        result.append(np.r_[h, sampled])
    return result

def ndcg_per_user(score_all, candidates, K=10):
    values = []
    for u, candidate in zip(users, candidates):
        order = ordered_items(score_all[u], candidate)
        rank = int(np.flatnonzero(order == held[u])[0]) + 1
        values.append(1/np.log2(rank+1) if rank <= K else 0.0)
    return np.array(values)

candidate_sets = {mode: make_candidates(mode) for mode in ("full", "uniform", "popular")}
full_scores = {}
print("model full uniform100 popular100")
for name, scores in models.items():
    full_scores[name] = ndcg_per_user(scores, candidate_sets["full"])
    metrics = [full_scores[name].mean(),
               ndcg_per_user(scores, candidate_sets["uniform"]).mean(),
               ndcg_per_user(scores, candidate_sets["popular"]).mean()]
    print(name, *(round(value, 4) for value in metrics))
# model full uniform100 popular100
# popularity 0.0854 0.3388 0.1101
# SVD-32 0.1226 0.4058 0.1926
# item-item 0.1171 0.4017 0.1474

SVD scores 0.1226 on the full eligible pool and 0.4058 with 100 uniform distractors, about 3.3 times as much. Popularity rises from 0.0854 to 0.3388, about fourfold. The multiplier is not common to all models. Removing distractors cannot push this single target farther down a fixed score ordering. A sampled NDCG can therefore be larger than its full-pool value even though the model is unchanged. The amount depends on which competing items are removed. Popularity weighting tends to include more high-count competitors than uniform sampling; it is not an exact popularity match to the held-out target or a universal representation of serving traffic.

Compare methods on the same users, targets, candidate rules, and tie policy. A score from a 100-distractor task should not be interpreted as a full-candidate score. Repeating the sampler measures sensitivity to that random draw; it does not remove the change in the evaluation task.

From offline ranking to a serving system

A large recommender often retrieves a manageable candidate set, ranks those candidates with a more detailed model, and applies availability or policy constraints before display. Retrieval can use fast popularity rules, item neighbors, or embedding search to find candidates. Retrieval recall measures the fraction of relevant items that reach this candidate set. Evaluate it as well as ranking quality: a ranker cannot recover a relevant item omitted by retrieval. For an end-to-end test, do not silently inject the held-out target into the retrieved set; a retrieval miss must count as a miss. Evaluation conditional on a retrieved target can be useful, but it answers a narrower question.

The relevant full set is the set eligible at the event time, not necessarily today’s entire catalog. If ranking within an actual retrieval stage is the deployment task, evaluate that stage’s candidates. Neither uniform nor popularity-weighted random distractors are guaranteed to reproduce them. Exact evaluation is preferable when feasible; sampled reports should disclose pool size, distribution, replacement, seeds, target inclusion, and seen-item filtering. Holding the protocol fixed makes experiments comparable under that protocol, but does not guarantee the same model winner under another one.

For future-event prediction, use a temporal design that respects availability across users as well as within each user’s history. A global cutoff can prevent training on interactions that occur after the evaluation event. Per-user “last interaction” splits alone can still expose future events from other users. Fit counts, similarities, embeddings, and transformations only from the permitted history, and retain the catalog available at that time. This timestamp-free experiment cannot measure the effect of a temporal split.

New users and new items create cold-start problems. With no interaction history, the personalized scores in these examples have little or no useful signal. Content features, onboarding choices, contextual popularity, and controlled exploration can provide starting points. An item–item score can be computed from one known item, but that does not guarantee useful neighbors or coverage of a new item. Evaluate these cases separately instead of removing them and treating the remaining score as representative of all users.

Measure behavior beyond the held-out target

Catalog coverage is the fraction of eligible catalog items appearing at least once in the evaluated recommendation lists. Head share is the fraction of recommendation slots occupied by a declared set of popular items; here that set is the top 5% by training count. Coverage depends on the number of users and list length, so compare models on the same population. It differs from diversity within one list, relevance, and fairness.

Offline interaction labels reflect how data were collected. In a deployed service, recommendations can change exposure, and later training can reinforce that exposure pattern. The static independent generator here contains no such feedback loop, so its head concentration does not demonstrate one. Online experiments can measure effects of changing recommendations, with engagement, satisfaction, retention, and relevant safeguards chosen for the application. Offline NDCG is useful evidence for ranking under its labels, not a direct estimate of every product outcome.

References

• Rendle, On Sampled Metrics for Item Recommendation studies how sampled ranking metrics can change relative model comparisons.
• TensorFlow Recommenders retrieval tutorial explains retrieval and ranking stages.
• scikit-learn documentation. TruncatedSVD describes the uncentered dimensionality reduction used here.