Diversity (in Recommender Systems)

Diversity in recommender systems is generally defined as the opposite of similarity. Shani and Gunawardana give an example in which a list of five vacation packages for the same location, differing only in hotel, may be less useful than five different locations, because the user can explore a wider range of options.

Three levels

• Within a list. How dissimilar the items in one recommendation list are. Shani and Gunawardana note that the most explored method uses item–item similarity, typically based on item content, and measures the diversity of a list by the sum, average, minimum, or maximum distance between item pairs. Intra-List Diversity (ILD) is the average-distance form.
• Between users’ lists. Zhou et al. (2010) call this personalization and measure it with an inter-list distance \(h_{ij}(L) = 1 – q_{ij}(L)/L\), where \(q_{ij}(L)\) is the number of items common to the two users’ top-\(L\) lists. Identical lists give 0 and completely different lists give 1. For two top-3 lists that share two items, \(h = 1 – 2/3 = 0.333\). The example is hypothetical.
• Across the catalog. How much of the catalog is recommended and how evenly. See Coverage.

What counts as similar

Every within-list measure depends on a distance between items. Distance can be built from genre tags (Jaccard index, genre overlap), from item text or content features, or from learned embeddings. A list that is diverse by genre can be uniform by another attribute, so the statement of a diversity result has to name the distance.

Vargas and Castells (2011) unify several novelty and diversity metrics in one framework and add item rank and relevance to a distance-based diversity measure, so that items at lower positions and non-relevant items count less.

Diversity and accuracy

Zhou et al. describe an apparent diversity–accuracy dilemma: the most useful individual recommendations may lie among diverse niche items, while the most reliably accurate methods recommend items based on user or item similarity. The title of their paper calls the dilemma apparent. The pull between the two goals is still what makes diversity something to measure next to relevance and not a goal to maximize alone. One common way to apply it is re-ranking: the final stage of a recommender can help ensure diversity, freshness, and fairness, as described in Google’s Machine Learning Crash Course on recommendation system stages.

Related entries: Novelty and Dominance and the Pareto front for comparing models on several objectives.

References:

• Shani and Gunawardana, Evaluating Recommendation Systems (Microsoft Research technical report)
• Zhou et al., Solving the Apparent Diversity-Accuracy Dilemma of Recommender Systems, arXiv:0808.2670 (published in PNAS, 2010)
• Vargas and Castells, Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems (RecSys 2011)
• Google for Developers, Machine Learning Crash Course: Recommendation system stages (Creative Commons Attribution 4.0)