Ranking Algorithms
A ranking algorithm turns candidates into an ordered list. Pairwise learning to rank trains from item preferences; lambda methods use the effect of swaps on a metric such as NDCG to guide training.
-
-
Long-Tail Items
Interaction data are usually dominated by a few popular items. Long-tail items are the many items with few interactions, and there is no universal cutoff for where the tail begins.
-
Self-Information of Popularity
Self-information is the surprise of an event, −log₂ of its probability. Applied to popularity, an item few users have chosen scores high, and the average over a list measures novelty.
-
Relevance in Recommender Systems
A ranking metric can only reward what the labels call relevant. Judged, rated, clicked, and held-out labels measure different things, and a missing label is not a negative label.
-
Genre Overlap
Genre overlap is not a single standard metric. It is a set-overlap measure applied to genre tags, and the choice of formula changes the value.
-
Jaccard index
The Jaccard index is the size of the intersection divided by the size of the union. It ignores weights, and two empty sets need an explicit rule.
-
Intra-List Diversity (ILD)
ILD is the average distance between the items in one recommendation list. The distance function and the list length decide what the number means.
-
Dominating in Recommender Systems
One model dominates another when it is at least as good on every metric and better on at least one. Most comparisons end with several non-dominated models and a trade-off to decide.
-
User Coverage
User coverage is the share of users who actually receive recommendations. A system that withholds low-confidence recommendations can gain accuracy and lose coverage.
-
Item Coverage
Item coverage counts how much of the catalog the system ever recommends. It grows with the number of users and the list length, and it can be high while a few items dominate.
-
DCG (Discounted Cumulative Gain)
DCG adds up the relevance of ranked items but discounts lower positions. Linear and exponential gains give different numbers, so the convention has to travel with the score.
-
Interleaving Tests
Instead of splitting users between two rankers, interleaving shows each request one blended list. The credit rule turns clicks into a preference, and some rules are biased.
-
Novelty (in Recommender Systems)
A novel recommendation is one the user did not know about. Offline, novelty is usually approximated from popularity, which is a proxy and not a measurement of what a user knows.
-
Diversity (in Recommender Systems)
Diversity is the opposite of similarity, and what counts as similar is a choice. It can be measured inside one list, between users’ lists, or across the whole catalog.
-
Coverage
Coverage is an umbrella for breadth: how much of the catalog gets recommended and how many users get recommendations at all. By itself it says nothing about relevance.
-
Hit Rate
Hit rate asks whether a relevant item appears in the top K at all. It ignores position inside the list and depends on how the candidates were chosen.
-
NDCG (Normalized Discounted Cumulative Gain)
NDCG rescales DCG by the ideal ordering’s DCG. A score is only comparable when the gain convention, cutoff, candidate set, and empty-query rule are the same.
-
Mean Average Precision (MAP)
Three metrics share the name AP. This is the retrieval one, where the denominator is every relevant item rather than every retrieved one.
-
Recommender Systems
The evaluation protocol moves the number more than the model does.