Relevance in Recommender Systems

In recommender evaluation, relevance is the label that marks which items count as good results for a user in a given context. Every ranking metric, including hit rate, precision, recall, and NDCG, rewards a recommender only for what the labels call relevant, so the definition of the label is part of the definition of the metric.

Where relevance labels come from

• Assessor judgments. People judge whether retrieved items satisfy a request. In TREC, NIST pools the results of the participating systems, judges the retrieved documents for correctness, and evaluates the results.
• Explicit feedback. Ratings, usually turned into relevant or not relevant by a threshold that the evaluator chooses.
• Implicit feedback. Clicks, purchases, plays, or completions. They are plentiful, but they reflect what was shown and what the user did, which is not the same as what the user would have liked.
• Held-out interactions. In leave-one-out evaluation, as in the protocol of He et al. (2017), each user’s latest interaction is held out as the test item, and that item is the relevant one.

Binary and graded relevance

Hit rate, precision, and recall treat relevance as binary. DCG and NDCG accept graded relevance, where a highly relevant item earns a larger gain than a marginally relevant one. With graded metrics the mapping from labels to grades, for example from star ratings to gains, has to be stated, because it changes the scores. DCG shows two common gain conventions.

A missing label is not a negative label

Observed feedback is not a random sample of all user–item pairs. Schnabel et al. observe that users typically rate the items they like and rarely rate the ones they do not, so the data are missing not at random, and they propose propensity-weighted estimators to correct evaluation for this selection. A protocol that treats every unobserved item as irrelevant penalizes a recommender for surfacing items the user was never shown or never tried.

The Recommender Systems article makes the same point about its own evaluation: items that are neither training items nor held-out targets act as distractors under incomplete labels, and calling them sampled negatives describes the protocol, not verified negative preferences.

Relevance is not the only goal

A relevant item can still be unhelpful, for example because the user already knows it or because the list repeats it. Novelty and diversity describe properties that a relevance label alone does not capture.

Three questions before reporting a result

1. What produced the relevance labels: judges, ratings above a threshold, clicks, or held-out items?
2. Is relevance binary or graded, and how were grades assigned?
3. How are unlabeled items treated, and is the protocol identical for every model being compared?

References:

• NIST, Text REtrieval Conference (TREC) overview
• He et al., Neural Collaborative Filtering (WWW 2017), arXiv:1708.05031
• Schnabel et al., Recommendations as Treatments: Debiasing Learning and Evaluation, arXiv:1602.05352