Novelty (in Recommender Systems)

Novel recommendations are recommendations for items that the user did not know about. This is the definition used by Shani and Gunawardana, who cite Konstan et al. (2006). Filtering out the items a user has already rated or used is an easy first step, but it does not remove everything the user knows, because users rarely report all the items they have used.

Measuring novelty

In a user study, novelty can be measured directly by asking whether the user was already familiar with a recommended item. Offline, the same report notes that some understanding of novelty can be gained from experiments, for example by splitting the data by time. The common offline proxy is popularity: an item that few users have interacted with is less likely to be known to a given user.

• Surprisal. Zhou et al. (2010) define novelty through the self-information of recommended objects, \(I_\alpha = \log_2(u/k_\alpha)\), where \(u\) is the number of users and \(k_\alpha\) is the number who collected the object. See Self-Information of Popularity.
• Popularity-based metrics with rank and relevance. Vargas and Castells (2011) define expected popularity complement (EPC), expected inverse popularity (EIP), and expected free discovery (EFD), and note that if rank and relevance are ignored, the popularity-based form reduces to the mean self-information of the recommended items.

A hypothetical example with 1,000 users: items collected by 500, 120, and 10 users have self-information of 1.00, 3.06, and 6.64 bits, and a list of these three has a mean of 3.57 bits.

Novelty, serendipity, and diversity

These terms are related and different. Serendipity, in Shani and Gunawardana’s words, measures how surprising the successful recommendations are; it can be thought of as the amount of relevant information that is new to the user. Diversity is dissimilarity among items (Diversity in Recommender Systems). A list of unknown items can be novel and uniform, and a varied list of well-known items can be diverse and not novel.

Cautions

• Popularity is a proxy. A popular item can be unknown to a particular user, and a rare one can be familiar.
• A novelty score that ignores relevance rewards obscure items whether or not they are useful. Relevance-aware variants exist for this reason; see Relevance in Recommender Systems.
• Shani and Gunawardana point out that random recommendations can be very surprising, so novelty and serendipity must be balanced with accuracy.
• Novelty is related to the exposure of long-tail items, but it is not the same as reaching them.

References:

• Shani and Gunawardana, Evaluating Recommendation Systems (Microsoft Research technical report)
• Zhou et al., Solving the Apparent Diversity-Accuracy Dilemma of Recommender Systems, arXiv:0808.2670 (published in PNAS, 2010)
• Vargas and Castells, Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems (RecSys 2011)