Self-Information of Popularity

The self-information, or surprisal, of an event with probability \(p\) is \(-\log_2 p\) bits. Rare events carry more surprise than common ones. For recommendation lists the event is that a user has collected an item, for example by clicking, rating, or buying it, and \(p\) is an estimate of the item’s popularity. Zhou et al. (2010) define the novelty of a recommended object through its surprisal relative to global popularity, \(I_\alpha = \log_2(u/k_\alpha)\), where \(u\) is the number of users and \(k_\alpha\) is the number of users who collected object \(\alpha\). This equals \(-\log_2(k_\alpha/u)\).

Two ways to define the probability

• The fraction of users who collected the item, \(k/u\), as in the form above. Fractions of this kind need not sum to 1 across items.
• The item’s share of all recorded interactions, which does sum to 1 across items.

Both are popularity estimates, and they give different values, so a result should say which one it uses.

A worked example

The numbers are hypothetical. With 1,000 users, an item collected by 500 users has self-information \(\log_2(1000/500) = 1\) bit. The values for 120 users, 10 users, and 1 user are 3.06, 6.64, and 9.97 bits. A list containing the items collected by 500, 120, and 10 users has a mean self-information of (1.00 + 3.06 + 6.64) / 3 = 3.57 bits.

The mean over a list is called the mean self-information (MSI). Vargas and Castells (2011) note that if rank and relevance are ignored, their popularity-based novelty metric reduces to MSI, a metric reported by Zhou et al.

How it behaves

• The value is unbounded for rare items, and an item nobody has collected has an undefined value because the logarithm of zero is undefined. Smoothing or excluding such items has to be specified.
• A few very rare items can dominate a list’s average.
• The unit depends on the logarithm base; base 2 gives bits.

What it does not tell you

Popularity is not the same as what a particular user knows, so a high value does not show that the user found the item new. The score also ignores relevance, so an obscure and useless item scores high. Relevance-aware variants and the wider question of what counts as new are covered in Novelty in Recommender Systems, and the skew of popularity in Long-Tail Items.

References:

• Zhou et al., Solving the Apparent Diversity-Accuracy Dilemma of Recommender Systems, arXiv:0808.2670 (published in PNAS, 2010)
• Vargas and Castells, Rank and Relevance in Novelty and Diversity Metrics for Recommender Systems (RecSys 2011)