Self-Information of Popularity

ByJu Yeon Eum

1. Self-Information (Information Content)

From information theory (Shannon, 1948):

  • The self-information of an event $x$ with probability $p(x)$ is:

$I(x) = -\log p(x)$

  • Interpretation: Rare events carry more information (surprise), common events carry less.
  • Example: If coin flip → “heads” with $p=0.5$, then $I=-\log(0.5)=1$ bit.
    If something very unlikely happens ($p=0.01$), $I=-\log(0.01)\approx 6.64$ bits → more surprising.

2. Popularity in Recommender Systems

  • In recommenders, each item’s popularity = probability of being chosen, clicked, rated, or consumed by users.
    • $p(i) = \frac{\text{\# interactions with item } i}{\text{total interactions}}$
  • Popular items → high probability $p(i)$.
  • Niche/rare items → low probability $p(i)$.

3. Self-Information of Popularity

Applying Shannon’s definition:

$I(i) = -\log p(i)$

  • If $i$ is very popular ($p(i)$ is large), then $I(i)$ is small → recommending it is not very informative.
  • If $i$ is rare $p(i)$ is small), then $I(i)$ is large → recommending it is informative, surprising, novel.

4. Intuition in Recommender Systems

  • High self-information items = “long-tail” recommendations (niche, less seen, more surprising).
  • Low self-information items = mainstream/popular recommendations (safe but less novel).

This connects popularity bias with novelty:

  • Purely popularity-based recommenders have low self-information (they keep showing what’s obvious).
  • Introducing high-self-information items can improve novelty, serendipity, and user satisfaction.

5. Example

Suppose:

  • Item A appears in 40% of interactions → $p(A)=0.4$

$I(A) = -\log_2(0.4) \approx 1.32 \ \text{bits}$

  • Item B appears in 0.5% of interactions → $p(B)=0.005$.

$I(B) = -\log_2(0.005) \approx 7.64 \ \text{bits}$

Item B is much more informative (rare, surprising) to recommend than Item A.

6. Applications

  • Novelty metrics: Use average self-information of recommended items to measure novelty.
  • Re-ranking: Combine predicted relevance with self-information to balance accuracy + novelty.
  • Diversity/serendipity: Encourage inclusion of items with higher information content.

Summary:
Self-Information of Popularity = $-\log p(i)$, where $p(i)$ is the probability (popularity) of an item. Popular items have low self-information (not surprising), while rare items have high self-information (novel, surprising). In recommender systems, this concept is used to measure novelty and counter popularity bias.

Similar Posts

Leave a Reply