User Coverage

User coverage is the proportion of users, or of user interactions, for which a recommender can produce recommendations. Shani and Gunawardana describe it as user space coverage, one of the distinct properties the word coverage can refer to.

Why some users are not served

In many applications the recommender may not produce recommendations for some users, for example because it has low confidence in its predictions for them. Coverage can also be measured by the richness of the user profile required to make a recommendation. In collaborative filtering, for instance, it can be the number of items a user must rate before receiving recommendations. According to Shani and Gunawardana, this can typically be evaluated in offline experiments.

Choices that define the number

• What counts as served. Any nonempty list, a full list of \(K\) items, or a list that passes a confidence threshold.
• The denominator. All registered users, users active in a period, or users present in the test set.
• The weight. Each user equally, or each interaction equally.

These choices can change the conclusion. Hypothetically, let four users have 100, 50, 10, and 2 interactions, and let the system serve all but the user with 2. User coverage is 3/4 = 0.75, while coverage weighted by interactions is 160/162 = 0.988. Which one matters depends on whether every user counts or the traffic does.

Coverage against accuracy

Shani and Gunawardana note that recommenders should be evaluated on the tradeoff between coverage and accuracy. Withholding uncertain recommendations can raise accuracy among the users who are served while lowering coverage, so the two numbers should be reported together, or models compared at the same coverage.

What to report

State the definition of a served user, the denominator, the period, and whether the value is weighted by interactions. Report user coverage next to the accuracy measured on the served users, so that a change in one can be read against the other.

Relation to cold start

In recommendation, cold start concerns new users or items with little or no interaction history. Measure the size of this group, the share that receives recommendations, and accuracy among those served; overall coverage can hide poor service for newcomers. For the catalog side, see Item Coverage, and for the umbrella term, Coverage.

References:

• Shani and Gunawardana, Evaluating Recommendation Systems (Microsoft Research technical report)